We Keep Coding

Preventing duplicate child entries in an ORM relationship

Basically I have a service that reads from a spreadsheet and inserts into database.
In SQLAlchemy I have the following relationship
class Customer(Base):
__tablename__ = 'customers'
id = Column(Integer, primary_key=True)
name = Column(String)
children = relationship('Email', backref=('customer')
class Email(Base):
__tablename__ = 'emails'
id = Column(Integer, primary_key=True)
customer = Column(Integer, ForeignKey('customer.id'))
email = Column(String)
primary = Column(Boolean)
Is it possible for SQLAlchemy to check for a duplicate entry between a fetched resource and one created in the ORM?
For example let's say customer 123 has an email some_email, and we try to add it again:
email_object = Email(customer=123, email='some_email', primary=True)
cust = connection.query(Customer).options(joinedload(Customer.emails)).filter_by(
id=123).first()
cust.emails.append(email_object)
Ideally I would like SQLAlchemy to either notice that such a combination exists and merge/ignore it, or throw some kind of exception.
But instead I'm getting the following result if I print out cust.emails
[<Email(id=1, email=some_email, primary=True, customer=123>),
<Email(customer=192071, email='some_email', primary=True, customers=<Employee(id=123, name='John', emails=['some_email', 'some_email']>>)]
and doing a merge and commit just seems to add an extra identical row in the database (except for the pk).
I think maybe it has to do with the unused primary key in Emails, but that is autogenerated when committing to the DB.
Any ideas?
Lemme know if I need to clarify anything.

Setting the Email class to have two primary keys doesn't seem to make SQLAlchemy stop from appending the extra email
That's correct. Using a composite primary key on (customer_id, email) does not prevent SQLAlchemy from trying to insert a new object that essentially duplicates an existing email — although it will warn you if an object with the same primary key already exists in the identity map — and the INSERT will fail (throw an exception and be rolled back) because of the duplicate PK, thus preventing the duplicate child record.
If you want to check whether an email exists before trying to add it you can either use session.get() …
with Session(engine) as session:
# retrieve John's object
john = (
session.execute(select(Customer).where(Customer.name == "John"))
.scalars()
.one()
)
print(john) # <Customer(id=123, name='John')>
# check if email already exists using .get()
email = session.get(Email, (john.id, "some_email"))
if email:
print(f"email already exists: {email}")
# email already exists: <Email(customer_id=123, email='some_email')>
else:
print("email does not already exist")
… or a relationship in Customer could provide the existing emails, allowing you to search for the one you want to add
# alternative (less efficient) method: check via relationship
e_list = [e for e in john.emails if e.email == "some_email"]
if e_list: # list not empty
print("email already exists")
else:
print("email does not already exist")

How to check if a table exists in Python?

I'm currently going over a course in Web design. What I want to do is to check if a table named portafolio exists in my database, if not I want to create one.
I'm using Python (flask) and sqlite3 to manage my database.
So far I the some of the logic in SQL to create a table if it doesn't exist:
# db is my database variable
db.execute('''create table if not exists portafolio(id INTEGER PRIMARY KEY AUTOINCREMENT,
stock TEXT,
shares INTEGER,
price FLOAT(2),
date TEXT
''');
But instead of using SQL commands I'd like to know how would I do the exact same checking in Python instead, since it would look a lot cleaner.
Any help would be appreciated.

Not sure about which way is cleaner but you can issue a simple select and handle the exception:
try:
cursor.execute("SELECT 1 FROM portafolio LIMIT 1;")
exists = True
except sqlite3.OperationalError as e:
message = e.args[0]
if message.startswith("no such table"):
print("Table 'portafolio' does not exist")
exists = False
else:
raise
Note that here we have to check what kind of OperationalError it is and, we have to have this "not pretty" message substring in a string check because there is currently no way to get the actual error code.
Or, a more SQLite specific approach:
table_name = "portafolio"
cursor.execute("""
SELECT name
FROM sqlite_master
WHERE type='table' AND name=?;
""", (table_name, ))
exists = bool(cursor.fetchone())

If you are looking for a neat job with Database operations, I advice you learn more about ORM(Object Relation Model).
I use Flask with SQLAlchemy. You can use python classes to manage SQL operations like this:
class Portafolio(db.Model):
id = db.Column(db.Integer, primary_key=True)
stock = db.Column(db.String(255), unique=True)
shares = db.Column(db.Integer, unique=True)
It does all the database checks and migration easily.

SQLAlchemy OperationalError due to Query-invoked autoflush

I have a table in a database which is created and accessed through SQLAlchemy:
I add a record to it using Flask-SQLAlchemy like so:
...
content = request.form['content']
date = datetime.today()
post = Post(date, content)
db.session.add(post)
db.session.commit()
...
This record is added to the table fine. Right after that code is executed, I query another table:
userID = session['userID']
posts = db.session.query(Post).filter_by(userID=userID).count()
However I receive an error during the query:
OperationalError: (raised as a result of Query-invoked autoflush;
consider using a session.no_autoflush block if this flush is occurring
prematurely) (_mysql_exceptions.OperationalError) (1292, "Incorrect
date value: '11/20' for column 'date' at row 1") [SQL: u'UPDATE
posts SET date=%s WHERE posts.id = %s'] [parameters: (('11/20',
1L))]
Why is the date of the post being updated when I have already specified it when adding the record to the table? Also what could the cause of this error be? Thanks.
Edit:
This is what the table model is like:
class Post(db.Model):
__tablename__ = 'posts'
id = db.Column(db.Integer, primary_key=True)
content = db.Column(db.String(500))
date = db.Column(db.Date, nullable=False)
def __init__(self, id, content, date):
self.id = id
self.content = content
self.date = date

Stephane is right, you are passing a wrong type to the model, either pass datetime.date object or change the definition of the model. As to the first part of the question, I recommend reading something about sessions and flushing. This is important:
All changes to objects maintained by a Session are tracked - before the database is queried again or before the current transaction is committed, it flushes all pending changes to the database.
So by creating the post object and adding it to the session, you made just a pending change, but there was no communication with the database at that point yet. That happens with flush(), which you can either call manually or automatically, for example, by calling commit().
(btw. you dont need to create your own init method for the model, see http://docs.sqlalchemy.org/en/latest/orm/tutorial.html#adding-and-updating-objects)

date = datetime.today() returns a datetime object (date AND time)
but the date attribute of the Post model is a db.Date (date WITHOUT time)
Try either :
from datetime import date
...
content = request.form['content']
date = date.today() #inject a Date object rather than a Datetime
or:
class Post(db.Model): #modify Post schema
...
date = db.Column(db.TIMESTAMP, nullable=False)

This error can cause from another query,
even if you solve it that exceptions will still occured if you not rollback previous session error
You can catch exception and rollback transaction
usually in my flask application, I commit session in end of request
#app_instance.after_request
def after(response):
try:
# commit transaction
db.session.commit()
except Exception:
db.session.rollback()
raise
return response

IntegrityError not caught in SQLAlchemy event listener

I'm building a simple database driven blog with Flask and SQLAlchemy. In the model for the blog postings I define title and slug attributes:
class BlogPost(Model):
...
title = Column(String(80))
slug = Column(String(80), unique=True)
Later I use an event listener to automatically create and insert a slug from the title:
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
target.slug = slugify(value)
As expected, if I try to add a post to the database, and the title of the post evaluates to the same slug as a previous post, then the transaction fails with an IntegrityError. I don't think in practice this will be a problem anyway. But just for giggles I tried something like this:
from sqlalchemy.exc import IntegrityError
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
try:
target.slug = slugify(value)
except IntegrityError:
target.slug = slugify(value) + random_string()
random_string could be anything, really, the point is that nothing that I've tried gets executed because the IntegrityError isn't getting caught, and I'm not sure why - attempting to add & commit a post to the database with the same title still raises an IntegrityError and aborts the transaction when I try to commit. I've seen a handful of other posts about it, but the answers are mostly pretty specific to Pyramid, which I'm not using.
Anybody know what I'm missing here?
Tech involved: Python3, Flask, Flask-Sqlalchemy, Sqlalchemy

SQLAlchemy will not flush changes to model objects to DB when setting. In order to get the error you have to do something like
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm.session import object_session
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
session = object_session(target)
try:
with session.begin_nested():
target.slug = slugify(value)
session.flush()
except IntegrityError:
target.slug = slugify(value) + random_string()
Note that you have to wrap your possible integrity violation in a nested transaction (a savepoint), or your whole transaction will fail even though you catch the IntegrityError. If your DB doesn't support savepoints or an SQLAlchemy implementation of the idea, you're out of luck.

how to catch A 'UNIQUE constraint failed' 404 in django

How do I specifically catch a UNIQUE constraint failed 404 in the following code, I know I have to add something in the ( here? ) section
try:
q = AnswerModel(user=user, yes_question=question_model)
q.save()
except ( here? ):
return HttpResponseRedirect('/user/already_exists')

from django.db import IntegrityError
except IntegrityError:
This is what you need.
EDITED for #mbrochh:
from django.db import IntegrityError
except IntegrityError as e:
if 'unique constraint' in e.message: # or e.args[0] from Django 1.10
#do something
Yes, you can be more precise but in question case UNIQUE failed is highly likely.

IMHO, I would recommend to resolve this situation by get_or_create().
new_obj, created = AnswerModel.objects.get_or_create(user=user, yes_question=question_model)
if created:
do_something_for_new_object(new_obj)
else:
logging.error("Duplicated item.")
return

Usually the "ask for forgiveness" principle is a good practice in programming but in this special case, I would not recommend it.
The exception you are looking for is IntegrityError. You could have easily figured that out yourself by simply removing the try-catch block and forcing that exception. The traceback shows the exception class.
The problem is, there are several different kinds of integrity errors, so inside your try-catch block you would have to check for something like if ex.pgcode == 23505 to see if this is actually a UNIQUE constraint error. This has been answered before here: IntegrityError: distinguish between unique constraint and not null violations
It gets worse: Each ORM has different error codes, the field name will not be pgcode but something else and some ORMs don't throw UNIQUE constraints at all. So if you are building a reusable app or if you are using a ORM that sucks (such as MySQL) or if you are not sure if you will change the database of your project at some time in the future, you should not do this!
The better way is simply removing the try-catch block and check if the object is already in the database before saving.
I don't know which field is UNIQUE in your case, so I will just assume that it is the user field. Your code would look something like this:
answers = AnswerModel.objects.filter(user=user)
if answers:
return HttpResponseRedirect('/user/already_exists')
obj = AnswerModel.objects.create(user=user, yes_question=question_model)
...
If you are dealing with a combined unique constraint, the first line would be this:
answers = AnswerModel.objects.filter(user=user, yes_question=question_model)

For Python > 3.5
You need:
except IntegrityError as e:
if 'unique constraint' in e.args:
Example:
from django.db import IntegrityError
for column in csv.reader(io_string, delimiter=',', quotechar="|"):
try:
_, created = Brand.objects.update_or_create(
system_id=column[0],
name=column[1],
slug=column[2],
description=column[3],
image=column[4]
)
except IntegrityError as e:
if 'unique constraint' in e.args:
continue

Found in django.db.models.query.QuerySet._create_object_from_params. And with some change:
from typing import Any, Dict, Type
from django.db import transaction, IntegrityError, Model
def get_or_create_obj(model: Type[Model], defaults: Dict[str, Any] = None, **kwargs: Any):
defaults = defaults or {}
try:
with transaction.atomic():
return model.objects.create(**kwargs, **defaults), True
except IntegrityError as e:
try:
return model.objects.using("default").get(**kwargs), False
except model.DoesNotExist:
pass
raise e

Unique constraint fail return :"('UNIQUE constraint failed: notepad_notepadform.title')" which is basically a tuple,So we can use below code to catch it and do whatever required:
from django.db import IntegrityError
try:
if form.is_valid():
form.save()
except IntegrityError as e:
if 'UNIQUE constraint' in str(e.args):
#your code here