I have a table in a database which is created and accessed through SQLAlchemy:
I add a record to it using Flask-SQLAlchemy like so:
...
content = request.form['content']
date = datetime.today()
post = Post(date, content)
db.session.add(post)
db.session.commit()
...
This record is added to the table fine. Right after that code is executed, I query another table:
userID = session['userID']
posts = db.session.query(Post).filter_by(userID=userID).count()
However I receive an error during the query:
OperationalError: (raised as a result of Query-invoked autoflush;
consider using a session.no_autoflush block if this flush is occurring
prematurely) (_mysql_exceptions.OperationalError) (1292, "Incorrect
date value: '11/20' for column 'date' at row 1") [SQL: u'UPDATE
posts SET date=%s WHERE posts.id = %s'] [parameters: (('11/20',
1L))]
Why is the date of the post being updated when I have already specified it when adding the record to the table? Also what could the cause of this error be? Thanks.
Edit:
This is what the table model is like:
class Post(db.Model):
__tablename__ = 'posts'
id = db.Column(db.Integer, primary_key=True)
content = db.Column(db.String(500))
date = db.Column(db.Date, nullable=False)
def __init__(self, id, content, date):
self.id = id
self.content = content
self.date = date
Stephane is right, you are passing a wrong type to the model, either pass datetime.date object or change the definition of the model. As to the first part of the question, I recommend reading something about sessions and flushing. This is important:
All changes to objects maintained by a Session are tracked - before the database is queried again or before the current transaction is committed, it flushes all pending changes to the database.
So by creating the post object and adding it to the session, you made just a pending change, but there was no communication with the database at that point yet. That happens with flush(), which you can either call manually or automatically, for example, by calling commit().
(btw. you dont need to create your own init method for the model, see http://docs.sqlalchemy.org/en/latest/orm/tutorial.html#adding-and-updating-objects)
date = datetime.today() returns a datetime object (date AND time)
but the date attribute of the Post model is a db.Date (date WITHOUT time)
Try either :
from datetime import date
...
content = request.form['content']
date = date.today() #inject a Date object rather than a Datetime
or:
class Post(db.Model): #modify Post schema
...
date = db.Column(db.TIMESTAMP, nullable=False)
This error can cause from another query,
even if you solve it that exceptions will still occured if you not rollback previous session error
You can catch exception and rollback transaction
usually in my flask application, I commit session in end of request
#app_instance.after_request
def after(response):
try:
# commit transaction
db.session.commit()
except Exception:
db.session.rollback()
raise
return response
Related
Description
I have an Flask application with original SQLalchemy. Application is intended to be used internally in a company for easier saving of measurement data with MySQL
On one page I have a table with all devices used for measurement and a form that is used to add, remove or modify measurement devices.
Problem
The problem is that when I enter a new device in the database, the page is automatically refreshed to fetch new data from DB and new device is sometimes shown and sometimes it is not when I refresh the page. In other words, added row in table is appearing and dissapearing even though the row is visible on database. Same goes when i try to delete the device from database. The row is sometimes shown, sometimes not when refreshing the page with row being deleted from DB.
The same problem appears for all examples similar to this one (adding, deleting and modifying data).
What i have tried
Bellow is the code for table model:
class DvDevice(Base):
__tablename__ = "dvdevice"
id = Column("device_id", Integer, primary_key=True, autoincrement=True)
name = Column("device_name", String(50), nullable=True)
code = Column("device_code", String(10), nullable=True, unique=True)
hw_ver = Column("hw_ver", String(10), nullable=True)
fw_ver = Column("fw_ver", String(10), nullable=True)
sw_ver = Column("sw_ver", String(10), nullable=True)
And here is the code that inserts/deletes data from table.
#Insertion
device = DvDevice()
device.code = self.device_code
device.name = self.device_name
device.hw_ver = self.hw_ver
device.fw_ver = self.fw_ver
device.sw_ver = self.sw_ver
ses.add(device)
ses.commit()
ses.expire_all() #Should this be here?
# Deletion
ses.query(DvDevice).filter_by(id=self.device_id).delete()
ses.commit()
ses.expire_all() # Should this be here?
I have read from some posts on stack to include the following decorator function in models.py
#app.teardown_appcontext
def shutdown_session(exception=None):
ses.expire_all() #ses being database session object.
I tried this and it still doesn't work as it should be. Should I put the decorator function somewhere else?
Second thing i tried is to put ses.expire_all() after all commits and it still doesnt work.
What should I do to prevent this from happening?
Edit 1
from sqlalchemy import create_engine, update
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import NullPool
from config import MYSQLCONNECT
engine = create_engine(MYSQLCONNECT)
Session = sessionmaker(bind=engine)
session = Session()
I solved the problem with the use of following function from http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it:
from contextlib import contextmanager
#contextmanager
def session_scope():
"""Provide a transactional scope around a series of operations."""
session = Session()
try:
yield session
session.commit()
except:
session.rollback()
raise
finally:
session.close()
with session_scope() as session:
... # code that uses session
The problem was that I created the session object in the beggining and then never closed it.
We have a resource table which has a field last_updated which we setup with mysql-workbench to have the following properties:
Datatype: TIMESTAMP
NN (NotNull) is checked
Default: CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
When I modify a row through the workbench and apply it, the last_updated field properly updates.
When I use the REST api we've setup, and issue a put:
update = requests.put('http://127.0.0.1:8000/api/resources/16',
data=json.dumps(dict(status="/api/status/4", timeout=timeout_time)),
headers=HEADER)
I can properly change any of the values (including status and timeout, and receive a 204 response), but last_updated does not update.
Django's model documentation says in this case it should be sending an UPDATE.
Anyone have and ideas on why it's missing these updates?
I can provide further details regarding our specific Django/tastypie setup, but as long as they are issuing an UPDATE, they should be triggering the databases ON UPDATE.
I suspect that the UPDATE statement issued by Django may be including an assignment to the last_updated column. This is just a guess, there's not enough information provided.
But if the Django model contains the last_updated column, and that column is fetched from the database into the model, I believe a save() will assign a value to the last_updated column, in the UPDATE statement.
https://docs.djangoproject.com/en/1.9/ref/models/instances/#specifying-which-fields-to-save
Consider the behavior when we issue an UPDATE statement like this:
UPDATE mytable
SET last_updated = last_updated
, some_col = 'some_value'
WHERE id = 42
Because the UPDATE statement is assigning a value to the last_updated column, the automatic assignment to the timestamp column won't happen. The value assigned in the statement takes precedence.
To get the automatic assignment to last_updated, that column has to be omitted from the SET clause, e.g.
UPDATE mytable
SET some_col = 'some_value'
WHERE id = 42
To debug this, you'd want to inspect the actual SQL statement.
With the added information from spencer7593's answer, I was able to track down how to do this through tastypie:
The BaseModelResource.save() (from tastypie/resources.py):
def save(self, bundle, skip_errors=False):
if bundle.via_uri:
return bundle
self.is_valid(bundle)
if bundle.errors and not skip_errors:
raise ImmediateHttpResponse(response=self.error_response(bundle.request, bundle.errors))
# Check if they're authorized.
if bundle.obj.pk:
self.authorized_update_detail(self.get_object_list(bundle.request), bundle)
else:
self.authorized_create_detail(self.get_object_list(bundle.request), bundle)
# Save FKs just in case.
self.save_related(bundle)
# Save the main object.
obj_id = self.create_identifier(bundle.obj)
if obj_id not in bundle.objects_saved or bundle.obj._state.adding:
bundle.obj.save()
bundle.objects_saved.add(obj_id)
# Now pick up the M2M bits.
m2m_bundle = self.hydrate_m2m(bundle)
self.save_m2m(m2m_bundle)
return bundle
Needs to be overridden in your class, so that you can change the Django save(), which has the update_fields parameter we want to modify:
if obj_id not in bundle.objects_saved or bundle.obj._state.adding:
bundle.obj.save()
To, for example:
class ResourcesResource(ModelResource):
# ...
def save(self, bundle, skip_errors=False):
# ...
if obj_id not in bundle.objects_saved or bundle.obj._state.adding:
resource_fields = [field.name for field in Resources._meta.get_fields()
if not field.name in ['id', 'last_updated']]
bundle.obj.save(update_fields=resource_fields)
# ...
This properly excludes the last_updated column from the sql UPDATE.
I'm building a simple database driven blog with Flask and SQLAlchemy. In the model for the blog postings I define title and slug attributes:
class BlogPost(Model):
...
title = Column(String(80))
slug = Column(String(80), unique=True)
Later I use an event listener to automatically create and insert a slug from the title:
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
target.slug = slugify(value)
As expected, if I try to add a post to the database, and the title of the post evaluates to the same slug as a previous post, then the transaction fails with an IntegrityError. I don't think in practice this will be a problem anyway. But just for giggles I tried something like this:
from sqlalchemy.exc import IntegrityError
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
try:
target.slug = slugify(value)
except IntegrityError:
target.slug = slugify(value) + random_string()
random_string could be anything, really, the point is that nothing that I've tried gets executed because the IntegrityError isn't getting caught, and I'm not sure why - attempting to add & commit a post to the database with the same title still raises an IntegrityError and aborts the transaction when I try to commit. I've seen a handful of other posts about it, but the answers are mostly pretty specific to Pyramid, which I'm not using.
Anybody know what I'm missing here?
Tech involved: Python3, Flask, Flask-Sqlalchemy, Sqlalchemy
SQLAlchemy will not flush changes to model objects to DB when setting. In order to get the error you have to do something like
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm.session import object_session
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
session = object_session(target)
try:
with session.begin_nested():
target.slug = slugify(value)
session.flush()
except IntegrityError:
target.slug = slugify(value) + random_string()
Note that you have to wrap your possible integrity violation in a nested transaction (a savepoint), or your whole transaction will fail even though you catch the IntegrityError. If your DB doesn't support savepoints or an SQLAlchemy implementation of the idea, you're out of luck.
I'm aware there are many other questions about the exact same issue, but I've tried their answers and none have worked so far.
I'm trying to delete records from a table which has relationships with other tables. The foreign key in those tables are nullable=false, so trying to delete a record which is in use by another table should raise an exception.
But even when surrounding the delete statement with a catchall try-except the error is still not caught, so I suspect the exception may be raised somewhere else.
I'm using SQLite with SQLAlchemy in a Pyramid framework, and my session is configured with the ZopeTransactionExtension.
This is how I'm trying to delete:
In views.py
from sqlalchemy.exc import IntegrityError
from project.app.models import (
DBSession,
foo)
#view_config(route_name='fooview', renderer='json', permission='view')
def fooview(request):
""" The fooview handles different cases for foo
depending on the http method
"""
if request.method == 'DELETE':
if not request.has_permission('edit'):
return HTTPForbidden()
deleteid = request.matchdict['id']
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
try:
qry = DBSession.delete(deletethis)
transaction.commit()
if qry == 0:
return HTTPNotFound(text=u'Foo not found')
except IntegrityError:
DBSession.rollback()
return HTTPConflict(text=u'Foo in use')
return HTTPOk()
In models.py I set up DBSession and my models:
from zope.sqlalchemy import ZopeTransactionExtension
from sqlalchemy.orm import (
scoped_session,
sessionmaker,
relationship,
backref,
)
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension('changed')))
Base = declarative_base()
class foo(Base):
""" foo defines a unit used by bar
"""
__tablename__ = 'foo'
id = Column(Integer, primary_key=True)
name = Column(Text(50))
bars = relationship('bar')
class bar(Base):
__tablename__ = 'bar'
id = Column(Integer, primary_key=True)
fooId = Column(Integer, ForeignKey('foo.id'), nullable=False)
foo = relationship('foo')
And in __init__.py I configure my session like so:
from project.app.models import (
DBSession,
Base,
)
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
engine = engine_from_config(settings, 'sqlalchemy.')
# fix for association_table cascade delete issues
engine.dialect.supports_sane_rowcount = engine.dialect.supports_sane_multi_rowcount = False
DBSession.configure(bind=engine)
Base.metadata.bind = engine
Using this setup I get
IntegrityError: (IntegrityError) NOT NULL constraint failed
Traceback here.
If I replace transaction.commit() with DBSession.flush(), I get
ResourceClosedError: This transaction is closed
And if I remove the transaction.commit(), I still get the same error, but without a clear point of origin.
UPDATE:
I ran some nose tests, and in some cases, but not all, the exception was handled correctly.
In my tests I import the session and configure it:
from optimate.app.models import (
DBSession,
Base,
foo)
def _initTestingDB():
""" Build a database with default data
"""
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
DBSession.configure(bind=engine)
with transaction.manager:
# add test data
class TestFoo(unittest.TestCase):
def setUp(self):
self.config = testing.setUp()
self.session = _initTestingDB()
def tearDown(self):
DBSession.remove()
testing.tearDown()
def _callFUT(self, request):
from project.app.views import fooview
return fooview(request)
def test_delete_foo_keep(self):
request = testing.DummyRequest()
request.method = 'DELETE'
request.matchdict['id'] = 1
response = self._callFUT(request)
# foo is used so it is not deleted
self.assertEqual(response.code, 409)
def test_delete_foo_remove(self):
_registerRoutes(self.config)
request = testing.DummyRequest()
request.method = 'DELETE'
request.matchdict['id'] = 2
response = self._callFUT(request)
# foo is not used so it is deleted
self.assertEqual(response.code, 200)
Does anyone know what's going on?
Might be you just "doing it wrong". Your question addresses two issues. Handling transaction level errors raised by database integrity errors and modelling application code/models/queries to implement business logic. My answer focuses on writing code that fits common patterns while using pyramid_tm for transaction management and sqlalchemy as an ORM.
In Pyramid, if you've configured your session (which the scaffold does for you automatically) to use the ZopeTransactionExtension, then session is not flushed/committed until after the view has executed. If you want to catch any SQL errors yourself in your view, you need to force a flush to send the SQL to the engine. DBSession.flush() should do it after the delete(...).
If you raise any of 4xx/5xx HTTP return codes like pyramid exception HTTPConflict the transaction will be aborted.
#view_config(route_name='fooview', renderer='json', permission='view')
def fooview(request):
""" The fooview handles different cases for foo
depending on the http method
"""
if request.method == 'DELETE':
if not request.has_permission('edit'):
return HTTPForbidden()
deleteid = request.matchdict['id']
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
if not deletethis:
raise HTTPNotFound()
try:
DBSession.delete(deletethis)
DBSession.flush()
except IntegrityError as e:
log.debug("delete operation not possible for id {0}".format(deleteid)
raise HTTPConflict(text=u'Foo in use')
return HTTPOk()
This excerpt from todopyramid/models.py highlights how to delete a collection item without using DBSession object.
def delete_todo(self, todo_id):
"""given a todo ID we delete it is contained in user todos
delete from a collection
http://docs.sqlalchemy.org/en/latest/orm/session.html#deleting-from-collections
https://stackoverflow.com/questions/10378468/deleting-an-object-from-collection-in-sqlalchemy"""
todo_item = self.todo_list.filter(
TodoItem.id == todo_id)
todo_item.delete()
This sample code from pyramid_blogr show clearly how simple pyramid view code to delete SQL database items could look like. Usually you do not have to interact with the transaction. This is a feature - as advertised as one the unique feature of pyramid. Just pick any of the available pyramid tutorials that use sqlalchemy and try to stick to the patterns as much as possible. If you address the problem at the application model level the transaction machinery will hide in the background unless you have a clear need for its services.
#view_config(route_name='blog_action', match_param="action=delete", permission='delete')
def blog_delete(request):
entry_id = request.params.get('id', -1)
entry = Entry.by_id(entry_id)
if not entry:
return HTTPNotFound()
DBSession.delete(entry)
return HTTPFound(location=request.route_url('home'))
To provide meaningful error messages to application users you either catch errors on database contraints at database model layer or at pyramid view layer. Catching sqlalchemy exceptions to provide error messages could look like in this sample code
from sqlalchemy.exc import OperationalError as SqlAlchemyOperationalError
#view_config(context=SqlAlchemyOperationalError)
def failed_sqlalchemy(exception, request):
"""catch missing database, logout and redirect to homepage, add flash message with error
implementation inspired by pylons group message
https://groups.google.com/d/msg/pylons-discuss/BUtbPrXizP4/0JhqB2MuoL4J
"""
msg = 'There was an error connecting to database'
request.session.flash(msg, queue='error')
headers = forget(request)
# Send the user back home, everything else is protected
return HTTPFound(request.route_url('home'), headers=headers)
References
Trying to catch integrity error with SQLAlchemy
pyramid_tm Usage
What the Zope Transaction Manager Means To Me (and you)
Not sure if this helps - I did not quite capture from the traceback what goes wrong, would need more time. But you can use transaction manager like this:
from sqlalchemy.exc import IntegrityError
try:
with transaction.manager:
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
qry = DBSession.delete(deletethis)
if qry == 0:
return HTTPNotFound()
# transaction.manager commits when with context manager exits here
except IntegrityError:
DBSession.rollback()
return HTTPConflict()
return HTTPOk()
I'm using Flask-SQLAlchemy to do a rather large bulk insert of 60k rows. I also have a many-to-many relationship on this table, so I can't use db.engine.execute for this. Before inserting, I need to find similar items in the database, and change the insert to an update if a duplicate item is found.
I could do this check beforehand, and then do a bulk insert via db.engine.execute, but I need the primary key of the row upon insertion.
Currently, I am doing a db.session.add() and db.session.commit() on each insert, and I get a measly 3-4 inserts per second.
I ran a profiler to see where the bottleneck is, and it seems that the db.session.commit() is taking 60% of the time.
Is there some way that would allow me to make this operation faster, perhaps by grouping commits, but which would give me primary keys back?
This is what my models looks like:
class Item(db.Model):
id = db.Column(db.Integer, primary_key=True)
title = db.Column(db.String(1024), nullable=True)
created = db.Column(db.DateTime())
tags_relationship = db.relationship('Tag', secondary=tags, backref=db.backref('items', lazy='dynamic'))
tags = association_proxy('tags_relationship', 'text')
class Tag(db.Model):
id = db.Column(db.Integer, primary_key=True)
text = db.Column(db.String(255))
My insert operation is:
for item in items:
if duplicate:
update_existing_item
else:
x = Item()
x.title = "string"
x.created = datetime.datetime.utcnow()
for tag in tags:
if not tag_already_exists:
y = Tag()
y.text = "tagtext"
x.tags_relationship.append(y)
db.session.add(y)
db.session.commit()
else:
x.tags_relationship.append(existing_tag)
db.session.add(x)
db.session.commit()
Perhaps you should try to db.session.flush() to send the data to the server, which means any primary keys will be generated. At the end you can db.session.commit() to actually commit the transaction.
I use the following code to quickly read the content of a pandas DataFrame into SQLite. Note that it circumvents the ORM features of SQLAlchemy. myClass in this context is a db.Model derived class that has a tablename assigned to it. As the code snippets mentions, I adapted
l = df.to_dict('records')
# bulk save the dictionaries, circumventing the slow ORM interface
# c.f. https://gist.github.com/shrayasr/5df96d5bc287f3a2faa4
connection.engine.execute(
myClass.__table__.insert(),
l
)
from app import db
data = [{"attribute": "value"}, {...}, {...}, ... ]
db.engine.execute(YourModel.__table__.insert(), data)
for more information refer https://gist.github.com/shrayasr/5df96d5bc287f3a2faa4