I'm aware there are many other questions about the exact same issue, but I've tried their answers and none have worked so far.
I'm trying to delete records from a table which has relationships with other tables. The foreign key in those tables are nullable=false, so trying to delete a record which is in use by another table should raise an exception.
But even when surrounding the delete statement with a catchall try-except the error is still not caught, so I suspect the exception may be raised somewhere else.
I'm using SQLite with SQLAlchemy in a Pyramid framework, and my session is configured with the ZopeTransactionExtension.
This is how I'm trying to delete:
In views.py
from sqlalchemy.exc import IntegrityError
from project.app.models import (
DBSession,
foo)
#view_config(route_name='fooview', renderer='json', permission='view')
def fooview(request):
""" The fooview handles different cases for foo
depending on the http method
"""
if request.method == 'DELETE':
if not request.has_permission('edit'):
return HTTPForbidden()
deleteid = request.matchdict['id']
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
try:
qry = DBSession.delete(deletethis)
transaction.commit()
if qry == 0:
return HTTPNotFound(text=u'Foo not found')
except IntegrityError:
DBSession.rollback()
return HTTPConflict(text=u'Foo in use')
return HTTPOk()
In models.py I set up DBSession and my models:
from zope.sqlalchemy import ZopeTransactionExtension
from sqlalchemy.orm import (
scoped_session,
sessionmaker,
relationship,
backref,
)
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension('changed')))
Base = declarative_base()
class foo(Base):
""" foo defines a unit used by bar
"""
__tablename__ = 'foo'
id = Column(Integer, primary_key=True)
name = Column(Text(50))
bars = relationship('bar')
class bar(Base):
__tablename__ = 'bar'
id = Column(Integer, primary_key=True)
fooId = Column(Integer, ForeignKey('foo.id'), nullable=False)
foo = relationship('foo')
And in __init__.py I configure my session like so:
from project.app.models import (
DBSession,
Base,
)
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
engine = engine_from_config(settings, 'sqlalchemy.')
# fix for association_table cascade delete issues
engine.dialect.supports_sane_rowcount = engine.dialect.supports_sane_multi_rowcount = False
DBSession.configure(bind=engine)
Base.metadata.bind = engine
Using this setup I get
IntegrityError: (IntegrityError) NOT NULL constraint failed
Traceback here.
If I replace transaction.commit() with DBSession.flush(), I get
ResourceClosedError: This transaction is closed
And if I remove the transaction.commit(), I still get the same error, but without a clear point of origin.
UPDATE:
I ran some nose tests, and in some cases, but not all, the exception was handled correctly.
In my tests I import the session and configure it:
from optimate.app.models import (
DBSession,
Base,
foo)
def _initTestingDB():
""" Build a database with default data
"""
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
DBSession.configure(bind=engine)
with transaction.manager:
# add test data
class TestFoo(unittest.TestCase):
def setUp(self):
self.config = testing.setUp()
self.session = _initTestingDB()
def tearDown(self):
DBSession.remove()
testing.tearDown()
def _callFUT(self, request):
from project.app.views import fooview
return fooview(request)
def test_delete_foo_keep(self):
request = testing.DummyRequest()
request.method = 'DELETE'
request.matchdict['id'] = 1
response = self._callFUT(request)
# foo is used so it is not deleted
self.assertEqual(response.code, 409)
def test_delete_foo_remove(self):
_registerRoutes(self.config)
request = testing.DummyRequest()
request.method = 'DELETE'
request.matchdict['id'] = 2
response = self._callFUT(request)
# foo is not used so it is deleted
self.assertEqual(response.code, 200)
Does anyone know what's going on?
Might be you just "doing it wrong". Your question addresses two issues. Handling transaction level errors raised by database integrity errors and modelling application code/models/queries to implement business logic. My answer focuses on writing code that fits common patterns while using pyramid_tm for transaction management and sqlalchemy as an ORM.
In Pyramid, if you've configured your session (which the scaffold does for you automatically) to use the ZopeTransactionExtension, then session is not flushed/committed until after the view has executed. If you want to catch any SQL errors yourself in your view, you need to force a flush to send the SQL to the engine. DBSession.flush() should do it after the delete(...).
If you raise any of 4xx/5xx HTTP return codes like pyramid exception HTTPConflict the transaction will be aborted.
#view_config(route_name='fooview', renderer='json', permission='view')
def fooview(request):
""" The fooview handles different cases for foo
depending on the http method
"""
if request.method == 'DELETE':
if not request.has_permission('edit'):
return HTTPForbidden()
deleteid = request.matchdict['id']
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
if not deletethis:
raise HTTPNotFound()
try:
DBSession.delete(deletethis)
DBSession.flush()
except IntegrityError as e:
log.debug("delete operation not possible for id {0}".format(deleteid)
raise HTTPConflict(text=u'Foo in use')
return HTTPOk()
This excerpt from todopyramid/models.py highlights how to delete a collection item without using DBSession object.
def delete_todo(self, todo_id):
"""given a todo ID we delete it is contained in user todos
delete from a collection
http://docs.sqlalchemy.org/en/latest/orm/session.html#deleting-from-collections
https://stackoverflow.com/questions/10378468/deleting-an-object-from-collection-in-sqlalchemy"""
todo_item = self.todo_list.filter(
TodoItem.id == todo_id)
todo_item.delete()
This sample code from pyramid_blogr show clearly how simple pyramid view code to delete SQL database items could look like. Usually you do not have to interact with the transaction. This is a feature - as advertised as one the unique feature of pyramid. Just pick any of the available pyramid tutorials that use sqlalchemy and try to stick to the patterns as much as possible. If you address the problem at the application model level the transaction machinery will hide in the background unless you have a clear need for its services.
#view_config(route_name='blog_action', match_param="action=delete", permission='delete')
def blog_delete(request):
entry_id = request.params.get('id', -1)
entry = Entry.by_id(entry_id)
if not entry:
return HTTPNotFound()
DBSession.delete(entry)
return HTTPFound(location=request.route_url('home'))
To provide meaningful error messages to application users you either catch errors on database contraints at database model layer or at pyramid view layer. Catching sqlalchemy exceptions to provide error messages could look like in this sample code
from sqlalchemy.exc import OperationalError as SqlAlchemyOperationalError
#view_config(context=SqlAlchemyOperationalError)
def failed_sqlalchemy(exception, request):
"""catch missing database, logout and redirect to homepage, add flash message with error
implementation inspired by pylons group message
https://groups.google.com/d/msg/pylons-discuss/BUtbPrXizP4/0JhqB2MuoL4J
"""
msg = 'There was an error connecting to database'
request.session.flash(msg, queue='error')
headers = forget(request)
# Send the user back home, everything else is protected
return HTTPFound(request.route_url('home'), headers=headers)
References
Trying to catch integrity error with SQLAlchemy
pyramid_tm Usage
What the Zope Transaction Manager Means To Me (and you)
Not sure if this helps - I did not quite capture from the traceback what goes wrong, would need more time. But you can use transaction manager like this:
from sqlalchemy.exc import IntegrityError
try:
with transaction.manager:
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
qry = DBSession.delete(deletethis)
if qry == 0:
return HTTPNotFound()
# transaction.manager commits when with context manager exits here
except IntegrityError:
DBSession.rollback()
return HTTPConflict()
return HTTPOk()
Related
I am working on testing my flask app model. I'm using mysql 5.7, sqlalchemy and pytest.
Within my model, I have a CRUD mixin that I used to manage creating, updating and deleting. Whenever I try to access the object in the Mixin before returning the object to the test function, SQLAlchemy hangs at db.drop_all in my tear down. When I look in mysql at PROCESSLIST, it shows 1 sleep query and 1 query waiting for table metadata lock.
I can fix this by calling db.session.commit in the create method in the mixin before returning the object. However, if I call it at the test teardown (or in the main test function), it doesn't work. I'd prefer not to add an extra commit just to make my tests work as it doesn't feel correct. Does anyone know why this is happening or have any suggested fixes?
models.py
class CRUDMixin(object):
#classmethod
def create(cls, **kwargs):
instance = cls(**kwargs)
saved_instance = instance.save()
# do stuff with saved_instance (i.e. add to full text search engine)
# db.drop_all in teardown works if add db.session.commit() here
return saved_instance
def save(self, commit=True):
db.session.add(self)
if commit:
try:
db.session.commit()
except Exception:
db.session.rollback()
raise
return self
class User(CRUDMixin, db.model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(50))
conftest.py
#pytest.fixture(scope='session')
def app():
app = create_app(TestConfig)
ctx = app.app_context()
ctx.push()
yield app
ctx.pop()
#pytest.fixture(scope='session')
def test_db(app):
db.drop_all()
db.create_all()
# add test db information
yield db
db.session.remove()
db.drop_all() # test hangs within drop all
#pytest.fixture(scope='function')
def db_session(test_db):
connection = db.engine.connect()
transaction = connection.begin()
options = dict(bind=connection, binds={})
session = db.create_scoped_session(options)
db.session = session
yield db
db.session.remove() # tables won't drop if I put db.session.commit before the remove call
transaction.rollback()
connection.close() # even though connection closes, mysql still shows process
test_models.py
class TestUser(object):
def test_add_new(self, db_session):
u = User.create(name='test_name')
assert u.name == 'test_name'
# if I put db.session.commit() here, tables won't drop
What I know:
We all know that flask has a useful query.get_or_404 we can call it to any class object and return the object or raise a 404 error if the object is not found.
The problem:
I have a very large application and using that function for my queries, but it becomes now a bit confusing when I'm sending data to the front end and other application consuming my APIs.
When an object they are querying is not found it returns 404 and the same behavior happens also when a page is not found.
What I want to achieve:
I would like to have a meaningful response that is different for every object that is not found, and different for the normal 404 error message.
Example :
if I have this query:
user = User.query.get_or_404(id)
if the user is not found I want to raise an HTTP error and return a message like user not found
What I have tried So far:
user = User.query.get(id)
if user:
#do something
else
return {'status':'01', 'description': 'user not found'}
# or raise a http error
The problem with this approach I can't maintain the application I'm working on, it will require me to change everywhere I was using get_or_404 with that code.
What I'm thinking about :
Creating a function like query.get_or_404but with another status message
like query.get_or_415 for example and add an error handler for 415 HTTP code so that if an object is not found it can return {'status':'0message:ge : 'object of the class is not found'}
How can I achieve it?
I have checked for that function in Flask but was unable to find it
Anyone with suggestions?
As noted by bereal, you should indeed use flask-SQLAlchemy's BaseQuery and add your own functionality from there. I'll check to see if I can add some messaging system to this approach as well.
import json
from flask import Flask, abort, Response
from flask_sqlalchemy import SQLAlchemy, BaseQuery
class CustomBaseQuery(BaseQuery):
def get_or_415(self, ident):
model_class_name = ''
try:
model_class_name = self._mapper_zero().class_.__name__
except Exception as e:
print(e)
rv = self.get(ident)
if not rv:
error_message = json.dumps({'message': model_class_name + ' ' + str(ident) + ' not found'})
abort(Response(error_message, 415))
return rv
app = Flask(__name__)
db = SQLAlchemy(app, query_class=CustomBaseQuery)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True, nullable=False)
db.create_all()
user = User(username='foo')
db.session.add(user)
db.session.commit()
#app.route('/')
def index():
User.query.get_or_415(1)
User.query.get_or_415(2)
return ''
When going to the index, it returns:
{"message": "User 2 not found"}
Based on this answer
You can write like this:
from sqlalchemy.orm import Query
class MyQuery(Query):
def get_or_415(self, pk):
instance = self.get(pk)
if not instance:
raise HttpException(code=415)
return instance
Then you can use flask error handling to handle your exception and response the way you want
Description
I have an Flask application with original SQLalchemy. Application is intended to be used internally in a company for easier saving of measurement data with MySQL
On one page I have a table with all devices used for measurement and a form that is used to add, remove or modify measurement devices.
Problem
The problem is that when I enter a new device in the database, the page is automatically refreshed to fetch new data from DB and new device is sometimes shown and sometimes it is not when I refresh the page. In other words, added row in table is appearing and dissapearing even though the row is visible on database. Same goes when i try to delete the device from database. The row is sometimes shown, sometimes not when refreshing the page with row being deleted from DB.
The same problem appears for all examples similar to this one (adding, deleting and modifying data).
What i have tried
Bellow is the code for table model:
class DvDevice(Base):
__tablename__ = "dvdevice"
id = Column("device_id", Integer, primary_key=True, autoincrement=True)
name = Column("device_name", String(50), nullable=True)
code = Column("device_code", String(10), nullable=True, unique=True)
hw_ver = Column("hw_ver", String(10), nullable=True)
fw_ver = Column("fw_ver", String(10), nullable=True)
sw_ver = Column("sw_ver", String(10), nullable=True)
And here is the code that inserts/deletes data from table.
#Insertion
device = DvDevice()
device.code = self.device_code
device.name = self.device_name
device.hw_ver = self.hw_ver
device.fw_ver = self.fw_ver
device.sw_ver = self.sw_ver
ses.add(device)
ses.commit()
ses.expire_all() #Should this be here?
# Deletion
ses.query(DvDevice).filter_by(id=self.device_id).delete()
ses.commit()
ses.expire_all() # Should this be here?
I have read from some posts on stack to include the following decorator function in models.py
#app.teardown_appcontext
def shutdown_session(exception=None):
ses.expire_all() #ses being database session object.
I tried this and it still doesn't work as it should be. Should I put the decorator function somewhere else?
Second thing i tried is to put ses.expire_all() after all commits and it still doesnt work.
What should I do to prevent this from happening?
Edit 1
from sqlalchemy import create_engine, update
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import NullPool
from config import MYSQLCONNECT
engine = create_engine(MYSQLCONNECT)
Session = sessionmaker(bind=engine)
session = Session()
I solved the problem with the use of following function from http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it:
from contextlib import contextmanager
#contextmanager
def session_scope():
"""Provide a transactional scope around a series of operations."""
session = Session()
try:
yield session
session.commit()
except:
session.rollback()
raise
finally:
session.close()
with session_scope() as session:
... # code that uses session
The problem was that I created the session object in the beggining and then never closed it.
I'm building a simple database driven blog with Flask and SQLAlchemy. In the model for the blog postings I define title and slug attributes:
class BlogPost(Model):
...
title = Column(String(80))
slug = Column(String(80), unique=True)
Later I use an event listener to automatically create and insert a slug from the title:
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
target.slug = slugify(value)
As expected, if I try to add a post to the database, and the title of the post evaluates to the same slug as a previous post, then the transaction fails with an IntegrityError. I don't think in practice this will be a problem anyway. But just for giggles I tried something like this:
from sqlalchemy.exc import IntegrityError
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
try:
target.slug = slugify(value)
except IntegrityError:
target.slug = slugify(value) + random_string()
random_string could be anything, really, the point is that nothing that I've tried gets executed because the IntegrityError isn't getting caught, and I'm not sure why - attempting to add & commit a post to the database with the same title still raises an IntegrityError and aborts the transaction when I try to commit. I've seen a handful of other posts about it, but the answers are mostly pretty specific to Pyramid, which I'm not using.
Anybody know what I'm missing here?
Tech involved: Python3, Flask, Flask-Sqlalchemy, Sqlalchemy
SQLAlchemy will not flush changes to model objects to DB when setting. In order to get the error you have to do something like
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm.session import object_session
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
session = object_session(target)
try:
with session.begin_nested():
target.slug = slugify(value)
session.flush()
except IntegrityError:
target.slug = slugify(value) + random_string()
Note that you have to wrap your possible integrity violation in a nested transaction (a savepoint), or your whole transaction will fail even though you catch the IntegrityError. If your DB doesn't support savepoints or an SQLAlchemy implementation of the idea, you're out of luck.
I'm starting to write tests with Flask-SQLAlchemy, and I'd like to add some fixtures for those. I have plenty of good data for that in my development database and a lot of tables so writing data manually would get annoying. I'd really like to just sample data from the dev database into fixtures and then use those. What's a good way to do this?
i would use factory boy
to create a model factory you just do:
import factory
from . import models
class UserFactory(factory.Factory):
class Meta:
model = models.User
first_name = 'John'
last_name = 'Doe'
admin = False
then to create instances:
UserFactory.create()
to add static data just give as kwarg to create
UserFactory.create(name='hank')
so to seed a bunch of stuff throw that in a for loop. :)
If you need to handle fixtures with SQLAlchemy or another ORM/backend then the Fixture package may be of use: Flask-Fixtures 0.3.3
That is a simple library that allows you to add database fixtures for your unit tests using nothing but JSON or YAML.
While Kyle's answer is correct, we still need to provide the model factory with a database session, otherwise we would never actually commit to the db. Also, factory boy has a dedicated class SQLAlchemyModelFactory for interacting with SQLAlchemy.
https://factoryboy.readthedocs.io/en/stable/orms.html#sqlalchemy
The whole setup could look something like this:
import pytest
import os
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from factory.alchemy import SQLAlchemyModelFactory
engine = create_engine( os.getenv("SQLALCHEMY_DATABASE_URI"))
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
# this resets our tables in between each test
def _reset_schema():
db = SessionLocal()
for table in Base.metadata.sorted_tables:
db.execute(
'TRUNCATE {name} RESTART IDENTITY CASCADE;'.format(name=table.name)
)
db.commit()
#pytest.fixture
def test_db():
yield engine
engine.dispose()
_reset_schema()
#pytest.fixture
def session(test_db):
connection = test_db.connect()
transaction = connection.begin()
db = scoped_session(sessionmaker(bind=engine))
try:
yield db
finally:
db.close()
transaction.rollback()
connection.close()
db.remove()
class UserFactory(SQLAlchemyModelFactory):
class Meta:
model = models.User
first_name = 'John'
last_name = 'Doe'
admin = False
#pytest.fixture(autouse=True)
def provide_session_to_factories(session):
# usually you'd have one factory for each db table
for factory in [UserFactory, ...]:
factory._meta.sqlalchemy_session = session