The following static method (taken from flasky) is a way to populate the database with fake data, and I want to import it to Django.
models.py
class User(UserMixin, db.Model):
# ......
#staticmethod
def generate_fake(count=100):
from sqlalchemy.exc import IntegrityError
from random import seed
import forgery_py
seed()
for i in range(count):
u = User(email=forgery_py.internet.email_address(),
username=forgery_py.internet.user_name(True),
password=forgery_py.lorem_ipsum.word(),
confirmed=True,
name=forgery_py.name.full_name(),
location=forgery_py.address.city(),
about_me=forgery_py.lorem_ipsum.sentence(),
member_since=forgery_py.date.date(True))
db.session.add(u)
try:
db.session.commit()
except IntegrityError:
db.session.rollback()
The problem is, while I can do something like User.objects.create(...), I don't know how to rollback the database in case an IntegrityError happens (presumable due to a duplicate primary key).
By default, Django makes SQL queries in auto-commit mode, where every query is wrapped in a transaction, so you should not worry about an integrity error messing up your database. If an integrity error happens, Django won't insert the data.
To be extra-safe, you can wrap the creation code in a transaction, that in Django, is implemented as a python context handler:
from django.db import transaction
with transaction.atomic():
User.objects.create(...)
But that would be unnecessary! You can read more about Transactions and how Django handles them in this documentation page.
Related
I would like to get some idea on how to improve (if any) my code in implementing transaction based query in Django.
This is how I understand the ATOMIC_REQUEST I read on django documentation. I have this function view:
from django.db import transaction
import sys
#transaction.atomic
def save_progress(request):
try:
with atomic.transaction():
qs = AllProgressModel()
qs.name = 'level up'
qs.level = 25
qs.description = 'Increased 2 level'
qs.save()
except:
print(sys.exc_info())
-Am I doing it right?
-Is the progress will be saved or not if the connection lost occur during saving?
Thank you in advance!
You don't need both decorator #transaction.atomic and with atomic.transaction(), one is usually enough.
While using with atomic.transaction(), catch IntegrityError exceptions instead of broadly handling all exceptions at once.
Edit: If you're handling exceptions outside the atomic block (like below) then it's a better practice to have an outer atomic wrap as well, for handling rollbacks and other database operations you might need in exception handling part.
from django.db import IntegrityError, transaction
def save_progress(request):
try:
# with atomic.transaction() -> produce error ( typo )
with transaction.atomic():
...
qs.name = 'level up'
qs.level = 25
qs.description = 'Increased 2 level'
qs.save()
except IntegrityError:
# You are here if something goes within the transaction, after rollback
# HANDLE exception
I'd like to notice whenever a model is saved and then do some processing and save another model. I need the model to already have an ID set by the database in the processing stage.
With Django one would override the .save() method of model or use signals like:
from django.db.models.signals import post_save
from django.dispatch import receiver
from .models import MyModel, OtherModel
#receiver(post_save, sender=MyModel)
def do_stuff(sender, instance, created, **kwargs):
assert instance.id is not None
...
OtherModel.create(related=instance, data=...)
How to do similar with SQLAlchemy and Flask? I looked up ORM Events and it seemed that expire IntanceEvent would fit the bill. It seems to fire whenever a model instance is saved but when I try to do the same kind of thing:
from sqlalchemy import event
from . import db
from .models import MyModel, OtherModel
#event.listens_for(MyModel, "expire")
def do_stuff(target, attrs):
assert target.id is not None
...
db.session.add(OtherModel(related=target, data=...))
db.session.commit()
It fails on assert instance.id is not None with:
InvalidRequestError: This session is in 'committed' state; no further SQL can be emitted within this transaction.
It might be that I'm just approaching this the wrong way or I'm missing something crucial but I cannot figure it out. The documentation is split among Flask, Flask-SQLAlchemy and SQLAlchemy and I have hard time piecing this together.
How should I make this kind of post save trigger with SQLAlchemy?
The event you want to listen for is 'after_insert', not 'expire':
#event.listens_for(MyModel, 'after_insert')
def do_stuff(mapper, connection, target):
assert target.id is not None
...
Also, after creating OtherModel inside the listener and calling db.session.add, don't call db.session.commit as it will throw a ResourceClosedError exception.
Have a look at the accepted answer to this question which gives an example of using SQLAlchemy's after_insert mapper event. It should do what you want, but using raw SQL rather than your session object is recommended.
I'm building a simple database driven blog with Flask and SQLAlchemy. In the model for the blog postings I define title and slug attributes:
class BlogPost(Model):
...
title = Column(String(80))
slug = Column(String(80), unique=True)
Later I use an event listener to automatically create and insert a slug from the title:
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
target.slug = slugify(value)
As expected, if I try to add a post to the database, and the title of the post evaluates to the same slug as a previous post, then the transaction fails with an IntegrityError. I don't think in practice this will be a problem anyway. But just for giggles I tried something like this:
from sqlalchemy.exc import IntegrityError
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
try:
target.slug = slugify(value)
except IntegrityError:
target.slug = slugify(value) + random_string()
random_string could be anything, really, the point is that nothing that I've tried gets executed because the IntegrityError isn't getting caught, and I'm not sure why - attempting to add & commit a post to the database with the same title still raises an IntegrityError and aborts the transaction when I try to commit. I've seen a handful of other posts about it, but the answers are mostly pretty specific to Pyramid, which I'm not using.
Anybody know what I'm missing here?
Tech involved: Python3, Flask, Flask-Sqlalchemy, Sqlalchemy
SQLAlchemy will not flush changes to model objects to DB when setting. In order to get the error you have to do something like
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm.session import object_session
#event.listens_for(BlogPost.title, 'set')
def autoslug(target, value, oldvalue, initiator):
session = object_session(target)
try:
with session.begin_nested():
target.slug = slugify(value)
session.flush()
except IntegrityError:
target.slug = slugify(value) + random_string()
Note that you have to wrap your possible integrity violation in a nested transaction (a savepoint), or your whole transaction will fail even though you catch the IntegrityError. If your DB doesn't support savepoints or an SQLAlchemy implementation of the idea, you're out of luck.
I'm aware there are many other questions about the exact same issue, but I've tried their answers and none have worked so far.
I'm trying to delete records from a table which has relationships with other tables. The foreign key in those tables are nullable=false, so trying to delete a record which is in use by another table should raise an exception.
But even when surrounding the delete statement with a catchall try-except the error is still not caught, so I suspect the exception may be raised somewhere else.
I'm using SQLite with SQLAlchemy in a Pyramid framework, and my session is configured with the ZopeTransactionExtension.
This is how I'm trying to delete:
In views.py
from sqlalchemy.exc import IntegrityError
from project.app.models import (
DBSession,
foo)
#view_config(route_name='fooview', renderer='json', permission='view')
def fooview(request):
""" The fooview handles different cases for foo
depending on the http method
"""
if request.method == 'DELETE':
if not request.has_permission('edit'):
return HTTPForbidden()
deleteid = request.matchdict['id']
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
try:
qry = DBSession.delete(deletethis)
transaction.commit()
if qry == 0:
return HTTPNotFound(text=u'Foo not found')
except IntegrityError:
DBSession.rollback()
return HTTPConflict(text=u'Foo in use')
return HTTPOk()
In models.py I set up DBSession and my models:
from zope.sqlalchemy import ZopeTransactionExtension
from sqlalchemy.orm import (
scoped_session,
sessionmaker,
relationship,
backref,
)
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension('changed')))
Base = declarative_base()
class foo(Base):
""" foo defines a unit used by bar
"""
__tablename__ = 'foo'
id = Column(Integer, primary_key=True)
name = Column(Text(50))
bars = relationship('bar')
class bar(Base):
__tablename__ = 'bar'
id = Column(Integer, primary_key=True)
fooId = Column(Integer, ForeignKey('foo.id'), nullable=False)
foo = relationship('foo')
And in __init__.py I configure my session like so:
from project.app.models import (
DBSession,
Base,
)
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
engine = engine_from_config(settings, 'sqlalchemy.')
# fix for association_table cascade delete issues
engine.dialect.supports_sane_rowcount = engine.dialect.supports_sane_multi_rowcount = False
DBSession.configure(bind=engine)
Base.metadata.bind = engine
Using this setup I get
IntegrityError: (IntegrityError) NOT NULL constraint failed
Traceback here.
If I replace transaction.commit() with DBSession.flush(), I get
ResourceClosedError: This transaction is closed
And if I remove the transaction.commit(), I still get the same error, but without a clear point of origin.
UPDATE:
I ran some nose tests, and in some cases, but not all, the exception was handled correctly.
In my tests I import the session and configure it:
from optimate.app.models import (
DBSession,
Base,
foo)
def _initTestingDB():
""" Build a database with default data
"""
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
DBSession.configure(bind=engine)
with transaction.manager:
# add test data
class TestFoo(unittest.TestCase):
def setUp(self):
self.config = testing.setUp()
self.session = _initTestingDB()
def tearDown(self):
DBSession.remove()
testing.tearDown()
def _callFUT(self, request):
from project.app.views import fooview
return fooview(request)
def test_delete_foo_keep(self):
request = testing.DummyRequest()
request.method = 'DELETE'
request.matchdict['id'] = 1
response = self._callFUT(request)
# foo is used so it is not deleted
self.assertEqual(response.code, 409)
def test_delete_foo_remove(self):
_registerRoutes(self.config)
request = testing.DummyRequest()
request.method = 'DELETE'
request.matchdict['id'] = 2
response = self._callFUT(request)
# foo is not used so it is deleted
self.assertEqual(response.code, 200)
Does anyone know what's going on?
Might be you just "doing it wrong". Your question addresses two issues. Handling transaction level errors raised by database integrity errors and modelling application code/models/queries to implement business logic. My answer focuses on writing code that fits common patterns while using pyramid_tm for transaction management and sqlalchemy as an ORM.
In Pyramid, if you've configured your session (which the scaffold does for you automatically) to use the ZopeTransactionExtension, then session is not flushed/committed until after the view has executed. If you want to catch any SQL errors yourself in your view, you need to force a flush to send the SQL to the engine. DBSession.flush() should do it after the delete(...).
If you raise any of 4xx/5xx HTTP return codes like pyramid exception HTTPConflict the transaction will be aborted.
#view_config(route_name='fooview', renderer='json', permission='view')
def fooview(request):
""" The fooview handles different cases for foo
depending on the http method
"""
if request.method == 'DELETE':
if not request.has_permission('edit'):
return HTTPForbidden()
deleteid = request.matchdict['id']
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
if not deletethis:
raise HTTPNotFound()
try:
DBSession.delete(deletethis)
DBSession.flush()
except IntegrityError as e:
log.debug("delete operation not possible for id {0}".format(deleteid)
raise HTTPConflict(text=u'Foo in use')
return HTTPOk()
This excerpt from todopyramid/models.py highlights how to delete a collection item without using DBSession object.
def delete_todo(self, todo_id):
"""given a todo ID we delete it is contained in user todos
delete from a collection
http://docs.sqlalchemy.org/en/latest/orm/session.html#deleting-from-collections
https://stackoverflow.com/questions/10378468/deleting-an-object-from-collection-in-sqlalchemy"""
todo_item = self.todo_list.filter(
TodoItem.id == todo_id)
todo_item.delete()
This sample code from pyramid_blogr show clearly how simple pyramid view code to delete SQL database items could look like. Usually you do not have to interact with the transaction. This is a feature - as advertised as one the unique feature of pyramid. Just pick any of the available pyramid tutorials that use sqlalchemy and try to stick to the patterns as much as possible. If you address the problem at the application model level the transaction machinery will hide in the background unless you have a clear need for its services.
#view_config(route_name='blog_action', match_param="action=delete", permission='delete')
def blog_delete(request):
entry_id = request.params.get('id', -1)
entry = Entry.by_id(entry_id)
if not entry:
return HTTPNotFound()
DBSession.delete(entry)
return HTTPFound(location=request.route_url('home'))
To provide meaningful error messages to application users you either catch errors on database contraints at database model layer or at pyramid view layer. Catching sqlalchemy exceptions to provide error messages could look like in this sample code
from sqlalchemy.exc import OperationalError as SqlAlchemyOperationalError
#view_config(context=SqlAlchemyOperationalError)
def failed_sqlalchemy(exception, request):
"""catch missing database, logout and redirect to homepage, add flash message with error
implementation inspired by pylons group message
https://groups.google.com/d/msg/pylons-discuss/BUtbPrXizP4/0JhqB2MuoL4J
"""
msg = 'There was an error connecting to database'
request.session.flash(msg, queue='error')
headers = forget(request)
# Send the user back home, everything else is protected
return HTTPFound(request.route_url('home'), headers=headers)
References
Trying to catch integrity error with SQLAlchemy
pyramid_tm Usage
What the Zope Transaction Manager Means To Me (and you)
Not sure if this helps - I did not quite capture from the traceback what goes wrong, would need more time. But you can use transaction manager like this:
from sqlalchemy.exc import IntegrityError
try:
with transaction.manager:
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
qry = DBSession.delete(deletethis)
if qry == 0:
return HTTPNotFound()
# transaction.manager commits when with context manager exits here
except IntegrityError:
DBSession.rollback()
return HTTPConflict()
return HTTPOk()
I'm using Flask, Flask-SQLAlchemy, Flask-Marshmallow + marshmallow-sqlalchemy, trying to implement REST api PUT method. I haven't found any tutorial using SQLA and Marshmallow implementing update.
Here is the code:
class NodeSchema(ma.Schema):
# ...
class NodeAPI(MethodView):
decorators = [login_required, ]
model = Node
def get_queryset(self):
if g.user.is_admin:
return self.model.query
return self.model.query.filter(self.model.owner == g.user)
def put(self, node_id):
json_data = request.get_json()
if not json_data:
return jsonify({'message': 'Invalid request'}), 400
# Here is part which I can't make it work for me
data, errors = node_schema.load(json_data)
if errors:
return jsonify(errors), 422
queryset = self.get_queryset()
node = queryset.filter(Node.id == node_id).first_or_404()
# Here I need some way to update this object
node.update(data) #=> raises AttributeError: 'Node' object has no attribute 'update'
# Also tried:
# node = queryset.filter(Node.id == node_id)
# node.update(data) <-- It doesn't if know there is any object
# Wrote testcase, when user1 tries to modify node of user2. Node doesn't change (OK), but user1 gets status code 200 (NOT OK).
db.session.commit()
return jsonify(), 200
UPDATED, 2022-12-08
Extending the ModelSchema from marshmallow-sqlalchemy instead of Flask-Marshmallow you can use the load method, which is defined like this:
load(data, *, session=None, instance=None, transient=False, **kwargs)
Putting that to use, it should look like that (or similar query):
node_schema.load(json_data, session= current_app.session, instance=Node().query.get(node_id))
And if you want to load without all required fields of Model, you can add the partial=True argument, like this:
node_schema.load(json_data, instance=Node().query.get(node_id), partial=True)
See the docs for more info (does not include definition of ModelSchema.load).
See the code for the load definition.
I wrestled with this issue for some time, and in consequence came back again and again to this post. In the end what made my situation difficult was that there was a confounding issue involving SQLAlchemy sessions. I figure this is common enough to Flask, Flask-SQLAlchemy, SQLAlchemy, and Marshmallow, to put down a discussion. I certainly, do not claim to be an expert on this, and yet I believe what I state below is essentially correct.
The db.session is, in fact, closely tied to the process of updating the DB with Marshmallow, and because of that decided to to give the details, but first the short of it.
Short Answer
Here is the answer I arrived at for updating the database using Marshmallow. It is a different approach from the very helpful post of Jair Perrut. I did look at the Marshmallow API and yet was unable to get his solution working in the code presented, because at the time I was experimenting with his solution I was not managing my SQLAlchemy sessions properly. To go a bit further, one might say that I wasn't managing them at all. The model can be updated in the following way:
user_model = user_schema.load(user)
db.session.add(user_model.data)
db.session.commit()
Give the session.add() a model with primary key and it will assume an update, leave the primary key out and a new record is created instead. This isn't all that surprising since MySQL has an ON DUPLICATE KEY UPDATE clause which performs an update if the key is present and creates if not.
Details
SQLAlchemy sessions are handled by Flask-SQLAlchemy during a request to the application. At the beginning of the request the session is opened, and when the request is closed that session is also closed. Flask provides hooks for setting up and tearing down the application where code for managing sessions and connections may be found. In the end, though, the SQLAlchemy session is managed by the developer, and Flask-SQLAlchemy just helps. Here is a particular case that illustrates the management of sessions.
Consider a function that gets a user dictionary as an argument and uses that with Marshmallow() to load the dictionary into a model. In this case, what is required is not the creation of a new object, but the update of an existing object. There are 2 things to keep in mind at the start:
The model classes are defined in a python module separate from any code, and these models require the session. Often the developer (Flask documentation) will put a line db = SQLAlchemy() at the head of this file to meet this requirement. This in fact, creates a session for the model.
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
In some other separate file there may be a need for a SQLAlchemy session as well. For example, the code may need to update the model, or create a new entry, by calling a function there. Here is where one might find db.session.add(user_model) and db.session.commit(). This session is created in the same way as in the bullet point above.
There are 2 SQLAlchemy sessions created. The model sits in one (SignallingSession) and the module uses its own (scoped_session). In fact, there are 3. The Marshmallow UserSchema has sqla_session = db.session: a session is attached to it. This then is the third, and the details are found in the code below:
from marshmallow_sqlalchemy import ModelSchema
from donate_api.models.donation import UserModel
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
class UserSchema(ModelSchema):
class Meta(object):
model = UserModel
strict = True
sqla_session = db.session
def some_function(user):
user_schema = UserSchema()
user['customer_id'] = '654321'
user_model = user_schema.load(user)
# Debug code:
user_model_query = UserModel.query.filter_by(id=3255161).first()
print db.session.object_session(user_model_query)
print db.session.object_session(user_model.data)
print db.session
db.session.add(user_model.data)
db.session.commit()
return
At the head of this module the model is imported, which creates its session, and then the module will create its own. Of course, as pointed out there is also the Marshmallow session. This is entirely acceptable to some degree because SQLAlchemy allows the developer to manage the sessions. Consider what happens when some_function(user) is called where user['id'] is assigned some value that exists in the database.
Since the user includes a valid primary key then db.session.add(user_model.data) knows that it is not creating a new row, but updating an existing one. This behavior should not be surprising, and is to be at least somewhat expected since from the MySQL documentation:
13.2.5.2 INSERT ... ON DUPLICATE KEY UPDATE Syntax
If you specify an ON DUPLICATE KEY UPDATE clause and a row to be inserted would cause a duplicate value in a UNIQUE index or PRIMARY KEY, an UPDATE of the old row occurs.
The snippet of code is then seen to be updating the customer_id on the dictionary for the user with primary key 32155161. The new customer_id is '654321'. The dictionary is loaded with Marshmallow and a commit done to the database. Examining the database it can be found that it was indeed updated. You might try two ways of verifying this:
In the code: db.session.query(UserModel).filter_by(id=325516).first()
In MySQL: select * from user
If you were to consider the following:
In the code: UserModel.query.filter_by(id=3255161).customer_id
You would find that the query brings back None. The model is not synchronized with the database. I have failed to manage our SQLAlchemy sessions correctly. In an attempt to bring clarity to this consider the output of the print statements when separate imports are made:
<sqlalchemy.orm.session.SignallingSession object at 0x7f81b9107b90>
<sqlalchemy.orm.session.SignallingSession object at 0x7f81b90a6150>
<sqlalchemy.orm.scoping.scoped_session object at 0x7f81b95eac50>
In this case the UserModel.query session is different from the Marshmallow session. The Marshmallow session is what gets loaded and added. This means that querying the model will not show our changes. In fact, if we do:
db.session.object_session(user_model.data).commit()
The model query will now bring back the updated customer_id! Consider the second alternative where the imports are done through flask_essentials:
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow
db = SQLAlchemy()
ma = Marshmallow()
<sqlalchemy.orm.session.SignallingSession object at 0x7f00fe227910>
<sqlalchemy.orm.session.SignallingSession object at 0x7f00fe227910>
<sqlalchemy.orm.scoping.scoped_session object at 0x7f00fed38710>
And the UserModel.query session is now the same as the user_model.data (Marshmallow) session. Now the UserModel.query does reflect the change in the database: the Marshmallow and UserModel.query sessions are the same.
A note: the signalling session is the default session that Flask-SQLAlchemy uses. It extends the default session system with bind selection and modification tracking.
I have rolled out own solution. Hope it helps someone else. Solution implements update method on Node model.
Solution:
class Node(db.Model):
# ...
def update(self, **kwargs):
# py2 & py3 compatibility do:
# from six import iteritems
# for key, value in six.iteritems(kwargs):
for key, value in kwargs.items():
setattr(self, key, value)
class NodeAPI(MethodView):
decorators = [login_required, ]
model = Node
def get_queryset(self):
if g.user.is_admin:
return self.model.query
return self.model.query.filter(self.model.owner == g.user)
def put(self, node_id):
json_data = request.get_json()
if not json_data:
abort(400)
data, errors = node_schema.load(json_data) # validate with marshmallow
if errors:
return jsonify(errors), 422
queryset = self.get_queryset()
node = queryset.filter(self.model.id == node_id).first_or_404()
node.update(**data)
db.session.commit()
return jsonify(message='Successfuly updated'), 200
Latest Update [2020]:
You might facing the issue of mapping keys to the database models. Your request body have only updated fields so, you want to change only those without affecting others. There is an option to write multiple if conditions but that's not a good approach.
Solution
You can implement patch or put methods using sqlalchemy library only.
For example:
YourModelName.query.filter_by(
your_model_column_id = 12 #change 12: where condition to find particular row
).update(request_data)
request_data should be dict object. For ex.
{
"your_model_column_name_1": "Hello",
"your_model_column_name_2": "World",
}
In above case, only two columns will be updated that is: your_model_column_name_1 and your_model_column_name_2
Update function maps request_data to the database models and creates update query for you. Checkout this: https://docs.sqlalchemy.org/en/13/core/dml.html#sqlalchemy.sql.expression.update
Previous answer seems to be outdated as ModelSchema is now deprecated.
You should instead SQLAlchemyAutoSchema with the proper options.
class NodeSchema(SQLAlchemyAutoSchema):
class Meta:
model = Node
load_instance = True
sqla_session = db.session
node_schema = NodeSchema()
# then when you need to update a Node orm instance :
node_schema.load(node_data, instance=node, partial=True)
db.session.update()
Below is my solution with Flask-Marshmallow + marshmallow-sqlalchemy bundle as the author requested initially.
schemas.py
from flask import current_app
from flask_marshmallow import Marshmallow
from app.models import Node
ma = Marshmallow(current_app)
class NodeSchema(ma.SQLAlchemyAutoSchema):
class Meta:
model = Node
load_instance = True
load_instance is a key point here to make an update further.
routes.py
from flask import jsonify, request
from marshmallow import ValidationError
from app import db
#bp.route("/node/<node_uuid>/edit", methods=["POST"])
def edit_node(node_uuid):
json_data = request.get_json(force=True, silent=True)
node = Node.query.filter_by(
node_uuid=node_uuid
).first()
if node:
try:
schema = NodeSchema()
json_data["node_uuid"] = node_uuid
node = schema.load(json_data, instance=node)
db.session.commit()
return schema.jsonify(node)
except ValidationError as err:
return jsonify(err.messages), 422
else:
return jsonify("Not found"), 404
You have to check for existence of Node first, otherwise the new instance will be created.