We Keep Coding

SQLAlchemy session.begin() giving transaction error when context manager doesn't

In short, why am I getting an "sqlalchemy.exc.InvalidRequestError: A transaction is already begun. Use subtransactions=True to allow subtransactions" error?
Following the best practices of separating and keeping the session external, I created foo(input) with a context manager instead of using the try / except / else. If I use foo(user) instead of it I get the above error. My guess is that foo() isn't committing and closing the connection. Howevere the documentation states otherwise.
Flask documentation uses a scoped_session but the SQLAlchemy documentation says "It is however strongly recommended that the integration tools provided with the web framework itself be used, if available, instead of scoped_session." Perhaps the scoped_session is causing errors across threads with the requests?
Here is my main code:
#__init__.py
import os
from flask import Flask, render_template, redirect, request, url_for
def create_app(test_config=None):
# create and configure the app
app = Flask(__name__, instance_relative_config=False)
app.config.from_object('config.DevelopmentConfig')
# set up extensions
# all flask extensions must support factory pattern
# can run these two steps from the cli
from app.database import init_db
init_db()
#app.route('/')
def index():
return render_template('index.html')
from app.auth import RegistrationForm
from app.models import User
from app.database import db_session, foo
#app.route('/register', methods=['GET', 'POST'])
def register():
form = RegistrationForm(request.form)
if request.method == 'POST' and form.validate():
user = User(form.name.data, form.email.data,
form.password.data)
foo(user)
# try:
# db_session.add(user)
# except:
# db_session.rollback()
# raise
# else:
# db_session.commit()
return redirect(url_for('login'))
return render_template('register.html', form=form)
#app.route('/login', methods=['GET'])
def login():
return render_template('login.html')
#app.teardown_appcontext
def shutdown_session(exception=None):
db_session.remove()
return app
Here is my database code:
#database.py
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
_database_uri = os.environ['DATABASE_URL'] engine = create_engine(_database_uri)
db_session = scoped_session(sessionmaker(autocommit=False,
autoflush=False,
bind=engine))
Base = declarative_base() Base.query = db_session.query_property()
def init_db():
# import all modules here that might define models so that
# they will be registered properly on the metadata. Otherwise
# you will have to import them first before calling init_db()
import app.models
Base.metadata.create_all(bind=engine)
def foo(input):
with db_session.begin() as session:
session.add(input)

I'm not sure whether this will actually answer your question or not but I think it worth mentioning.
Near the last line in your database.py file, I suggest you to not alias db_session.begin() as session because you'll then be confused thinking that session is an object of Session class while it's an object of SessionTransaction class which is:
largely an internal object that in modern use provides a context manager for session transactions. SessionTransaction
You can switch to either:
with db_session() as session, session.begin():
session.add(input)
or shorter version
with db_session.begin():
db_session.add(input)
Also you need to wrap your User object creation with Session.begin() context like below:
def register():
form = RegistrationForm(request.form)
if request.method == 'POST' and form.validate():
with db_session.begin():
user = User(form.name.data, form.email.data,
form.password.data)
foo(user)
Because User model is just a proxy object that will actually execute database query under the hood. Therefore A transaction is already begun. in the creation process. The exception itself will be raised upon the next transactional query calls.
When using a Session, it’s useful to consider the ORM mapped objects that it maintains as proxy objects to database rows, which are local to the transaction being held by the Session. In order to maintain the state on the objects as matching what’s actually in the database, there are a variety of events that will cause objects to re-access the database in order to keep synchronized. It is possible to “detach” objects from a Session, and to continue using them, though this practice has its caveats. It’s intended that usually, you’d re-associate detached objects with another Session when you want to work with them again, so that they can resume their normal task of representing database state.
Session Basic
As additional answer to your last question: Perhaps the scoped_session is causing errors across threads with the requests?
No. SQLAlchemy scoped_session is actually a helper function that act as Registry for the global Session object. It is very useful in multithreading application, helping to ensure the same Session object is being used accross threads while each keeping their own data to local via threading.local api provided by python. Most web framework uses threading strategies to cope with many web requests at once hence most of them provide some integration with/without this helper.

Can I use Flask-SQLAlchemy and SQLAlchemy at the same time?

I've been using Flask-SQLAlchemy for a project for about a year. I like that it abstract away the sessioning for me. But now I need more granular control over my sessioning, namely to make a DB connection in a thread after the user has left my application. Is it possible / are there any dangers to use both Flask-SQLAlchemy and SQLAlchemy at the same time?
Bonus: if I must revert to just SQLAlchemy, what must I know? Is it just session scope?
EDIT trying detached session:
(pdb) db
<SQLAlchemy engine=None>
(Pdb) db.session
<sqlalchemy.orm.scoping.scoped_session object at 0x104b81210>
(Pdb) db.session()
*** RuntimeError: application not registered on db instance and no application bound to current context

You have an app like:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
db = SQLAlchemy(app)
Currently you use:
#app.route('/add-some-item', method=['POST'])
def add_some_item():
some_item = SomeItem(foo=request.form.get('foo'))
db.session.add(some_item)
db.session.commit()
return 'The new id is: {}'.format(some_item.id)
But you can also use:
def i_run_in_some_other_thread():
# no need for a Flask request context, just use:
session = db.session() # a bare SQLAlchemy session
...
some_item = SomeItem(foo=bar)
session.add(some_item)
session.commit()
#app.route('/do-a-cron-job')
def do_a_cron_job()
Thread(target=i_run_in_some_other_thread).start()
return 'Thread started.'
By default a session is bound to a thread. For this simple case you don't need to do any changes to your code at all, but if sessions are shared between threads, then you would need to do a few changes: “Session and sessionmaker()”.
Just don't share sessions or objects between threads I'd say, or things will get messy. Share IDs and you're fine.

How to handle multiple requests without SQLAlchemy crashing/raising exceptions?

Context: I'm working on a Flask app, running on CherryPy, DB handled using SQLAlchemy ORM.
Problem:
The app runs fine and does everything I want, however, if I have a page which fetches some data from DB and displays, and I press and hold "Ctrl + R" or "F5". That is, continuously refresh a page, making that many DB requests. First few goes fine, and then it breaks.
The following errors are logged:
(OperationalError) (2013, 'Lost connection to MySQL server during query')
Can't reconnect until invalid transaction is rolled back (original cause:
InvalidRequestError: Can't reconnect until invalid transaction is rolled back)
This result object does not return rows. It has been closed automatically.
(ProgrammingError) (2014, "Commands out of sync; you can't run this command now")
There's also another error which bothers me (but not logged this time), it's
dictionary changed size during iteration
This happens when I'm iterating through a query, using values obtained to populate a dictionary. The dictionary is local (scope of the dict) to the function.
More info:
How I am handling sessions:
A new session is created when you enter any page, use that session to perform all the DB transactions, and the session is closed right before rendering the HTML. Technically, that means, the scope of session is the same as the HTTP request.
I do a session.rollback() only when there's an exception raised during updating table or inserting into a table. No rollback() during any query() operations.
I'm pretty sure I've made some silly mistakes or am not doing things the right way.
Unlimited refreshes like that is not really a probably scenario, but can't be overlooked.
Also, I think the behavior would be similar when there a lot of users using it at the same time.
How the SQLAlchemy engine, sessionmaker was handled:
sql_alchemy_engine = create_engine(self.db_string, echo=False, encoding="utf8", convert_unicode=True, pool_recycle=9)
sqla_session = sessionmaker(bind=sql_alchemy_engine)
It's done only ONCE like it's recommended in the SQLA documentation, and a new session is created and returned sqla_session() whenever required.

If you're using Flask, you should be using flask-sqlalchemy, and let the Flask request context manage your session, and not handling your engine and sessions by hand. This is how SQLAlchemy recommends it:
Most web frameworks include infrastructure to establish a single Session, associated with the request, which is correctly constructed and torn down corresponding torn down at the end of a request. Such infrastructure pieces include products such as Flask-SQLAlchemy, for usage in conjunction with the Flask web framework, and Zope-SQLAlchemy, for usage in conjunction with the Pyramid and Zope frameworks. SQLAlchemy strongly recommends that these products be used as available.
http://docs.sqlalchemy.org/en/rel_0_9/orm/session.html?highlight=flask
Then you create your engine simply by:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'your db uri'
db = SQLAlchemy(app)
Or, if you're using app factory:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
def create_app():
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'your db uri'
db.init_app(app)
With that, the base declarative model you should be using will be at db.Model and the session you should be using will be at db.session.

what is the recommended way of setting up an sqlalchemy connection for view callables?

I am using the sqlalchemy expression language for its notation and connection pooling to create dao objects for communicating with the persistence layer. I wanted to get some opinions on how I should approach setting up the metadata and engine so that they are available to the applications view callables. According to sqlalchemy's documentation http://docs.sqlalchemy.org/en/rel_0_7/core/connections.html, they are typically bound and declared global, however I've neither this or the singleton approach are good ideas. Any thoughts would be appreciated...
This is what my __init__.py file looks like inside my project's directory:
from pyramid.config import Configurator
from sqlalchemy import engine_from_config, MetaData, create_engine
from pyramid_beaker import session_factory_from_settings
db_url = 'postgresql://user:password#localhost/dbname'
engine = create_engine(db_url)
meta = MetaData()
def main(global_config, **settings):
meta.bind = engine
.
.
.
[other configuration settings]

The Pyramid documentation includes a tutorial on integrating Pyramid with SQLAlchemy.
There are two special packages that integrate SQLAlchemy transactions and session management with Pyramid, pyramid_tm and zope.sqlalchemy. These together take care of your sessions:
from sqlalchemy import engine_from_config
from .models import DBSession
def main(global_config, **settings):
"""This function returns a Pyramid WSGI application."""
engine = engine_from_config(settings, 'sqlalchemy.')
DBSession.configure(bind=engine)
# Configuration setup
Here we take the configuration settings from your .ini configuration file; and in models.py:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import (
scoped_session,
sessionmaker,
)
from zope.sqlalchemy import ZopeTransactionExtension
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base()
class YourModel(Base):
# Define your model
Note the use of a scoped_session there, using the transaction extension to integrate with Pyramid.
Then in views, all you need to do is use the DBSession session factory to get your sessions:
from pyramid.view import view_config
from .models import (
DBSession,
YourModel,
)
#view_config(...)
def aview(request):
result = DBSession.query(YourModel).filter(...).first()
Committing and rolling back will be integrated with the request; commit on 2xx and 3xx, rollback on exceptions, for example.

I think the sqlalchemy doc examples declare them as global for succinctness and not to indicate that they recommend that.
I think the only thing you really want to pass around to the different parts of your application is a Session object. The simpler option there is to use a scoped session (which I seem to recall the O'Reilly sqlalchemy book in fact recommends for simpler web based applications; your code suggests it's a web app). I think there's very few applications for needing the engine or metadata in any location other than when you're instantiating the database connection.
The scoped session would also be created when the engine and metadata are created, upon app startup (in the case of pyramid, in the main function here). Then you'd pass it as a parameter to the various parts of your application that need database access.

SqlAlchemy - When should new session creation occur?

Being new to SQLAlchemy, I was wondering at what time would Session() should be called, in say, a view. Should it be defined as a global variable, or should a new session be created for each request.

I strongly recommend you follow the Pyramid SQLAlchemy tutorial. It teaches you how to use SQLAlchemy with Pyramid in a simple Wiki application.
You'll note that tutorial a models.py module is created which defines a DBSession item. This gives you access to the SQLAlchemy session, scoped to a Pyramid thread and tied to the Pyramid transaction model, and which is imported whenever you need the session:
from pyramid.view import view_config
from .models import (
DBSession,
MyModel,
)
#view_config(route_name='home', renderer='templates/mytemplate.pt')
def my_view(request):
one = DBSession.query(MyModel).filter(MyModel.name=='one').first()
return {'one':one, 'project':'tutorial'}
In models.py the DBSession variable is defined as follows:
from sqlalchemy.orm import (
scoped_session,
sessionmaker,
)
from zope.sqlalchemy import ZopeTransactionExtension
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Note the ZopeTransactionExtension; Pyramid automatically starts a new transaction with each request, committing it on successful responses, aborting it when an exception occurs. This relieves you of most transaction handling duties. Just remember to .flush your session when you need to see updates to the database (such as auto-incrementing primary keys).
Again, the tutorial expands on all this and more.

Actually, after reading some more of the sqlalchemy docs, I found the correct answer to my question, where a new contextual session should be created with each request. See http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html#lifespan-of-a-contextual-session