I am using SQlAlchemy in my web project. What should I use - scoped_session(sessionmaker()) or plain sessionmaker() - and why? Or should I use something else?
## model.py
from sqlalchemy import *
from sqlalchemy.orm import *
engine = create_engine('mysql://dbUser:dbPassword#dbServer:dbPort/dbName',
pool_recycle=3600, echo=False)
metadata = MetaData(engine)
Session = scoped_session(sessionmaker())
Session.configure(bind=engine)
user = Table('user', metadata, autoload=True)
class User(object):
pass
usermapper = mapper(User, user)
## some other python file called abc.py
from models import *
def getalluser():
session = Session()
session.query(User).all()
session.flush()
session.close()
## onemore file defg.py
from models import *
def updateuser():
session = Session()
session.query(User).filter(User.user_id == '4').update({User.user_lname: 'villkoo'})
session.commit()
session.flush()
session.close()
I create a session = Session() object for each request and I close it. Am I doing the right thing or is there a better way to do it?
Reading the documentation is recommended:
the scoped_session() function is provided which produces a thread-managed registry of Session objects. It is commonly used in web applications so that a single global variable can be used to safely represent transactional sessions with sets of objects, localized to a single thread.
In short, use scoped_session() for thread safety.
Scoped_session at every method will give you a thread of local session which you cannot obtain beforehand (like at the module level).It's not needed to open a new session in every method, You can use a global session , Create a session only when the global session is not available. i.e you can write a method which returns a session and add it to the init.py inside your package.
Don't use scoped_session and don't use Flask-SQLAlchemy.
Just use Session = sessionmaker() held in a singleton/service class, and use session = Session() on every HTTP request to guarantee that a fresh connection is provided.
Thread Local storage is clumsy and involves holding state which doesn't play nicely with different web-server threading models. Better to stay stateless. See for example SqlAlchemy's documentation here mentioning not to forget to call .remove() if you are using scoped_session. Will anyone remember to do that?
Below is an excerpt from https://docs.sqlalchemy.org/en/14/orm/contextual.html#using-thread-local-scope-with-web-applications:
Using the above flow, the process of integrating the Session with the web application has exactly two requirements:
Create a single scoped_session registry when the web application first starts, ensuring that this object is accessible by the rest of the application.
Ensure that scoped_session.remove() is called when the web request ends, usually by integrating with the web framework’s event system to establish an “on request end” event.
FYI, when using flask-sqlalchemy, the session object provided is by default a scoped session object.
http://flask-sqlalchemy.pocoo.org/2.3/quickstart/#road-to-enlightenment
I am looking into this myself, but I am not an expert.
My three points are:
SQLAlchemy docs provide a proposed approach using scoped_session, per Mr. Kluev's comment above, at this link: http://docs.sqlalchemy.org/en/rel_0_9/orm/session.html#using-thread-local-scope-with-web-applications.
At that web location, the SQLAlchemy docs also say that it is "...strongly recommended that the integration tools provided with the web framework itself be used, if available, instead of scoped_session."
Flask-SQLAlchemy, for example, appears to claim that it takes care of this: http://pythonhosted.org/Flask-SQLAlchemy/quickstart.html#a-minimal-application
Related
I'm tring to make a clean Flask App which use SQLAlchemy and Multi-Threading.
I've read the doc : https://docs.sqlalchemy.org/en/14/orm/contextual.html#thread-local-scope but can't manage to make it work successfully.
SQL Alchemy is initate directly at the app init.py file with something like that :
db = SQLAlchemy(app)
session_factory = sessionmaker( bind=db.engine, autocommit=False, autoflush=False)
DBSession = scoped_session( session_factory )
And in another file, which is not some Flask routes, on a Class :
from blabla import DBSession
class Worker(Thread):
def __init__(self):
Thread.__init__(self)
self.dbsession = DBSession()
def run(self):
self.dbsession.query(...)
When I run my app, multiple Worker Class is running at the same time.
Then I'm facing lot of errors like :
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) Packet sequence number wrong - got 54 expected 1
What I am doing wrong ?
Thanks a lot in advance for your time !
Looks like maybe you are duplicating the scoped session already provided by flask sqlalchemy. You can probably use db.session directly. Although you have to clean it up yourself when the thread ends or at regular intervals with db.session.remove() because it isn't within a request like flask expects. Why are you adding custom threading ?
https://flask-sqlalchemy.palletsprojects.com/en/2.x/quickstart/#road-to-enlightenment
a preconfigured scoped session called session
Thanks to comments and answer, I've made it worky pretty smoothly. Just need to thinks it from scratch.
On my app.py :
db = SQLAlchemy(app, session_options={"autocommit":False, "autoflush":False, "expire_on_commit":False })
Ian highlighted that the Object db.session is already a scoped session. So no need to create another one manually.
Moreover, I didn't know that it's possible to add session_options directly at this step. Obviously it's easier to directly configure the already integrated one.
On my others files :
dbsession = db.session
dbsession.query(...)
...
dbsession.commit()
dbsession.close()
Each new db.session seems to create a "real" session, I can see them directly on my MySQL instance. So, It's very important to close() and limit the number of concurent threads.
Did you see something wrong in this way ? The answer is YES.
EDIT : With Flask-SQLALchemy V3 comes a new way of deals whith context. The session scope to use the current app context instead of the thread. That is a problem when used out of context.
It's possible to add with app.app_context(): at some points. But it's not clear for me where it has to be done. And I feel like this will overload my code.
Maybe it's possible to come back to the previous behaviour directly from the params ? Don't know how to for the moment.
I created a rest API with openapi generator that contains all the requests necessary for selecting, inserting, and updating my SQL database.
I use from my database generation and manipulation SQLAlchemy and I'm not sure how to use the session to interact with the database in this context.
My project looks like this:
DB
| openapi_server (generated)
| __init__.py
| request.py
| database.py
In database.py I keep my database structure.
In request.py I have all the functions that need to be processed on every request(to interact with the database).
My way of handling this situation is: I create a session variable at the beginning of each function and after the operation is complete I close it.
Any other methods that are more scalable and easy to maintain or which are the best practices?
My understanding is that the sqlalchemy session is different from the client session in that the client session stores information about authorization & permissions whereas the sqlalchemy session is a gil-bound transaction state which associates your code / machine to an external database.
Assuming you're not utilizing multithreading or parallel processing, a single sqlalchemy session shared between your application would be appropriate. In the case where your users have different levels of database permissions, I would establish those rules in your application authorization, rather than the database user-permission schema. (That should be reserved for system-users.)
Bear in mind, multiple sqlalchemy sessions are appropriate in many scenarios and there are advantages for creating and closing sessions on the fly. But there are also potential downsides, such as write collisions (2 processes try to write the same record) and so on. In these more fine grained cases, I'd suggest a queuing process as a central orchestrator.
For implementation:I usually create a file create_session.py which has a function to create a new db session with the appropriate DB URI. I then call that function in the main __init__.py like so: session = create_session() --> importing that session throughout the application is done by importing session from the main module ex: from database import session.
In cases where you need to create new / multiple sessions, do so with:
# Getting the path right here isn't always straightforward tbh
# basically, import the function from the module directly
from create_session import create_session
def do_something():
# Always create your session in a method
# otherwise your db will open many unnecessary connections
my_session = create_session()
print('Done')
# Close the session when you're done
my_session.close()
I'm using SQLAlchemy in a Pyramid system. I use a transaction manager (zope) and add a request_method called 'db_session', so request.session can be used to do queries.
This is fine in views, but for functions on my SQLAlchemy objects themselves I'm using inspect(self).session to get the session. This is fairly common as far as I can tell.
Now I want to be able to access my zope transaction session from classmethods on my SQLAlchemy objects. After trying various things, I realize I have to set my metaclass Base.query to Session.query_property() inside the includeme() function in init.py. Basically this:-
def includeme(config):
settings = config.get_settings()
# use pyramid_tm to hook the transaction lifecycle to the request
config.include('pyramid_tm')
session_factory = get_session_factory(get_engine(settings))
factory = scoped_session(session_factory)
config.registry['dbsession_factory'] = factory
Base.query = factory.query_property()
# make request.dbsession available for use in Pyramid
config.add_request_method(
# r.tm is the transaction manager used by pyramid_tm
lambda r: get_tm_session(session_factory, r.tm),
'dbsession',
reify=True
)
'Base' here is my metaclass.
This does what I want, but I realize what I'm effectively doing when using this is using a python global variable. I haven't proceeded very far yet, but is this going to cause problems (especially with testability) in future? I'm pretty sure it 'works' for what I want to do (classmethods have inherent access to the current session without having to have it passed to them).
I'm struggling with SQLAlchemy and py.test.
In my __init__.py I create a engine and a session using:
engine = create_engine('sqlite://')
Session = sessionmaker(bind=engine)
session = Session()
I also have a entity.py and a test_entity.py. In both files I import session
from __init__ import session
In conftest.py I defined a function which sets up the database and create a schema from Base.metadata.
The point is that all transactions inside my test module all pass, but all transactions in my class being tested fails with errors like Object already bound to session (when adding and commiting an object) or with OperationalError: no such table (when fetching an object).
How do I fix it?
After some trial and error I found out that all works fine when I use a database on disk.
engine = create_engine('sqlite:////path/to/db')
It is documented:
Pysqlite’s default behavior is to prohibit the usage of a single
connection in more than one thread. [...] Pysqlite does include a
now-undocumented flag known as check_same_thread which will disable
this check, however note that pysqlite connections are still not safe
to use in concurrently in multiple threads.
I'm using sqlalchemy in a pyqt desktop application. When I execute and update to the database the changes are not reflected in the database, if I inspect the session object 'sqlalchemy.orm.scoping.ScopedSession', it tell's me that my object is not present in the session, but if I try to added It tells me that is already present.
The way I'm managing the connections is the following, when the application starts I open a connection and keep it open all the user session, when the application is closed I close the connection. Thus all queries are performed opening only one connection.
selected_mine = Mine.query.filter_by(mine_name="some_name").first()
''' updating the object attributes '''
selected_mine.region = self.ui.region.text()
self.sqlite_model.conn.commit()
I've inspect the sessions, and there are two different objects (I don't know why).
s=self.sqlite_model.conn()
>>> s
<sqlalchemy.orm.session.Session object at 0x30fb210>
s.object_session(selected_mine)
>>> <sqlalchemy.orm.session.Session object at 0x30547d0>
how do I solve this? why the commit it's not working?
I'm creating the session in the class SqliteModel (class of the object self.sqlite_model)
Session = scoped_session(sessionmaker(autocommit=False, bind=self.engine))
self.conn = Session()
Base.query = Session.query_property()
''' connect to database '''
Base.metadata.create_all(bind=self.engine)
You don't show what the Mine class is, but I believe your issue lies with this line:
selected_mine = Mine.query.filter_by(mine_name="some_name").first()
You are not actually using the session that you originally setup to query the database thus a new session is being created for you. I believe your code should look like this instead:
selected_mine = self.conn.query(Mine).filter_by(mine_name="some_name").first()
If you haven't done so yet, you should definitely glance through the excellent documentation that SQLAlchemy has made available explaining what a Session is and how to use it.