I'm struggling with SQLAlchemy and py.test.
In my __init__.py I create a engine and a session using:
engine = create_engine('sqlite://')
Session = sessionmaker(bind=engine)
session = Session()
I also have a entity.py and a test_entity.py. In both files I import session
from __init__ import session
In conftest.py I defined a function which sets up the database and create a schema from Base.metadata.
The point is that all transactions inside my test module all pass, but all transactions in my class being tested fails with errors like Object already bound to session (when adding and commiting an object) or with OperationalError: no such table (when fetching an object).
How do I fix it?
After some trial and error I found out that all works fine when I use a database on disk.
engine = create_engine('sqlite:////path/to/db')
It is documented:
Pysqlite’s default behavior is to prohibit the usage of a single
connection in more than one thread. [...] Pysqlite does include a
now-undocumented flag known as check_same_thread which will disable
this check, however note that pysqlite connections are still not safe
to use in concurrently in multiple threads.
Related
I created a rest API with openapi generator that contains all the requests necessary for selecting, inserting, and updating my SQL database.
I use from my database generation and manipulation SQLAlchemy and I'm not sure how to use the session to interact with the database in this context.
My project looks like this:
DB
| openapi_server (generated)
| __init__.py
| request.py
| database.py
In database.py I keep my database structure.
In request.py I have all the functions that need to be processed on every request(to interact with the database).
My way of handling this situation is: I create a session variable at the beginning of each function and after the operation is complete I close it.
Any other methods that are more scalable and easy to maintain or which are the best practices?
My understanding is that the sqlalchemy session is different from the client session in that the client session stores information about authorization & permissions whereas the sqlalchemy session is a gil-bound transaction state which associates your code / machine to an external database.
Assuming you're not utilizing multithreading or parallel processing, a single sqlalchemy session shared between your application would be appropriate. In the case where your users have different levels of database permissions, I would establish those rules in your application authorization, rather than the database user-permission schema. (That should be reserved for system-users.)
Bear in mind, multiple sqlalchemy sessions are appropriate in many scenarios and there are advantages for creating and closing sessions on the fly. But there are also potential downsides, such as write collisions (2 processes try to write the same record) and so on. In these more fine grained cases, I'd suggest a queuing process as a central orchestrator.
For implementation:I usually create a file create_session.py which has a function to create a new db session with the appropriate DB URI. I then call that function in the main __init__.py like so: session = create_session() --> importing that session throughout the application is done by importing session from the main module ex: from database import session.
In cases where you need to create new / multiple sessions, do so with:
# Getting the path right here isn't always straightforward tbh
# basically, import the function from the module directly
from create_session import create_session
def do_something():
# Always create your session in a method
# otherwise your db will open many unnecessary connections
my_session = create_session()
print('Done')
# Close the session when you're done
my_session.close()
I have a caching problem when I use sqlalchemy.
I use sqlalchemy to insert data into a MySQL database. Then, I have another application process this data, and update it directly.
But sqlalchemy always returns the old data rather than the updated data. I think sqlalchemy cached my request ... so ... how should I disable it?
The usual cause for people thinking there's a "cache" at play, besides the usual SQLAlchemy identity map which is local to a transaction, is that they are observing the effects of transaction isolation. SQLAlchemy's session works by default in a transactional mode, meaning it waits until session.commit() is called in order to persist data to the database. During this time, other transactions in progress elsewhere will not see this data.
However, due to the isolated nature of transactions, there's an extra twist. Those other transactions in progress will not only not see your transaction's data until it is committed, they also can't see it in some cases until they are committed or rolled back also (which is the same effect your close() is having here). A transaction with an average degree of isolation will hold onto the state that it has loaded thus far, and keep giving you that same state local to the transaction even though the real data has changed - this is called repeatable reads in transaction isolation parlance.
http://en.wikipedia.org/wiki/Isolation_%28database_systems%29
This issue has been really frustrating for me, but I have finally figured it out.
I have a Flask/SQLAlchemy Application running alongside an older PHP site. The PHP site would write to the database and SQLAlchemy would not be aware of any changes.
I tried the sessionmaker setting autoflush=True unsuccessfully
I tried db_session.flush(), db_session.expire_all(), and db_session.commit() before querying and NONE worked. Still showed stale data.
Finally I came across this section of the SQLAlchemy docs: http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#transaction-isolation-level
Setting the isolation_level worked great. Now my Flask app is "talking" to the PHP app. Here's the code:
engine = create_engine(
"postgresql+pg8000://scott:tiger#localhost/test",
isolation_level="READ UNCOMMITTED"
)
When the SQLAlchemy engine is started with the "READ UNCOMMITED" isolation_level it will perform "dirty reads" which means it will read uncommited changes directly from the database.
Hope this helps
Here is a possible solution courtesy of AaronD in the comments
from flask.ext.sqlalchemy import SQLAlchemy
class UnlockedAlchemy(SQLAlchemy):
def apply_driver_hacks(self, app, info, options):
if "isolation_level" not in options:
options["isolation_level"] = "READ COMMITTED"
return super(UnlockedAlchemy, self).apply_driver_hacks(app, info, options)
Additionally to zzzeek excellent answer,
I had a similar issue. I solved the problem by using short living sessions.
with closing(new_session()) as sess:
# do your stuff
I used a fresh session per task, task group or request (in case of web app). That solved the "caching" problem for me.
This material was very useful for me:
When do I construct a Session, when do I commit it, and when do I close it
This was happening in my Flask application, and my solution was to expire all objects in the session after every request.
from flask.signals import request_finished
def expire_session(sender, response, **extra):
app.db.session.expire_all()
request_finished.connect(expire_session, flask_app)
Worked like a charm.
I have tried session.commit(), session.flush() none worked for me.
After going through sqlalchemy source code, I found the solution to disable caching.
Setting query_cache_size=0 in create_engine worked.
create_engine(connection_string, convert_unicode=True, echo=True, query_cache_size=0)
First, there is no cache for SQLAlchemy.
Based on your method to fetch data from DB, you should do some test after database is updated by others, see whether you can get new data.
(1) use connection:
connection = engine.connect()
result = connection.execute("select username from users")
for row in result:
print "username:", row['username']
connection.close()
(2) use Engine ...
(3) use MegaData...
please folowing the step in : http://docs.sqlalchemy.org/en/latest/core/connections.html
Another possible reason is your MySQL DB is not updated permanently. Restart MySQL service and have a check.
As i know SQLAlchemy does not store caches, so you need to looking at logging output.
I'm working on a Flask project and I am using Flask-SQLAlchemy.
I need to work with multiple already existing databases.
I created the "app" object and the SQLAlchemy one:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
db = SQLAlchemy(app)
In the configuration I set the default connection and the additional binds:
SQLALCHEMY_DATABASE_URI = 'postgresql://pg_user:pg_pwd#pg_server/pg_db'
SQLALCHEMY_BINDS = {
'oracle_bind': 'oracle://oracle_user:oracle_pwd#oracle_server/oracle_schema',
'mssql_bind': 'mssql+pyodbc://msssql_user:mssql_pwd#mssql_server/mssql_schema?driver=FreeTDS'
}
Then I created the table models using the declarative system and, where needed, I set the
__bind_key__ parameter to indicate in which database the table is located.
For example:
class MyTable(db.Model):
__bind_key__ = 'mssql_bind'
__tablename__ = 'my_table'
id = db.Column(db.Integer, nullable=False, primary_key=True)
val = db.Column(db.String(50), nullable=False)
in this way everything works correctly, when I do a query it is made on the right database.
Reading the SQLAlchemy documentation and the Flask-SQLALchemy documentation I understand these things
(i write them down to check I understand correctly):
You can handle the transactions through the session.
In SQLAlchemy you can bind a session with a specific engine.
Flask-SQLAlchemy automatically creates the session (scoped_session) at the request start and it destroys it at the request end
so I can do:
record = MyTable(1, 'some text')
db.session.add(record)
db.session.commit()
I can not understand what happens when we use multiple databases, regarding the session, in Flask-SqlAlchemy.
I verified that the system is able to bind the table correctly at the right database through the __bind_key__ parameter,
I can, therefore, insert data on different databases through db.session and, at the commit, everything is saved.
I can't, however, understand if Flask-SQLAlchemy create multiple sessions (one for each engine) or if manages the thing in a different way.
In both cases, how is it possible refer to the session/transaction of a specific database?
If I use db.session.commit() the system does the commit on all involved databases, but how can I do if I want to commit only for a single database?
I would do something like:
db.session('mssql_bind').commit()
but I can not figure out how to do this.
I also saw a Flask-SQLAlchemy implementation which should ease the management of these situations:
Issue: https://github.com/mitsuhiko/flask-sqlalchemy/issues/107
Implementation: https://github.com/mitsuhiko/flask-sqlalchemy/pull/249
but I can not figure out how to use it.
In Flask-SQLAlchemy how can I manage sessions specifically for each single engine?
Flask-SQLAlchemy uses a customized session that handles bind routing according to given __bind_key__ attribute in mapped class. Under the hood it actually adds that key as info to the created table. In other words, Flask does not create multiple sessions, one for each bind, but a single session that routes to correct connectable (engine/connection) according to the bind key. Note that vanilla SQLAlchemy has similar functionality out of the box.
In both cases, how is it possible refer to the session/transaction of a specific database?
If I use db.session.commit() the system does the commit on all involved databases, but how can I do if I want to commit only for a single database?
It might not be a good idea to subvert and issue commits to specific databases mid session using the connections owned by the session. The session is a whole and keeps track of state for object instances, flushing changes to databases when needed etc. That means that the transaction handled by the session is not just the database transactions, but the session's own transaction as well. All that should commit and rollback as one.
You could on the other hand create new SQLAlchemy (or Flask-SQLAlchemy) sessions that possibly join the ongoing transaction in one of the binds:
session = db.create_scoped_session(
options=dict(bind=db.get_engine(app, 'oracle_bind'),
binds={}))
This is what the pull request is about. It allows using an existing transactional connection as the bind for a new Flask-SQLAlchemy session. This is very useful for example in testing, as can be seen in the rationale for that pull request. That way you can have a "master" transaction that can for example rollback everything done in testing.
Note that the SignallingSession always consults the db.get_engine() method if a bind_key is present. This means that the example session is unable to query tables without a bind key and which don't exist on your oracle DB, but would still work for tables with your mssql_bind key.
The issue you linked to on the other hand does list ways to issue SQL to specific binds:
rows = db.session.execute(query, params,
bind=db.get_engine(app, 'oracle_bind'))
There were other less explicit methods listed as well, but explicit is better than implicit.
I'm using sqlalchemy in a pyqt desktop application. When I execute and update to the database the changes are not reflected in the database, if I inspect the session object 'sqlalchemy.orm.scoping.ScopedSession', it tell's me that my object is not present in the session, but if I try to added It tells me that is already present.
The way I'm managing the connections is the following, when the application starts I open a connection and keep it open all the user session, when the application is closed I close the connection. Thus all queries are performed opening only one connection.
selected_mine = Mine.query.filter_by(mine_name="some_name").first()
''' updating the object attributes '''
selected_mine.region = self.ui.region.text()
self.sqlite_model.conn.commit()
I've inspect the sessions, and there are two different objects (I don't know why).
s=self.sqlite_model.conn()
>>> s
<sqlalchemy.orm.session.Session object at 0x30fb210>
s.object_session(selected_mine)
>>> <sqlalchemy.orm.session.Session object at 0x30547d0>
how do I solve this? why the commit it's not working?
I'm creating the session in the class SqliteModel (class of the object self.sqlite_model)
Session = scoped_session(sessionmaker(autocommit=False, bind=self.engine))
self.conn = Session()
Base.query = Session.query_property()
''' connect to database '''
Base.metadata.create_all(bind=self.engine)
You don't show what the Mine class is, but I believe your issue lies with this line:
selected_mine = Mine.query.filter_by(mine_name="some_name").first()
You are not actually using the session that you originally setup to query the database thus a new session is being created for you. I believe your code should look like this instead:
selected_mine = self.conn.query(Mine).filter_by(mine_name="some_name").first()
If you haven't done so yet, you should definitely glance through the excellent documentation that SQLAlchemy has made available explaining what a Session is and how to use it.
I am using SQlAlchemy in my web project. What should I use - scoped_session(sessionmaker()) or plain sessionmaker() - and why? Or should I use something else?
## model.py
from sqlalchemy import *
from sqlalchemy.orm import *
engine = create_engine('mysql://dbUser:dbPassword#dbServer:dbPort/dbName',
pool_recycle=3600, echo=False)
metadata = MetaData(engine)
Session = scoped_session(sessionmaker())
Session.configure(bind=engine)
user = Table('user', metadata, autoload=True)
class User(object):
pass
usermapper = mapper(User, user)
## some other python file called abc.py
from models import *
def getalluser():
session = Session()
session.query(User).all()
session.flush()
session.close()
## onemore file defg.py
from models import *
def updateuser():
session = Session()
session.query(User).filter(User.user_id == '4').update({User.user_lname: 'villkoo'})
session.commit()
session.flush()
session.close()
I create a session = Session() object for each request and I close it. Am I doing the right thing or is there a better way to do it?
Reading the documentation is recommended:
the scoped_session() function is provided which produces a thread-managed registry of Session objects. It is commonly used in web applications so that a single global variable can be used to safely represent transactional sessions with sets of objects, localized to a single thread.
In short, use scoped_session() for thread safety.
Scoped_session at every method will give you a thread of local session which you cannot obtain beforehand (like at the module level).It's not needed to open a new session in every method, You can use a global session , Create a session only when the global session is not available. i.e you can write a method which returns a session and add it to the init.py inside your package.
Don't use scoped_session and don't use Flask-SQLAlchemy.
Just use Session = sessionmaker() held in a singleton/service class, and use session = Session() on every HTTP request to guarantee that a fresh connection is provided.
Thread Local storage is clumsy and involves holding state which doesn't play nicely with different web-server threading models. Better to stay stateless. See for example SqlAlchemy's documentation here mentioning not to forget to call .remove() if you are using scoped_session. Will anyone remember to do that?
Below is an excerpt from https://docs.sqlalchemy.org/en/14/orm/contextual.html#using-thread-local-scope-with-web-applications:
Using the above flow, the process of integrating the Session with the web application has exactly two requirements:
Create a single scoped_session registry when the web application first starts, ensuring that this object is accessible by the rest of the application.
Ensure that scoped_session.remove() is called when the web request ends, usually by integrating with the web framework’s event system to establish an “on request end” event.
FYI, when using flask-sqlalchemy, the session object provided is by default a scoped session object.
http://flask-sqlalchemy.pocoo.org/2.3/quickstart/#road-to-enlightenment
I am looking into this myself, but I am not an expert.
My three points are:
SQLAlchemy docs provide a proposed approach using scoped_session, per Mr. Kluev's comment above, at this link: http://docs.sqlalchemy.org/en/rel_0_9/orm/session.html#using-thread-local-scope-with-web-applications.
At that web location, the SQLAlchemy docs also say that it is "...strongly recommended that the integration tools provided with the web framework itself be used, if available, instead of scoped_session."
Flask-SQLAlchemy, for example, appears to claim that it takes care of this: http://pythonhosted.org/Flask-SQLAlchemy/quickstart.html#a-minimal-application