I have a caching problem when I use sqlalchemy.
I use sqlalchemy to insert data into a MySQL database. Then, I have another application process this data, and update it directly.
But sqlalchemy always returns the old data rather than the updated data. I think sqlalchemy cached my request ... so ... how should I disable it?
The usual cause for people thinking there's a "cache" at play, besides the usual SQLAlchemy identity map which is local to a transaction, is that they are observing the effects of transaction isolation. SQLAlchemy's session works by default in a transactional mode, meaning it waits until session.commit() is called in order to persist data to the database. During this time, other transactions in progress elsewhere will not see this data.
However, due to the isolated nature of transactions, there's an extra twist. Those other transactions in progress will not only not see your transaction's data until it is committed, they also can't see it in some cases until they are committed or rolled back also (which is the same effect your close() is having here). A transaction with an average degree of isolation will hold onto the state that it has loaded thus far, and keep giving you that same state local to the transaction even though the real data has changed - this is called repeatable reads in transaction isolation parlance.
http://en.wikipedia.org/wiki/Isolation_%28database_systems%29
This issue has been really frustrating for me, but I have finally figured it out.
I have a Flask/SQLAlchemy Application running alongside an older PHP site. The PHP site would write to the database and SQLAlchemy would not be aware of any changes.
I tried the sessionmaker setting autoflush=True unsuccessfully
I tried db_session.flush(), db_session.expire_all(), and db_session.commit() before querying and NONE worked. Still showed stale data.
Finally I came across this section of the SQLAlchemy docs: http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#transaction-isolation-level
Setting the isolation_level worked great. Now my Flask app is "talking" to the PHP app. Here's the code:
engine = create_engine(
"postgresql+pg8000://scott:tiger#localhost/test",
isolation_level="READ UNCOMMITTED"
)
When the SQLAlchemy engine is started with the "READ UNCOMMITED" isolation_level it will perform "dirty reads" which means it will read uncommited changes directly from the database.
Hope this helps
Here is a possible solution courtesy of AaronD in the comments
from flask.ext.sqlalchemy import SQLAlchemy
class UnlockedAlchemy(SQLAlchemy):
def apply_driver_hacks(self, app, info, options):
if "isolation_level" not in options:
options["isolation_level"] = "READ COMMITTED"
return super(UnlockedAlchemy, self).apply_driver_hacks(app, info, options)
Additionally to zzzeek excellent answer,
I had a similar issue. I solved the problem by using short living sessions.
with closing(new_session()) as sess:
# do your stuff
I used a fresh session per task, task group or request (in case of web app). That solved the "caching" problem for me.
This material was very useful for me:
When do I construct a Session, when do I commit it, and when do I close it
This was happening in my Flask application, and my solution was to expire all objects in the session after every request.
from flask.signals import request_finished
def expire_session(sender, response, **extra):
app.db.session.expire_all()
request_finished.connect(expire_session, flask_app)
Worked like a charm.
I have tried session.commit(), session.flush() none worked for me.
After going through sqlalchemy source code, I found the solution to disable caching.
Setting query_cache_size=0 in create_engine worked.
create_engine(connection_string, convert_unicode=True, echo=True, query_cache_size=0)
First, there is no cache for SQLAlchemy.
Based on your method to fetch data from DB, you should do some test after database is updated by others, see whether you can get new data.
(1) use connection:
connection = engine.connect()
result = connection.execute("select username from users")
for row in result:
print "username:", row['username']
connection.close()
(2) use Engine ...
(3) use MegaData...
please folowing the step in : http://docs.sqlalchemy.org/en/latest/core/connections.html
Another possible reason is your MySQL DB is not updated permanently. Restart MySQL service and have a check.
As i know SQLAlchemy does not store caches, so you need to looking at logging output.
I am using MSSQL as my database. It is taking too long to respond back for some queries and sometimes it is resulting in deadlocks. Hence i want to set custom timeout for queries. I got some custom solution here but it is for psycopg2 and sqlalchemy but i need it for MSSQL and sqlalchemy. Is there any way to accomplish this?
Thanks in advance.
I'm trying to automatically create a test database before running my tests without shelling out, but I'm struggling to connect to the postgres server without connecting to a database.
If I give create_engine() a URL with .database None it tries to connect to a default database name.
Is what I'm trying to do possible?
I was going a long the lines of this answer: https://stackoverflow.com/a/28784334/311220
One way to achieve this is to connect to the default postgres database which should always be present. Then once connected you can create the desired databases.
I am facing a problem where data from MySQL retrieved using PySQLPool is returned as the db was at start of process, INSERT or UPDATE queries from python or MySQL client do not show up until a kill and re-run of the python process.
Would appreciate any help regarding this.
Ref: Why are some mysql connections selecting old data the mysql database after a delete + insert?
MySQL's isolation level was causing this. Somehow only python clients get affected and never stumbled across this issue earlier. It is a valid problem and has a detailed solution. My question was targeting python and pySQLPool because it did not occur to me that MySQL could be the one causing this. Now my deployment procedure includes details on altering global isolation level for MySQL to be "READ-COMMITTED".
SET GLOBAL tx_isolation='READ-COMMITTED';
How can I execute raw SQL after connect to database?
I need to run script once, after connect to DB.
Thanks.
UPD: question is not how to run raw SQL.
Just visit the docs here.
It was I think the second match on google with "Django ORM".
EDIT See comments
If you look at this Django page you see that MySQLdb (the underlying layer) also accepts an init_command option which is run immediately after a connection is established. That's a feature of MySQLdb, and not so much of Django itself.
If you are using django >= 1.2:
from django.db import connection, transaction
query = "SELECT foo FROM bar;"
connection.cursor().execute(query)
transaction.commit_unless_managed()