I'm using Flask-SQLAlchemy 1.0, Flask 0.10, SQLAlchemy 0.8.2, and Python 2.7.5. I'm connecting to MySQL 5.6 with Oracle's MySQL Connector/Python 1.0.12.
When I restart my web server (either Apache2 or Flask's built-in), I receive the exception OperationalError: MySQL Connection not available after MySQL's wait_timeout expires (default 8 hours).
I've found people with similar problems and explicitly set SQLALCHEMY_POOL_RECYCLE = 7200, even though that's Flask-SQLAlchemy's default. When I put a breakpoint here, I see that the teardown function is successfully calling session.remove() after each request. Any ideas?
Update 7/21/2014:
Since this question continues to receive attention, I must add that I did try some of the proposals. Two of my attempts looked like the following:
First:
#contextmanager
def safe_commit():
try:
yield
db.session.commit()
except:
db.session.rollback()
raise
This allowed me to wrap my commit calls like so:
with safe_commit():
model = Model(prop=value)
db.session.add(model)
I am 99% certain that I did not miss any db.session.commit calls with this method and I still had problems.
Second:
def managed_session():
def decorator(f):
#wraps(f)
def decorated_function(*args, **kwargs):
try:
response = f(*args, **kwargs)
db.session.commit()
return response
except:
db.session.rollback()
raise
finally:
db.session.close()
return decorated_function
return decorator
To further ensure I wasn't missing any commit calls, I made a Flask wrapper that enabled code such as (if I remember correctly):
#managed_session()
def hello(self):
model = Model(prop=value)
db.session.add(model)
return render_template(...
Unfortunately, neither method worked. I also recall trying to issue SELECT(1) calls in an attempt to re-establish the connection, but I don't have that code anymore.
To me, the bottom line is MySQL/SQL Alchemy has issues. When I migrated to Postgres, I didn't have to worry about my commits. Everything just worked.
I was having this problem and it was driving me nuts. I tried playing with SQLALCHEMY_POOL_RECYCLE but this didn't seem to fix the problem.
I finally found http://docs.sqlalchemy.org/en/latest/orm/session.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it , and adapted for flask-sqlalchemy.
After I started using the following pattern, I haven't seen the problem. The key seems to be always assuring that a commit() or rollback() is executed. So if there is if-then-else which doesn't flow through commit() (e.g., for detected error), also do commit() or rollback() before redirect, abort, render_template call.
class DoSomething(MethodView):
def get(self):
try:
# do stuff
db.session.commit()
return flask.render_template('sometemplate.html')
except:
db.session.rollback()
raise
app.add_url_rule('/someurl',view_func=DoSomething.as_view('dosomething'),methods=['GET'])
UPDATE 7/22/2014
I discovered that I also had to change the SQLALCHEMY_POOL_RECYCLE to be less than the MySQL interactive_timeout. On the godaddy server, interactive_timeout was set to 60, so I set SQLALCHEMY_POOL_RECYCLE to 50. I think both the pattern I used, and this timeout were necessary to make the problem go away, but at this point I'm not positive. However, I'm pretty sure that when SQLALCHEMY_POOL_RECYCLE was greater than interactive_timeout, I was still getting the operational error.
I ran across the same issue recently - the first request to the MYSQL database after a long period of FLASK & SQLAlchemy application inactivity (at least 8 hours) results in an unhandled exception, which in turn implies 500 Internal Server Error: Connection Unavailable. All subsequent requests are just fine.
I managed to boil down the problem to MYSQL connection by decreasing the ##session.wait_timeout (and ##global just in case) value to 5 seconds. Then every odd request was just alright, while every second after 5-plus-second pause failed. The conclusion was obvious - SQLAlchemy was using open, but timeouted on the database end connection.
Solution
In my case it turned out the solution is spelled out in the SQLAlchemy – MYSQL has gone away blog post:
The first thing to make sure of is [...] the value of pool_recycle should be less than your MYSQLs wait_timeout value.
In MYSQL documentation you can find wait_timeout defaults to 8 hours (28 800 seconds), while SQLAlchemy engine's pool_recycle default value is -1, that entails no connection recycle whatsoever. I simply passed the value of 21 600 (6 hours) to the create_engine function and the error is gone.
sqlalchemy provides 2 ways of handling with disconnections, details in the documentation
Short version:
Optimistically
use try...except block to catch disconnection exceptions. This will return a 500 on the failing request, then the web application continues as normal. So use this one if disconnection happens infrequently. Note: you'll need to wrap each potential-to-fail operations in the try...except block.
Pessimistically (the one I'm using)
Basically do an extra ping operation (something like SELECT 1) each time a connection is checked out from the pool. If the ping fails raise DisconnectionError, upon which the host pool will attempt to force a new connection to be created (in fact the pool will try 3 times before officially give up). In this way your application won't see 500 error. The tradeoff is the extra SQL executed, although according to the doc the overhead is small.
Related
So, basically,someone was telling me today that the way I Query my database was bad before I never close the connection after finishing and thus I probably had many open connection with the database running simultaneously.
I looked online however, I did not get a straight answer. I am using MySQL Mariadb.
and this is how I am doing it
db = mysql.connector.connect(host='localhost', username, passwd, db='mydb')
# and actually call this function and pass query
def db_execute(query):
cursor = db.cursor()
cursor.execute(query)
db.commit()
is this decent? how can I close the connection after I finish? what's better?
db = mysql.connector.connect(host='localhost', username, passwd, db='mydb')
# and actually call this function and pass query
def db_execute(query):
cursor = db.cursor()
try:
cursor.execute(query)
db.commit()
#You can also catch any exceptions you are interested in here
finally:
cursor.close()
db.close()
Finally block gets executed even if you code throws an exception, hence your connection is always closed even if something unexpected happens. If you do not use finally and there is an exception in your code before the close statement, the connection remains open, which is undesired.
Promptly close anything inside the database. This applies to "cursors", "transactions", etc.
As for disconnect() (however it is spelled), the rules are different.
Do not do a connect-disconnect around each SQL statement; this is terribly inefficient.
In a web application, do connect at early in the "web page" and either disconnect when finished, or let the containing code (PHP, etc) deal with closing when it goes away. Web pages are short-lived and cannot leave "state" for the next page when the user clicks a link.
In a long-running application, be aware that after some number of minutes (configurable) of not touching the database, the connection to the database will be closed. If you then try to do some SQL, you will get some form of error. For such rare applications, you may need to connect+disconnect more than once. And you may as well do them at times convenient to the application.
Generally having lots of idle connections is "not a problem". However, if it exceeds max_connections, newcomers will get an error about 'cannot connect'.
"Connection pooling" adds another dimension to this discussion. I won't go there.
cursor.close() and then db.close()
We are having issues recently with our prod servers connecting to Oracle. Intermittently we are getting "DatabaseError: ORA-12502: TNS:listener received no CONNECT_DATA from client". This issue is completely random and goes away in a second by itself and it's not a Django problem, can replicate it with SQLPlus from the servers.
We opened ticket with Oracle support but in the meantime i'm wondering if it's possible to simply retry any DB-related operation when it fails.
The problem is that i can't use try/catch blocks in the code to handle this since this can happen on ANY DB interaction in the entire codebase. I have to do this at a lower level so that i do it only once. Is there any way to install an error handler or something like that directly on django.db.backends.oracle level so that it will cover all the codebase? Basically, all i want to do is this:
try:
execute_sql()
catch:
if code == ORA-12502:
sleep 1 second
#re-try the same operation
exexute_sql()
Is this even possible or I'm out of luck?
Thanks!
I am using psycopg2 (2.6.1) to connect to Amazon's Redshift.
I have a query that should last about 1 second, but about 1 time out of every 20 concurrent tries it just hangs forever (I manually kill them after 1 hour). To address this, I configured the statement_timeout setting before my query, as such:
rcur.execute("SET statement_timeout TO 60000")
rcur.execute(query)
so that after 1 minute the query will give up, and I can try again (the second try does complete quickly as expected), but even with this (which I confirmed worked by setting the timeout to 1 ms and seeing it raise an Exception), sometimes the Python code hangs instead of raising an Exception (it never reaches the print directly after the rcur.execute(query)). And I can see in the Redshift AWS dashboard that the query has been "terminated" after 59 seconds, but my code still hangs for an hour instead of raising an Exception.
Does anyone know how to resolve this, or have a better method of dealing with typically short queries that occasionally take unnaturally long and simply need to be cancelled and retried?
I think you need to configure your keepAlive settings for the Redshift connection.
Follow the steps in this AWS doc to do that,
http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html
I have a Flask web application that uses SQLAlchemy to access a PostgreSQL database.
When I start the application, there is instantly created an " in transaction" connection in PostgreSQL.
When the application has been used for some time, several of these connections appear in pg_stat_activity.
After some time, it appears that deadlocks occurs on some resources, and I have to restart the application to get it working again.
I have read that this can happen, if I return from a view function that uses database before closing the db session. So in order to avoid this problem, I have created the following decorator:
#app.teardown_appcontext
def shotdown_session(exception=None):
db.session.remove()
This should cause all sessions to be closed after each request and effectively avoid the problem of having " in transaction" connections.
Unfortunately, it does not seem to have any effect.
So, how do I really solve this problem?
UPDATE:
I should probably add, that I have verified that my decorator function is actually run.
I verified this by adding a print to it:
#app.teardown_appcontext
def shotdown_session(exception=None):
print "#app.teardown_appcontext: shotdown_session()"
db.session.remove()
I have also verified that it is indeed run AFTER return of the view function by adding a print to the view function as well:
[...]
products = db.session.query(...).all()
print "BEFORE RETURN"
return render_template("show_products.html", products=products)
This produces log lines like these:
* Running on http://0.0.0.0:5000/
* Restarting with reloader
BEFORE RETURN
#app.teardown_appcontext: shotdown_session()
10.0.0.100 - - [03/Dec/2014 13:41:30] "GET /product/51 HTTP/1.1" 200 -
I also went through the code and added a db.session.remove() call before each return in each function using db.session.
This does get rid of the in transaction, however, it also causes problems. I pass SQLAlchemy model objects from the database along to the templates. Some templates then do some operations on these model objects that causes the application to fail since the object is no longer attached to a session.
EDIT 2014-12-08:
Connection set up can be seen here:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
from flask_sqlalchemy import SQLAlchemy
from config import cfg
engine = create_engine(cfg["db_uri"], echo=False, pool_size=10)
db = SQLAlchemy()
Base = db.Model
Session = scoped_session(sessionmaker(bind=engine))
The structure of the entire application can be found here: http://paste.yt/p3219.html
I've seen this situation occur when you run Flask in Debug mode. If your code throws an exception and the debugger kicks in, the transaction will never get "rolled back" or "removed". As a result, the session that was used on the request that failed never gets returned to the pool.
The solution is to disable debug mode.
EDIT:
There's another circumstance where I've seen this happen. If you have code that runs autonomously (i.e. not part of an HTTP transaction – like an independent thread started and spawned off at launch of the Flask app), it will usually involve a sleep. If you access the session before the sleep, then you'll end up with a hung transaction like this during the sleep.
Another possibility is you are accessing a session from the create app function. If you do so, make sure to .remove() it. Otherwise, that session could remain hung on the main thread in a gevent app.
from sqlalchemy.pool import NullPool
use the NullPoll as poolclass solved the problem for me. Not sure why.
EDIT(Mar 23 2021):
Despite I got downvote, If using uWSGI, this probably the only choice. Check the sqlachemy official doc
https://docs.sqlalchemy.org/en/14/core/pooling.html#using-connection-pools-with-multiprocessing-or-os-fork
In my case I had async functions that started creating connections that would become idle or idle in transaction (based on whether I was using isolation_level=AUTOMATIC or not). They would continue to pile up until the # of db connections would reach the pool_size limit. When I removed the async nature from these handlers, the transactions pooled properly and the connection space no longer clogged itself. ¯\_(ツ)_/¯
#user.route("/<string:user_id>", methods=["GET"])
# async def user_get(user_id): // doesn't work
def user_get(user_id):
user = db.session.query(User).filter_by(id=user_id).first()
return jsonify({"data": prl.to_dict()}), 200
I have built little custom web framework on top of Python 3.2 using Cherrypy to built WSGI application and SQLAlchemy Core (just for connection pooling and executing text SQL statements).
Versions I am using:
Python: 3.2.3
CherryPy: 3.2.2
SQL Alchemy: 0.7.5
Psycopg2: 2.4.5
For every request, a DB connection is retrieved from pool using sqlalchemy.engine.base.Engine´s connect method. After request handler finishes, the connection is closed using close method. Pseudocode for example:
with db.connect() as db:
handler(db)
Where db.connect() is context manager defined like this:
#contextmanager
def connect(self):
conn = self.engine.connect()
try:
yield conn
finally:
conn.close()
I hope that this is correct practice for doing this task. It worked until things went more complicated in page handlers.
I am getting weird behavior. Because of uknown reason, connection is sometimes closed before the handler finishes it´s work. But not every time!
By observation, this happens only when making requests quickly consecutively. If I make small pause between requests, the connection is not closed and request is finished successfully. But anyway, this does not happen every time. I have not found more specific pattern in failures/successes of requests.
I observed that the connection is not closed by my context manager. It is already closed at that point.
My question:
How to figure out when, why and by what code is my connection closed?
I tried debugging. I put breakpoint on sqlalchemy.engine.base.Connection´s close method but the connection is closed before it reach this code. Which is weird.
I will appreciate any tips or help.
*edit *
Information requested by zzzeek:
symptom of the "connection being closed":
Sorry for not clarifying this before. It is the sqlalchemy.engine.Connection that is closed.
In handlers I am calling sqlalchemy.engine.base.Connection´s execute method to get data from database (select statements). I can say that sqlalchemy.engine.Connection is closed, because I am checking it's closed property before calling execute.
I can post here traceback, but only thing that you will probably see in it is that Exception is raised before the execute in my DB wrapper library (because connection is closed).
If I remove this check (and let the execute method execute), SQLAlchemy raises this exception: http://pastebin.com/H6052yca
Regarding the concurency problem that zzzeek mentioned. I must apologize. After more observation the situation is slightly different.
This is exact procedure how to invoke the error:
Request for HandlerA. Everything ok.
Wait moment (about 10-20s).
Request for HandlerB. Everything ok.
Request for HandlerA. Everything ok.
Immediate request for HandlerB. Error!
Immediate request for HandlerB. Error!
Immediate request for HandlerB. Error!
Wait moment (about 10-20s).
Request for HandlerB. Everything ok.
I am using default SQLAlchemy pooling class with pool_size = 5.
I know that you cannot do miracles when you don't have the actual code. But unfortunately, I cannot share it. Is there any best practice for debugging this type of error? Or the only option is to debug more deeply step by step and try to figure it out?
Another observation:
When I start the server in debugger (WingIDE), I cannot bring up the error. Probably because the the debugger is so slow when interpreting the code, that the connection is somehow "repaired" before second request (RequestB) is handled.
After daylong debugging. I found out the problem.
Unfortunatelly it was not related to SQLAlchemy directly. So the question should be deleted. But you guys tried to help me, so I will answer my own question. And maybe, somebody will find this helpfull some day.
Basically, Error was caused by my custom publish/subscribe methods which did not play nicely in multi threaded enviorment.
I tried stepping code line by line... which was not working (as i described in the question). So I started generating very detailed log of what is going on.
Even then, everything looked normal, until I noticed that few lines before crash, the address of Connection object referenced in the model changed. Which practically meant that something assigned another Connection object to model and that connection object was already closed.
So the lesson is. When everything looks correct, print out / log the repr() of objects which are problematic.
Thanks to commenters for their time.