We're developing a website which requires database access. Accessing such a page works fine; accessing multiple in a row is also fine. However, if you wait for a long amount of time (15 minutes seems to be enough), accessing another page will hang for a long time (10-20 minutes has been observed). Afterwards, this error will print.
Here's the relevant code:
if __name__ == "__main__":
conf = load_conf(sys.argv[1])
engine = create_engine('postgresql://%s:%s#%s:%s/%s' %
(conf['db']['user'], conf['db']['pw'], conf['db']['address'],
conf['db']['port'], conf['db']['database']), echo=False)
Session = sessionmaker(bind=engine)
session = Session()
app = make_app(session, conf)
app.listen(8888)
tornado.ioloop.IOLoop.current().start()
The database is on a different server. My personal machine is in the Netherlands, while the database is in a server in Germany. My partner's personal machine is in Italy.
Most notably, this issue is only present on my machine, running Arch Linux. We've tested this on two other machines, running Windows and some other Linux (I assume Ubuntu, can check if needed). At this point, we have no clue on how to continue debugging.
Of course, I will provide any extra needed information on request.
It's unclear where this 15-minute timeout is coming from, although as other commenters have indicated it's likely coming from something in the network between your computer and the server. However, wherever it is coming from, there are a couple of options to work around it in SQLAlchemy.
The pool_pre_ping=True option will issue a simple test query before any attempt to reuse a connection, allowing it to detect this problem and reconnect transparently (at a small performance cost)
The pool_recycle=600 option tells sqlalchemy not to reuse a connection that has been idle for longer than 10 minutes. This is a more efficient solution to the problem since it doesn't add any new queries, but it requires you to work out the best recycle timeout to use.
Related
Everything worked great until today, while I did not change anything in the code.
I can't connect to the database not from the application, not from the IDE, did something go wrong on the heroku side? I have not seen news with global updates over the past couple of days on the heroku website. Can anyone advise how to solve the current problem?
I use the free version of dyno and postgresql, I definitely still have a lot of free space (less than 1 thousand fields). It looks like blocking access to the database locally, not from the service side.
What I would try in your situation:
Go to https://data.heroku.com/, select your Datastore and check everything there: Health, number of connections, number of rows, data size.
If everything is fine: Go to settings -> database credentials and try setting up a connection from any desktop tool such as Navicat or Pgadmin. What error message do you get?
Set-up another database on Heroku and try the same. If the second DB works, there is an issue with the first one. If it does not, it's rather about your setup/settings.
Hope that helps
I am developing a web-based application using Python, Flask, MySQL, and uWSGI. However, I am not using SQL Alchemy or any other ORM. I am working with a preexisting database from an old PHP application that wouldn't play well with an ORM anyway, so I'm just using mysql-connector and writing queries by hand.
The application works correctly when I first start it up, but when I come back the next morning I find that it has become broken. I'll get errors like mysql.connector.errors.InterfaceError: 2013: Lost connection to MySQL server during query or the similar mysql.connector.errors.OperationalError: 2055: Lost connection to MySQL server at '10.0.0.25:3306', system error: 32 Broken pipe.
I've been researching it and I think I know what the problem is. I just haven't been able to find a good solution. As best as I can figure, the problem is the fact that I am keeping a global reference to the database connection, and since the Flask application is always running on the server, eventually that connection expires and becomes invalid.
I imagine it would be simple enough to just create a new connection for every query, but that seems like a far from ideal solution. I suppose I could also build some sort of connection caching mechanism that would close the old connection after an hour or so and then reopen it. That's the best option I've been able to come up with, but I still feel like there ought to be a better one.
I've looked around, and most people that have been receiving these errors have huge or corrupted tables, or something to that effect. That is not the case here. The old PHP application still runs fine, the tables all have less than about 50,000 rows, and less than 30 columns, and the Python application runs fine until it has sat for about a day.
So, here's to hoping someone has a good solution for keeping a continually open connection to a MySQL database. Or maybe I'm barking up the wrong tree entirely, if so hopefully someone knows.
I have it working now. Using pooled connections seemed to fix the issue for me.
mysql.connector.connect(
host='10.0.0.25',
user='xxxxxxx',
passwd='xxxxxxx',
database='xxxxxxx',
pool_name='batman',
pool_size = 3
)
def connection():
"""Get a connection and a cursor from the pool"""
db = mysql.connector.connect(pool_name = 'batman')
return (db, db.cursor())
I call connection() before each query function and then close the cursor and connection before returning. Seems to work. Still open to a better solution though.
Edit
I have since found a better solution. (I was still occasionally running into issues with the pooled connections). There is actually a dedicated library for Flask to handle mysql connections, which is almost a drop-in replacement.
From bash: pip install Flask-MySQL
Add MYSQL_DATABASE_HOST, MYSQL_DATABASE_USER, MYSQL_DATABASE_PASSWORD, MYSQL_DATABASE_DB to your Flask config. Then in the main Python file containing your Flask App object:
from flaskext.mysql import MySQL
mysql = MySQL()
mysql.init_app(app)
And to get a connection: mysql.get_db().cursor()
All other syntax is the same, and I have not had any issues since. Been using this solution for a long time now.
I've built a small python REST service using Flask, with Flask-SQLAlchemy used for talking to the MySQL DB.
If I connect directly to the MySQL server everything is good, no problems at all. If I use HAproxy (handles HA/failover, though in this dev environment there is only one DB server) then I constantly get MySQL server has gone away errors if the application doesn't talk to the DB frequently enough.
My HAproxy client timeout is set to 50 seconds, so what I think is happening is it cuts the stream, but the application isn't aware and tries to make use of an invalid connection.
Is there a setting I should be using when using services like HAproxy?
Also it doesn't seem to reconnect automatically, but if I issue a request manually I get Can't reconnect until invalid transaction is rolled back, which is odd since it is just a select() call I'm making, so I don't think it is a commit() I'm missing - or should I be calling commit() after every ORM based query?
Just to tidy up this question with an answer I'll post what I (think I) did to solve the issues.
Problem 1: HAproxy
Either increase the HAproxy client timeout value (globally, or in the frontend definition) to a value longer than what MySQL is set to reset on (see this interesting and related SF question)
Or set SQLALCHEMY_POOL_RECYCLE = 30 (30 in my case was less than HAproxy client timeout) in Flask's app.config so that when the DB is initialised it will pull in those settings and recycle connections before HAproxy cuts them itself. Similar to this issue on SO.
Problem 2: Can't reconnect until invalid transaction is rolled back
I believe I fixed this by tweaking the way the DB is initialised and imported across various modules. I basically now have a module that simply has:
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
Then in my main application factory I simply:
from common.database import db
db.init_app(app)
Also since I wanted to easily load table structures automatically I initialised the metadata binds within the app context, and I think it was this which cleanly handled the commit() issue/error I was getting, as I believe the database sessions are now being correctly terminated after each request.
with app.app_context():
# Setup DB binding
db.metadata.bind = db.engine
I have a django application, which is making use of SQLAlchemy to connect to a SQL Server instance on Windows Azure. The app has worked perfectly for 3 months on a local SQL Server instance, and for over a month on an Azure instance. The issues appeared this monday, after a week without any code changes.
The site uses:
Python 2.7
Django 1.6
Apache/Nginx
SQLAlchemy 0.9.3
pyODBC 3.0.7
FreeTDS
The application appears to lock up right after a connection is pulled out of the Pool (I have setup verbose logging at every point in the workflow). I assumed this had something to do with the connections going stale. So we tried making the pool_recycle incredibly short (5 secs), all the way up to an hour. That did not help.
We also tried using the NullPool to force a new connection on every page view. However that does not help either. After about 15 minutes the site will completely lock up again (meaning no pages that use the database are viewable).
The weird thing is, half the computers that experience the "hang", will end up loading the page about 15 minutes later.
Has anyone had any experience with SQL Azure and SQLAlchemy?
I found a workaround for this issue. Please note that this is definitely not a fix, since the site worked perfectly fine before. We could not determine what the actual issue is because SQL Azure has no error log (one of the 100 reasons I would suggest never considering SQL Azure over a real database server).
I got around the problem by turning off all Connection Pooling, at the application level, AND at the driver level.
Things started consistently working after making my /etc/odbcinst.ini look like:
[FreeTDS]
Description = TDS driver (Sybase/MS SQL)
# Some installations may differ in the paths
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
CPReuse =
CPTimeout = 0
FileUsage = 1
Pooling = No
The key being setting CPTimeout (Connection Pool Timeout) to 0, and Pooling to No. Just turning pooling off at the application level (in SQL Alchemy) did not work, only after setting it at the driver level did things start working smoothly.
I am now at 4 days without a problem after changing that setting.
There is a multi-million record database in MS SQL Server 2008 R2 under Windows XP SP3.
My co-worker has written a .Net application, which directly connects to this db and runs a reasonable amount of queries. I dont know .Net, but I'm sure this app does not connect to DB using ODBC.
On the other hand I've written a command-line python (CPython version 2.7.5) application, which connects to this database and runs simple queries on it to send the data over internet to somewhere else. Database connection is made using pyodbc 3.0.7 (installer from http://www.lfd.uci.edu/~gohlke/pythonlibs/) and a DSN which uses SQL Server Native Client 10.0 driver. I've tried both disabling and enabling connection pools for this driver in Connection Pooling tab of windows Data Sources (ODBC) applet. The script sends 100 records from the db and then closes the connection and sleeps for 2 minutes and then runs again.
Both of these programs run constantly on the same machine as db.
The question is the .Net app runs nicely when I remove the defined DSN ( and of course the python script is not running). When I define the DSN again and start the python script to run side by side .Net app, there is no problem for roughly 5 hours. but then gradually while python script is mostly fine, the .Net app starts to get timeouts from the db.
What could go wrong that this happens?
EDIT:
python script (which connects using ODBC) runs fine, all the time. but .Net app falls behind of usual performance after couple of hours. When I close the python script, .Net app still stays behind. but when I remove the ODBC DSN which I defined for python script, .Net app goes back to normal performance. It's very strange. As I said I don't know anything about .Net, so this could be result of non-standard code on the part of .Net app, maybe open transactions, locks, too many connections, etc. To make the case stranger, cutting database size to half by deleting records and rebuilding indexes, seems to have solved the .Net app issue so far.
EDIT 2:
The only two queries that python script runs, are:
SELECT TOP 100 FROM tbl_data WHERE id > ? ORDER BY id
and
SELECT * FROM tbl_data WHERE id = ?
The first query usually only runs once in each run of the python script. The second one runs at most 100 times. id is primary key, and so is indexed. As you can see the queries could not be simpler. For the first query I read the entire result set in the program to not keep a cursor open on DB server. Also I've turned off connection pooling in ODBC applet for the driver I'm using, so after each run of the script, DB connection should have been disposed and all resources on DB server should have been freed. The script sleeps for 2 minutes and then repeats this.
The queries that .Net app runs are far more complex, combined with some triggers on the database. And the strange thing is it runs mostly fine, on itself. but when DSN is defined, it starts to get long waits on a single insert statement, which sometimes results in timeout.
Also I should have said that Windows and MSSQL are not updated with latest patches from microsoft, so if it's a bug in ODBC driver or MSSQL itself, it could have been already solved for others.
EDIT 3
Table is clustered on PK index. the data table contains about 1.5M records now. DB size is about 160GB. Server does not have a high spec. Intel Core i7 2600, 4GB RAM, plain 1TB SATA disk drive.
There are a number of things that can be affecting performance. There could be resource contention, sub-optimal SQL Server configuration, slow disks, missing indexes etc...
I would start by monitoring the system resources when both processes are running. You can use perfmon to monitor the OS and SQL counters. I would start by looking at
ProcessorInformation/%ProcessorTime
LogicalDisk/Avg Disk sec/read
LogicalDisk/Avg Disk sec/write
Memory/Available MBytes
SQLServerBufferManager/BufferCacheHitRatio
SQLServerBufferManager/PageLifeExpectency
Here is a great article on using perfmon, http://www.brentozar.com/archive/2006/12/dba-101-using-perfmon-for-sql-performance-tuning/
The next step is optimizing query and SQL Server performance.
Query Performance, http://technet.microsoft.com/en-us/magazine/2007.11.sqlquery.aspx
SQL Server Performance,
-Separate data and log files on different volumes (raid 10 is preferable)
-tempdb (http://technet.microsoft.com/en-us/library/ms175527(v=SQL.105).aspx)
-configure min/max memory
-configure lock pages in memory and instant file initialization
-Check out this link, http://www.brentozar.com/sql/sql-server-performance-tuning/
The point is that there are so many possibilities that could be affecting your database, that it is nearly impossible to identify your problem based on the provided information. I recommend that you go through each of these items to identify why your system is not running optimally.