We're occasionally getting the following error when doing queries:
OperationFailure: database error: error querying server
There is no specific query causing this, and when repeating the process things work. Has anybody else seen this error?
Our setup is a cluster of Ubuntu VMs on Amazon EC2, we're using Python 2.7.3 and pymongo v2.3. We're also using Mongoengine, however we still get this exception from non-Mongoengine code.
To those discovering this question:
We were never able to fully diagnose the problem with this, our hunch is that the database connection tends to fail every once in a while for whatever reason. From our research into distributed computing, this is a common problem and needs to be handled explicitly.
In the end, we adapted our system to become robust to DB connection failures by catching OperationFailure exceptions along with similar ones and re-establishing the database connection. This resolved the problem along with a number of similar ones we were having.
Seems the query failed on the server - to diagnose you'd need to check the server logs.
Related
I am currently trying to update a lot of objects in my mongodb database that is saved on Azure. My code is
mongo_client = mongoengine.connect(
db=db,
host=h,
port=p,
username=config('AZURE_MONGO_USERNAME'),
password=config('AZURE_MONGO_PASSWORD'),
authentication_source="admin",
ssl=True,
ssl_cert_reqs=ssl.CERT_NONE,
retrywrites=False)
instruments=collection.objects()
for i in instruments:
perform update
Now, I am constantly getting a pymongo.errors.CursorNotFound error after a while of running it. From what I understand from different answers is that the problem is a time out of the mongodb connection. I looked into https://docs.mongoengine.org/ and tried
instruments=collection.objects().timeout(enabled=False)
but it didnt solve the problem. I could try to reduce the batch size but my problem is that I cannot sort the query - the function .order_by('name') gives me an error that seems to be related to cosmodb on azure (locally it works fine).
Is there a standard way of handling this problem and solving the error?
Thank you!
I am developing a web-based application using Python, Flask, MySQL, and uWSGI. However, I am not using SQL Alchemy or any other ORM. I am working with a preexisting database from an old PHP application that wouldn't play well with an ORM anyway, so I'm just using mysql-connector and writing queries by hand.
The application works correctly when I first start it up, but when I come back the next morning I find that it has become broken. I'll get errors like mysql.connector.errors.InterfaceError: 2013: Lost connection to MySQL server during query or the similar mysql.connector.errors.OperationalError: 2055: Lost connection to MySQL server at '10.0.0.25:3306', system error: 32 Broken pipe.
I've been researching it and I think I know what the problem is. I just haven't been able to find a good solution. As best as I can figure, the problem is the fact that I am keeping a global reference to the database connection, and since the Flask application is always running on the server, eventually that connection expires and becomes invalid.
I imagine it would be simple enough to just create a new connection for every query, but that seems like a far from ideal solution. I suppose I could also build some sort of connection caching mechanism that would close the old connection after an hour or so and then reopen it. That's the best option I've been able to come up with, but I still feel like there ought to be a better one.
I've looked around, and most people that have been receiving these errors have huge or corrupted tables, or something to that effect. That is not the case here. The old PHP application still runs fine, the tables all have less than about 50,000 rows, and less than 30 columns, and the Python application runs fine until it has sat for about a day.
So, here's to hoping someone has a good solution for keeping a continually open connection to a MySQL database. Or maybe I'm barking up the wrong tree entirely, if so hopefully someone knows.
I have it working now. Using pooled connections seemed to fix the issue for me.
mysql.connector.connect(
host='10.0.0.25',
user='xxxxxxx',
passwd='xxxxxxx',
database='xxxxxxx',
pool_name='batman',
pool_size = 3
)
def connection():
"""Get a connection and a cursor from the pool"""
db = mysql.connector.connect(pool_name = 'batman')
return (db, db.cursor())
I call connection() before each query function and then close the cursor and connection before returning. Seems to work. Still open to a better solution though.
Edit
I have since found a better solution. (I was still occasionally running into issues with the pooled connections). There is actually a dedicated library for Flask to handle mysql connections, which is almost a drop-in replacement.
From bash: pip install Flask-MySQL
Add MYSQL_DATABASE_HOST, MYSQL_DATABASE_USER, MYSQL_DATABASE_PASSWORD, MYSQL_DATABASE_DB to your Flask config. Then in the main Python file containing your Flask App object:
from flaskext.mysql import MySQL
mysql = MySQL()
mysql.init_app(app)
And to get a connection: mysql.get_db().cursor()
All other syntax is the same, and I have not had any issues since. Been using this solution for a long time now.
I'm using a Python script that does certain flow control on our outgoing mail messages, mostly checking whether a user is sending spam.
The script establishes a persistent connection with a database via a SQLObject. Under certain circumstances, the connection is dropped by a third-party (e.g. our firewall closes the connection due to excess idle), and the SQLObject doesn't notice it has been closed and it continues sending queries on a dead TCP handler, resulting in log entries like these:
Feb 06 06:56:07 mailsrv2 flow: ERROR Processing request error: [Failure instance: Traceback: <class 'psycopg2.InterfaceError'>: connection already closed#012/usr/lib/python2.7/threading.py:524:__bootstrap#012/usr/lib/python2.7/threading.py:551:__bootstrap_inner#012/usr/lib/python2.7/threading.py:504:run#012--- <exception caught here>---#012
/opt/scripts/virtualenv/local/lib/python2.7/site-packages/twisted/python/threadpool.py:191:_worker#012
/opt/scripts/virtualenv/local/lib/python2.7/site-packages/twisted/python/context.py:118:callWithContext#012
/opt/scripts/virtualenv/local/lib/python2.7/site-packages/twisted/python/context.py:81:callWithContext#012
/opt/scripts/flow/server.py:91:check#012
/opt/scripts/flow/flow.py:252:check#012
/opt/scripts/flow/flow.py:155:append_to_log#012
/opt/scripts/virtualenv/local/lib/python2.7/site-packages/SQLObject-1.3.1-py2.7.egg/sqlobject/main.py:1226:__init__#012
/opt/scripts/virtualenv/local/lib/python2.7/site-packages/SQLObject-1.3.1-py2.7.egg/sqlobject/main.py:1274:_create#012
/opt/scripts/virtualenv/local/lib/python2.7/site-packages/SQLObject-1.3.1-py2.7.egg/sqlobject/main.py:1298:_SO_finishCreate#012
/opt/scripts/virtualenv/local/lib/python2.7/site-packages/SQLObject-1.3.1-py2.7.egg/sqlobject/dbconnection.py:468:queryInsertID#012
/opt/scripts/virtualenv/local/lib/python2.7/site-packages/SQLObject-1.3.1-py2.7.egg/sqlobject/dbconnection.py:327:_runWithConnection#012
/opt/scripts/virtualenv/local/lib/python2.7/site-packages/SQLObject-1.3.1-py2.7.egg/sqlobject/postgres/pgconnection.py:191:_queryInsertID#012]
This makes me think that indeed there must be some callback for this kind of situation, otherwise that log entry wouldn't be written. I'd use that callback to establish a new connection to the database. I've been unable to find any piece of documentation about that.
Does anyone know if it's even possible to implement that callback and how to declare it?
Thanks.
We're more regular users of SQLAlchemy rather than SQLObject. According to this thread from 2010 (http://sourceforge.net/p/sqlobject/mailman/message/26460439), SQLObject does not support reconnection logic for PostgreSQL. It's an old thread, but there does not appear to be any discussion about solving this from within SQLObject.
I have three suggested solutions.
The first solution is to explore Connection Pools. It might provide a way to open a new connection object when SQLObject detects the psycopg2 has disconnected. I can't guarantee it will, but if it does this solution would be your best best as it requires the least amount of changes on your part.
The second solution is to switch your backend from Postgres to MySQL. The SQLObject docs provide information on how use the reconnection logic of the mysql driver - http://sourceforge.net/p/mysql-python/feature-requests/9
The third solution is to switch to SQLAlchemy as your ORM and to use their version of Connection Pools. According to the core event documentation, when using pools if a connection is dropped or closed a new connection will opened -- http://docs.sqlalchemy.org/en/rel_0_9/core/exceptions.html#sqlalchemy.exc.DisconnectionError
Best of luck
I am hosting a web app at pythonanywhere.com and experiencing a strange problem. Every half-hour or so I am getting the OperationalError: (2006, 'MySQL server has gone away'). However, if I resave my wsgi.py file, the error disappears. And then appears again some half-an-hour later...
During the loading of the main page, my app checks a BOOL field in a 1x1 table (basically whether sign-ups should be open or closed). The only other MySQL actions are inserts into another small table, but none of these appear to be associated with the problem.
Any ideas for how I can fix this? I can provide more information as is necessary. Thanks in advance for your help.
EDIT
Problem turned out to be a matter of knowing when certain portions of code run. I assumed that every time a page loaded a new connection was opened. This was not the case; however, I have fixed it now.
It normally because your mysql network connect be disconnected, may by your network gateway/router, so you have two options. One is always build a mysql connect before every query (not using connect pool etc). Second is try and catch this error, then get connect and query db again.
I'm running a Django project on Postgresql 8.1.21 (using Django 1.1.1, Python2.5, psycopg2, Apache2 with mod_wsgi 3.2). We've recently encountered this lovely error:
OperationalError: FATAL: connection limit exceeded for non-superusers
I'm not the first person to run up against this. There's a lot of discussion about this error, specifically with psycopg, but much of it centers on older versions of Django and/or offer solutions involving edits to code in Django itself. I've yet to find a succinct explanation of how to solve the problem of the Django ORM (or psycopg, whichever is really responsible, in this case) leaving open Postgre connections.
Will simply adding connection.close() at the end of every view solve this problem? Better yet, has anyone conclusively solved this problem and kicked this error's ass?
Edit: we later upped Postgresql's limit to 500 connections; this prevented the error from cropping up, but replaced it with excessive memory usage.
This could be caused by other things. For example, configuring Apache/mod_wsgi in a way that theoretically it could accept more concurrent requests than what the database itself may be able to accept at the same time. Have you reviewed your Apache/mod_wsgi configuration and compared limit on maximum clients to that of PostgreSQL to make sure something like that hasn't been done. Obviously this presumes though that you have managed to reach that limit in Apache some how and also depends on how any database connection pooling is set up.