We are having issues recently with our prod servers connecting to Oracle. Intermittently we are getting "DatabaseError: ORA-12502: TNS:listener received no CONNECT_DATA from client". This issue is completely random and goes away in a second by itself and it's not a Django problem, can replicate it with SQLPlus from the servers.
We opened ticket with Oracle support but in the meantime i'm wondering if it's possible to simply retry any DB-related operation when it fails.
The problem is that i can't use try/catch blocks in the code to handle this since this can happen on ANY DB interaction in the entire codebase. I have to do this at a lower level so that i do it only once. Is there any way to install an error handler or something like that directly on django.db.backends.oracle level so that it will cover all the codebase? Basically, all i want to do is this:
try:
execute_sql()
catch:
if code == ORA-12502:
sleep 1 second
#re-try the same operation
exexute_sql()
Is this even possible or I'm out of luck?
Thanks!
Related
In my program there are multiple asynchronous functions updating data in a database.
There can be some cases where the functions are executed parallelly.
My question is :
In my case, do I need to create a new connection each time in a function or having a single connection throughout the program will work fine.
Second, in case of a single connection is it necessary to close it at the end?
Also, please recommend the best tool to access .db file outside the code [Just in case], that shouldn't interrupt the connections of the code with the database even if I make some changes personally outside the code.
Note : Am on windows
Thanks!
I made a python web app and deployed it to heroku successfully and it works well to an extent.
The problem starts when once in a while the worker process throws a connection reset by peer error once in a while for which i have to go in and redeploy again only for it to happen again.
This process affects the entire web app as those small glitches cause the entire program to malfunction and produce inconsistent if not wrong information, so I'm trying to validate if an exception handling statement in the following format would work:
def conti():
opens the connection to the site
performs the operations needed
closes the connection
try:
conti()
except:
conti()
How can i make the try statement sort of recursive so that whenever the error happens it would still continue.
Do i need to put the try statement in a recursive function to handle the error.
Thank you.
My recommendation is to consider a connection pool. If you are on Heroku and using PostgreSQL, you are probably already using psycopg2, which has a pool built in. See psycopg2 and infinite python script
This will avoid either recursion or explicit connection state/error detection in your code.
I have a python script running on my server which accessed a database, executes a fetch query and runs a learning algorithm to classify and updates certain values and means depending on the query.
I want to know if for some reason my server shuts down in between then my python script would shut down and my query lost.
How do i get to know where to continue from once I re-run the script and i want to carry on the updated means from the previous queries that have happened.
First of all: the question is not really related to Python at all. It's a general problem.
And the answer is simple: keep track of what your script does (in a file or directly in db). If it crashes continue from the last step.
I have a one year production site configured with django.contrib.sessions.backends.cached_db backend with a MySQL database backend. The reason why I chose cached_db is a mix of security with read performance.
The problem is, the cleanup command, responsible to delete all expired sessions, was never executed, resulting in a 2.3GB session table data length, 6 million rows and 500Mb index length.
When I try to run the ./manage.py cleanup (in Django 1.3) command, or ./manage.py clearsessions (Django`s 1.5 correspondent), the process never ends (or my patience doesn't complete 3 hours).
The code that Django use's to do this is:
Session.objects.filter(expire_date__lt=timezone.now()).delete()
In a first impression, I think that's normal because the table has 6M rows, but, after I inspect System's monitor, I discover that all memory and cpu was used by the python process, not mysqld, fullfilling my machine's resources. I think that's something terrible wrong with this command code. It seems that python iterates over all founded expired session rows before deleting each of them, one by one. In this case, a code refactoring to just raw a DELETE FROM command can resolve my problem and helps Django community, right? But, if this is the case, a Queryset delete command is acting weird and none optimized in my opinion. Am I right?
I have built little custom web framework on top of Python 3.2 using Cherrypy to built WSGI application and SQLAlchemy Core (just for connection pooling and executing text SQL statements).
Versions I am using:
Python: 3.2.3
CherryPy: 3.2.2
SQL Alchemy: 0.7.5
Psycopg2: 2.4.5
For every request, a DB connection is retrieved from pool using sqlalchemy.engine.base.Engine´s connect method. After request handler finishes, the connection is closed using close method. Pseudocode for example:
with db.connect() as db:
handler(db)
Where db.connect() is context manager defined like this:
#contextmanager
def connect(self):
conn = self.engine.connect()
try:
yield conn
finally:
conn.close()
I hope that this is correct practice for doing this task. It worked until things went more complicated in page handlers.
I am getting weird behavior. Because of uknown reason, connection is sometimes closed before the handler finishes it´s work. But not every time!
By observation, this happens only when making requests quickly consecutively. If I make small pause between requests, the connection is not closed and request is finished successfully. But anyway, this does not happen every time. I have not found more specific pattern in failures/successes of requests.
I observed that the connection is not closed by my context manager. It is already closed at that point.
My question:
How to figure out when, why and by what code is my connection closed?
I tried debugging. I put breakpoint on sqlalchemy.engine.base.Connection´s close method but the connection is closed before it reach this code. Which is weird.
I will appreciate any tips or help.
*edit *
Information requested by zzzeek:
symptom of the "connection being closed":
Sorry for not clarifying this before. It is the sqlalchemy.engine.Connection that is closed.
In handlers I am calling sqlalchemy.engine.base.Connection´s execute method to get data from database (select statements). I can say that sqlalchemy.engine.Connection is closed, because I am checking it's closed property before calling execute.
I can post here traceback, but only thing that you will probably see in it is that Exception is raised before the execute in my DB wrapper library (because connection is closed).
If I remove this check (and let the execute method execute), SQLAlchemy raises this exception: http://pastebin.com/H6052yca
Regarding the concurency problem that zzzeek mentioned. I must apologize. After more observation the situation is slightly different.
This is exact procedure how to invoke the error:
Request for HandlerA. Everything ok.
Wait moment (about 10-20s).
Request for HandlerB. Everything ok.
Request for HandlerA. Everything ok.
Immediate request for HandlerB. Error!
Immediate request for HandlerB. Error!
Immediate request for HandlerB. Error!
Wait moment (about 10-20s).
Request for HandlerB. Everything ok.
I am using default SQLAlchemy pooling class with pool_size = 5.
I know that you cannot do miracles when you don't have the actual code. But unfortunately, I cannot share it. Is there any best practice for debugging this type of error? Or the only option is to debug more deeply step by step and try to figure it out?
Another observation:
When I start the server in debugger (WingIDE), I cannot bring up the error. Probably because the the debugger is so slow when interpreting the code, that the connection is somehow "repaired" before second request (RequestB) is handled.
After daylong debugging. I found out the problem.
Unfortunatelly it was not related to SQLAlchemy directly. So the question should be deleted. But you guys tried to help me, so I will answer my own question. And maybe, somebody will find this helpfull some day.
Basically, Error was caused by my custom publish/subscribe methods which did not play nicely in multi threaded enviorment.
I tried stepping code line by line... which was not working (as i described in the question). So I started generating very detailed log of what is going on.
Even then, everything looked normal, until I noticed that few lines before crash, the address of Connection object referenced in the model changed. Which practically meant that something assigned another Connection object to model and that connection object was already closed.
So the lesson is. When everything looks correct, print out / log the repr() of objects which are problematic.
Thanks to commenters for their time.