Python 2.7 Multi-threaded DB-Connection crashes randomly - python

We use cx_Oracle to connect multiple threads to the database and issue various selects and updates.
However, for unknown reasons, the script is being killed by the system on random database connections. There are no informations in the syslog or messages-files.
Due to the error handling we try to write tracebacks in the logfile. Unfortunately we have no information about the crash of the script in the logfile. Only in stdout is a printout with "PID killed" at the last line.
Could it be a problem to make database connections with multiple threads at the same time? There are also other scripts running at the same time that also talk to the database (not multi-threaded) but access other tables.
This is our Function that is called for every select. For the updates we have other Functions.
def ora_send_sql( logger, statement):
try:
dsn = cx_Oracle.makedsn(SQL_IP, SQL_PORT, SQL_SID)
con = cx_Oracle.connect(SQL_USR, SQL_PWD, dsn)
cur = con.cursor()
cur.execute(statement)
con.commit()
con.close()
return 0
except cx_Oracle.Warning as w:
logger.warning(" Oracle-Warning: "+ str(e).strip())
pass
except cx_Oracle.Error as e:
error, = exc.args
logger.error(" Oracle-Error-Code:", error.code)
logger.error(" Oracle-Error-Message:", error.message)
return -1
except:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
logger.exception(" Got Traceback in ora_send_sql: " + str(exc_type) + " Fname=" + str(fname) + " Lineno=" + str(exc_tb.tb_lineno))
return -2

I don't suppose you tried turning except: into except Exception as e and then trying to see if the exception is somewhat special. Another possible thing to try perhaps is removing the exception handling completely and letting it crash, then investigating the output. That way you could spot the actual exception that is thrown, because I simply cannot believe it would just "crash". Finally, try investigating dmesg for any segfaults.

I think we have fixed the problem by updating cx_Oracle to the latest version.
They fixed a few memory-leaks in the updates.....
But that still does not explain why we do not see any information about the killing....

You almost certainly need to use threaded=True in the connect() call, see http://cx-oracle.readthedocs.io/en/latest/module.html#cx_Oracle.Connection

Related

Catch any of the errors in psycopg2 without listing them explicitly

I have a try and except block where I would like to catch only the errors in the psycopg2.errors and not any other error.
The explicit way would be:
try:
# execute a query
cur = connection.cursor()
cur.execute(sql_query)
except psycopg2.errors.SyntaxError, psycopg2.errors.GroupingError as err:
# handle in case of error
The query will always be some SELECT statement. If the execution fails it should be handled. Any other exception not belonging to psycopg, e.g. like ZeroDivisionError, should not be caught from the except clause. However, I would like to avoid to list all errors after the except clause. In fact, if you list the psycopg errors, you get a quite extensive list:
from psycopg2 import errors
dir(errors)
I have searched quite extensively and am not sure if this question has been asked already.
You can you use the base class psycopg2.Error it catch all psycopg2 related errors
import psycopg2
try:
cur = connection.cursor()
cur.execute(sql_query)
except psycopg2.Error as err:
# handle in case of error
see official documentation
Meanwhile, I have implemented by catching a generic Exception and checking if the exception belongs to the list returned by dir(errors). The solution proposed by Yannick looks simpler, though.
The function that I use prints the error details and checks using the name of the exception err_type.__name__ whether it is in any of the psycopg errors:
from psycopg2 import errors
def is_psycopg2_exception(_err):
err_type, err_obj, traceback = sys.exc_info()
print ("\npsycopg2 ERROR:", _err, "on line number:", traceback.tb_lineno)
print ("psycopg2 traceback:", traceback, "-- type:", err_type)
return err_type.__name__ in dir(errors)
Then, I use this function in the try/except clause:
try:
# execute a query
cur = connection.cursor()
cur.execute(sql_query)
except Exception as err:
if is_psycopg2_exception(err):
# handle in case of psycopg error
else:
# other type of error
sys.exit(1) # quit
For my very specific case, where I need to check for other other exceptions as well, I can readapt Yannick solution as follows:
try:
# execute a query
cur = connection.cursor()
cur.execute(sql_query)
except psycopg2.OperationalError as err:
# handle some connection-related error
except psycopg2.Error as err:
# handle in case of other psycopg error
except Exception as err:
# any other error
sys.exit(1) # quit

Identify a specific exception in python

I have a problem with identifying an exception.
Im writing a scraper that scrapes a lot of different websites, and some errors I want to handle and some I only want to ignore.
I except my exceptions like this:
except Exception as e:
most of the exceptions I can identify like this:
type(e).__name__ == "IOError"
But I have one exception "[Errno 10054] An existing connection was forcibly closed by the remote host"
that has the name "error" which is too vague and Im guessing other errors also have that name. Im guessing I can somehow get the errno number from my exception and thus identify it. But I don't know how.
First, you should not rely on the exception's class name, but on the class itself - two classes from two different modules can have the same value for the __name__ attribute while being different exceptions. So what you want is:
try:
something_that_may_raise()
except IOError as e:
handle_io_error(e)
except SomeOtherError as e:
handle_some_other_error(e)
etc...
Then you have two kind of exceptions: the one that you can actually handle one way or another, and the other ones. If the program is only for your personal use, the best way to handle "the other ones" is usually to not handle them at all - the Python runtime will catch them, display a nice traceback with all relevant informations (so you know what happened and where and can eventually add some handling for this case).
If it's a "public" program and/or if you do have some things to clean up before the program crash, you can add a last "catch all" except clause at the program's top level that will log the error and traceback somewhere so it isn't lost (logging.exception is your friend), clean what has to be cleaned and terminate with a more friendly error message.
There are very few cases where one would really want to just ignore an exception (I mean pretending nothing wrong or unexpected happened and happily continue). At the very least you will want to notify the user one of the actions failed and why - in your case that might be a top-level loop iterating over a set of sites to scrap, with an inner try/except block catching "expected" error cases, ie:
# config:
config = [
# ('url', {params})
('some.site.tld', {"param1" : value1, "param2" : value2}),
('some.other.tld', {"param1" : value1, "answer" : 42}),
# etc
]
def run():
for url, params in config:
try:
results = scrap(url, **params)
except (SomeKnownError, SomeOtherExceptedException) as e:
# things that are to be expected and mostly harmless
#
# you configured your logger so that warnings only
# go to stderr
logger.warning("failed to scrap %s : %s - skipping", url, e)
except (MoreSeriousError, SomethingIWannaKnowAbout) as e:
# things that are more annoying and you want to know
# about but that shouldn't prevent from continuing
# with the remaining sites
#
# you configured your logger so that exceptions goes
# to both stderr and your email.
logger.exception("failed to scrap %s : %s - skipping", url, e)
else:
do_something_with(results)
Then have a top-level handler around the call to run() that takes care of unexpected errors :
def main(argv):
parse_args()
try:
set_up_everything()
run()
return 0
except Exception as e:
logger.exception("oops, something unexpected happened : %s", e)
return 1
finally:
do_some_cleanup()
if __name__ == "__main__":
sys.exit(main(sys.argv))
Note that the logging module has an SMTPHandler - but since mail can easily fail too you'd better still have a reliable log (stderr and tee to a file ?) locally. The logging module takes some time to learn but it really pays off in the long run.

how to properly use try/except/with inside functions and main

I am a relative python newbie and I am getting confused with how to properly handle exceptions. Apologies for the dumb question.
In my main() I iterate through a list of dates and for each date I call a function, which downloads a csv file from a public web server. I want to properly catch exceptions for obvious reasons but especially because I do not know when the files of interest will be available for download. My program will execute as part of a cron job and will attempt to download these files every 3 hours if available.
What I want is to download the first file in the list of dates and if that results in a 404 then the program shouldn't proceed to the next file because the assumption is if the oldest date in the list is not available then none of the others that come after it will be available either.
I have the following python pseudo code. I have try/except blocks inside the function that attempts to download the files but if an exception occurred inside the function how do I properly handle it in the main() so I can make decisions whether to proceed to the next date or not. The reason why I created a function to perform the download is because I want to re-use that code later on in the same main() block for other file types.
def main():
...
...
# datelist is a list of date objects
for date in datelist:
download_file(date)
def download_file(date):
date_string = str(date.year) + str(date.strftime('%m')) + str(date.strftime('%d'))
request = HTTP_WEB_PREFIX+ date_string + FILE_SUFFIX
try:
response = urllib2.urlopen(request)
except urllib2.HTTPError, e:
print "HTTPError = " + str(e)
except urllib2.URLError, e:
print "URLError = " + str(e)
except httplib.HTTPException, e:
print "HTTPException = " + str(e)
except IOError:
print "IOError = " + str(e)
except Exception:
import traceback
print "Generic exception: " + traceback.format_exc()
else:
print "No problem downloading %s - continue..." % (response)
try:
with open(TMP_DOWNLOAD_DIRECTORY + response, 'wb') as f:
except IOError:
print "IOError = " + str(e)
else:
f.write(response.read())
f.close()
The key concept here is, if you can fix the problem, you should trap the exception; if you can't, it's the caller's problem to deal with. In this case, the downloader can't fix things if the file isn't there, so it should bubble up its exceptions to the caller; the caller should know to stop the loop if there's an exception.
So let's move all the exception handling out of the function into the loop, and fix it so it craps out if there's a failure downloading the file, as the spec requires:
for date in datelist:
date_string = str(date.year) +
str(date.strftime('%m')) +
str(date.strftime('%d'))
try:
download_file(date_string)
except:
e = sys.exc_info()[0]
print ( "Error downloading for date %s: %s" % (date_string, e) )
break
download_file should now, unless you want to put in retries or something like that, simply not trap the exceptions at all. Since you've decoded the date as you like in the caller, that code can come out of download_file as well, giving the much simpler
def download_file(date_string):
request = HTTP_WEB_PREFIX + date_string + FILE_SUFFIX
response = urllib2.urlopen(request)
print "No problem downloading %s - continue..." % (response)
with open(TMP_DOWNLOAD_DIRECTORY + response, 'wb') as f:
f.write(response.read())
f.close()
I would suggest that the print statement is superfluous, but that if you really want it, using logger is a more flexible way forward, as that will allow you to turn it on or off as you prefer later by changing a config file instead of the code.
From my understanding of your question... you should just insert code into the except blocks that you want to execute when you encounter the particular exception. You don't have to print out the encountered error, you can do whatever you feel is necessary when it is raised... provide a popup box with information/options or otherwise direct your program to the next step. Your else section should isolate that portion, so it will only execute if none of your exceptions are raised.

Python and MySQLdb - Using DROP TABLE IF EXISTS seems to throw exception

I got this code..
.....
try:
task_db.cursor.execute('DROP TABLE IF EXISTS `tasks`')
print "Affected: %d" % task_db.cursor.rowcount
except MySQLdb.Error, e:
print "Error ocurred: %s " % e.args[0]
print e
If the tasks table doesn't exist, then I get a warning like
create_database.py:11: Warning: Unknown table 'tasks'
But if the table does exist then I wont get that warning.
Odd?
Catching the MySQLdb.Warning didn't work for me, so I found another way to suppress warnings:
import warnings
warnings.filterwarnings("ignore", "Unknown table.*")
And you can edit the second parameter with whatever you want to suppress.
The most elegant way to avoid Mysql warnings :
from warnings import filterwarnings
import MySQLdb
filterwarnings('ignore', category = MySQLdb.Warning)
It's perfectly correct behaviour. If the table exists, then it's dropped. If it doesn't exist then you get Warning exception which is not the same as Error - you are free to ignore it and nothing bad should happen, but you have to catch it to allow your script to proceed.
EDIT:
To prevent Warning from bubbling up, just catch it as any other exception:
try:
[some code]
except MySQLdb.Warning:
[exception handling code]

Writing Pythons Errors to a txt file

I am having trouble getting this to work correctly (obviously) - I am ALMOST there, and I have a good idea of WHY it is not working - just not sure how to make it work.
This is suppose to attempt to read a file into memory, if it fails it goes to the "except" clause of the block of code (that part is 'duh'). The error file prints: "<main.DebugOutput instance at 0x04021EB8>". What I want it to do is print the actual Error. Like a FileIOError or TraceBackError or whatever it is to that error file. This is just the beginning stages and I plan to add things like date stamps and to have it append, not write/create - I just need the actual error printed to the file. Advice?
import os, sys
try:
myPidFile = "Zeznadata.txt"
myOpenPID_File = open(myPidFile, "r") #Attempts to open the file
print "Sucessfully opened the file: \"" + myPidFile + "\"."
except:
print "This file, \"" + myPidFile + "\", does not exist. Please check the file name and try again. "
myFileErr = open("PIDErrorlog.txt", "w")
myStdError = str(sys.stderr)
myFileErr.write(myStdError)
myFileErr.close()
print "\nThis error was logged in the file (and stored in the directory): "
First, you should use a logging library. It will help you deal with different logging levels (info/warn/error), timestamps and more.
http://docs.python.org/library/logging.html
Second, you need to catch the error, and then you can log details about it. This example comes from the Python documentation.
import sys
try:
f = open('myfile.txt')
s = f.readline()
i = int(s.strip())
except IOError as (errno, strerror):
print "I/O error({0}): {1}".format(errno, strerror)
except ValueError:
print "Could not convert data to an integer."
except:
print "Unexpected error:", sys.exc_info()[0]
raise
See how it catches an IOError and assigns the error number and error message to variables? You can have an except block for each type of error you want to deal with. In the last (generic error) block, it uses sys.exc_info()[0] to get the error details.
http://docs.python.org/tutorial/errors.html
the problem is here:
myStdError = str(sys.stderr)
myFileErr.write(myStdError)
sys.stderr is a file-like interface defined in POSIX standards, called standard error, not "error text" that you can write to a file. So what you (probably) wanted to do is:
sys.stderr = myFileErr
This is the python equivalent to python your_python_sctipt.py 2> PIDErrorLog.txt in unix shell.
Use the logging module. The logging module have exception formatters that will help you print exceptions in a pretty way.
http://docs.python.org/library/logging.html
Like Kimvais says, your problem is:
myStdError = str(sys.stderr)
myFileErr.write(myStdError)
sys.stderr is a file handle to stderr (stdout is what print writes to).
You should do something like:
try:
...open file...
except IOError as (errno, strerror):
...open errfile...
errfile.write(strerror)
errfile.close()
Alison's answer is also very good and well written.

Categories