We Keep Coding

Is it ever okay to catch a generic exception in Python? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I was just reading through this article from Real Python about the evils of catching generic exceptions.
I'm working on a large-ish program in Python at the moment. This program spawns some subprocesses (using Popen) and then calls some other code over and over again. Being a large program, it has the potential to throw a variety of classes of exception, many of which I haven't even thought of yet. This is a problem because it means that the subprocesses aren't killed properly.
I want to do something like this:
while keep_going_signal():
try:
do_the_thing()
except Exception as e:
kill_subprocesses()
raise Exception(e)
However, given the dire warnings about catching generic exceptions in the aforementioned article, I'm a bit nervous about doing so.
How should I proceed?

There is nothing wrong with a catch-all exception handler as long as it does not catch more exceptions than the action in the handler is meant to handle. The action in your exception handler is meant to perform a cleanup for all errors, so it justifies a catch-all exception handler.
That said, to re-raise an exception in an exception handler, you should instead use the raise statement without passing to it a new Exception object in order to preserve the call stack in the associated traceback object for debugging purposes:
try:
do_the_thing()
except Exception:
kill_subprocesses()
raise

If you're concerned about cleaning the process but not catching everything you could try... finally:
while keep_going_signal():
try:
do_the_thing()
except SomePrettyLikelySpecificException e:
handle_it()
finally:
if process_is_still_running():
kill_subprocesses()
You'd have to supply process_is_still_running() yourself, of course.
There may be some exception(al) situations which take the whole of the runtime down with them, in which case your subprocesses are probably beyond saving.
(Edit: this example catches the predictable exception types which we believe we can handle. Others have pointed out the significance and merits of this and of re-throwing ones we can't handle.)
https://docs.python.org/3/tutorial/errors.html

Catching an exception means that you are prepared to handle certain kinds of problems which may occur and that you have a plan for what to do when that happens. That should usually be as narrow as possible, so you're really only handling the specific problems you're prepared for and for which you have a specific remedy.
Now, if your code is very broad, then it also makes sense to have a broad error handling. Say you're writing a web framework like Flask or Tornado. You'll be making calls into user supplied code, over which you have zero influence and which may raise any kind of error. But just because the user code raised an error, you don't want that to affect the web server. So you'd always encapsulate any and all calls into unknown user code with the broadest possible exception handler, because your primary goal is to keep the server running, regardless of what issues the user code may have.
So, yes, it's perfectly fine to have a generic exception handler in the right circumstances.

How to clean up upon a crash?

I would like some clean-up activities to occur in case of crash of my program. I understand that some situations cannot be handled (a SIGKILL for instance) but I would like to cover as much as possible.
The atexit module was a good candidate but the docs explicitely state that
The functions registered via this module are not called when the
program is killed by a signal not handled by Python, when a Python
fatal internal error is detected, or when os._exit() is called.
Are there functions or python features which allow to handle sys.exit() and unhandled exceptions program terminations? (these are the main one I am concerned with)

SIGKILL cannot be handled, no matter what, your program is just terminated (killed violently), and you can do nothing about this.
The only thing you can do about SIGKILL is to look for data that needs to be cleaned up during the next launch of your program.
For other cases use atexit to handle Python's interpreter normal termination. If you've got some unhandled exceptions, see where they can occur and wrap these pieces of code in try/except blocks:
try:
pass
except ValueError as e:
pass
except:
# catch other exceptions
pass
To deal with sys.exit calls, you can wrap the entire program's starting point in a try/except block and catch the SystemExit exception:
try:
# your program goes here
# you're calling your functions from here, etc
except SystemExit:
# do cleanup
raise

Elegant way to create a failed deferred with a stacktrace?

I'm implementing a function that is supposed to return a deferred. Inside that function I decide that an error happened.
I could just raise the error:
raise ValueError("...")
but then the function is not returning a deferred anymore. I don't want to use maybeDeferred anytime I call it.
Alternatively, I could return a deferred like this:
return defer.fail(Failure(ValueError(...)))
This works, but the Failure won't include a stacktrace, making it hard to track the error down.
The best thing I found so far is this:
try:
raise ValueError("...")
except:
return defer.fail()
I get a deferred back and the Failure contains the stacktrace. But this is rather verbose.
Is there a better way that I'm missing?
I'm a bit surprised that such a common thing hasn't an elegant solution.

This question surprised me a little, and I had to think for a while about why.
If all you want is for the failure with traceback to bubble up and get logged by the global error handler, you needn't return a defer.fail(). Raising the exception will get that behavior.
The difference would be in a situation like this:
foo().addErrback(fooErrorHandler)
in that case, fooErrorHandler would get called when the result of the deferred was a failure but not when it was an exception raised synchronously. That would require this more cumbersome form:
try:
foo().addErrback(fooErrorHandler)
except Exception, err:
fooErrorHandler(Failure(err))
which admittedly looks pretty bad. But the situation we're talking about is now a case where
you define an explicit error handler for this call
the error handler does something with the failure's traceback
the function can fail synchronously or asynchronously
You use the same error handler for both the synchronous and asynchronous failure modes.
which is maybe why it hasn't come up as such a common thing as you might think.
Of course, the other reason it might not have come up as a common thing is that people are often lazy about defining error handlers, and often lazy about documenting the sorts of exceptions their code may raise.
Ah-hah, something else that may have added to my confusion: Failure does know how to to store its stack, but it was explicitly changed to not do this unless there's a traceback for performance reasons. The method described by that commit message to get a traceback is the same as the four-line try/except example in your post.
I guess that could be an option to Failure() (as captureVars is), or an alternate constructor method, if this does come up enough to warrant it.

What is the risk of not catching exceptions in a CGI script

I have a python CGI script that takes several query strings as arguments.
The query strings are generated by another script, so there is little possibility to get illegal arguments,
unless some "naughty" user changes them intentionally.
A illegal argument may throw a exception (e.g. int() function get non-numerical inputs),
but does it make sense to write some codes to catch such rare errors? Any security risk or performance penalty if not caught?
I know the page may go ugly if exceptions are not nicely handled, but a naughty user deserves it, right?

Any unhandled exception causes program to terminate.
That means if your program is doing some thing and exception occurs it will shutdown in an unclean fashion without releasing resources.
Any ways CGI is obsolete use Django, Flask, Web2py or something.

What is the proper way to handle (in python) IOError: [Errno 4] Interrupted system call, raised by multiprocessing.Queue.get

When I use multiprocessing.Queue.get I sometimes get an exception due to EINTR.
I know definitely that sometimes this happens for no good reason (I open another pane in a tmux buffr), and in such a case I would want to continue working and retry the operation.
I can imagine that in some other cases The error would be due to a good reason and I should stop running or fix some error.
How can I distinguish the two?
Thanks in advance

The EINTR error can be returned from many system calls when the application receives a signal while waiting for other input. Typically these signals can be quite benign and already handled by Python, but the underlying system call still ends up being interrupted. When doing C/C++ coding this is one reason why you can't entirely rely on functions like sleep(). The Python libraries sometimes handle this error code internally, but obviously in this case they're not.
You might be interested to read this thread which discusses this problem.
The general approach to EINTR is to simply handle the error and retry the operation again - this should be a safe thing to do with the get() method on the queue. Something like this could be used, passing the queue as a parameter and replacing the use of the get() method on the queue:
import errno
def my_queue_get(queue, block=True, timeout=None):
while True:
try:
return queue.get(block, timeout)
except IOError, e:
if e.errno != errno.EINTR:
raise
# Now replace instances of queue.get() with my_queue_get(queue), with other
# parameters passed as usual.
Typically you shouldn't need to worry about EINTR in a Python program unless you know you're waiting for a particular signal (for example SIGHUP) and you've installed a signal handler which sets a flag and relies on the main body of the code to pick up the flag. In this case, you might need to break out of your loop and check the signal flag if you receive EINTR.
However, if you're not using any signal handling then you should be able to just ignore EINTR and repeat your operation - if Python itself needs to do something with the signal it should have already dealt with it in the signal handler.

Old question, modern solution: as of Python 3.5, the wonderful PEP 475 - Retry system calls failing with EINTR has been implemented and solves the problem for you. Here is the abstract:
System call wrappers provided in the standard library should be retried automatically when they fail with EINTR , to relieve application code from the burden of doing so.
By system calls, we mean the functions exposed by the standard C library pertaining to I/O or handling of other system resources.
Basically, the system will catch and retry for you a piece of code that failed with EINTR so you don't have to handle it anymore. If you are targeting an older release, the while True loop still is the way to go. Note however that if you are using Python 3.3 or 3.4, you can catch the dedicated exception InterruptedError instead of catching IOError and checking for EINTR.