Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I was just reading through this article from Real Python about the evils of catching generic exceptions.
I'm working on a large-ish program in Python at the moment. This program spawns some subprocesses (using Popen) and then calls some other code over and over again. Being a large program, it has the potential to throw a variety of classes of exception, many of which I haven't even thought of yet. This is a problem because it means that the subprocesses aren't killed properly.
I want to do something like this:
while keep_going_signal():
try:
do_the_thing()
except Exception as e:
kill_subprocesses()
raise Exception(e)
However, given the dire warnings about catching generic exceptions in the aforementioned article, I'm a bit nervous about doing so.
How should I proceed?
There is nothing wrong with a catch-all exception handler as long as it does not catch more exceptions than the action in the handler is meant to handle. The action in your exception handler is meant to perform a cleanup for all errors, so it justifies a catch-all exception handler.
That said, to re-raise an exception in an exception handler, you should instead use the raise statement without passing to it a new Exception object in order to preserve the call stack in the associated traceback object for debugging purposes:
try:
do_the_thing()
except Exception:
kill_subprocesses()
raise
If you're concerned about cleaning the process but not catching everything you could try... finally:
while keep_going_signal():
try:
do_the_thing()
except SomePrettyLikelySpecificException e:
handle_it()
finally:
if process_is_still_running():
kill_subprocesses()
You'd have to supply process_is_still_running() yourself, of course.
There may be some exception(al) situations which take the whole of the runtime down with them, in which case your subprocesses are probably beyond saving.
(Edit: this example catches the predictable exception types which we believe we can handle. Others have pointed out the significance and merits of this and of re-throwing ones we can't handle.)
https://docs.python.org/3/tutorial/errors.html
Catching an exception means that you are prepared to handle certain kinds of problems which may occur and that you have a plan for what to do when that happens. That should usually be as narrow as possible, so you're really only handling the specific problems you're prepared for and for which you have a specific remedy.
Now, if your code is very broad, then it also makes sense to have a broad error handling. Say you're writing a web framework like Flask or Tornado. You'll be making calls into user supplied code, over which you have zero influence and which may raise any kind of error. But just because the user code raised an error, you don't want that to affect the web server. So you'd always encapsulate any and all calls into unknown user code with the broadest possible exception handler, because your primary goal is to keep the server running, regardless of what issues the user code may have.
So, yes, it's perfectly fine to have a generic exception handler in the right circumstances.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I usually use try-except blocks when I eg need input data from the user or try to start a thread. But is there some rule of thumb telling when one should definitely use a try-except block? As technically speaking nothing prohibits you of doing something as "clever" as:
try:
print("Hello world")
except:
print("Bye bye world")
Should I implement a try-except block whenever I for example feel like one of the following errors could arise?
$ python -m pydoc builtins
BaseException
Exception
ArithmeticError
FloatingPointError
OverflowError
ZeroDivisionError
AssertionError
AttributeError
BufferError
EOFError
ImportError
ModuleNotFoundError
LookupError
IndexError
KeyError
MemoryError
NameError
UnboundLocalError
OSError
BlockingIOError
ChildProcessError
ConnectionError
BrokenPipeError
ConnectionAbortedError
ConnectionRefusedError
ConnectionResetError
FileExistsError
FileNotFoundError
InterruptedError
IsADirectoryError
NotADirectoryError
PermissionError
ProcessLookupError
TimeoutError
ReferenceError
RuntimeError
NotImplementedError
RecursionError
StopAsyncIteration
StopIteration
SyntaxError
IndentationError
TabError
SystemError
TypeError
ValueError
UnicodeError
UnicodeDecodeError
UnicodeEncodeError
UnicodeTranslateError
Warning
BytesWarning
DeprecationWarning
FutureWarning
ImportWarning
PendingDeprecationWarning
ResourceWarning
RuntimeWarning
SyntaxWarning
UnicodeWarning
UserWarning
GeneratorExit
KeyboardInterrupt
SystemExit
It really should only be used when there are errors that might happen in a program that you want to handle a certain way. There might be errors you know could potentially happen, such as a memory error, but unless you want your program to react a certain way, you shouldn't use a try-except block.
For a smooth user experience, it might also be good to catch certain exceptions that are out of your control (like a Connection Error) so that you can tell your user what happened and they can try to remedy it.
Exceptions are raised when called code encounters a problem that it cannot solve itself. For example when arguments are invalid, or when resources it attempts to access are not responding properly. Exceptions are generally meant to be exceptional and while they may occur when performing things, the normal control flow would be without exceptions.
You should catch exceptions, whenever called code could potentially raise an exception which you can recover from. That part is very important: There is no use catching an exception when you cannot work around that failure. Only catch exceptions that you expect to be raised.
That may seem counter intuitive after I having said that exceptions are exceptional, so expecting them seems weird. But the point is that code could raise any exception. For example, there are a lot of different external factors that could cause perfectly working code to suddenly raise an exception that it would usually never do. So you don’t just catch any exception. Instead you catch those exceptions explicitly that you expect to be eventually raised from the code and that you can work with without affecting your overall program.
I go into a lot more detail about this in my answer to another question: Why is “except: pass” a bad programming practice?
So basically, catch a specific exception when the code you are calling could raise that one and you could recover from it. Asking the user for input and want to parse this? Catch the exception from the parser and ask the user to correct it. Performing a network request to some API? Catch a network exception and maybe retry it. Writing a library for consuming an API that then performs a network request? Do not catch the network exception but let the consumer of your code decide how to recover from it.
Also, if you don’t know what exceptions code could raise, check the documentation. Usually the relevant exceptions are documented. There’s always a possibility that some exceptions may occur outside of the control of the called code, e.g. MemoryError, but those are usually neither to be expected nor really recoverable anyway, so you shouldn’t really check for those.
I'm implementing a function that is supposed to return a deferred. Inside that function I decide that an error happened.
I could just raise the error:
raise ValueError("...")
but then the function is not returning a deferred anymore. I don't want to use maybeDeferred anytime I call it.
Alternatively, I could return a deferred like this:
return defer.fail(Failure(ValueError(...)))
This works, but the Failure won't include a stacktrace, making it hard to track the error down.
The best thing I found so far is this:
try:
raise ValueError("...")
except:
return defer.fail()
I get a deferred back and the Failure contains the stacktrace. But this is rather verbose.
Is there a better way that I'm missing?
I'm a bit surprised that such a common thing hasn't an elegant solution.
This question surprised me a little, and I had to think for a while about why.
If all you want is for the failure with traceback to bubble up and get logged by the global error handler, you needn't return a defer.fail(). Raising the exception will get that behavior.
The difference would be in a situation like this:
foo().addErrback(fooErrorHandler)
in that case, fooErrorHandler would get called when the result of the deferred was a failure but not when it was an exception raised synchronously. That would require this more cumbersome form:
try:
foo().addErrback(fooErrorHandler)
except Exception, err:
fooErrorHandler(Failure(err))
which admittedly looks pretty bad. But the situation we're talking about is now a case where
you define an explicit error handler for this call
the error handler does something with the failure's traceback
the function can fail synchronously or asynchronously
You use the same error handler for both the synchronous and asynchronous failure modes.
which is maybe why it hasn't come up as such a common thing as you might think.
Of course, the other reason it might not have come up as a common thing is that people are often lazy about defining error handlers, and often lazy about documenting the sorts of exceptions their code may raise.
Ah-hah, something else that may have added to my confusion: Failure does know how to to store its stack, but it was explicitly changed to not do this unless there's a traceback for performance reasons. The method described by that commit message to get a traceback is the same as the four-line try/except example in your post.
I guess that could be an option to Failure() (as captureVars is), or an alternate constructor method, if this does come up enough to warrant it.
I have a python CGI script that takes several query strings as arguments.
The query strings are generated by another script, so there is little possibility to get illegal arguments,
unless some "naughty" user changes them intentionally.
A illegal argument may throw a exception (e.g. int() function get non-numerical inputs),
but does it make sense to write some codes to catch such rare errors? Any security risk or performance penalty if not caught?
I know the page may go ugly if exceptions are not nicely handled, but a naughty user deserves it, right?
Any unhandled exception causes program to terminate.
That means if your program is doing some thing and exception occurs it will shutdown in an unclean fashion without releasing resources.
Any ways CGI is obsolete use Django, Flask, Web2py or something.
I've been using GAE for more than a year now, and one of the most difficult things for me to deal with is the fact that my otherwise well written code may occasionally raise an exception because of a GAE hiccup.
I already have a decent procedure for unhandled exceptions. My custom request handler presents a nice error page and administrators get an email. This, however, is a bad user experience.
What I want to do is to handle exceptions so I can immediately take the appropriate action and prevent some generic error page.
My questions are:
What exceptions should I catch?
Where should I catch them?
I realize that a full answer to this is not practical, but I'm looking for some best practices for the most common situations.
For exceptions that I should catch, I sometimes see exceptions that are not on the official list. For example, I've received an UnknownError.
For where to catch exceptions, I wonder if I should catch them in each get/post method. Something like this:
def get(self):
try:
# normal get processing
except SomeException:
# redirect to the same page to try again and fix any data if necessary
I'm surprised there is not more information out there about this as this is an important aspect of any GAE app. There are some good articles here and here, but these don't answer my questions.
What exceptions should I catch?
That depends upon what level of error catching you're going for. From my experience catching the errors in the official list and linked articles will get you a very high level of error catching. If you need to go above and beyond that putting in a generic except would be easier than trying to predict unknown errors.
Where should I catch them?
The most likely place(s) for GAE errors is when interacting with the db, so setting some try-except blocks around there if you haven't will give you a good return on your effort for dealing with GAE-issue error handling.
Besides the advice of your linked articles you can also think about putting the failed operations into a task queue. Each task will automatically retry 5 times before failing which can give you some ability to ride out datastore switches or other service interruptions if you don't need immediate feedback from the operation.
I've been doing amateur coding in Python for a while now and feel quite comfortable with it. Recently though I've been writing my first Daemon and am trying to come to terms with how my programs should flow.
With my past programs, exceptions could be handled by simply aborting the program, perhaps after some minor cleaning up. The only consideration I had to give to program structure was the effective handling of non-exception input. In effect, "Garbage In, Nothing Out".
In my Daemon, there is an outside loop that effectively never ends and a sleep statement within it to control the interval at which things happen. Processing of valid input data is easy but I'm struggling to understand the best practice for dealing with exceptions. Sometimes the exception may occur within several levels of nested functions and each needs to return something to its parent, which must, in turn, return something to its parent until control returns to the outer-most loop. Each function must be capable of handling any exception condition, not only for itself but also for all its subordinates.
I apologise for the vagueness of my question but I'm wondering if anyone could offer me some general pointers into how these exceptions should be handled. Should I be looking at spawning sub-processes that can be terminated without impact to the parent? A (remote) possibility is that I'm doing things correctly and actually do need all that nested handling. Another very real possibility is that I haven't got a clue what I'm talking about. :)
Steve
Exceptions are designed for the purpose of (potentially) not being caught immediately-- that's how they differ from when a function returns a value that means "error". Each exception can be caught at the level where you want to (and can) do something about it.
At a minimum, you could start by catching all exceptions at the main loop and logging a message. This is simple and ensures that your daemon won't die. At the main loop it's probably too late to fix most problems, so you can catch specific exceptions sooner. E.g. if a file has the wrong format, catch the exception in the routine that opens and tries to use the file, not deep in the parsing code where the problem is discovered; perhaps you can try another format. Basically if there's a place where you could recover from a particular error condition, catch it there and do so.
The answer will be "it depends".
If an exception occurs in some low-level function, it may be appropriate to catch it there if there is enough information available at this level to let the function complete successfully in spite of the exception. E.g. when reading triangles from an .stl file, the normal vector of the triangle it both explicitly given and implicitly given by the sequence of the three points that make up the triangle. So if the normal vector is given as (0,0,0), which is a 0-length vector and should trigger an exception in the constructor of a Normal vector class, that can be safely caught in the constructor of a Triangle class, because it can still be calculated by other means.
If there is not enough information available to handle an exception, it should trickle upwards to a level where it can be handled. E.g. if you are writing a module to read and interpret a file format, it should raise an exception if the file it was given doesn't match the file format. In this case it is probably the top level of the program using that module that should handle the exception and communicate with the user. (Or in case of a daemon, log the error and carry on.)