Is there a way to create a middleware that will catch every raised exception and print the stacktrace both to log and stdout (possibly with some additional information) in Pylons framework?
Standard paste.exceptions.errormiddleware.ErrorMiddleware already does this, and even a little more.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I was just reading through this article from Real Python about the evils of catching generic exceptions.
I'm working on a large-ish program in Python at the moment. This program spawns some subprocesses (using Popen) and then calls some other code over and over again. Being a large program, it has the potential to throw a variety of classes of exception, many of which I haven't even thought of yet. This is a problem because it means that the subprocesses aren't killed properly.
I want to do something like this:
while keep_going_signal():
try:
do_the_thing()
except Exception as e:
kill_subprocesses()
raise Exception(e)
However, given the dire warnings about catching generic exceptions in the aforementioned article, I'm a bit nervous about doing so.
How should I proceed?
There is nothing wrong with a catch-all exception handler as long as it does not catch more exceptions than the action in the handler is meant to handle. The action in your exception handler is meant to perform a cleanup for all errors, so it justifies a catch-all exception handler.
That said, to re-raise an exception in an exception handler, you should instead use the raise statement without passing to it a new Exception object in order to preserve the call stack in the associated traceback object for debugging purposes:
try:
do_the_thing()
except Exception:
kill_subprocesses()
raise
If you're concerned about cleaning the process but not catching everything you could try... finally:
while keep_going_signal():
try:
do_the_thing()
except SomePrettyLikelySpecificException e:
handle_it()
finally:
if process_is_still_running():
kill_subprocesses()
You'd have to supply process_is_still_running() yourself, of course.
There may be some exception(al) situations which take the whole of the runtime down with them, in which case your subprocesses are probably beyond saving.
(Edit: this example catches the predictable exception types which we believe we can handle. Others have pointed out the significance and merits of this and of re-throwing ones we can't handle.)
https://docs.python.org/3/tutorial/errors.html
Catching an exception means that you are prepared to handle certain kinds of problems which may occur and that you have a plan for what to do when that happens. That should usually be as narrow as possible, so you're really only handling the specific problems you're prepared for and for which you have a specific remedy.
Now, if your code is very broad, then it also makes sense to have a broad error handling. Say you're writing a web framework like Flask or Tornado. You'll be making calls into user supplied code, over which you have zero influence and which may raise any kind of error. But just because the user code raised an error, you don't want that to affect the web server. So you'd always encapsulate any and all calls into unknown user code with the broadest possible exception handler, because your primary goal is to keep the server running, regardless of what issues the user code may have.
So, yes, it's perfectly fine to have a generic exception handler in the right circumstances.
Will calls to logger block request processing in a Django views?
For instance, in the example below, will the logger.info() at line 2 block the response on line 3?
def random_view(request):
xyz.do_something()
logger.info("Does this block the next line execution?")
return HttpResponse("Done")
I've looked at the Django logging documentation and the python logging module docs but I can't find a concrete answer to this question.
I understand the the Handler decides where each log message goes, but I am not sure if the Handler itself is non-blocking. Would appreciate if someone could clarify or point to documentation explaining this.
From Django Docs
Django uses Python’s builtin logging module to perform system logging.
The usage of this module is discussed in detail in Python’s own
documentation.
From Python Docs
Sometimes you have to get your logging handlers to do their work
without blocking the thread you’re logging from. This is common in Web
applications, though of course it also occurs in other scenarios.
So, although not explicitly mentioned yet logging does seem to be blocking. For details see Python Docs.
I have a python CGI script that takes several query strings as arguments.
The query strings are generated by another script, so there is little possibility to get illegal arguments,
unless some "naughty" user changes them intentionally.
A illegal argument may throw a exception (e.g. int() function get non-numerical inputs),
but does it make sense to write some codes to catch such rare errors? Any security risk or performance penalty if not caught?
I know the page may go ugly if exceptions are not nicely handled, but a naughty user deserves it, right?
Any unhandled exception causes program to terminate.
That means if your program is doing some thing and exception occurs it will shutdown in an unclean fashion without releasing resources.
Any ways CGI is obsolete use Django, Flask, Web2py or something.
In the app I'm currently working on (2.7 runtime), I'm trying to make sure that exceptions at the API level (i.e. not my code) are handled correctly within my application. However, it appears that Google/AppEngine handle those exceptions internally and doesn't bubble them up. For instance, using Thing which is a previously defined ndb.Model
t = Thing(id=1,name='thingy')
try:
t.put()
except Exception as e:
self.log(e)
self.abort(500)
In the unlikely event that something goes awry with the put() I have no way to catch/log that event -- or is there?
A similar thing happens with storing data to the blobstore where exceptions are, apparently, caught and raised internally and leaving no chance for me to log those.
Perhaps I'm missing a key point? I've looked through the API docs but the exceptions raised by services and how to catch them doesn't seem to be a priority for documentation team.
Actually App Engine logs every single request. Just go to the application's dashboard and click on Logs.
If you want to log something on your own you should use the logging library and you can read more about it in the documentation.
So instead of self.log you should use logging.error.
I've been using GAE for more than a year now, and one of the most difficult things for me to deal with is the fact that my otherwise well written code may occasionally raise an exception because of a GAE hiccup.
I already have a decent procedure for unhandled exceptions. My custom request handler presents a nice error page and administrators get an email. This, however, is a bad user experience.
What I want to do is to handle exceptions so I can immediately take the appropriate action and prevent some generic error page.
My questions are:
What exceptions should I catch?
Where should I catch them?
I realize that a full answer to this is not practical, but I'm looking for some best practices for the most common situations.
For exceptions that I should catch, I sometimes see exceptions that are not on the official list. For example, I've received an UnknownError.
For where to catch exceptions, I wonder if I should catch them in each get/post method. Something like this:
def get(self):
try:
# normal get processing
except SomeException:
# redirect to the same page to try again and fix any data if necessary
I'm surprised there is not more information out there about this as this is an important aspect of any GAE app. There are some good articles here and here, but these don't answer my questions.
What exceptions should I catch?
That depends upon what level of error catching you're going for. From my experience catching the errors in the official list and linked articles will get you a very high level of error catching. If you need to go above and beyond that putting in a generic except would be easier than trying to predict unknown errors.
Where should I catch them?
The most likely place(s) for GAE errors is when interacting with the db, so setting some try-except blocks around there if you haven't will give you a good return on your effort for dealing with GAE-issue error handling.
Besides the advice of your linked articles you can also think about putting the failed operations into a task queue. Each task will automatically retry 5 times before failing which can give you some ability to ride out datastore switches or other service interruptions if you don't need immediate feedback from the operation.