Question
How do I write a custom handler for a python logging logger that will catch an IOError caused by a sudden inability to write logging information to file and re-raise the error as a different type?
Further information
I'm reading and converting a large repository of raw data and it's important for me to record the process (python logging module). If logging fails because of some error, e.g. IOError, I would like the application to gracefully exit. Over the weekend, the network location where I write my log files crashed and the conversion process gamely continued.
There can be an IOError thrown by other components in my code (e.g. a raw file to convert for a given hour is not present indicating that the detector was down for that given period in time) that are not important and can be ignored. I catch these errors and continue with the next hour of data.
If the logger threw a different, custom exception I could handle it separately. Something like (disclaimer: this doesn't work):
class WarningFileHandler(logging.FileHandler):
"""File handler for only writing WARNING log information to disk"""
def __init__(self,filename,mode='a',encoding=None,delay=False):
logging.FileHandler.__init__(self, filename, mode, encoding, delay)
def emit(self, record):
# Filter out all record objects that are not at the logging.WARNING level.
if not record.levelno == logging.WARNING:
return
try:
logging.FileHandler.emit(self, record)
except IOError as e:
# The logger cannot write successfully to disk (e.g. a network
# resource becomes unavailable).
raise LoggerIOError("[ERROR NO {0}: {1}]. Logger write "
"failure.".format(e.errno,e.strerror))
LoggerIOError is a simple, custom exception.
class LoggerIOError(Exception):
"""Logger IOError"""
pass
Can anyone provide some help or advice? I don't have much experience with the logging module and so I may be overlooking something simple.
logging.raiseExceptions should be its default value of True.
Details
OS/Python details:
Windows 7
Python 2.7
Thanks!
Exceptions are handled by the handleError method of a handler, documented here. Subclass a relevant handler to implement custom error processing. The handleError method is called from the exception clause, so sys.exc_info() should return the current exception.
Related
I use logging. The logger writes to file only. (and not to standard output.) In some cases logger raises exception from inside a logger method.
My goal is say to logger not to write anything on to standard out/err, in case of any error at all. All internal exception should be handled internally, and should be written into the log file.
Details:
My logger raised charmap error. This particular issue has been solved based on this thread.
But I afraid of that in a specific case, other exception can occur (file IO error, etc), on the field, which is very frustrating, that the logger fails, while the system works. I want to ensure that the logger don't print anything on standard out/err at all.
My expected behavior is something like this:
try:
logger.debug('Some error maker')
except:
try:
logger.debug('Error during log')
except:
pass
Just, of course, I don't want to write try-except around all of my logger statement.
Is there anything similar feature in the logger? Something quiet mode?
What about a function like this?
def my_logger(thing_to_log):
try:
logger.debug(thing_to_log)
except Exception:
try:
logger.debug('Error during log')
except Exception:
pass
In the rest of the code you'll only need:
my_logger(thing_to_log)
To improve over #mdgm answer you can use the built-in features of logging to log stack trace and details of the exception:
def my_logger(thing_to_log):
try:
logger.debug(thing_to_log)
except Exception:
try:
logger.debug('Error during log', exc_info=True, stack_info=True)
except Exception:
pass
I am scratching my head about what is the best-practice to get the traceback in the logfile only once. Please note that in general I know how to get the traceback into the log.
Let's assume I have a big program consisting of various modules and functions that are imported, so that it can have quite some depth and the logger is set up properly.
Whenever an exception may occur I do the following:
try:
do_something()
except MyError as err:
log.error("The error MyError occurred", exc_info=err)
raise
Note that the traceback is written to the log via the option exc_info=err.
My Problem is now that when everything gets a bit more complex and nested I loose control about how often this traceback is written to the log and it gets quite messy.
An example of the situation with my current solution for this problem is as follows:
from other_module import other_f
def main():
try:
# do something
val = other_f()
except (AlreadyLoggedError1, AlreadyLoggedError2, AlreadyLoggedError3):
# The error was caught within other_f() or deeper and
# already logged with traceback info where it occurred
# After logging it was raised like in the above example
# I do not want to log it again, so it is just raised
raise
except BroaderException as err:
# I cannot expect to have thought of all exceptions
# So in case something unexpected happened
# I want to have the traceback logged here
# since the error is not logged yet
log.error("An unecpected error occured", exc_info=err)
raise
The problem with this solution is, that I need to to keep track of all Exceptions that are already logged by myself and the line except (AlreadyLoggedError1, AlreadyLoggedError2, ...) gets arbitrary long and has to be put at any level between main() and the position the error actually occured.
So my question is: Is there some better (pythonic) way handling this? To be more specific: I want to raise the information that the exception was already logged together with the exception so that I do not have to account for that via an extra except block like in my above example.
The solution normally used for larger applications is for the low-level code to not actually do error handling itself if it's just going to be logged, but to put exception logging/handling at the highest level in the code possible, since exceptions will bubble up as far as needed. For example, libraries that send errors to a service like New Relic and Sentry don't need you to instrument each small part of your code that might throw an error, they are set up to just catch any exception and send it to a remote service for aggregation and tracking.
Is there anyway for the warnings.warn() function to be caught be a caller while still executing the rest of the code after the warn() call? The problem I am having is that function b will warnings.warn() if something happens, and then I want the rest of that function to finish its job and return a list of what it actually did. If a warning was thrown, I want to catch it, email it to someone, and continue on when I call that function from another module, but that isn't happening. here is what it looks like in code:
import warnings
def warn_function(arg_1):
if arg_1 > 10:
warnings.warn("Your argument was greater than 10.")
return arg_1 - 5
with warnings.catch_warnings():
warnings.filterwarnings("error")
try:
answer = warn_function(20)
except Warning:
print("A warning was thrown")
finally:
print(answer)
Yes, warnings can warn without exiting out of a function. But the way you're trying to do things just isn't going to work.
Using catch_warnings with the "error" action means you're explicitly asking Python to raise every warning as an exception. And the Python exception model doesn't have any way to resume from the point where an exception was thrown.
You can reorganize your code to provide explicit ways to "do the rest" after each possible warnings, but for non-trivial cases you either end up doing a ton of work, or building a hacky continuation-passing mechanism.
The right way to handle your use case is logging.captureWarnings. This way, all warnings go to a logger named 'py.warnings' instead of through the normal warning path. You can then configure a log handler that sends these warnings to someone via email, and you're done.
And of course once you've built this, you can use the exact same handler to get emails sent from high-severity log messages to other loggers, or to add in runtime configuration so you can turn up and down the email threshold without deploying a whole new build of the server, and so on.
If you're not already using logging, it may be easier to hook warnings manually. As the warnings introduction explains:
The printing of warning messages is done by calling showwarning(), which may be overridden; the default implementation of this function formats the message by calling formatwarning(), which is also available for use by custom implementations.
Yes, Python is encouraging you to monkeypatch a stdlib module. The code to do this looks something like:
def showwarning(message, category, filename, lineno, file=None, line=None):
fmsg = warning.formatwarning(message, category, filename, lineno, line)
# send fmsg by email
warning.showwarning = showwarning
I am using the basic python logger in django and it seems to be workng well. I have the logging setup in my setting.py as;
logging.baseConfig(level = logging.NOTSET,
format='a format',
datemt=' a datefmt',
filename='path to log',
filemode = 'a')
logging.getLogger('').setLevel(logging.NOTSET)
My question is with regard to propagating exceptions. In my code if I have a try/except clause and catch the exception so I can log it, what is the best way to then propagate that error so that I can redirect to my 500 page. I have been using
try:
do stuff
except Exception, e:
logging.error(e)
raise
but I find that this causes the exeption to be logged twice. Is there another way to do this or am I doing something wrong?
Regards
Andrew
There's no need to catch the exception just so you can log it. You can log it and handle it, or else let it bubble up to some higher level which will log it and handle it. If you want to log exceptions which occur in some view, which you don't want to handle, then you can install some exception middleware which logs the exception and either returns a custom response which you determine, or None (to return whatever response Django would normally return).
There's an example of extensible exception middleware here, which doesn't actually use logging but whose log_exception() method you could subclass to log the exception, or just use that snippet as a guide to provide your own exception middleware - it's basically just a class with a method called process_exception:
class MyExceptionMiddleware:
def process_exception(self, request, exception):
#Do your logging here
Also, note that loggers have an exception() method which works like error() but includes traceback information in the log.
There's a recipe: http://code.activestate.com/recipes/466332/
In any somewhat complex application, you likely want to log and handle most exceptions. The recipe shows a way to separate logging from handling, so that it is not necessary to explicitly invoke the logging mechanism in each try-except clause.
I'm using a PyQt4 user interface. I've redirected stderr to a log file for easy debugging and trouble-shooting, but now I need to display error messages to the user when an error occurs.
My issue is that I need to catch an exception when it happens and let the user know that it happened, but still let the traceback propagate to stderr (i.e. the log file).
If I do something like this:
def updateResults(self):
try:
#code that updates the results
except:
#display error message box
This will catch the exception and not propogate to the error log.
Is there some way to show the user the message and then continue to propogate the error?
Would this work?
except, e:
#display error message box
raise e
Is there a better way to accomplish my goal?
I think you are thinking about this in the wrong way. You shouldn't be re-raising the error simply to log it further down the line. The cannonical way of doing this in Python is to use the logging module. Adapted from the docs:
import logging
LOG_FILENAME = '/tmp/logging_example.out'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG,)
...
try:
# code
except:
logging.debug('Something bad happened', exc_info=True)
# display message box
# raise (if necessary)
This gives a far more flexible logging system than relying on errors produced on sys.stdout. You may not need to re-raise the exception if you can recover from the exception in some way.
Exactly, but you can just
raise
which will re-raise the currently handled exception.
Some additional information:
(With PyQt4) you will also need to rebind sys.excepthook to your own function to catch all uncaught exceptions. Otherwise PyQt will just print them to the console, which may not be what you need...
import sys
def excepthook(exc_type, exc_val, tracebackobj):
# do something useful with the uncaught exception
...
def main():
# rebind excepthook
sys.excepthook = excepthook
...