Related
How do you cause uncaught exceptions to output via the logging module rather than to stderr?
I realize the best way to do this would be:
try:
raise Exception, 'Throwing a boring exception'
except Exception, e:
logging.exception(e)
But my situation is such that it would be really nice if logging.exception(...) were invoked automatically whenever an exception isn't caught.
Here's a complete small example that also includes a few other tricks:
import sys
import logging
logger = logging.getLogger(__name__)
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logger.error("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = handle_exception
if __name__ == "__main__":
raise RuntimeError("Test unhandled")
Ignore KeyboardInterrupt so a console python program can exit with Ctrl + C.
Rely entirely on python's logging module for formatting the exception.
Use a custom logger with an example handler. This one changes the unhandled exception to go to stdout rather than stderr, but you could add all sorts of handlers in this same style to the logger object.
As Ned pointed out, sys.excepthook is invoked every time an exception is raised and uncaught. The practical implication of this is that in your code you can override the default behavior of sys.excepthook to do whatever you want (including using logging.exception).
As a straw man example:
import sys
def foo(exctype, value, tb):
print('My Error Information')
print('Type:', exctype)
print('Value:', value)
print('Traceback:', tb)
Override sys.excepthook:
>>> sys.excepthook = foo
Commit obvious syntax error (leave out the colon) and get back custom error information:
>>> def bar(a, b)
My Error Information
Type: <type 'exceptions.SyntaxError'>
Value: invalid syntax (<stdin>, line 1)
Traceback: None
For more information about sys.excepthook, read the docs.
Why not:
import sys
import logging
import traceback
def log_except_hook(*exc_info):
text = "".join(traceback.format_exception(*exc_info()))
logging.error("Unhandled exception: %s", text)
sys.excepthook = log_except_hook
None()
Here is the output with sys.excepthook as seen above:
$ python tb.py
ERROR:root:Unhandled exception: Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
Here is the output with the sys.excepthook commented out:
$ python tb.py
Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
The only difference is that the former has ERROR:root:Unhandled exception: at the beginning of the first line.
The method sys.excepthook will be invoked if an exception is uncaught: http://docs.python.org/library/sys.html#sys.excepthook
When an exception is raised and uncaught, the interpreter calls sys.excepthook with three arguments, the exception class, exception instance, and a traceback object. In an interactive session this happens just before control is returned to the prompt; in a Python program this happens just before the program exits. The handling of such top-level exceptions can be customized by assigning another three-argument function to sys.excepthook.
To build on Jacinda's answer, but using a logger object:
def catchException(logger, typ, value, traceback):
logger.critical("My Error Information")
logger.critical("Type: %s" % typ)
logger.critical("Value: %s" % value)
logger.critical("Traceback: %s" % traceback)
# Use a partially applied function
func = lambda typ, value, traceback: catchException(logger, typ, value, traceback)
sys.excepthook = func
In my case (using python 3) when using #Jacinda 's answer the content of the traceback was not printed. Instead, it just prints the object itself: <traceback object at 0x7f90299b7b90>.
Instead I do:
import sys
import logging
import traceback
def custom_excepthook(exc_type, exc_value, exc_traceback):
# Do not print exception when user cancels the program
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.error("An uncaught exception occurred:")
logging.error("Type: %s", exc_type)
logging.error("Value: %s", exc_value)
if exc_traceback:
format_exception = traceback.format_tb(exc_traceback)
for line in format_exception:
logging.error(repr(line))
sys.excepthook = custom_excepthook
Wrap your app entry call in a try...except block so you'll be able to catch and log (and perhaps re-raise) all uncaught exceptions. E.g. instead of:
if __name__ == '__main__':
main()
Do this:
if __name__ == '__main__':
try:
main()
except Exception as e:
logger.exception(e)
raise
Although #gnu_lorien's answer gave me good starting point, my program crashes on first exception.
I came with a customised (and/or) improved solution, which silently logs Exceptions of functions that are decorated with #handle_error.
import logging
__author__ = 'ahmed'
logging.basicConfig(filename='error.log', level=logging.DEBUG)
def handle_exception(exc_type, exc_value, exc_traceback):
import sys
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.critical(exc_value.message, exc_info=(exc_type, exc_value, exc_traceback))
def handle_error(func):
import sys
def __inner(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception, e:
exc_type, exc_value, exc_tb = sys.exc_info()
handle_exception(exc_type, exc_value, exc_tb)
finally:
print(e.message)
return __inner
#handle_error
def main():
raise RuntimeError("RuntimeError")
if __name__ == "__main__":
for _ in xrange(1, 20):
main()
To answer the question from Mr.Zeus discussed in the comment section of the accepted answer, I use this to log uncaught exceptions in an interactive console (tested with PyCharm 2018-2019). I found out sys.excepthook does not work in a python shell so I looked deeper and found that I could use sys.exc_info instead. However, sys.exc_info takes no arguments unlike sys.excepthook that takes 3 arguments.
Here, I use both sys.excepthook and sys.exc_info to log both exceptions in an interactive console and a script with a wrapper function. To attach a hook function to both functions, I have two different interfaces depending if arguments are given or not.
Here's the code:
def log_exception(exctype, value, traceback):
logger.error("Uncaught exception occurred!",
exc_info=(exctype, value, traceback))
def attach_hook(hook_func, run_func):
def inner(*args, **kwargs):
if not (args or kwargs):
# This condition is for sys.exc_info
local_args = run_func()
hook_func(*local_args)
else:
# This condition is for sys.excepthook
hook_func(*args, **kwargs)
return run_func(*args, **kwargs)
return inner
sys.exc_info = attach_hook(log_exception, sys.exc_info)
sys.excepthook = attach_hook(log_exception, sys.excepthook)
The logging setup can be found in gnu_lorien's answer.
Maybe you could do something at the top of a module that redirects stderr to a file, and then logg that file at the bottom
sock = open('error.log', 'w')
sys.stderr = sock
doSomething() #makes errors and they will log to error.log
logging.exception(open('error.log', 'r').read() )
I have a class in which I log errors and raise them. However, I have different functions which expect that error.
Now, even if these functions except the error properly, it is still logged. This leads to confusing log files in which multiple conflicting entries can be seen. For example:
import logging
logging.basicConfig(filename = "./log")
logger = logging.getLogger()
logger.setLevel(logging.INFO)
class Foo:
def __init__(self):
pass
def foo_error(self):
logger.error("You have done something very stupid!")
raise RuntimeError("You have done something very stupid!")
def foo_except(self):
try:
self.foo_error()
except RuntimeError as error:
logger.info("It was not so stupid after all!")
Foo = Foo()
Foo.foo_except()
Here, both messages show up in "./log". Preferably, I would like to suppress the first error log message if it is caught later on.
I have not seen an answer anywhere else. Maybe the way I am doing things suggests bad design. Any ideas?
You cannot really ask Python whether an exception will be caught lateron. So your only choice is to log only after you know whether the exception was caught or not.
One possible solution (though I'm not sure if this will work in your context):
import logging
logging.basicConfig(filename = "./log")
logger = logging.getLogger()
class Foo:
def __init__(self):
pass
def foo_error(self):
# logger.error("You have done something very stupid!")
raise RuntimeError("You have done something very stupid!")
def foo_except(self):
try:
self.foo_error()
except RuntimeError as error:
logger.warning("It was not so stupid after all!")
try:
Foo = Foo()
Foo.foo_except()
Foo.foo_error()
except Exception as exc:
if isinstance(exc, RuntimeError):
logger.error("%s", exc)
raise
After some more thinking and several failed attempts, i arrived at the following answer.
Firstly, as #gelonida mentioned:
You cannot really ask Python whether an exception will be caught later on.
This implies that a log entry which also raises an exception has to be written, because if the exception is not caught later on, the log entry is wiped out and missing from the file.
So instead of trying to control which log message gets written to file, we should implement a way to delete voided log messages from the file.
import logging
logging.basicConfig(filename = "./log")
logger = logging.getLogger()
logger.setLevel(logging.INFO)
class Foo:
def __init__(self):
pass
def foo_error(self):
logger.error("You have done something very stupid!")
raise RuntimeError("You have done something very stupid!")
def foo_except(self):
try:
self.foo_error()
except RuntimeError as error:
logger.info("It was not so stupid after all!")
Foo = Foo()
Foo.foo_except()
Following that logic we should replace in the original example above the line logger.info("It was not so stupid after all!") with a function that deletes the last committed log message and logs the correct one instead!
One way to achieve this is to modify the logging class and add two components. Namely a log record history and a FileHandler which supports deletion of log records. Let's start with the log record history.
class RecordHistory:
def __init__(self):
self._record_history = []
def write(self, record):
self._record_history.append(record)
def flush(self):
pass
def get(self):
return self._record_history[-1]
def pop(self):
return self._record_history.pop()
This is basically a data container which implements the write and flush methods alongside some other conveniences. The write and flush methods are required by logging.StreamHandler. For more information visit the logging.handlers documentation.
Next, we modify the existing logging.FileHandler to support the revoke method. This method allows us to delete a specific log record.
import re
class RevokableFileHandler(logging.FileHandler):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def revoke(self, record):
with open(self.baseFilename, mode="r+") as log:
substitute = re.sub(rf"{record}", "", log.read(), count=1)
log.seek(0)
log.write(substitute)
log.truncate()
Finally, we modify the Logger class. Note however that we cannot inherit from logging.Logger directly as stated here. Additionally, we add a logging.StreamHandler which pushes log records to our RecordHistory object. Also we implement a addRevokableHandler method which registers all handlers that support revoking records.
import logging
class Logger(logging.getLoggerClass()):
def __init__(self, name):
super().__init__(name)
self.revokable_handlers = []
self.record_history = RecordHistory()
stream_handler = logging.StreamHandler(stream=self.record_history)
stream_handler.setLevel(logging.INFO)
self.addHandler(stream_handler)
def addRevokableHandler(self, handler):
self.revokable_handlers.append(handler)
super().addHandler(handler)
def pop_and_log(self, level, msg):
record = self.record_history.pop()
for handler in self.revokable_handlers:
handler.revoke(record)
self.log(level, msg)
This leads to the following solution in the original code:
logging.setLoggerClass(Logger)
logger = logging.getLogger("root")
logger.setLevel(logging.INFO)
file_handler = RevokableFileHandler("./log")
file_handler.setLevel(logging.INFO)
logger.addRevokableHandler(file_handler)
class Foo:
def __init__(self):
pass
def foo_error(self):
logger.error("You have done something very stupid!")
raise RuntimeError("You have done something very stupid!")
def foo_except(self):
try:
self.foo_error()
except KeyError as error:
logger.pop_and_log(logging.INFO, "It was not so stupid after all!")
Foo = Foo()
Foo.foo_except()
Hopefully this lengthy answer can be of use to someone. Though it is still not clear to me if logging errors and info messages in such a way is considered bad code design.
I want to check for errors in a particular background file, but the standard error stream is being controlled by the program in the foreground and the errors in the file in the question are not being displayed. I can use the logging module and write output to a file, though. I was wondering how I can use this to log all exceptions, errors and their tracebacks.
It's probably a bad idea to log any exception thrown within the program, since Python uses exceptions also for normal control flow.
Therefore you should only log uncaught exceptions. You can easily do this using a logger's exception() method, once you have an exception object.
To handle all uncaught exceptions, you can either wrap your script's entry point in a try...except block, or by installing a custom exception handler by re-assigning sys.excepthook():
import logging
import sys
logger = logging.getLogger('mylogger')
# Configure logger to write to a file...
def my_handler(type, value, tb):
logger.exception("Uncaught exception: {0}".format(str(value)))
# Install exception handler
sys.excepthook = my_handler
# Run your main script here:
if __name__ == '__main__':
main()
import sys
import logging
import traceback
# log uncaught exceptions
def log_exceptions(type, value, tb):
for line in traceback.TracebackException(type, value, tb).format(chain=True):
logging.exception(line)
logging.exception(value)
sys.__excepthook__(type, value, tb) # calls default excepthook
sys.excepthook = log_exceptions
Inspired of the main answer + How to write to a file, using the logging Python module? + Print original exception in excepthook, here is how to write the full traceback into a file test.log, like it would be printed in the console:
import logging, sys, traceback
logger = logging.getLogger('logger')
fh = logging.FileHandler('test.log')
logger.addHandler(fh)
def exc_handler(exctype, value, tb):
logger.exception(''.join(traceback.format_exception(exctype, value, tb)))
sys.excepthook = exc_handler
print("hello")
1/0
How do you cause uncaught exceptions to output via the logging module rather than to stderr?
I realize the best way to do this would be:
try:
raise Exception, 'Throwing a boring exception'
except Exception, e:
logging.exception(e)
But my situation is such that it would be really nice if logging.exception(...) were invoked automatically whenever an exception isn't caught.
Here's a complete small example that also includes a few other tricks:
import sys
import logging
logger = logging.getLogger(__name__)
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logger.error("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = handle_exception
if __name__ == "__main__":
raise RuntimeError("Test unhandled")
Ignore KeyboardInterrupt so a console python program can exit with Ctrl + C.
Rely entirely on python's logging module for formatting the exception.
Use a custom logger with an example handler. This one changes the unhandled exception to go to stdout rather than stderr, but you could add all sorts of handlers in this same style to the logger object.
As Ned pointed out, sys.excepthook is invoked every time an exception is raised and uncaught. The practical implication of this is that in your code you can override the default behavior of sys.excepthook to do whatever you want (including using logging.exception).
As a straw man example:
import sys
def foo(exctype, value, tb):
print('My Error Information')
print('Type:', exctype)
print('Value:', value)
print('Traceback:', tb)
Override sys.excepthook:
>>> sys.excepthook = foo
Commit obvious syntax error (leave out the colon) and get back custom error information:
>>> def bar(a, b)
My Error Information
Type: <type 'exceptions.SyntaxError'>
Value: invalid syntax (<stdin>, line 1)
Traceback: None
For more information about sys.excepthook, read the docs.
Why not:
import sys
import logging
import traceback
def log_except_hook(*exc_info):
text = "".join(traceback.format_exception(*exc_info()))
logging.error("Unhandled exception: %s", text)
sys.excepthook = log_except_hook
None()
Here is the output with sys.excepthook as seen above:
$ python tb.py
ERROR:root:Unhandled exception: Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
Here is the output with the sys.excepthook commented out:
$ python tb.py
Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
The only difference is that the former has ERROR:root:Unhandled exception: at the beginning of the first line.
The method sys.excepthook will be invoked if an exception is uncaught: http://docs.python.org/library/sys.html#sys.excepthook
When an exception is raised and uncaught, the interpreter calls sys.excepthook with three arguments, the exception class, exception instance, and a traceback object. In an interactive session this happens just before control is returned to the prompt; in a Python program this happens just before the program exits. The handling of such top-level exceptions can be customized by assigning another three-argument function to sys.excepthook.
To build on Jacinda's answer, but using a logger object:
def catchException(logger, typ, value, traceback):
logger.critical("My Error Information")
logger.critical("Type: %s" % typ)
logger.critical("Value: %s" % value)
logger.critical("Traceback: %s" % traceback)
# Use a partially applied function
func = lambda typ, value, traceback: catchException(logger, typ, value, traceback)
sys.excepthook = func
In my case (using python 3) when using #Jacinda 's answer the content of the traceback was not printed. Instead, it just prints the object itself: <traceback object at 0x7f90299b7b90>.
Instead I do:
import sys
import logging
import traceback
def custom_excepthook(exc_type, exc_value, exc_traceback):
# Do not print exception when user cancels the program
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.error("An uncaught exception occurred:")
logging.error("Type: %s", exc_type)
logging.error("Value: %s", exc_value)
if exc_traceback:
format_exception = traceback.format_tb(exc_traceback)
for line in format_exception:
logging.error(repr(line))
sys.excepthook = custom_excepthook
Wrap your app entry call in a try...except block so you'll be able to catch and log (and perhaps re-raise) all uncaught exceptions. E.g. instead of:
if __name__ == '__main__':
main()
Do this:
if __name__ == '__main__':
try:
main()
except Exception as e:
logger.exception(e)
raise
Although #gnu_lorien's answer gave me good starting point, my program crashes on first exception.
I came with a customised (and/or) improved solution, which silently logs Exceptions of functions that are decorated with #handle_error.
import logging
__author__ = 'ahmed'
logging.basicConfig(filename='error.log', level=logging.DEBUG)
def handle_exception(exc_type, exc_value, exc_traceback):
import sys
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.critical(exc_value.message, exc_info=(exc_type, exc_value, exc_traceback))
def handle_error(func):
import sys
def __inner(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception, e:
exc_type, exc_value, exc_tb = sys.exc_info()
handle_exception(exc_type, exc_value, exc_tb)
finally:
print(e.message)
return __inner
#handle_error
def main():
raise RuntimeError("RuntimeError")
if __name__ == "__main__":
for _ in xrange(1, 20):
main()
To answer the question from Mr.Zeus discussed in the comment section of the accepted answer, I use this to log uncaught exceptions in an interactive console (tested with PyCharm 2018-2019). I found out sys.excepthook does not work in a python shell so I looked deeper and found that I could use sys.exc_info instead. However, sys.exc_info takes no arguments unlike sys.excepthook that takes 3 arguments.
Here, I use both sys.excepthook and sys.exc_info to log both exceptions in an interactive console and a script with a wrapper function. To attach a hook function to both functions, I have two different interfaces depending if arguments are given or not.
Here's the code:
def log_exception(exctype, value, traceback):
logger.error("Uncaught exception occurred!",
exc_info=(exctype, value, traceback))
def attach_hook(hook_func, run_func):
def inner(*args, **kwargs):
if not (args or kwargs):
# This condition is for sys.exc_info
local_args = run_func()
hook_func(*local_args)
else:
# This condition is for sys.excepthook
hook_func(*args, **kwargs)
return run_func(*args, **kwargs)
return inner
sys.exc_info = attach_hook(log_exception, sys.exc_info)
sys.excepthook = attach_hook(log_exception, sys.excepthook)
The logging setup can be found in gnu_lorien's answer.
Maybe you could do something at the top of a module that redirects stderr to a file, and then logg that file at the bottom
sock = open('error.log', 'w')
sys.stderr = sock
doSomething() #makes errors and they will log to error.log
logging.exception(open('error.log', 'r').read() )
Here's the thing:
I'm trying to combine the logging module with wx.App()'s redirect feature. My intention is to log to a file AND to stderr. But I want stderr/stdout redirected to a separate frame as is the feature of wx.App.
My test code:
import logging
import wx
class MyFrame(wx.Frame):
def __init__(self):
self.logger = logging.getLogger("main.MyFrame")
wx.Frame.__init__(self, parent = None, id = wx.ID_ANY, title = "MyFrame")
self.logger.debug("MyFrame.__init__() called.")
def OnExit(self):
self.logger.debug("MyFrame.OnExit() called.")
class MyApp(wx.App):
def __init__(self, redirect):
self.logger = logging.getLogger("main.MyApp")
wx.App.__init__(self, redirect = redirect)
self.logger.debug("MyApp.__init__() called.")
def OnInit(self):
self.frame = MyFrame()
self.frame.Show()
self.SetTopWindow(self.frame)
self.logger.debug("MyApp.OnInit() called.")
return True
def OnExit(self):
self.logger.debug("MyApp.OnExit() called.")
def main():
logger_formatter = logging.Formatter("%(name)s\t%(levelname)s\t%(message)s")
logger_stream_handler = logging.StreamHandler()
logger_stream_handler.setLevel(logging.INFO)
logger_stream_handler.setFormatter(logger_formatter)
logger_file_handler = logging.FileHandler("test.log", mode = "w")
logger_file_handler.setLevel(logging.DEBUG)
logger_file_handler.setFormatter(logger_formatter)
logger = logging.getLogger("main")
logger.setLevel(logging.DEBUG)
logger.addHandler(logger_stream_handler)
logger.addHandler(logger_file_handler)
logger.info("Logger configured.")
app = MyApp(redirect = True)
logger.debug("Created instance of MyApp. Calling MainLoop().")
app.MainLoop()
logger.debug("MainLoop() ended.")
logger.info("Exiting program.")
return 0
if (__name__ == "__main__"):
main()
Expected behavior is:
- a file is created named test.log
- the file contains logging messages with level DEBUG and INFO/ERROR/WARNING/CRITICAL
- messages from type INFO and ERROR/WARNING/CRITICAL are either shown on the console or in a separate frame, depending on where they are created
- logger messages that are not inside MyApp or MyFrame are displayed at the console
- logger messages from inside MyApp or MyFrame are shown in a separate frame
Actual behavior is:
- The file is created and contains:
main INFO Logger configured.
main.MyFrame DEBUG MyFrame.__init__() called.
main.MyFrame INFO MyFrame.__init__() called.
main.MyApp DEBUG MyApp.OnInit() called.
main.MyApp INFO MyApp.OnInit() called.
main.MyApp DEBUG MyApp.__init__() called.
main DEBUG Created instance of MyApp. Calling MainLoop().
main.MyApp DEBUG MyApp.OnExit() called.
main DEBUG MainLoop() ended.
main INFO Exiting program.
- Console output is:
main INFO Logger configured.
main.MyFrame INFO MyFrame.__init__() called.
main.MyApp INFO MyApp.OnInit() called.
main INFO Exiting program.
- No separate frame is opened, although the lines
main.MyFrame INFO MyFrame.__init__() called.
main.MyApp INFO MyApp.OnInit() called.
shouldget displayed within a frame and not on the console.
It seems to me that wx.App can't redirect stderr to a frame as soon as a logger instance uses stderr as output. wxPythons Docs claim the wanted behavior though, see here.
Any ideas?
Uwe
When wx.App says it will redirect stdout/stderr to a popup window, what it means really is that it will redirect sys.stdout and sys.stderr, so if you directly write to sys.stdout or sys.stderr it will be redirected to a popup window e.g. try this
print "this will go to wx msg frame"
sys.stdout.write("yes it goes")
sys.stderr.write("... and this one too")
Problem here is that if wxApp is created after creating streamhandler, streamhandler is pointing to old(original) sys.stderr and sys.stdout not to the new ones which wxApp has set, so a simpler solution is to create wx.App before creating streap handler e.g. in code move app = MyApp(redirect = True) before logging initialization code.
Alternatively create a custom logging handler and write data to sys.stdout and sys.stderr or better create you own window and add data there. e.g. try this
class LogginRedirectHandler(logging.Handler):
def __init__(self,):
# run the regular Handler __init__
logging.Handler.__init__(self)
def emit(self, record):
sys.stdout.write(record.message)
loggingRedirectHandler = LogginRedirectHandler()
logger_file_handler.setLevel(logging.DEBUG)
logger.addHandler(loggingRedirectHandler)
The way I do this, which I think is more elegant, is to create a custom logging Handler subclass that posts its messages to a specific logging frame.
This makes it easier to turn GUI logging on/off at runtime.