I want to save ALL ERRORS into a file which means the errors outside an exception, also get in the log file.
for example import some module and in the begining of a script, set it to save the errors; something like this:
import blob
blob.save_errors(filename)
try:
lst = ['a']
print(lst[2]) # save this Error
except:
pass
print(2/0) # also save this error
There is no easy automatic way to process all the exception, it contradict with exception handling idea.
One of solutions: we can log all uncatched exceptions like this way:
import sys
import logging
logger = logging.getLogger(__name__)
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
else:
logger.critical("Exception occured:", exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = handle_exception
# Just to test
if __name__ == "__main__":
raise Exception("Something happend!")
And to log all handled exception we need to all logging in except block:
try:
...
except:
logger.error("Exception occured:", exc_info=(exc_type, exc_value, exc_traceback))
...
Other way to deal with exception is to handle them in top-level function. All non-handled exceptions from downstream functions will be processed in top-level except block:
def my_main_func():
try:
run_my_application_logic()
except:
# Logging
logger.error("Exception occured:", exc_info=(exc_type, exc_value, exc_traceback))
def run_my_application_logic():
first_step_of_app()
second_step_of_app()
third_step_of_app()
if __name__ == "__main__":
my_main_func()
Related
How do you cause uncaught exceptions to output via the logging module rather than to stderr?
I realize the best way to do this would be:
try:
raise Exception, 'Throwing a boring exception'
except Exception, e:
logging.exception(e)
But my situation is such that it would be really nice if logging.exception(...) were invoked automatically whenever an exception isn't caught.
Here's a complete small example that also includes a few other tricks:
import sys
import logging
logger = logging.getLogger(__name__)
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logger.error("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = handle_exception
if __name__ == "__main__":
raise RuntimeError("Test unhandled")
Ignore KeyboardInterrupt so a console python program can exit with Ctrl + C.
Rely entirely on python's logging module for formatting the exception.
Use a custom logger with an example handler. This one changes the unhandled exception to go to stdout rather than stderr, but you could add all sorts of handlers in this same style to the logger object.
As Ned pointed out, sys.excepthook is invoked every time an exception is raised and uncaught. The practical implication of this is that in your code you can override the default behavior of sys.excepthook to do whatever you want (including using logging.exception).
As a straw man example:
import sys
def foo(exctype, value, tb):
print('My Error Information')
print('Type:', exctype)
print('Value:', value)
print('Traceback:', tb)
Override sys.excepthook:
>>> sys.excepthook = foo
Commit obvious syntax error (leave out the colon) and get back custom error information:
>>> def bar(a, b)
My Error Information
Type: <type 'exceptions.SyntaxError'>
Value: invalid syntax (<stdin>, line 1)
Traceback: None
For more information about sys.excepthook, read the docs.
Why not:
import sys
import logging
import traceback
def log_except_hook(*exc_info):
text = "".join(traceback.format_exception(*exc_info()))
logging.error("Unhandled exception: %s", text)
sys.excepthook = log_except_hook
None()
Here is the output with sys.excepthook as seen above:
$ python tb.py
ERROR:root:Unhandled exception: Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
Here is the output with the sys.excepthook commented out:
$ python tb.py
Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
The only difference is that the former has ERROR:root:Unhandled exception: at the beginning of the first line.
The method sys.excepthook will be invoked if an exception is uncaught: http://docs.python.org/library/sys.html#sys.excepthook
When an exception is raised and uncaught, the interpreter calls sys.excepthook with three arguments, the exception class, exception instance, and a traceback object. In an interactive session this happens just before control is returned to the prompt; in a Python program this happens just before the program exits. The handling of such top-level exceptions can be customized by assigning another three-argument function to sys.excepthook.
To build on Jacinda's answer, but using a logger object:
def catchException(logger, typ, value, traceback):
logger.critical("My Error Information")
logger.critical("Type: %s" % typ)
logger.critical("Value: %s" % value)
logger.critical("Traceback: %s" % traceback)
# Use a partially applied function
func = lambda typ, value, traceback: catchException(logger, typ, value, traceback)
sys.excepthook = func
In my case (using python 3) when using #Jacinda 's answer the content of the traceback was not printed. Instead, it just prints the object itself: <traceback object at 0x7f90299b7b90>.
Instead I do:
import sys
import logging
import traceback
def custom_excepthook(exc_type, exc_value, exc_traceback):
# Do not print exception when user cancels the program
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.error("An uncaught exception occurred:")
logging.error("Type: %s", exc_type)
logging.error("Value: %s", exc_value)
if exc_traceback:
format_exception = traceback.format_tb(exc_traceback)
for line in format_exception:
logging.error(repr(line))
sys.excepthook = custom_excepthook
Wrap your app entry call in a try...except block so you'll be able to catch and log (and perhaps re-raise) all uncaught exceptions. E.g. instead of:
if __name__ == '__main__':
main()
Do this:
if __name__ == '__main__':
try:
main()
except Exception as e:
logger.exception(e)
raise
Although #gnu_lorien's answer gave me good starting point, my program crashes on first exception.
I came with a customised (and/or) improved solution, which silently logs Exceptions of functions that are decorated with #handle_error.
import logging
__author__ = 'ahmed'
logging.basicConfig(filename='error.log', level=logging.DEBUG)
def handle_exception(exc_type, exc_value, exc_traceback):
import sys
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.critical(exc_value.message, exc_info=(exc_type, exc_value, exc_traceback))
def handle_error(func):
import sys
def __inner(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception, e:
exc_type, exc_value, exc_tb = sys.exc_info()
handle_exception(exc_type, exc_value, exc_tb)
finally:
print(e.message)
return __inner
#handle_error
def main():
raise RuntimeError("RuntimeError")
if __name__ == "__main__":
for _ in xrange(1, 20):
main()
To answer the question from Mr.Zeus discussed in the comment section of the accepted answer, I use this to log uncaught exceptions in an interactive console (tested with PyCharm 2018-2019). I found out sys.excepthook does not work in a python shell so I looked deeper and found that I could use sys.exc_info instead. However, sys.exc_info takes no arguments unlike sys.excepthook that takes 3 arguments.
Here, I use both sys.excepthook and sys.exc_info to log both exceptions in an interactive console and a script with a wrapper function. To attach a hook function to both functions, I have two different interfaces depending if arguments are given or not.
Here's the code:
def log_exception(exctype, value, traceback):
logger.error("Uncaught exception occurred!",
exc_info=(exctype, value, traceback))
def attach_hook(hook_func, run_func):
def inner(*args, **kwargs):
if not (args or kwargs):
# This condition is for sys.exc_info
local_args = run_func()
hook_func(*local_args)
else:
# This condition is for sys.excepthook
hook_func(*args, **kwargs)
return run_func(*args, **kwargs)
return inner
sys.exc_info = attach_hook(log_exception, sys.exc_info)
sys.excepthook = attach_hook(log_exception, sys.excepthook)
The logging setup can be found in gnu_lorien's answer.
Maybe you could do something at the top of a module that redirects stderr to a file, and then logg that file at the bottom
sock = open('error.log', 'w')
sys.stderr = sock
doSomething() #makes errors and they will log to error.log
logging.exception(open('error.log', 'r').read() )
I have a pyqt5 GUI which I start with ./mygui.py. In myguy.py I added a class to handle exceptions via a popup window (copied from http://timlehr.com/python-exception-hooks-with-qt-message-box/). However, while the exceptions are now in the popup window, they keep being written in the terminal. How can I remove this? Here is the class from the link above, maybe something to change/add in there:
import sys
import traceback
import logging
# basic logger functionality
log = logging.getLogger(__name__)
handler = logging.StreamHandler(stream=sys.stdout)
log.addHandler(handler)
def show_exception_box(log_msg):
"""Checks if a QApplication instance is available and shows a messagebox with the exception message.
If unavailable (non-console application), log an additional notice.
"""
if QtWidgets.QApplication.instance() is not None:
errorbox = QtWidgets.QMessageBox()
errorbox.setText("Oops. An unexpected error occured:\n{0}".format(log_msg))
errorbox.exec_()
else:
log.debug("No QApplication instance available.")
class UncaughtHook(QtCore.QObject):
_exception_caught = QtCore.Signal(object)
def __init__(self, *args, **kwargs):
super(UncaughtHook, self).__init__(*args, **kwargs)
# this registers the exception_hook() function as hook with the Python interpreter
sys.excepthook = self.exception_hook
# connect signal to execute the message box function always on main thread
self._exception_caught.connect(show_exception_box)
def exception_hook(self, exc_type, exc_value, exc_traceback):
"""Function handling uncaught exceptions.
It is triggered each time an uncaught exception occurs.
"""
if issubclass(exc_type, KeyboardInterrupt):
# ignore keyboard interrupt to support console applications
sys.__excepthook__(exc_type, exc_value, exc_traceback)
else:
exc_info = (exc_type, exc_value, exc_traceback)
log_msg = '\n'.join([''.join(traceback.format_tb(exc_traceback)),
'{0}: {1}'.format(exc_type.__name__, exc_value)])
log.critical("Uncaught exception:\n {0}".format(log_msg), exc_info=exc_info)
# trigger message box show
self._exception_caught.emit(log_msg)
According to your logic you are using as a handler the StreamHandler that sends the log to sys.stdout (the terminal in this case), if you don't want it to be sent then use NullHandler:
log = logging.getLogger(__name__)
handler = logging.NullHandler() # StreamHandler(stream=sys.stdout)
log.addHandler(handler)
I have this code and I'm using python 3:
import cmd
class myShell(cmd.Cmd):
def do_bad(self, arg):
raise Exception('something bad happened')
if __name__ == '__main__':
sh = myShell()
sh.cmdloop()
and I want to return to the shell-prompt after the exception was thrown. How to do that?
From the code, the functions are called from Cmd.onecmd (even in loop).
You can simply override it:
def onecmd(self, line):
try:
return super().onecmd(line)
except:
# display error message
return False # don't stop
The advantage is that you don't stop the command loop.
well you could override the cmdloop to wrap the original cmdLoop call:
class myShell(cmd.Cmd):
def do_bad(self, arg):
raise Exception('something bad happened')
def cmdLoop(self):
try:
cmd.Cmd.cmdLoop(self)
except Exception as e:
print("recovered from exception {}".format(e))
the wrap will let SystemExit and KeyboardInterrupt pass.
Here is my code:
import sys
class App(object):
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, tb):
print sys.exc_info()
app = App()
with app:
try:
1/0
except:
print 'Caught you'
#sys.exc_clear()
and another example using flask app context:
from flask import Flask
app = Flask(__name__)
#app.teardown_appcontext
def teardown(exception):
print exception
with app.app_context():
try:
1 / 0
except:
pass
It's weird that if i didn't call sys.exc_clear when i handle exception, then when exiting app, sys.exc_info will still return exception info.
Is there any way to avoid this case?
In my project which is based on flask, i will rollback transaction when there is a exception. Although i handled exception, app context can still get it using sys.exc_info like code below showed:
#AppContext pop method
def pop(self, exc=None):
"""Pops the app context."""
self._refcnt -= 1
if self._refcnt <= 0:
if exc is None:
exc = sys.exc_info()[1]
self.app.do_teardown_appcontext(exc)
rv = _app_ctx_stack.pop()
assert rv is self, 'Popped wrong app context. (%r instead of %r)' \
% (rv, self)
appcontext_popped.send(self.app)
I cannot ask everyone in my team to call sys.exc_info when handling every single one exception.
How should i do to avoid this situation?
This is really a bug in how the AppContext handles your case.
The exception is automatically cleared the moment the current frame exits (so outside the with statement). If you called another function from within the with statement frame and handled the exception there, the sys.exc_info() information would be cleared again when that other function exits.
The AppContext.__exit__() method is correctly notified that you handled the exception and passes the exception value on to the AppContext.pop() method.
As such the Flask AppContext.pop() method should use a different sentinel value to detect that no exception was passed in; it could detect that None was passed in not as a default but as an actual value, indicating no exceptions were raised or were properly handled.
I've filed an issue with the project requesting that this is implemented, with accompanying pull request. This was merged and will be part of a future release of Flask.
You could use a monkeypatch to backport this to the current Flask version:
from flask import app, ctx
import sys
if ctx.AppContext.pop.__defaults__ == (None,):
# unpatched
_sentinel = object()
def do_teardown_appcontext(self, exc=_sentinel):
if exc is _sentinel:
exc = sys.exc_info()[1]
for func in reversed(self.teardown_appcontext_funcs):
func(exc)
app.appcontext_tearing_down.send(self, exc=exc)
app.Flask.do_teardown_appcontext = do_teardown_appcontext
def pop(self, exc=_sentinel):
"""Pops the app context."""
self._refcnt -= 1
if self._refcnt <= 0:
if exc is _sentinel:
exc = sys.exc_info()[1]
self.app.do_teardown_appcontext(exc)
rv = ctx._app_ctx_stack.pop()
assert rv is self, 'Popped wrong app context. (%r instead of %r)' \
% (rv, self)
ctx.appcontext_popped.send(self.app)
ctx.AppContext.pop = pop
How do you cause uncaught exceptions to output via the logging module rather than to stderr?
I realize the best way to do this would be:
try:
raise Exception, 'Throwing a boring exception'
except Exception, e:
logging.exception(e)
But my situation is such that it would be really nice if logging.exception(...) were invoked automatically whenever an exception isn't caught.
Here's a complete small example that also includes a few other tricks:
import sys
import logging
logger = logging.getLogger(__name__)
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logger.error("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = handle_exception
if __name__ == "__main__":
raise RuntimeError("Test unhandled")
Ignore KeyboardInterrupt so a console python program can exit with Ctrl + C.
Rely entirely on python's logging module for formatting the exception.
Use a custom logger with an example handler. This one changes the unhandled exception to go to stdout rather than stderr, but you could add all sorts of handlers in this same style to the logger object.
As Ned pointed out, sys.excepthook is invoked every time an exception is raised and uncaught. The practical implication of this is that in your code you can override the default behavior of sys.excepthook to do whatever you want (including using logging.exception).
As a straw man example:
import sys
def foo(exctype, value, tb):
print('My Error Information')
print('Type:', exctype)
print('Value:', value)
print('Traceback:', tb)
Override sys.excepthook:
>>> sys.excepthook = foo
Commit obvious syntax error (leave out the colon) and get back custom error information:
>>> def bar(a, b)
My Error Information
Type: <type 'exceptions.SyntaxError'>
Value: invalid syntax (<stdin>, line 1)
Traceback: None
For more information about sys.excepthook, read the docs.
Why not:
import sys
import logging
import traceback
def log_except_hook(*exc_info):
text = "".join(traceback.format_exception(*exc_info()))
logging.error("Unhandled exception: %s", text)
sys.excepthook = log_except_hook
None()
Here is the output with sys.excepthook as seen above:
$ python tb.py
ERROR:root:Unhandled exception: Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
Here is the output with the sys.excepthook commented out:
$ python tb.py
Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
The only difference is that the former has ERROR:root:Unhandled exception: at the beginning of the first line.
The method sys.excepthook will be invoked if an exception is uncaught: http://docs.python.org/library/sys.html#sys.excepthook
When an exception is raised and uncaught, the interpreter calls sys.excepthook with three arguments, the exception class, exception instance, and a traceback object. In an interactive session this happens just before control is returned to the prompt; in a Python program this happens just before the program exits. The handling of such top-level exceptions can be customized by assigning another three-argument function to sys.excepthook.
To build on Jacinda's answer, but using a logger object:
def catchException(logger, typ, value, traceback):
logger.critical("My Error Information")
logger.critical("Type: %s" % typ)
logger.critical("Value: %s" % value)
logger.critical("Traceback: %s" % traceback)
# Use a partially applied function
func = lambda typ, value, traceback: catchException(logger, typ, value, traceback)
sys.excepthook = func
In my case (using python 3) when using #Jacinda 's answer the content of the traceback was not printed. Instead, it just prints the object itself: <traceback object at 0x7f90299b7b90>.
Instead I do:
import sys
import logging
import traceback
def custom_excepthook(exc_type, exc_value, exc_traceback):
# Do not print exception when user cancels the program
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.error("An uncaught exception occurred:")
logging.error("Type: %s", exc_type)
logging.error("Value: %s", exc_value)
if exc_traceback:
format_exception = traceback.format_tb(exc_traceback)
for line in format_exception:
logging.error(repr(line))
sys.excepthook = custom_excepthook
Wrap your app entry call in a try...except block so you'll be able to catch and log (and perhaps re-raise) all uncaught exceptions. E.g. instead of:
if __name__ == '__main__':
main()
Do this:
if __name__ == '__main__':
try:
main()
except Exception as e:
logger.exception(e)
raise
Although #gnu_lorien's answer gave me good starting point, my program crashes on first exception.
I came with a customised (and/or) improved solution, which silently logs Exceptions of functions that are decorated with #handle_error.
import logging
__author__ = 'ahmed'
logging.basicConfig(filename='error.log', level=logging.DEBUG)
def handle_exception(exc_type, exc_value, exc_traceback):
import sys
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.critical(exc_value.message, exc_info=(exc_type, exc_value, exc_traceback))
def handle_error(func):
import sys
def __inner(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception, e:
exc_type, exc_value, exc_tb = sys.exc_info()
handle_exception(exc_type, exc_value, exc_tb)
finally:
print(e.message)
return __inner
#handle_error
def main():
raise RuntimeError("RuntimeError")
if __name__ == "__main__":
for _ in xrange(1, 20):
main()
To answer the question from Mr.Zeus discussed in the comment section of the accepted answer, I use this to log uncaught exceptions in an interactive console (tested with PyCharm 2018-2019). I found out sys.excepthook does not work in a python shell so I looked deeper and found that I could use sys.exc_info instead. However, sys.exc_info takes no arguments unlike sys.excepthook that takes 3 arguments.
Here, I use both sys.excepthook and sys.exc_info to log both exceptions in an interactive console and a script with a wrapper function. To attach a hook function to both functions, I have two different interfaces depending if arguments are given or not.
Here's the code:
def log_exception(exctype, value, traceback):
logger.error("Uncaught exception occurred!",
exc_info=(exctype, value, traceback))
def attach_hook(hook_func, run_func):
def inner(*args, **kwargs):
if not (args or kwargs):
# This condition is for sys.exc_info
local_args = run_func()
hook_func(*local_args)
else:
# This condition is for sys.excepthook
hook_func(*args, **kwargs)
return run_func(*args, **kwargs)
return inner
sys.exc_info = attach_hook(log_exception, sys.exc_info)
sys.excepthook = attach_hook(log_exception, sys.excepthook)
The logging setup can be found in gnu_lorien's answer.
Maybe you could do something at the top of a module that redirects stderr to a file, and then logg that file at the bottom
sock = open('error.log', 'w')
sys.stderr = sock
doSomething() #makes errors and they will log to error.log
logging.exception(open('error.log', 'r').read() )