We Keep Coding

Is there a builtin way to log any exception? [duplicate]

How do you cause uncaught exceptions to output via the logging module rather than to stderr?
I realize the best way to do this would be:
try:
raise Exception, 'Throwing a boring exception'
except Exception, e:
logging.exception(e)
But my situation is such that it would be really nice if logging.exception(...) were invoked automatically whenever an exception isn't caught.

Here's a complete small example that also includes a few other tricks:
import sys
import logging
logger = logging.getLogger(__name__)
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logger.error("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = handle_exception
if __name__ == "__main__":
raise RuntimeError("Test unhandled")
Ignore KeyboardInterrupt so a console python program can exit with Ctrl + C.
Rely entirely on python's logging module for formatting the exception.
Use a custom logger with an example handler. This one changes the unhandled exception to go to stdout rather than stderr, but you could add all sorts of handlers in this same style to the logger object.

As Ned pointed out, sys.excepthook is invoked every time an exception is raised and uncaught. The practical implication of this is that in your code you can override the default behavior of sys.excepthook to do whatever you want (including using logging.exception).
As a straw man example:
import sys
def foo(exctype, value, tb):
print('My Error Information')
print('Type:', exctype)
print('Value:', value)
print('Traceback:', tb)
Override sys.excepthook:
>>> sys.excepthook = foo
Commit obvious syntax error (leave out the colon) and get back custom error information:
>>> def bar(a, b)
My Error Information
Type: <type 'exceptions.SyntaxError'>
Value: invalid syntax (<stdin>, line 1)
Traceback: None
For more information about sys.excepthook, read the docs.

Why not:
import sys
import logging
import traceback
def log_except_hook(*exc_info):
text = "".join(traceback.format_exception(*exc_info()))
logging.error("Unhandled exception: %s", text)
sys.excepthook = log_except_hook
None()
Here is the output with sys.excepthook as seen above:
$ python tb.py
ERROR:root:Unhandled exception: Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
Here is the output with the sys.excepthook commented out:
$ python tb.py
Traceback (most recent call last):
File "tb.py", line 11, in <module>
None()
TypeError: 'NoneType' object is not callable
The only difference is that the former has ERROR:root:Unhandled exception: at the beginning of the first line.

The method sys.excepthook will be invoked if an exception is uncaught: http://docs.python.org/library/sys.html#sys.excepthook
When an exception is raised and uncaught, the interpreter calls sys.excepthook with three arguments, the exception class, exception instance, and a traceback object. In an interactive session this happens just before control is returned to the prompt; in a Python program this happens just before the program exits. The handling of such top-level exceptions can be customized by assigning another three-argument function to sys.excepthook.

To build on Jacinda's answer, but using a logger object:
def catchException(logger, typ, value, traceback):
logger.critical("My Error Information")
logger.critical("Type: %s" % typ)
logger.critical("Value: %s" % value)
logger.critical("Traceback: %s" % traceback)
# Use a partially applied function
func = lambda typ, value, traceback: catchException(logger, typ, value, traceback)
sys.excepthook = func

In my case (using python 3) when using #Jacinda 's answer the content of the traceback was not printed. Instead, it just prints the object itself: <traceback object at 0x7f90299b7b90>.
Instead I do:
import sys
import logging
import traceback
def custom_excepthook(exc_type, exc_value, exc_traceback):
# Do not print exception when user cancels the program
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.error("An uncaught exception occurred:")
logging.error("Type: %s", exc_type)
logging.error("Value: %s", exc_value)
if exc_traceback:
format_exception = traceback.format_tb(exc_traceback)
for line in format_exception:
logging.error(repr(line))
sys.excepthook = custom_excepthook

Wrap your app entry call in a try...except block so you'll be able to catch and log (and perhaps re-raise) all uncaught exceptions. E.g. instead of:
if __name__ == '__main__':
main()
Do this:
if __name__ == '__main__':
try:
main()
except Exception as e:
logger.exception(e)
raise

Although #gnu_lorien's answer gave me good starting point, my program crashes on first exception.
I came with a customised (and/or) improved solution, which silently logs Exceptions of functions that are decorated with #handle_error.
import logging
__author__ = 'ahmed'
logging.basicConfig(filename='error.log', level=logging.DEBUG)
def handle_exception(exc_type, exc_value, exc_traceback):
import sys
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logging.critical(exc_value.message, exc_info=(exc_type, exc_value, exc_traceback))
def handle_error(func):
import sys
def __inner(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception, e:
exc_type, exc_value, exc_tb = sys.exc_info()
handle_exception(exc_type, exc_value, exc_tb)
finally:
print(e.message)
return __inner
#handle_error
def main():
raise RuntimeError("RuntimeError")
if __name__ == "__main__":
for _ in xrange(1, 20):
main()

To answer the question from Mr.Zeus discussed in the comment section of the accepted answer, I use this to log uncaught exceptions in an interactive console (tested with PyCharm 2018-2019). I found out sys.excepthook does not work in a python shell so I looked deeper and found that I could use sys.exc_info instead. However, sys.exc_info takes no arguments unlike sys.excepthook that takes 3 arguments.
Here, I use both sys.excepthook and sys.exc_info to log both exceptions in an interactive console and a script with a wrapper function. To attach a hook function to both functions, I have two different interfaces depending if arguments are given or not.
Here's the code:
def log_exception(exctype, value, traceback):
logger.error("Uncaught exception occurred!",
exc_info=(exctype, value, traceback))
def attach_hook(hook_func, run_func):
def inner(*args, **kwargs):
if not (args or kwargs):
# This condition is for sys.exc_info
local_args = run_func()
hook_func(*local_args)
else:
# This condition is for sys.excepthook
hook_func(*args, **kwargs)
return run_func(*args, **kwargs)
return inner
sys.exc_info = attach_hook(log_exception, sys.exc_info)
sys.excepthook = attach_hook(log_exception, sys.excepthook)
The logging setup can be found in gnu_lorien's answer.

Maybe you could do something at the top of a module that redirects stderr to a file, and then logg that file at the bottom
sock = open('error.log', 'w')
sys.stderr = sock
doSomething() #makes errors and they will log to error.log
logging.exception(open('error.log', 'r').read() )

how to log all errors in python

I want to save ALL ERRORS into a file which means the errors outside an exception, also get in the log file.
for example import some module and in the begining of a script, set it to save the errors; something like this:
import blob
blob.save_errors(filename)
try:
lst = ['a']
print(lst[2]) # save this Error
except:
pass
print(2/0) # also save this error

There is no easy automatic way to process all the exception, it contradict with exception handling idea.
One of solutions: we can log all uncatched exceptions like this way:
import sys
import logging
logger = logging.getLogger(__name__)
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
else:
logger.critical("Exception occured:", exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = handle_exception
# Just to test
if __name__ == "__main__":
raise Exception("Something happend!")
And to log all handled exception we need to all logging in except block:
try:
...
except:
logger.error("Exception occured:", exc_info=(exc_type, exc_value, exc_traceback))
...
Other way to deal with exception is to handle them in top-level function. All non-handled exceptions from downstream functions will be processed in top-level except block:
def my_main_func():
try:
run_my_application_logic()
except:
# Logging
logger.error("Exception occured:", exc_info=(exc_type, exc_value, exc_traceback))
def run_my_application_logic():
first_step_of_app()
second_step_of_app()
third_step_of_app()
if __name__ == "__main__":
my_main_func()

Raising errors without traceback

I would like to use raise without printing the traceback on the screen. I know how to do that using try ..catch but doesn't find a way with raise.
Here is an example:
def my_function(self):
resp = self.resp
if resp.status_code == 404:
raise NoSuchElementError('GET'+self.url+'{}'.format(resp.status_code))
elif resp.status_code == 500:
raise ServerErrorError('GET'+self.url+'{}'.format(resp.status_code))
When executing this, if I have a 404, the traceback will print on the screen.
Traceback (most recent call last):
File "test.py", line 32, in <module>
print ins.my_function()
File "api.py", line 820, in my_function
raise NoSuchElementError('GET ' + self.url + ' {} '.format(resp.status_code))
This is an API wrapper and I don't want users to see the traceback but to see the API response codes and error messages instead.
Is there a way to do it ?

I ran into a similar problem where a parent class was using the exception value on raise to pass messages through but where I didn't want to dump the traceback. #lejlot gives a great solution using sys.excepthook but I needed to apply it with a more limited scope. Here's the modification:
import sys
from contextlib import contextmanager
#contextmanager
def except_handler(exc_handler):
"Sets a custom exception handler for the scope of a 'with' block."
sys.excepthook = exc_handler
yield
sys.excepthook = sys.__excepthook__
Then, to use it:
def my_exchandler(type, value, traceback):
print(': '.join([str(type.__name__), str(value)]))
with except_handler(my_exchandler):
raise Exception('Exceptional!')
# -> Exception: Exceptional!
That way, if an exception isn't raised in the block, default exception handling will resume for any subsequent exceptions:
with except_handler(my_exchandler):
pass
raise Exception('Ordinary...')
# -> Traceback (most recent call last):
# -> File "raise_and_suppress_traceback.py", line 22, in <module>
# -> raise Exception('Ordinary...')
# -> Exception: Ordinary...

The problem is not with raising anything, but with what python interpreter does, when your program terminates with an exception (and it simply prints the stack trace). What you should do if you want to avoid it, is to put try except block around everything that you want to "hide" the stack trace, like:
def main():
try:
actual_code()
except Exception as e:
print(e)
The other way around is to modify the exeption handler, sys.excepthook(type, value, traceback), to do your own logic, like
def my_exchandler(type, value, traceback):
print(value)
import sys
sys.excepthook = my_exchandler
you can even condition of exception type and do the particular logic iff it is your type of exception, and otherwise - backoff to the original one.

Modified #Alec answer:
#contextmanager
def disable_exception_traceback():
"""
All traceback information is suppressed and only the exception type and value are printed
"""
default_value = getattr(sys, "tracebacklimit", 1000) # `1000` is a Python's default value
sys.tracebacklimit = 0
yield
sys.tracebacklimit = default_value # revert changes
Usage:
with disable_exception_traceback():
raise AnyYourCustomException()
Use this if you only need to hide a traceback without modifying an exception message. Tested on Python 3.8
UPD: code improved by #DrJohnAStevenson comment

Catch the exception, log it and return something that indicates something went wrong to the consumer (sending a 200 back when a query failed will likely cause problems for your client).
try:
return do_something()
except NoSuchElementError as e:
logger.error(e)
return error_response()
The fake error_response() function could do anything form returning an empty response or an error message. You should still make use of proper HTTP status codes. It sounds like you should be returning a 404 in this instance.
You should handle exceptions gracefully but you shouldn't hide errors from clients completely. In the case of your NoSuchElementError exception it sounds like the client should be informed (the error might be on their end).

You can create a class that takes two values; Type and code for a custom Exception Message. Afterwards, you can just pass the class in a try/except statement.
class ExceptionHandler(Exception):
def __init__(self, exceptionType, code):
self.exceptionType = exceptionType
self.code = code
print(f"Error logged: {self.exceptionType}, Code: {self.code}")
try:
raise(ExceptionHandler(exceptionType=KeyboardInterrupt, code=101))
except Exception:
pass

how to avoid to use sys.exc_clear

Here is my code:
import sys
class App(object):
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, tb):
print sys.exc_info()
app = App()
with app:
try:
1/0
except:
print 'Caught you'
#sys.exc_clear()
and another example using flask app context:
from flask import Flask
app = Flask(__name__)
#app.teardown_appcontext
def teardown(exception):
print exception
with app.app_context():
try:
1 / 0
except:
pass
It's weird that if i didn't call sys.exc_clear when i handle exception, then when exiting app, sys.exc_info will still return exception info.
Is there any way to avoid this case?
In my project which is based on flask, i will rollback transaction when there is a exception. Although i handled exception, app context can still get it using sys.exc_info like code below showed:
#AppContext pop method
def pop(self, exc=None):
"""Pops the app context."""
self._refcnt -= 1
if self._refcnt <= 0:
if exc is None:
exc = sys.exc_info()[1]
self.app.do_teardown_appcontext(exc)
rv = _app_ctx_stack.pop()
assert rv is self, 'Popped wrong app context. (%r instead of %r)' \
% (rv, self)
appcontext_popped.send(self.app)
I cannot ask everyone in my team to call sys.exc_info when handling every single one exception.
How should i do to avoid this situation?

This is really a bug in how the AppContext handles your case.
The exception is automatically cleared the moment the current frame exits (so outside the with statement). If you called another function from within the with statement frame and handled the exception there, the sys.exc_info() information would be cleared again when that other function exits.
The AppContext.__exit__() method is correctly notified that you handled the exception and passes the exception value on to the AppContext.pop() method.
As such the Flask AppContext.pop() method should use a different sentinel value to detect that no exception was passed in; it could detect that None was passed in not as a default but as an actual value, indicating no exceptions were raised or were properly handled.
I've filed an issue with the project requesting that this is implemented, with accompanying pull request. This was merged and will be part of a future release of Flask.
You could use a monkeypatch to backport this to the current Flask version:
from flask import app, ctx
import sys
if ctx.AppContext.pop.__defaults__ == (None,):
# unpatched
_sentinel = object()
def do_teardown_appcontext(self, exc=_sentinel):
if exc is _sentinel:
exc = sys.exc_info()[1]
for func in reversed(self.teardown_appcontext_funcs):
func(exc)
app.appcontext_tearing_down.send(self, exc=exc)
app.Flask.do_teardown_appcontext = do_teardown_appcontext
def pop(self, exc=_sentinel):
"""Pops the app context."""
self._refcnt -= 1
if self._refcnt <= 0:
if exc is _sentinel:
exc = sys.exc_info()[1]
self.app.do_teardown_appcontext(exc)
rv = ctx._app_ctx_stack.pop()
assert rv is self, 'Popped wrong app context. (%r instead of %r)' \
% (rv, self)
ctx.appcontext_popped.send(self.app)
ctx.AppContext.pop = pop

Log exception with traceback in Python

How can I log my Python exceptions?
try:
do_something()
except:
# How can I log my exception here, complete with its traceback?

Use logging.exception from within the except: handler/block to log the current exception along with the trace information, prepended with a message.
import logging
LOG_FILENAME = '/tmp/logging_example.out'
logging.basicConfig(filename=LOG_FILENAME, level=logging.DEBUG)
logging.debug('This message should go to the log file')
try:
run_my_stuff()
except:
logging.exception('Got exception on main handler')
raise
Now looking at the log file, /tmp/logging_example.out:
DEBUG:root:This message should go to the log file
ERROR:root:Got exception on main handler
Traceback (most recent call last):
File "/tmp/teste.py", line 9, in <module>
run_my_stuff()
NameError: name 'run_my_stuff' is not defined

Use exc_info options may be better, remains warning or error title:
try:
# coode in here
except Exception as e:
logging.error(e, exc_info=True)

My job recently tasked me with logging all the tracebacks/exceptions from our application. I tried numerous techniques that others had posted online such as the one above but settled on a different approach. Overriding traceback.print_exception.
I have a write up at http://www.bbarrows.com/ That would be much easier to read but Ill paste it in here as well.
When tasked with logging all the exceptions that our software might encounter in the wild I tried a number of different techniques to log our python exception tracebacks. At first I thought that the python system exception hook, sys.excepthook would be the perfect place to insert the logging code. I was trying something similar to:
import traceback
import StringIO
import logging
import os, sys
def my_excepthook(excType, excValue, traceback, logger=logger):
logger.error("Logging an uncaught exception",
exc_info=(excType, excValue, traceback))
sys.excepthook = my_excepthook
This worked for the main thread but I soon found that the my sys.excepthook would not exist across any new threads my process started. This is a huge issue because most everything happens in threads in this project.
After googling and reading plenty of documentation the most helpful information I found was from the Python Issue tracker.
The first post on the thread shows a working example of the sys.excepthook NOT persisting across threads (as shown below). Apparently this is expected behavior.
import sys, threading
def log_exception(*args):
print 'got exception %s' % (args,)
sys.excepthook = log_exception
def foo():
a = 1 / 0
threading.Thread(target=foo).start()
The messages on this Python Issue thread really result in 2 suggested hacks. Either subclass Thread and wrap the run method in our own try except block in order to catch and log exceptions or monkey patch threading.Thread.run to run in your own try except block and log the exceptions.
The first method of subclassing Thread seems to me to be less elegant in your code as you would have to import and use your custom Thread class EVERYWHERE you wanted to have a logging thread. This ended up being a hassle because I had to search our entire code base and replace all normal Threads with this custom Thread. However, it was clear as to what this Thread was doing and would be easier for someone to diagnose and debug if something went wrong with the custom logging code. A custome logging thread might look like this:
class TracebackLoggingThread(threading.Thread):
def run(self):
try:
super(TracebackLoggingThread, self).run()
except (KeyboardInterrupt, SystemExit):
raise
except Exception, e:
logger = logging.getLogger('')
logger.exception("Logging an uncaught exception")
The second method of monkey patching threading.Thread.run is nice because I could just run it once right after __main__ and instrument my logging code in all exceptions. Monkey patching can be annoying to debug though as it changes the expected functionality of something. The suggested patch from the Python Issue tracker was:
def installThreadExcepthook():
"""
Workaround for sys.excepthook thread bug
From
http://spyced.blogspot.com/2007/06/workaround-for-sysexcepthook-bug.html
(https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230540&group_id=5470).
Call once from __main__ before creating any threads.
If using psyco, call psyco.cannotcompile(threading.Thread.run)
since this replaces a new-style class method.
"""
init_old = threading.Thread.__init__
def init(self, *args, **kwargs):
init_old(self, *args, **kwargs)
run_old = self.run
def run_with_except_hook(*args, **kw):
try:
run_old(*args, **kw)
except (KeyboardInterrupt, SystemExit):
raise
except:
sys.excepthook(*sys.exc_info())
self.run = run_with_except_hook
threading.Thread.__init__ = init
It was not until I started testing my exception logging I realized that I was going about it all wrong.
To test I had placed a
raise Exception("Test")
somewhere in my code. However, wrapping a a method that called this method was a try except block that printed out the traceback and swallowed the exception. This was very frustrating because I saw the traceback bring printed to STDOUT but not being logged. It was I then decided that a much easier method of logging the tracebacks was just to monkey patch the method that all python code uses to print the tracebacks themselves, traceback.print_exception.
I ended up with something similar to the following:
def add_custom_print_exception():
old_print_exception = traceback.print_exception
def custom_print_exception(etype, value, tb, limit=None, file=None):
tb_output = StringIO.StringIO()
traceback.print_tb(tb, limit, tb_output)
logger = logging.getLogger('customLogger')
logger.error(tb_output.getvalue())
tb_output.close()
old_print_exception(etype, value, tb, limit=None, file=None)
traceback.print_exception = custom_print_exception
This code writes the traceback to a String Buffer and logs it to logging ERROR. I have a custom logging handler set up the 'customLogger' logger which takes the ERROR level logs and send them home for analysis.

You can log all uncaught exceptions on the main thread by assigning a handler to sys.excepthook, perhaps using the exc_info parameter of Python's logging functions:
import sys
import logging
logging.basicConfig(filename='/tmp/foobar.log')
def exception_hook(exc_type, exc_value, exc_traceback):
logging.error(
"Uncaught exception",
exc_info=(exc_type, exc_value, exc_traceback)
)
sys.excepthook = exception_hook
raise Exception('Boom')
If your program uses threads, however, then note that threads created using threading.Thread will not trigger sys.excepthook when an uncaught exception occurs inside them, as noted in Issue 1230540 on Python's issue tracker. Some hacks have been suggested there to work around this limitation, like monkey-patching Thread.__init__ to overwrite self.run with an alternative run method that wraps the original in a try block and calls sys.excepthook from inside the except block. Alternatively, you could just manually wrap the entry point for each of your threads in try/except yourself.

You can get the traceback using a logger, at any level (DEBUG, INFO, ...). Note that using logging.exception, the level is ERROR.
# test_app.py
import sys
import logging
logging.basicConfig(level="DEBUG")
def do_something():
raise ValueError(":(")
try:
do_something()
except Exception:
logging.debug("Something went wrong", exc_info=sys.exc_info())
DEBUG:root:Something went wrong
Traceback (most recent call last):
File "test_app.py", line 10, in <module>
do_something()
File "test_app.py", line 7, in do_something
raise ValueError(":(")
ValueError: :(
EDIT:
This works too (using python 3.6)
logging.debug("Something went wrong", exc_info=True)

What I was looking for:
import sys
import traceback
exc_type, exc_value, exc_traceback = sys.exc_info()
traceback_in_var = traceback.format_tb(exc_traceback)
See:
https://docs.python.org/3/library/traceback.html

Uncaught exception messages go to STDERR, so instead of implementing your logging in Python itself you could send STDERR to a file using whatever shell you're using to run your Python script. In a Bash script, you can do this with output redirection, as described in the BASH guide.
Examples
Append errors to file, other output to the terminal:
./test.py 2>> mylog.log
Overwrite file with interleaved STDOUT and STDERR output:
./test.py &> mylog.log

Here is a version that uses sys.excepthook
import traceback
import sys
logger = logging.getLogger()
def handle_excepthook(type, message, stack):
logger.error(f'An unhandled exception occured: {message}. Traceback: {traceback.format_tb(stack)}')
sys.excepthook = handle_excepthook

This is how I do it.
try:
do_something()
except:
# How can I log my exception here, complete with its traceback?
import traceback
traceback.format_exc() # this will print a complete trace to stout.

maybe not as stylish, but easier:
#!/bin/bash
log="/var/log/yourlog"
/path/to/your/script.py 2>&1 | (while read; do echo "$REPLY" >> $log; done)

To key off of others that may be getting lost in here, the way that works best with capturing it in logs is to use the traceback.format_exc() call and then split this string for each line in order to capture in the generated log file:
import logging
import sys
import traceback
try:
...
except Exception as ex:
# could be done differently, just showing you can split it apart to capture everything individually
ex_t = type(ex).__name__
err = str(ex)
err_msg = f'[{ex_t}] - {err}'
logging.error(err_msg)
# go through the trackback lines and individually add those to the log as an error
for l in traceback.format_exc().splitlines():
logging.error(l)

Heres a simple example taken from the python 2.6 documentation:
import logging
LOG_FILENAME = '/tmp/logging_example.out'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG,)
logging.debug('This message should go to the log file')