We Keep Coding

Why doesn't python logging.exception method log traceback by default?

When writing defensive code in python (e.g. you're handling some user input or whatever), I find it useful to return Exception objects alongside regular computation results, so they can be discarded/logged or processed in some other way. Consider the following snippet:
import logging
from traceback import TracebackException
from typing import Union
logging.basicConfig(level=logging.INFO)
def _compute(x) -> int:
return len(x)
def compute(x) -> Union[int, Exception]:
try:
return _compute(x)
except Exception as e:
return e
inputs = [
'whatever',
1,
'ooo',
None,
]
outputs = []
for i in inputs:
r = compute(i)
outputs.append(r)
for i, r in zip(inputs, outputs):
logging.info('compute(%s)', i)
if isinstance(r, Exception):
logging.exception(r)
else:
logging.info(r)
This results in the following output
INFO:root:compute(whatever)
INFO:root:8
INFO:root:compute(1)
ERROR:root:object of type 'int' has no len()
NoneType: None
INFO:root:compute(ooo)
INFO:root:3
INFO:root:compute(None)
ERROR:root:object of type 'NoneType' has no len()
NoneType: None
So you can see that useful exception information like stacktrace is lost, which makes it a bit hard to debug the cause of exception.
This can be fixed by logging exception as logging.exception(r, exc_info=r):
INFO:root:compute(whatever)
INFO:root:8
INFO:root:compute(1)
ERROR:root:object of type 'int' has no len()
Traceback (most recent call last):
File "/tmp/test.py", line 15, in compute
return _compute(x)
File "/tmp/test.py", line 10, in _compute
return len(x)
TypeError: object of type 'int' has no len()
INFO:root:compute(ooo)
INFO:root:3
INFO:root:compute(None)
ERROR:root:object of type 'NoneType' has no len()
Traceback (most recent call last):
File "/tmp/test.py", line 15, in compute
return _compute(x)
File "/tmp/test.py", line 10, in _compute
return len(x)
TypeError: object of type 'NoneType' has no len()
My question is -- why doesn't logging.exception method do this by default, if the argument passed to it happens to be an Exception? I tried searching in PEPs/etc but wasn't really fruitful.
My only guess is that logging.exception is essentially just a special case of logging.error, so in principle logging.exception method doesn't know whether is' passed an Exception object or something else. So supporting this would require some code, e.g. checking whether isinstance(msg, Exception), and perhaps the authors of logging library decided it's a bit too specific. But IMO it makes sense considering in practice in most cases logging.exception is passed an Exception object.

logging.exception does log the traceback by default. However, you're using it wrong. As the docs say,
This function should only be called from an exception handler.
logging.exception does not expect to be passed an exception instance, or any sort of exception information whatsoever. It uses sys.exc_info to gather info about the exception currently being handled, which only works if an exception is currently being handled. If you call it outside of an exception handler, it breaks.

Get class and attribute names from AttributeError

How do I get the class and missing attribute from AttributeError, e.g.
when AttributeError says: 'NoneType' object has not attribute a, I would like to get "NoneType" and "a".

Looks like the only thing you can retrieve from an AttributeError is the string with the error message:
try:
str.s
except AttributeError as err:
error_message, = err.args
print("Error message:", error_message)
raise

For getting the type information out of an AttributeError, you could probably use regex, since you know the format of the error message:
import re
try:
None.attr
except AttributeError as e:
matches = re.match(r"'([^']*)' object has no attribute '([^']*)'", str(e))
obj_type = matches.group(1)
attr_name = matches.group(2)
print(f"Object type: {obj_type}, attribute name: {attr_name}")
# Object type: NoneType, attribute name: attr

import re
try:
# access attribute
except AttributeError as e:
obj_type, attr_name = re.match(r"\'([a-zA-Z0-9\_]+)\' object has no attribute \'([a-zA-Z0-9\_]+)\'", str(e)).groups()
First cast the error into text with str(e)
Then, using regular expression pattern read the object type and the name of the attribute you're trying to access. The groups() method will return all the captured groups from the regex which are marked with parenthesis.

Python TypeError when printing traceback of AttributeError using traceback.print_exc()

A reproducible example:
import traceback
X = None
try:
X.text
except (TypeError, AttributeError) as e:
traceback.print_exc(e)
This will raise an error at traceback.print_exc(e):
TypeError: '>=' not supported between instances of 'AttributeError' and 'int'
Any suggestion why this happens?

print_exc doesn't take the exception object as an argument, it uses sys.exc_info() to obtain exception information. When you're passing it e, it's interpreting it as a positional argument for limit which expects a type int. I believe if you just remove the argument you'll get the result you're looking for.
traceback.print_exc documentation

Based on the documentation : Python Docs - traceback module
The first argument to traceback.print_exc isn't the exception, it is a depth limit of how many deep the traceback goes. You are hitting an exception within the traceback module itselse, since it expects the first argument to be a limit.
Your code needs to be :
import traceback
X = None
try:
X.text
except (TypeError, AttributeError) as e:
traceback.print_exc()
The exception data is kept as a thread global in sys.exc_info() which is what traceback.print_exc() uses.

TypeError when chaining exception from a warning

I am trying to catch a warning that is raised as an error by applying the 'error' filter of warnings.simplefilter. A minimum working example is given below:
>>> import warnings
>>> warnings.simplefilter('error')
>>> try:
... warnings.warn('test')
... except UserWarning:
... raise ValueError
...
ValueError:
This works fine, but if I want to chain this so that the traceback from the warning is included, I get the following TypeError:
>>> import sys
>>> try:
... warnings.warn('test')
... except UserWarning:
... raise ValueError from sys.exc_info()[2]
...
TypeError: exception causes must derive from BaseException
It seems that even though I am raising a class derived from BaseException (i.e. the ValueError), the information from the traceback from the UserWarning seems to be tricking Python into thinking I am still raising the UserWarning.
Is this the expected behavior? If so, is there a good workaround?
I am on Python 3.4.3.

You are trying to use the traceback object as the exception:
raise ValueError from sys.exc_info()[2]
sys.exc_info() returns a tuple of (ExceptionType, exception_instance, traceback); you'd want index 1 here (remember Python counts indices from 0!):
raise ValueError from sys.exc_info()[1]
Much better, capture the exception instance in a variable directly:
try:
warnings.warn('test')
except UserWarning as warning:
raise ValueError from warning
That way you don't have to count out tuple indices.
When you set the warnings filter to 'error', the exceptions are subclasses of the Warning exception class, which inherits from Exception, see the Exception Hierachy section.

Python : Another 'NoneType' object has no attribute error

For a newbie exercise , I am trying to find the meta tag in a html file and extract the generator so I did like this :
Version = soup.find("meta", {"name":"generator"})['content']
and since I had this error :
TypeError: 'NoneType' object has no attribute '__getitem__'
I was thinking that working with exception would correct it, so I wrote :
try: Version = soup.find("meta", {"name":"generator"})['content']
except NameError,TypeError:
print "Not found"
and what I got is the same error.
What should I do then ?

The soup.find() method did not find a matching tag, and returned None.
The [...] item access syntax looks for a __getitem__ method, which is the source of the AttributeError here:
>>> None[1]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object has no attribute '__getitem__'
Test for None explicitly:
Version = soup.find("meta", {"name":"generator"})
if Version is not None:
Version = Version['content']
else:
print "Not found"
Your exception handling would work too, provided you use parenthesis to group the exceptions:
try:
Version = soup.find("meta", {"name":"generator"})['content']
except (NameError, TypeError):
print "Not found"
Without parenthesis you are telling Python to catch NameError exceptions and assign the resulting exception object to the local name TypeError. This except Exception, name: syntax has been deprecated because it can lead to exactly your situation, where you think you are catching two exceptions.
However, your code here should not throw a NameError exception; that'd be a separate problem better solved by instantiating your variables properly; the following would work just as well here:
try:
Version = soup.find("meta", {"name":"generator"})['content']
except TypeError:
# No such meta tag found.
print "Not found"

Try this:
content = None
Version = soup.find("meta", {"name":"generator"})
if Version:
content = Version.get('content')
#or even
#Version = Version.get('content')
else:
print "Not found"
The issue is, soup.find returns a None if match was not found, and extracting data out of None results in error.