When writing defensive code in python (e.g. you're handling some user input or whatever), I find it useful to return Exception objects alongside regular computation results, so they can be discarded/logged or processed in some other way. Consider the following snippet:
import logging
from traceback import TracebackException
from typing import Union
logging.basicConfig(level=logging.INFO)
def _compute(x) -> int:
return len(x)
def compute(x) -> Union[int, Exception]:
try:
return _compute(x)
except Exception as e:
return e
inputs = [
'whatever',
1,
'ooo',
None,
]
outputs = []
for i in inputs:
r = compute(i)
outputs.append(r)
for i, r in zip(inputs, outputs):
logging.info('compute(%s)', i)
if isinstance(r, Exception):
logging.exception(r)
else:
logging.info(r)
This results in the following output
INFO:root:compute(whatever)
INFO:root:8
INFO:root:compute(1)
ERROR:root:object of type 'int' has no len()
NoneType: None
INFO:root:compute(ooo)
INFO:root:3
INFO:root:compute(None)
ERROR:root:object of type 'NoneType' has no len()
NoneType: None
So you can see that useful exception information like stacktrace is lost, which makes it a bit hard to debug the cause of exception.
This can be fixed by logging exception as logging.exception(r, exc_info=r):
INFO:root:compute(whatever)
INFO:root:8
INFO:root:compute(1)
ERROR:root:object of type 'int' has no len()
Traceback (most recent call last):
File "/tmp/test.py", line 15, in compute
return _compute(x)
File "/tmp/test.py", line 10, in _compute
return len(x)
TypeError: object of type 'int' has no len()
INFO:root:compute(ooo)
INFO:root:3
INFO:root:compute(None)
ERROR:root:object of type 'NoneType' has no len()
Traceback (most recent call last):
File "/tmp/test.py", line 15, in compute
return _compute(x)
File "/tmp/test.py", line 10, in _compute
return len(x)
TypeError: object of type 'NoneType' has no len()
My question is -- why doesn't logging.exception method do this by default, if the argument passed to it happens to be an Exception? I tried searching in PEPs/etc but wasn't really fruitful.
My only guess is that logging.exception is essentially just a special case of logging.error, so in principle logging.exception method doesn't know whether is' passed an Exception object or something else. So supporting this would require some code, e.g. checking whether isinstance(msg, Exception), and perhaps the authors of logging library decided it's a bit too specific. But IMO it makes sense considering in practice in most cases logging.exception is passed an Exception object.
logging.exception does log the traceback by default. However, you're using it wrong. As the docs say,
This function should only be called from an exception handler.
logging.exception does not expect to be passed an exception instance, or any sort of exception information whatsoever. It uses sys.exc_info to gather info about the exception currently being handled, which only works if an exception is currently being handled. If you call it outside of an exception handler, it breaks.
Related
I am using python unittest to test a web app using selenium. In my teardownClass, I am calling cls.fail but it returns "AttributeError" saying that "failureException" not found in the "string".
Here is what my teardownClass method looks like:
#classmethod
def tearDownClass(cls):
browser_logs = cls.driver.get_log("browser")
errors = [log_entry['message'] for log_entry in browser_logs if logentry['level'] == 'SEVERE']
if errors:
cls.fail(errors)
this returns the following attributeError:
======================================================================
ERROR: tearDownClass (__main__.unitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/shahrukh/Desktop/apollotest/test_users.py", line 239, in tearDownClass
cls.fail("error")
File "/usr/lib/python3.6/unittest/case.py", line 670, in fail
raise self.failureException(msg)
AttributeError: 'str' object has no attribute 'failureException'
----------------------------------------------------------------------
"fail" method from /usr/lib/python3.6/unittest/case.py file:
def fail(self, msg=None):
"""Fail immediately, with the given message."""
raise self.failureException(msg)
This method is part of "Testcase" class
The problem is that fail method above does not find the reference to self. "errors" in my code is a string object. So when I call cls.fail(errors) it considers self=errors and msg remains None.
I don't get this problem if I change my code statement to cls.fail(cls, errors).
I want to understand why am I experiencing this behavior? Am I missing anything because according to my understanding cls.fail should mean that self in fail method is equal to cls.
Would really appreciate some help.
I am using AWS and use AWS cloudwatch to view logs. While things should not break on AWS, they could. I just had such a case. Then I searched for Traceback and just got the lines
Traceback (most recent call last):
without the actual traceback. I have a working structured logging setup (see other question) and I would like to get tracebacks in a similar way.
So instead of:
Traceback (most recent call last):
File "/home/math/Desktop/test.py", line 32, in <module>
adf
NameError: name 'adf' is not defined
something like
{"message": "Traceback (most recent call last):\n File \"/home/math/Desktop/test.py\", line 32, in <module>\n adf\n NameError: name 'adf' is not defined", "lineno": 35, "pathname": "/home/math/Desktop/test.py"}
or even better also with the string in a JSON format.
The only way to achieve this I can think of is a giant try-except block. Pokemon-style. Is there a better solution?
You can use sys.excepthook. It is invoked whenever an exception occurs in your script.
import logging
import sys
import traceback
def exception_logging(exctype, value, tb):
"""
Log exception by using the root logger.
Parameters
----------
exctype : type
value : NameError
tb : traceback
"""
write_val = {'exception_type': str(exctype),
'message': str(traceback.format_tb(tb, 10))}
logging.exception(str(write_val))
Then in your script you have to override the value of sys.excepthook.
sys.excepthook = exception_logging
Now whenever an exception occurs it will be logged with your logger handler.
Note: Don't forget to setup logger before running this
In case somebody wants the exception logged in its default format, but in one line (for any reason), based on the accepted answer:
def exception_logging(exctype, value, tb):
"""
Log exception in one line by using the root logger.
Parameters
----------
exctype : exception type
value : seems to be the Exception object (with its message)
tb : traceback
"""
logging.error(''.join(traceback.format_exception(exctype, value, tb)))
Please also note, that it uses logging.error() instead of logging.exception() which also printed some extra "NoneType: None" line.
Also note that it only seems to work with uncaught exceptions.
For logging caught exceptions, visit How do I can format exception stacktraces in Python logging? and see also my answer.
A slight variation: If you run a Flask application, you can do this:
#app.errorhandler(Exception)
def exception_logger(error):
"""Log the exception."""
logger.exception(str(error))
return str(error)
I am creating tests for a python project. The normal tests work just fine, however I want to test if in a certain condition my function raises a self-defined exception. Therefor I want to use assertRaises(Exception, Function). Any ideas?
The function that raises the exception is:
def connect(comp1, comp2):
if comp1 == comp2:
raise e.InvalidConnectionError(comp1, comp2)
...
The exception is:
class InvalidConnectionError(Exception):
def __init__(self, connection1, connection2):
self._connection1 = connection1
self._connection2 = connection2
def __str__(self):
string = '...'
return string
The test method is the following:
class TestConnections(u.TestCase):
def test_connect_error(self):
comp = c.PowerConsumer('Bus', True, 1000)
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
However I get the following error:
Error
Traceback (most recent call last):
File "C:\Users\t5ycxK\PycharmProjects\ElectricPowerDesign\test_component.py", line 190, in test_connect_error
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
File "C:\Users\t5ycxK\PycharmProjects\ElectricPowerDesign\component.py", line 428, in connect
raise e.InvalidConnectionError(comp1, comp2)
InvalidConnectionError: <unprintable InvalidConnectionError object>
assertRaises expects to actually perform the call. Yet, you already perform it by yourself, thereby throwing the error before assertRaises actually executes.
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
# run this ^ with first static argument ^ and second argument ^ from `c.connect(comp, comp)`
Use either of those instead:
self.assertRaises(e.InvalidConnectionError, c.connect, comp, comp)
with self.assertRaises(e.InvalidConnectionError):
c.connect(comp, comp)
I would like to implement a deferred exception in Python that is OK to store somewhere but as soon as it is used in any way, it raises the exception that was deferred. Something like this:
# this doesn't work but it's a start
class DeferredException(object):
def __init__(self, exc):
self.exc = exc
def __getattr__(self, key):
raise self.exc
# example:
mydict = {'foo': 3}
try:
myval = obtain_some_number()
except Exception as e:
myval = DeferredException(e)
mydict['myval'] = myval
def plus_two(x):
print x+2
# later on...
plus_two(mydict['foo']) # prints 5
we_dont_use_this_val = mydict['myval'] # Always ok to store this value if not used
plus_two(mydict['myval']) # If obtain_some_number() failed earlier,
# re-raises the exception, otherwise prints the value + 2.
The use case is that I want to write code to analyze some values from incoming data; if this code fails but the results are never used, I want it to fail quietly; if it fails but the results are used later, then I'd like the failure to propagate.
Any suggestions on how to do this? If I use my DeferredException class I get this result:
>>> ke = KeyError('something')
>>> de = DeferredException(ke)
>>> de.bang # yay, this works
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in __getattr__
KeyError: 'something'
>>> de+2 # boo, this doesn't
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'DeferredException' and 'int'
Read section 3.4.12 of the docs, "Special method lookup for new-style classes." It explains exactly the problem you have encountered. The normal attribute lookup is bypassed by the interpreter for certain operators, such as addition (as you found out the hard way). Thus the statement de+2 in your code never calls your getattr function.
The only solution, according to that section, is to insure that "the special method must be set on the class object itself in order to be consistently invoked by the interpreter."
Perhaps you'd be better off storing all your deferred exceptions in a global list, wrapping your entire program in a try:finally: statement, and printing out the whole list in the finally block.
I am running into the following rather strange problem:
I am developing a django app and in my models class I am defining an exception that should be raised when a validation fails:
class MissingValueException(Exception):
"""Raise when a required attribute is missing."""
def __init__(self, message):
super(MissingValueException, self).__init__()
self.message = message
def __str__(self):
return repr(self.message)
This code is called from a publication class in a validation method:
def validate_required_fields(self):
# Here is the validation code.
if all_fields_present:
return True
else:
raise MissingValueException(errors)
In my unit test I create a case where the exception should be raised:
def test_raise_exception_incomplete_publication(self):
publication = Publication(publication_type="book")
self.assertRaises(MissingValueException, publication.validate_required_fields)
This produces the following output:
======================================================================
ERROR: test_raise_exception_incomplete_publication (core_knowledge_platform.core_web_service.tests.logic_tests.BusinessLogicTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/media/data/Dokumente/Code/master_project/core_knowledge_platform/../core_knowledge_platform/core_web_service/tests/logic_tests.py", line 45, in test_raise_exception_incomplete_publication
self.assertRaises(MissingValueException, method, )
File "/usr/lib/python2.7/unittest/case.py", line 465, in assertRaises
callableObj(*args, **kwargs)
File "/media/data/Dokumente/Code/master_project/core_knowledge_platform/../core_knowledge_platform/core_web_service/models.py", line 150, in validate_required_fields
raise MissingValueException(errors)
MissingValueException: 'Publication of type book is missing field publisherPublication of type book is missing field titlePublication of type book is missing field year'
So it looks like the exception is raised (which is the case - I even checked it in an interactive IPython session), but it seems that assertRaises is not catching it.
Anyone has any idea why this might happen?
Thanks
This could happen if your tests and your product code are importing your exception class through two different paths, so asserRaises doesn't realize that the exception you got was the one you were looking for.
Look at your imports, make sure that they are the same in both places. Having the same directories available in two different ways in your PYTHONPATH can make this happen. Symbolic links in those entries can also confuse things.