We Keep Coding

Can I make Python output exceptions in one line / via logging?

I am using AWS and use AWS cloudwatch to view logs. While things should not break on AWS, they could. I just had such a case. Then I searched for Traceback and just got the lines
Traceback (most recent call last):
without the actual traceback. I have a working structured logging setup (see other question) and I would like to get tracebacks in a similar way.
So instead of:
Traceback (most recent call last):
File "/home/math/Desktop/test.py", line 32, in <module>
adf
NameError: name 'adf' is not defined
something like
{"message": "Traceback (most recent call last):\n File \"/home/math/Desktop/test.py\", line 32, in <module>\n adf\n NameError: name 'adf' is not defined", "lineno": 35, "pathname": "/home/math/Desktop/test.py"}
or even better also with the string in a JSON format.
The only way to achieve this I can think of is a giant try-except block. Pokemon-style. Is there a better solution?

You can use sys.excepthook. It is invoked whenever an exception occurs in your script.
import logging
import sys
import traceback
def exception_logging(exctype, value, tb):
"""
Log exception by using the root logger.
Parameters
----------
exctype : type
value : NameError
tb : traceback
"""
write_val = {'exception_type': str(exctype),
'message': str(traceback.format_tb(tb, 10))}
logging.exception(str(write_val))
Then in your script you have to override the value of sys.excepthook.
sys.excepthook = exception_logging
Now whenever an exception occurs it will be logged with your logger handler.
Note: Don't forget to setup logger before running this

In case somebody wants the exception logged in its default format, but in one line (for any reason), based on the accepted answer:
def exception_logging(exctype, value, tb):
"""
Log exception in one line by using the root logger.
Parameters
----------
exctype : exception type
value : seems to be the Exception object (with its message)
tb : traceback
"""
logging.error(''.join(traceback.format_exception(exctype, value, tb)))
Please also note, that it uses logging.error() instead of logging.exception() which also printed some extra "NoneType: None" line.
Also note that it only seems to work with uncaught exceptions.
For logging caught exceptions, visit How do I can format exception stacktraces in Python logging? and see also my answer.

A slight variation: If you run a Flask application, you can do this:
#app.errorhandler(Exception)
def exception_logger(error):
"""Log the exception."""
logger.exception(str(error))
return str(error)

Error while testing the raise of self-defined exceptions (using assertRaises())

I am creating tests for a python project. The normal tests work just fine, however I want to test if in a certain condition my function raises a self-defined exception. Therefor I want to use assertRaises(Exception, Function). Any ideas?
The function that raises the exception is:
def connect(comp1, comp2):
if comp1 == comp2:
raise e.InvalidConnectionError(comp1, comp2)
...
The exception is:
class InvalidConnectionError(Exception):
def __init__(self, connection1, connection2):
self._connection1 = connection1
self._connection2 = connection2
def __str__(self):
string = '...'
return string
The test method is the following:
class TestConnections(u.TestCase):
def test_connect_error(self):
comp = c.PowerConsumer('Bus', True, 1000)
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
However I get the following error:
Error
Traceback (most recent call last):
File "C:\Users\t5ycxK\PycharmProjects\ElectricPowerDesign\test_component.py", line 190, in test_connect_error
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
File "C:\Users\t5ycxK\PycharmProjects\ElectricPowerDesign\component.py", line 428, in connect
raise e.InvalidConnectionError(comp1, comp2)
InvalidConnectionError: <unprintable InvalidConnectionError object>

assertRaises expects to actually perform the call. Yet, you already perform it by yourself, thereby throwing the error before assertRaises actually executes.
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
# run this ^ with first static argument ^ and second argument ^ from `c.connect(comp, comp)`
Use either of those instead:
self.assertRaises(e.InvalidConnectionError, c.connect, comp, comp)
with self.assertRaises(e.InvalidConnectionError):
c.connect(comp, comp)

My class is callable at the begining of my code, but not while my tests loops

I've got 2 files : main.py and batsol.py
batsol.py contains a class and main.py is creating some instances from the class Batsol.
So I'll show you a concise version of my code...
class Batsol:
def __init__(self, addressCan = None, name = None) :
self.addressCan = addressCan
self.name = name
#other stuff ...
Then my main.py :
from batsol import Batsol
# other import and code ...
print(callable(Batsol))
bs1 = Batsol()
# code...
if len(listener.ring_buffer) == 0 :
for Batsol in tab_BS :
try:
print(tab_BS[Batsol])
except (IndexError):
pass
# code...
while(True) :
# for and if interlocked
print(callable(Batsol))
bs2 = Batsol()
The console shows :
True
False
Traceback (most recent call last):
File "./main.py", line 135, in <module>
bs2 = Batsol()
TypeError: 'int' object is not callable
the second part of the traceback is not linked to other stuff i'm doing in my code (thread not terminated properly... something like this) , in my opinion
Exception ignored in: <module 'threading' from '/usr/lib/python3.4/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.4/threading.py", line 1292, in _shutdown
t = _pickSomeNonDaemonThread()
File "/usr/lib/python3.4/threading.py", line 1300, in _pickSomeNonDaemonThread
if not t.daemon and t.is_alive():
TypeError: 'bool' object is not callable
WHY my object is not callable inside my tests loops ???
It drives me crazy...

Your shadowing occurs in this code fragment:
if len(listener.ring_buffer) == 0 :
for Batsol in tab_BS :
try:
print(tab_BS[Batsol])
except (IndexError):
pass
time.sleep(4)
for-in construct on sequences works as following:
Sequence is asked for next (first, second, ... last) element. Internal pointer keeps track of element in current iteration.
Element gets assigned to name on left side of "in".
Go to 1.
After loop finishes, Batsol is no longer your class, but last element from tab_BS.
I'd suggest getting a better IDE, or using good static code analysis tool (Pylint / Flake8 etc.) as this kind of error is easily detected by e.g. PyCharm (your code shadows name from outer scope).
Related: How bad is shadowing names defined in outer scopes?

implementing a deferred exception in Python

I would like to implement a deferred exception in Python that is OK to store somewhere but as soon as it is used in any way, it raises the exception that was deferred. Something like this:
# this doesn't work but it's a start
class DeferredException(object):
def __init__(self, exc):
self.exc = exc
def __getattr__(self, key):
raise self.exc
# example:
mydict = {'foo': 3}
try:
myval = obtain_some_number()
except Exception as e:
myval = DeferredException(e)
mydict['myval'] = myval
def plus_two(x):
print x+2
# later on...
plus_two(mydict['foo']) # prints 5
we_dont_use_this_val = mydict['myval'] # Always ok to store this value if not used
plus_two(mydict['myval']) # If obtain_some_number() failed earlier,
# re-raises the exception, otherwise prints the value + 2.
The use case is that I want to write code to analyze some values from incoming data; if this code fails but the results are never used, I want it to fail quietly; if it fails but the results are used later, then I'd like the failure to propagate.
Any suggestions on how to do this? If I use my DeferredException class I get this result:
>>> ke = KeyError('something')
>>> de = DeferredException(ke)
>>> de.bang # yay, this works
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in __getattr__
KeyError: 'something'
>>> de+2 # boo, this doesn't
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'DeferredException' and 'int'

Read section 3.4.12 of the docs, "Special method lookup for new-style classes." It explains exactly the problem you have encountered. The normal attribute lookup is bypassed by the interpreter for certain operators, such as addition (as you found out the hard way). Thus the statement de+2 in your code never calls your getattr function.
The only solution, according to that section, is to insure that "the special method must be set on the class object itself in order to be consistently invoked by the interpreter."
Perhaps you'd be better off storing all your deferred exceptions in a global list, wrapping your entire program in a try:finally: statement, and printing out the whole list in the finally block.

How can I get the traceback object ( sys.exc_info()[2] , same as sys.exc_traceback ) as a string?

I have a function which catches all exceptions, and I want to be able to get the traceback as a string within this function.
So far this is not working:
def handle_errors(error_type, error_message, error_traceback):
"""catch errors"""
import traceback
error = {}
error['type'] = error_type.__name__
error['message'] = str(error_message)
error['file'] = os.path.split(error_traceback.tb_frame.f_code.co_filename)[1]
error['line'] = error_traceback.tb_lineno
error['traceback'] = repr(traceback.print_tb(error_traceback))
### finalise error handling and exit ###
sys.excepthook = handle_errors
It's the error['traceback'] line which is wrong. Do i even need to use the traceback module?
As per this other vaguely similar question, I have tried:
error['traceback'] = repr(error_traceback.print_exc())
...but this gives an error:
Error in sys.excepthook:
Traceback (most recent call last):
File "xxxxxxxxxxx", line 54, in handle_errors
error['traceback'] = repr(error_traceback.print_exc())
AttributeError: 'traceback' object has no attribute 'print_exc'

Use traceback.format_tb() instead of print_tb() to get the formatted stack trace (as a list of lines):
error['traceback'] = ''.join(traceback.format_tb(error_traceback))
print_tb() directly prints the traceback, that's why you get None as a result (that's the default for any Python function that doesn't return anything explicitely).

traceback.format_exc([limit])
This is like print_exc(limit) but
returns a string instead of printing to a file.
New in version 2.4.
error['traceback'] = traceback.format_exc(error_traceback)