We Keep Coding

Why don't I get a full traceback from a saved exception - and how do I get and save the full trace?

I have a user submitted data validation interface for a scientific site in django, and I want the user to be able to submit files of scientific data that will aid them in resolving simple problems with their data before they're allowed to make a formal submission (to reduce workload on the curators who actually load the data into our database).
The validation interface re-uses the loading code, which is good for code re-use. It has a "validate mode" that doesn't change the database. Everything is in an atomic transaction block and it gets rolled back in any case when it runs in validate mode.
I'm in the middle of a refactor to alleviate a problem. The problem is that the user has to submit the files multiple times, each time, getting the next error. So I've been refining the code to be able to "buffer" the exceptions in an array and only really stop if any error makes further processing impossible. So far, it's working great.
Since unexpected errors are expected in this interface (because the data is complex and lab users are continually finding new ways to screw up the data), I am catching and buffering any exception and intend to write custom exception classes for each case as I encounter them.
The problem is that when I'm adding new features and encounter a new error, the tracebacks in the buffered exceptions aren't being fully preserved, which makes it annoying to debug - even when I change the code to raise and immediately catch the exception so I can add it to the buffer with the traceback. For example, in my debugging, I may get an exception from a large block of code, and I can't tell what line it is coming from.
I have worked around this problem by saving the traceback as a string inside the buffered exception object, which just feels wrong. I had to play around in the shell to get it to work. Here is my simple test case to demonstrate what's happening. It's reproducible for me, but apparently not for others who try this toy example - and I don't know why:
import traceback
class teste(Exception):
"""This is an exception class I'm going to raise to represent some unanticipated exception - for which I will want a traceback."""
pass
def buf(exc, args):
"""This represents my method I call to buffer an exception, but for this example, I just return the exception and keep it in main in a variable. The actual method in my code appends to a data member array in the loader object."""
try:
raise exc(*args)
except Exception as e:
# This is a sanity check that prints the trace that I will want to get from the buffered exception object later
print("STACK:")
traceback.print_stack()
# This is my workaround where I save the trace as a string in the exception object
e.past_tb = "".join(traceback.format_stack())
return e
The above example raises the exception inside buf. (My original code supports both raising the exception for the first time and buffering an already raised and caught exception. In both cases, I wasn't getting a saved full traceback, so I'm only providing the one example case (where I raise it inside the buf method).
And here's what I see when I use the above code in the shell. This first call shows my sanity check - the whole stack, which is what I want to be able to access later:
In [5]: es = buf(teste, ["This is a test"])
STACK:
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/manage.py", line 22, in <module>
main()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/commands/shell.py", line 100, in handle
return getattr(self, shell)(options)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/commands/shell.py", line 36, in ipython
start_ipython(argv=[])
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/__init__.py", line 126, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
app.start()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/terminal/ipapp.py", line 356, in start
self.shell.mainloop()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/terminal/interactiveshell.py", line 566, in mainloop
self.interact()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/terminal/interactiveshell.py", line 557, in interact
self.run_cell(code, store_history=True)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2914, in run_cell
result = self._run_cell(
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2960, in _run_cell
return runner(coro)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 78, in _pseudo_sync_runner
coro.send(None)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3185, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3377, in run_ast_nodes
if (await self.run_code(code, result, async_=asy)):
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3457, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-5-92f4a0db918d>", line 1, in <module>
es = buf(teste, ["This is a test"])
File "<ipython-input-2-86e515dc1ec1>", line 6, in buf
traceback.print_stack()
But this is what I see when I want to see the original traceback from the es object (i.e. the buffered exception) later. It only has the last item from the traceback. This is exactly what I see in the original source code - a single item for the line of code inside the buffer method:
In [8]: traceback.print_exception(type(es), es, es.__traceback__)
Traceback (most recent call last):
File "<ipython-input-2-86e515dc1ec1>", line 3, in buf
raise exc(*args)
teste: This is a test
My workaround suffices for now, but I'd like to have a proper traceback object.
I debugged the issue by re-cloning our repo in a second directory to make sure I hadn't messed up my sandbox. I guess I should try this on another computer too - my office mac. But can anyone point me in the right direction to debug this issue? What could be the cause for losing the full traceback?

Python has a really weird way of building exception tracebacks. You might expect it to build the traceback when the exception is created, or when it's raised, but that's not how it works.
Python builds a traceback as an exception propagates. Every time the exception propagates up to a new stack frame, a traceback entry for that stack frame is added to the exception's traceback.
This means that an exception's traceback only goes as far as the exception itself propagates. If you catch it (and don't reraise it), the traceback only goes up to the point where it got caught.
Unfortunately, your workaround is about as good as it gets. You're not really losing the full traceback, because a full traceback was never created. If you want full stack info, you need to record it yourself, with something like the traceback.format_stack() function you're currently using.

How to throw exception messages inside the multiprocessing code?

For example, I am using multiprocessing pool to process files:
with Pool(5) as pool:
results = pool.starmap(self.process_file, zip(files, repeat(channel)))
When an exception occurs inside the function process_file, the exception message indicates that it occurs at the pool.starmap line, but not the actual place inside process_file function.
I am using PyCharm to develop and debug. Is there a way to change this behavior? The current error message doesn't give the correct position of the error occurred.

Multiprocessing transfers the errors between processes using the pickle module, but pickle doesn't know how to preserve the tracebacks of exceptions by default.
I found tblib to be a very convenient way to address this shortcoming. Based on this example I suggest you try adding this code to the main module of your code:
from tblib import pickling_support
# all your setup code
pickling_support.install()
if __name__ == "__main__":
# your pool work

The exception has the original exception info but PyCharm is not ferreting it out.
Assuming there are no PyCharm configuration options to enhance its ability to ferret out all the exception information, and not just the outer exception as your are seeing, you need to programmatically extract it out.
For good in-program error handling, you probably want to do that anyway. Especially with sub-processes, I very often will catch Exception and log it and re-raise if not considered handled (depends), or if I catch a specific exception I'm expecting and consider it handled, I won't re-raise it.
Note, it's not only PyCharm showing the outer exception... I see the same thing with other tools.
Below will show you the original problem (see "Line 7" below) and re-raise. Again, re-raising or not is really context dependent so below is just an example. But the point is that the exception you are seeing has more data that PyCharm by default is apparently not showing you.
from itertools import repeat
from multiprocessing import Pool
import traceback
def process(a,b):
print(a,b)
raise Exception("not good")
if __name__ == '__main__':
with Pool(5) as pool:
try:
results = pool.starmap(process, zip([1,2,3,4,5], repeat('A')))
except Exception as ex:
print("starmap failure:")
for error_line in traceback.format_exception(ex, ex, ex.__traceback__):
error_line = error_line.strip()
if not error_line:
continue
print(f" {error_line}")
raise # re-raise if we do not consider this handled.
Gives me this output:
starmap failure:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\...\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "C:\Users\...\multiprocessing\pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "...\starmap_exception.py", line 7, in process
raise Exception("not good")
Exception: not good
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...\starmap_exception.py", line 12, in <module>
results = pool.starmap(process, zip([1,2,3,4,5], repeat('A')))
File "C:\Users\...\multiprocessing\pool.py", line 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "C:\Users\...\multiprocessing\pool.py", line 771, in get
raise self._value
Exception: not good

Ignore and log error with contextlib contextmanager

I want a context manager to catch an exception, print the stack trace, and then allow execution to continue.
I want to know if I can do this with the contextlib contextmanager decorator. If not, how can I do it?
Documentation suggests the following:
At the point where the generator yields, the block nested in the with statement is executed. The generator is then resumed after the block is exited. If an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred. Thus, you can use a try…except…finally statement to trap the error (if any), or ensure that some cleanup takes place. If an exception is trapped merely in order to log it or to perform some action (rather than to suppress it entirely), the generator must reraise that exception.
So I try the obvious approach that the documentation leads me to:
import contextlib
import logging
#contextlib.contextmanager
def log_error():
try:
yield
except Exception as e:
logging.exception('hit exception')
finally:
print 'done with contextmanager'
def something_inside_django_app():
with log_error():
raise Exception('alan!')
something_inside_django_app()
print 'next block of code'
This produces the output
ERROR:root:hit exception
Traceback (most recent call last):
File "exception_test.py", line 8, in log_error
yield
File "exception_test.py", line 17, in something_inside_django_app
raise Exception('alan!')
Exception: alan!
done with contextmanager
next block of code
This loses critical information about where the exception was raised from. Consider what you get when you adjust the context manager to not supress the exception:
Traceback (most recent call last):
File "exception_test.py", line 20, in <module>
something_inside_django_app()
File "exception_test.py", line 17, in something_inside_django_app
raise Exception('alan!')
Exception: alan!
Yes, it was able to tell me that the exception was raised from line 17, thank you very much, but the prior call at line 20 is lost information. How can I have the context manager give me the actual full call stack and not its truncated version of it? To recap, I want to fulfill two requirements:
have a python context manager suppress an exception raised in the code it wraps
print the stack trace that would have been generated by that code, had I not been using the context manager
If this cannot be done with the decorator, then I'll use the other style of context manager instead. If this cannot be done with context managers, period, I would like to know what a good pythonic alternative is.

I have updated my solution for this problem here:
https://gist.github.com/AlanCoding/288ee96b60e24c1f2cca47326e2c0af1
There was more context that the question missed. In order to obtain the full stack at the point of exception, we need both the traceback returned to the context manager, and the current context. Then we can glue together the top of the stack with the bottom of the stack.
To illustrate the use case better, consider this:
def err_method1():
print [1, 2][4]
def err_method2():
err_method1()
def outside_method1():
with log_error():
err_method2()
def outside_method2():
outside_method1()
outside_method2()
To really accomplish what this question is looking for, we want to see both outer methods, and both inner methods in the call stack.
Here is a solution that does appear to work for this:
class log_error(object):
def __enter__(self):
return
def __exit__(self, exc_type, exc_value, exc_traceback):
if exc_value:
# We want the _full_ traceback with the context, so first we
# get context for the current stack, and delete the last 2
# layers of context, saying that we're in the __exit__ method...
top_stack = StringIO.StringIO()
tb.print_stack(file=top_stack)
top_lines = top_stack.getvalue().strip('\n').split('\n')[:-4]
top_stack.close()
# Now, we glue that stack to the stack from the local error
# that happened within the context manager
full_stack = StringIO.StringIO()
full_stack.write('Traceback (most recent call last):\n')
full_stack.write('\n'.join(top_lines))
full_stack.write('\n')
tb.print_tb(exc_traceback, file=full_stack)
full_stack.write('{}: {}'.format(exc_type.__name__, str(exc_value)))
sinfo = full_stack.getvalue()
full_stack.close()
# Log the combined stack
logging.error('Log message\n{}'.format(sinfo))
return True
The traceback looks like:
ERROR:root:Log message
Traceback (most recent call last):
File "exception_test.py", line 71, in <module>
outside_method2()
File "exception_test.py", line 69, in outside_method2
outside_method1()
File "exception_test.py", line 65, in outside_method1
err_method2()
File "exception_test.py", line 60, in err_method2
err_method1()
File "exception_test.py", line 56, in err_method1
print [1, 2][4]
IndexError: list index out of range
This is the same information that you would expect from doing logging.exception in a try-except over the same code that you wrap in the context manager.

What is the simplest way of using Python pdb to inspect the cause of an unhandled exception?

I just converted all my unit test data from JSON to YAML, and now an exception is raised somewhere in my code. More specifically, this is printed traceback:
Traceback (most recent call last):
File "tests/test_addrtools.py", line 95, in test_validate_correctable_addresses
self.assertTrue(self.validator(addr), msg)
File "/Users/tomas/Dropbox/Broadnet/broadpy/lib/broadpy/addrtools.py", line 608, in __call__
self.validate(addr)
File "/Users/tomas/Dropbox/Broadnet/broadpy/lib/broadpy/addrtools.py", line 692, in validate
if self._correction_citytypo(addr): return
File "/Users/tomas/Dropbox/Broadnet/broadpy/lib/broadpy/addrtools.py", line 943, in _correction_citytypo
ratio = lev_ratio(old_city, city)
TypeError: ratio expected two Strings or two Unicodes
Now, the file "addrtools.py" on line 943 contains the answer to my problem. I want to see the type and values of old_city and city in the scope where the exception is raised. I have this sort of issue all the time, and a quick and painless method of using pdb to inspect the locals in the scope where the exception is raised would save me tons of time in the future.
I did try the solution posted in the answer to this question, but the post-mortem function places me in python2.7/unittest/main.py(231)runTests() which doesn't help me a whole lot. I guess this is because the exception is caught and re-raised from the unittest code.

Wrap it with that:
def debug_on(*exceptions):
if not exceptions:
exceptions = (AssertionError, )
def decorator(f):
#functools.wraps(f)
def wrapper(*args, **kwargs):
try:
return f(*args, **kwargs)
except exceptions:
pdb.post_mortem(sys.exc_info()[2])
return wrapper
return decorator
Example:
#debug_on(TypeError)
def buggy_function()
....
raise TypeError

The unittest superset nose has an option that drops you to pdb when a test fails, if it's okay for you to use nose as your test runner:
--pdb Drop into debugger on errors
--pdb-failures Drop into debugger on failures

Windows Python 2.5 Traceback.py Nonetype error

Hello I am running Python 2.5 on Windows and whenever my application gets an exception rather than seeing the debug information I get an error inside of the traceback.py file itself. Anyone know a fix for this mb a patch or replacement file.
Traceback (most recent call last):
File "C:\Python25\lib\logging\__init__.py", line 744, in emit
msg = self.format(record)
File "C:\Python25\lib\logging\__init__.py", line 630, in format
return fmt.format(record)
File "C:\Python25\lib\logging\__init__.py", line 426, in format
record.exc_text = self.formatException(record.exc_info)
File "C:\Python25\lib\logging\__init__.py", line 398, in formatException
traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
File "C:\Python25\lib\traceback.py", line 126, in print_exception
lines = format_exception_only(etype, value)
File "C:\Python25\lib\traceback.py", line 176, in format_exception_only
stype = etype.__name__
AttributeError: 'NoneType' object has no attribute '__name__'
===EDIT===
Found same error in mailing list here outdated answer it seems
http://mail.python.org/pipermail/python-dev/2006-September/068975.html

Possible causes:
Calling logging.exception() when there is no active exception
Calling a logging function with exc_info=1, when there is no active exception.
Calling a logging function with exc_info=(None, None, None) to a logging function (e.g. if doing the exception logging manually).
You should not use logging.exception outside of an except block.
The exception is caused by a None exception type passed to traceback.print_exception, meaning that there is no active exception to process.
Meanwhile, the newsgroup posting you linked to indicates that it was a regression in the standard library that resulted in that particular traceback. You may want to try upgrading your Python to 2.5.1, which fixed this particular problem.