We Keep Coding

Why don't I get a full traceback from a saved exception - and how do I get and save the full trace?

I have a user submitted data validation interface for a scientific site in django, and I want the user to be able to submit files of scientific data that will aid them in resolving simple problems with their data before they're allowed to make a formal submission (to reduce workload on the curators who actually load the data into our database).
The validation interface re-uses the loading code, which is good for code re-use. It has a "validate mode" that doesn't change the database. Everything is in an atomic transaction block and it gets rolled back in any case when it runs in validate mode.
I'm in the middle of a refactor to alleviate a problem. The problem is that the user has to submit the files multiple times, each time, getting the next error. So I've been refining the code to be able to "buffer" the exceptions in an array and only really stop if any error makes further processing impossible. So far, it's working great.
Since unexpected errors are expected in this interface (because the data is complex and lab users are continually finding new ways to screw up the data), I am catching and buffering any exception and intend to write custom exception classes for each case as I encounter them.
The problem is that when I'm adding new features and encounter a new error, the tracebacks in the buffered exceptions aren't being fully preserved, which makes it annoying to debug - even when I change the code to raise and immediately catch the exception so I can add it to the buffer with the traceback. For example, in my debugging, I may get an exception from a large block of code, and I can't tell what line it is coming from.
I have worked around this problem by saving the traceback as a string inside the buffered exception object, which just feels wrong. I had to play around in the shell to get it to work. Here is my simple test case to demonstrate what's happening. It's reproducible for me, but apparently not for others who try this toy example - and I don't know why:
import traceback
class teste(Exception):
"""This is an exception class I'm going to raise to represent some unanticipated exception - for which I will want a traceback."""
pass
def buf(exc, args):
"""This represents my method I call to buffer an exception, but for this example, I just return the exception and keep it in main in a variable. The actual method in my code appends to a data member array in the loader object."""
try:
raise exc(*args)
except Exception as e:
# This is a sanity check that prints the trace that I will want to get from the buffered exception object later
print("STACK:")
traceback.print_stack()
# This is my workaround where I save the trace as a string in the exception object
e.past_tb = "".join(traceback.format_stack())
return e
The above example raises the exception inside buf. (My original code supports both raising the exception for the first time and buffering an already raised and caught exception. In both cases, I wasn't getting a saved full traceback, so I'm only providing the one example case (where I raise it inside the buf method).
And here's what I see when I use the above code in the shell. This first call shows my sanity check - the whole stack, which is what I want to be able to access later:
In [5]: es = buf(teste, ["This is a test"])
STACK:
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/manage.py", line 22, in <module>
main()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/commands/shell.py", line 100, in handle
return getattr(self, shell)(options)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/commands/shell.py", line 36, in ipython
start_ipython(argv=[])
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/__init__.py", line 126, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
app.start()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/terminal/ipapp.py", line 356, in start
self.shell.mainloop()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/terminal/interactiveshell.py", line 566, in mainloop
self.interact()
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/terminal/interactiveshell.py", line 557, in interact
self.run_cell(code, store_history=True)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2914, in run_cell
result = self._run_cell(
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2960, in _run_cell
return runner(coro)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 78, in _pseudo_sync_runner
coro.send(None)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3185, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3377, in run_ast_nodes
if (await self.run_code(code, result, async_=asy)):
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3457, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-5-92f4a0db918d>", line 1, in <module>
es = buf(teste, ["This is a test"])
File "<ipython-input-2-86e515dc1ec1>", line 6, in buf
traceback.print_stack()
But this is what I see when I want to see the original traceback from the es object (i.e. the buffered exception) later. It only has the last item from the traceback. This is exactly what I see in the original source code - a single item for the line of code inside the buffer method:
In [8]: traceback.print_exception(type(es), es, es.__traceback__)
Traceback (most recent call last):
File "<ipython-input-2-86e515dc1ec1>", line 3, in buf
raise exc(*args)
teste: This is a test
My workaround suffices for now, but I'd like to have a proper traceback object.
I debugged the issue by re-cloning our repo in a second directory to make sure I hadn't messed up my sandbox. I guess I should try this on another computer too - my office mac. But can anyone point me in the right direction to debug this issue? What could be the cause for losing the full traceback?

Python has a really weird way of building exception tracebacks. You might expect it to build the traceback when the exception is created, or when it's raised, but that's not how it works.
Python builds a traceback as an exception propagates. Every time the exception propagates up to a new stack frame, a traceback entry for that stack frame is added to the exception's traceback.
This means that an exception's traceback only goes as far as the exception itself propagates. If you catch it (and don't reraise it), the traceback only goes up to the point where it got caught.
Unfortunately, your workaround is about as good as it gets. You're not really losing the full traceback, because a full traceback was never created. If you want full stack info, you need to record it yourself, with something like the traceback.format_stack() function you're currently using.

MySQL Type Converter Issue

I am new to tkinter and I am trying to connect python with MySQL. When I Try to insert the records via a new window in python I get the following error message
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\sapna\anaconda3\lib\tkinter\__init__.py", line 1705, in __call__
return self.func(*args)
File "C:\Users\sapna\Desktop\SAMARTH.py", line 141, in dynamic_data_entry
libcur.execute("INSERT INTO tcl VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)" ,
(id1,name1,auth1,doi1,dor1,reiss,rname,Address,gen1,mil1))
File "C:\Users\sapna\anaconda3\lib\site-packages\mysql\connector\cursor_cext.py", line 248, in
execute
prepared = self._cnx.prepare_for_mysql(params)
File "C:\Users\sapna\anaconda3\lib\site-packages\mysql\connector\connection_cext.py", line 626, in
prepare_for_mysql
result = self._cmysql.convert_to_mysql(*params)
_mysql_connector.MySQLInterfaceError: Python type method cannot be converted
Here is the snip of my insert command.
def dynamic_data_entry():
bname.delete(0,END)
bid.delete(0,END)
author.delete(0,END)
DOI.delete(0,END)
DOR.delete(0,END)
RES.delete(0,END)
RNM.delete(0,END)
AD.delete(0,END)
mail.delete(0,END)
id1=ln.get()
name1=fn.get()
auth1=dn.get()
doi1=an.get()
dor1=gn.get()
rname=kn.get()
Address=pn.get
reiss=mn.get()
gen1=var.get()
mil1=ll.get()
libcur.execute("INSERT INTO tcl VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)" ,
(id1,name1,auth1,doi1,dor1,reiss,rname,Address,gen1,mil1))
mydb.commit()
messagebox.showinfo("INSERT STATUS","RECORD HAS BEEN ADDED SUCCESSFULLY")
libcur.close()
The datatype that I have used is varchar() in Book_id,DOI,DOR and char() in others for MySQL.
All the dataypes that I have used in Python are as follows
var=StringVar()
fn=StringVar()
yn=StringVar()
ln=StringVar()
dn=StringVar()
an=StringVar()
kn=StringVar()
gn=StringVar()
mn=StringVar()
pn=StringVar()
ll=StringVar()
Thanks in advance

Looks like you missed some parentheses in your assignment to Address. Try Address=pn.get().
The error implies that you're passing a method rather than a value. This is consistent with Address=pn.get, i.e. you're passing the pn.get method rather than the return value of pn.get().

Ending a TKinter class method without killing the main thread

I've scoured the internet on this one and haven't come up with anything, so please excuse me if this is a dumb question, but...
I've written a Tkinter class that takes a .xlsx file as input for one of its processes. You press a button and it uses TKFileDialog.askopenfile() to pop the familiar-looking file input dialog. However, if a user hits cancel, the input is NoneType, the file doesn't open, and the TKinter window terminates with a TypeError.
def GUIDocPullRename(self):
indir = self.e1.get()
Tab = askopenfile()
try:
tBook = xlrd.open_workbook(Tab)
except TypeError:
self.text.insert(END, "Canceled")
#Insert method-ender here
I want to insert something in the exception catch that stops the GUIDocPullRename from continuing, but allows the parent TKinter process to continue running so the user can specify another file without having to re-run the whole program. Is it possible or am I just being a silly newb?
Error is this, when the try/except is excluded:
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Python27\lib\lib-tk\Tkinter.py", line 1470, in __call__
return self.func(*args)
File "P:/Administrative/IT/Scripting/BillsPython/PdfMerger.py", line 102, in GUIDocPullRename
tBook = xlrd.open_workbook(Tab)
File "C:\Python27\lib\site-packages\xlrd\__init__.py", line 394, in open_workbook
f = open(filename, "rb")
TypeError: coercing to Unicode: need string or buffer, NoneType found

Traceback of exceptions: Upper frames missing

In the following stacktrace I miss the upper frames.
Who called callback() in ioloop.py line 458?
The stacktrace comes from a unittest TestCase. All tests pass but this traceback is in the logs and reproducible.
I can't see in which test of the TestCase the exception was raised.
ERROR [25950] Exception in callback <functools.partial object at 0x5358368>
Traceback (most recent call last):
File "/home/modwork_foo_dtg/lib/python2.7/site-packages/tornado/ioloop.py", line 458, in _run_callback
callback()
File "/home/modwork_foo_dtg/lib/python2.7/site-packages/tornado/stack_context.py", line 331, in wrapped
raise_exc_info(exc)
File "/home/modwork_foo_dtg/lib/python2.7/site-packages/tornado/stack_context.py", line 302, in wrapped
ret = fn(*args, **kwargs)
File "/home/modwork_foo_dtg/src/websocketrpc/websocketrpc/client.py", line 71, in connect
self.ws = websocket_connect(self.args.url)
File "/home/modwork_foo_dtg/src/websocketrpc/websocketrpc/client.py", line 179, in websocket_connect
conn = websocket.WebSocketClientConnection(io_loop, request)
File "/home/modwork_foo_dtg/lib/python2.7/site-packages/tornado/websocket.py", line 777, in __init__
raise Exception('%s %s' % (request, request.url))
Exception: <tornado.httpclient._RequestProxy object at 0x535cb10> None
How could I use tornado to see the upper stacktrace frames?
The exception itself is not the problem.

You can use IOLoop.handle_callback_exception to print from sys.exc_info to see what specifically is breaking.

The callback was invoked by ioloop.py:458, just like it says. No outer stack frames are shown because the exception didn't escape that frame. The thing that's confusing you is that the callback goes on to re-raise an exception that was captured earlier.
In Python 2, preserving tracebacks to re-raise later is messy (it gets better in Python 3). Tornado usually does the right thing here, but there are some gaps where a traceback will get truncated. The main problem I'm aware of in current versions is that AsyncHTTPClient tends to throw away tracebacks (and there are some annoying backwards-compatibility issues with fixing this).
As a crude workaround while debugging, you can try printing traceback.format_stack just before throwing an exception (at least where it's feasible to modify the code, as you've done here to add an exception to websocket.py).

Finding exception in python multiprocessing

I have a bit of python code that looks like this:
procs = cpu_count()-1
if serial or procs == 1:
results = map(do_experiment, experiments)
else:
pool = Pool(processes=procs)
results = pool.map(do_experiment, experiments)
It runs fine when I set the serial flag, but it gives the following error when the Pool is used. When I try to print something from do_experiment nothing shows up, so I can't try/catch there and print a stack trace.
Exception in thread Thread-2:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 530, in __bootstrap_inner
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 483, in run
self.__target(*self.__args, **self.__kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 285, in _handle_tasks
put(task)
TypeError: 'NoneType' object is not callable
What is a good way to proceed debugging this?

I went back in my git history until I found a commit where things were still working.
I added a class to my code that extends dict so that keys can be accessed with a . (so dict.foo in stead of dict["foo"]. Multiprocessing did not take kindly to this, using an ordinary dict solved the problem.