Using Python 3.7, I am trying to catch an exception and re-raise it by following an example I found on StackOverflow. While the example does work, it doesn't seem to work for all situations. Below I have two asynchronous Python scripts that try to re-raise exceptions. The first example works, it will print both the inner and outer exception.
import asyncio
class Foo:
async def throw_exception(self):
raise Exception("This is the inner exception")
async def do_the_thing(self):
try:
await self.throw_exception()
except Exception as e:
raise Exception("This is the outer exception") from e
async def run():
await Foo().do_the_thing()
def main():
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
if __name__ == "__main__":
main()
Running this will correctly output the following exception stack trace:
$ py test.py
Traceback (most recent call last):
File "test.py", line 9, in do_the_thing
await self.throw_exception()
File "test.py", line 5, in throw_exception
raise Exception("This is the inner exception")
Exception: This is the inner exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test.py", line 21, in <module>
main()
File "test.py", line 18, in main
loop.run_until_complete(run())
File "C:\Python37\lib\asyncio\base_events.py", line 584, in run_until_complete
return future.result()
File "test.py", line 14, in run
await Foo().do_the_thing()
File "test.py", line 11, in do_the_thing
raise Exception("This is the outer exception") from e
Exception: This is the outer exception
However, in my next Python script, I have multiple tasks that I queue up that I want to get a similar exception stack trace from. Essentially, I except the above stack trace to be printed 3 times (once for each task in the following script). The only difference between the above and below scripts is the run() function.
import asyncio
class Foo:
async def throw_exception(self):
raise Exception("This is the inner exception")
async def do_the_thing(self):
try:
await self.throw_exception()
except Exception as e:
raise Exception("This is the outer exception") from e
async def run():
tasks = []
foo = Foo()
tasks.append(asyncio.create_task(foo.do_the_thing()))
tasks.append(asyncio.create_task(foo.do_the_thing()))
tasks.append(asyncio.create_task(foo.do_the_thing()))
results = await asyncio.gather(*tasks, return_exceptions=True)
for result in results:
if isinstance(result, Exception):
print(f"Unexpected exception: {result}")
def main():
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
if __name__ == "__main__":
main()
The above code snippet produces the disappointingly short exceptions lacking stack traces.
$ py test.py
Unexpected exception: This is the outer exception
Unexpected exception: This is the outer exception
Unexpected exception: This is the outer exception
If I change return_exceptions to be False, I will get the exceptions and stack trace printed out once and then execution stops and the remaining two tasks are cancelled. The output is identical to the output from the first script. The downside of this approach is, I want to continue processing tasks even when exceptions are encountered and then display all the exceptions at the end when all the tasks are completed.
asyncio.gather will stop at the first exception if you do not provide a return_exceptions=True argument, so your approach is the right one: you need to gather all the results and exceptions first, then display them.
To get the full stacktrace that you are missing, you will need to do more than just "print" the exception. Have a look at the traceback module in the stdlib which has all you need for that: https://docs.python.org/3/library/traceback.html
You can also use logging.exception, that would do more or less the same.
Related
For example, I am using multiprocessing pool to process files:
with Pool(5) as pool:
results = pool.starmap(self.process_file, zip(files, repeat(channel)))
When an exception occurs inside the function process_file, the exception message indicates that it occurs at the pool.starmap line, but not the actual place inside process_file function.
I am using PyCharm to develop and debug. Is there a way to change this behavior? The current error message doesn't give the correct position of the error occurred.
Multiprocessing transfers the errors between processes using the pickle module, but pickle doesn't know how to preserve the tracebacks of exceptions by default.
I found tblib to be a very convenient way to address this shortcoming. Based on this example I suggest you try adding this code to the main module of your code:
from tblib import pickling_support
# all your setup code
pickling_support.install()
if __name__ == "__main__":
# your pool work
The exception has the original exception info but PyCharm is not ferreting it out.
Assuming there are no PyCharm configuration options to enhance its ability to ferret out all the exception information, and not just the outer exception as your are seeing, you need to programmatically extract it out.
For good in-program error handling, you probably want to do that anyway. Especially with sub-processes, I very often will catch Exception and log it and re-raise if not considered handled (depends), or if I catch a specific exception I'm expecting and consider it handled, I won't re-raise it.
Note, it's not only PyCharm showing the outer exception... I see the same thing with other tools.
Below will show you the original problem (see "Line 7" below) and re-raise. Again, re-raising or not is really context dependent so below is just an example. But the point is that the exception you are seeing has more data that PyCharm by default is apparently not showing you.
from itertools import repeat
from multiprocessing import Pool
import traceback
def process(a,b):
print(a,b)
raise Exception("not good")
if __name__ == '__main__':
with Pool(5) as pool:
try:
results = pool.starmap(process, zip([1,2,3,4,5], repeat('A')))
except Exception as ex:
print("starmap failure:")
for error_line in traceback.format_exception(ex, ex, ex.__traceback__):
error_line = error_line.strip()
if not error_line:
continue
print(f" {error_line}")
raise # re-raise if we do not consider this handled.
Gives me this output:
starmap failure:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\...\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "C:\Users\...\multiprocessing\pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "...\starmap_exception.py", line 7, in process
raise Exception("not good")
Exception: not good
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...\starmap_exception.py", line 12, in <module>
results = pool.starmap(process, zip([1,2,3,4,5], repeat('A')))
File "C:\Users\...\multiprocessing\pool.py", line 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "C:\Users\...\multiprocessing\pool.py", line 771, in get
raise self._value
Exception: not good
I have nested try except blocks that appear to print the wrong exception.
This is on python 2.7.16
import logging
try:
raise ValueError('1')
except Exception as e:
try:
raise KeyError('2')
except KeyError:
logging.exception(e)
The output is
ERROR:root:1
Traceback (most recent call last):
File "<stdin>", line 5, in <module>
KeyError: '2'
But I would expect the output to be for the first exception as that is what is assigned to e.
Follow up question
If I wanted to save the execution info from the first exception, how would I do that?
Oh, self-answer here: It's only the execution info that is for the second exception. The actual message is from the first one.
How can we exit a script with an unhandled exception trace-back, going through every catch-statement in the outer try-catch blocks we are nested inside of?
Supposed we define the following functions:
import sys
def foo():
try:
bar()
list()[592]
except IndexError:
print('`foo()` be handling error generated by list()[592]')
return
def bar():
try:
list()[345]
except IndexError:
pass
### MAGIC GOES HERE ###
return
Now, we make the call:
foo()
What I would like, is for the catch-statement in bar() to kill the process.
We do not want foo() to catch the IndexError, and definitely do not want to print 'foo()' be handling error generated by list()[592].
If we replace ### MAGIC GOES HERE ### with something like sys.exit(-20374290374), then we successfully kill the process, but there is no trace-back, and there is no information regarding an "uncaught" exception.
How can we exit as if the exception was uhandled, without first going through all of the outer try-catching blocks?
You raise something that is not catched:
import sys
class MyException(Exception):
pass
def foo():
try:
bar()
list()[592]
except IndexError:
print('`foo()` be handling error generated by list()[592]')
return
def bar():
try:
list()[345]
except IndexError:
raise MyException("step out")
return
foo()
Output:
Traceback (most recent call last):
File "main.py", line 26, in <module>
foo()
File "main.py", line 10, in foo
bar()
File "main.py", line 21, in bar
raise MyException("step out")
__main__.MyException: step out
In case you somewhere catch the base Exception (-you should not-) you need to check for your special kind and reraise it:
except Exception as e:
if isinstance(e,MyException):
raise # e not needed, raise rethrows last one
print('`foo()` be handling error generated by list()[592]')
I'm using the Pool function of the multiprocessing module in order to run the same code in parallel on different data.
It turns out that on some data my code raises an exception, but the precise line in which this happens is not given:
Traceback (most recent call last):
File "my_wrapper_script.py", line 366, in <module>
main()
File "my_wrapper_script.py", line 343, in main
results = pool.map(process_function, folders)
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 148, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 422, in get
raise self._value
KeyError: 'some_key'
I am aware of multiprocessing.log_to_stderr() , but it seems that it is useful when concurrency issues arise, which is not my case.
Any ideas?
If you're using a new enough version of Python, you'll actually see the real exception get printed prior to that one. For example, here's a sample that fails:
import multiprocessing
def inner():
raise Exception("FAIL")
def f():
print("HI")
inner()
p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()
Here's the exception when running this with python 3.4:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "test.py", line 9, in f
inner()
File "test.py", line 4, in inner
raise Exception("FAIL")
Exception: FAIL
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test.py", line 13, in <module>
p.apply(f)
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 253, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
Exception: FAIL
If using a newer version isn't an option, the easiest thing to do is to wrap your worker function in a try/except block that will print the exception prior to re-raising it:
import multiprocessing
import traceback
def inner():
raise Exception("FAIL")
def f():
try:
print("HI")
inner()
except Exception:
print("Exception in worker:")
traceback.print_exc()
raise
p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()
Output:
HI
Exception in worker:
Traceback (most recent call last):
File "test.py", line 11, in f
inner()
File "test.py", line 5, in inner
raise Exception("FAIL")
Exception: FAIL
Traceback (most recent call last):
File "test.py", line 18, in <module>
p.apply(f)
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 244, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
Exception: FAIL
You need to implement your own try/except block in the worker. Depending on how you want to organize your code, you could log to stderr as you mention above, log to some other place like a file, return some sort of error code or even tag the exception with the current traceback and re-raise. Here's an example of the last technique:
import traceback
import multiprocessing as mp
class MyError(Exception):
pass
def worker():
try:
# your real code here
raise MyError("boom")
except Exception, e:
e.traceback = traceback.format_exc()
raise
def main():
pool = mp.Pool()
try:
print "run worker"
result = pool.apply_async(worker)
result.get()
# handle exceptions you expect
except MyError, e:
print e.traceback
# re-raise the rest
except Exception, e:
print e.traceback
raise
if __name__=="__main__":
main()
It returns
run worker
Traceback (most recent call last):
File "doit.py", line 10, in worker
raise MyError("boom")
MyError: boom
This question already has answers here:
"Inner exception" (with traceback) in Python?
(9 answers)
Closed 8 years ago.
Is there a way to provide information about the cause of an inner exception when passing it up the chain (like it's possible in java with the cause property of Exception class).
Please consider the following "python pseudo code" (without 100 % correct and invented function and class names)
try:
clientlib.receive_data_chunk()
except ClientException as clientException:
raise RuntimeError("reading from client failed"
+ " (see nested exceptions for details)", cause=clientException)
and in clientlib.py
def receive_data_chunk():
try:
chunk = socket.read(20)
return chunk
except IOException as iOException:
raise ClientException("couldn't read from client", cause = iOException)
If not within native python what would be best practice to achieve what I want to do?
Please note that I want to preserve both stacktraces of the inner and the outer exception, i.e. the following solution is not satisfying:
import sys
def function():
try:
raise ValueError("inner cause")
except Exception:
_, ex, traceback = sys.exc_info()
message = "outer explanation (see nested exception for details)"
raise RuntimeError, message, traceback
if __name__ == "__main__":
function()
produces only the following output:
Traceback (most recent call last):
File "a.py", line 13, in <module>
function()
File "a.py", line 6, in function
raise ValueError("inner cause")
RuntimeError: outer explanation (see nested exception for details)
I cannot see where the RuntimeError occured, so in my understanding the outer stacktrace is lost.
In Python 3, you can use the from keyword to specify an inner exception:
raise ClientException(...) from ioException
You get a traceback that looks like this:
Traceback (most recent call last):
...
IOException: timeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
...
ClientException: couldn't read from client