I'm trying to raise an exception with a stacktrace that doesn't include the last Frame.
In python 2 it was possible to remove the last entry from the strack like this:
def exceptionCatcher(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception:
excInfo = sys.exc_info()
# some custom logging stuff
raise excInfo[0], excInfo[1], excInfo[2].tb_next
return wrapper
def a():
raise Exception("My Exception")
#exceptionCatcher
def test():
a()
This worked in python and excluded the last exceptionCatcher call itself from the traceback e.g.
Traceback (most recent call last):
File "test.py", line 15, in <module>
test()
File "test.py", line 12, in a
raise Exception("a")
Exception: a
instead of
Traceback (most recent call last):
File "test.py", line 12, in <module>
test()
File "test.py", line 9, in wrapper
raise e
File "test.py", line 5, in wrapper
return func(*args, **kwargs)
File "test.py", line 10, in test
a()
File "test.py", line 5, in a
raise Exception("a")
Exception: a
Switching to python3 it's not possible to raise an exception like:
raise excInfo[0], excInfo[1], excInfo[2].tb_next
and i had to use:
def exceptionCatcher(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception as e:
excInfo = sys.exc_info()
# some custom logging stuff
raise e.with_traceback(excInfo[2].tb_next)
...
Which added the following line to my traceback:
Traceback (most recent call last):
File "test.py", line 12, in <module>
test()
File "test.py", line 9, in wrapper
raise e.with_traceback(excInfo[2].tb_next)
File "test.py", line 10, in test
a()
File "test.py", line 5, in a
raise Exception("a")
Exception: a
Is there any way to get rid of this part of the traceback?
I know exceptionCatching can be done with the exceptionhook but in the scenario i'm using this it's not possible to use the exceptionhook as it's another third party application (in cpp) which seems to catch the exception so the exceptionhook won't catch any. So i came up with the idea of using a wrapper decorator as shown above eventho it's a hacky way.
Instead of a decorator, use a context manager, which doesn't add a stack frame in the first place.
import contextlib
#contextlib.contextmanager
def exceptionCatcher():
try:
yield
except Exception as e:
print("custom logging")
raise
def a():
raise Exception("My Exception")
def test():
with exceptionCatcher():
a()
test()
Related
I want to add a sentence to every error message my Python program raise. Something like this:
Traceback (most recent call last):
File "test.py", line 1, in <module>
raise Exception
Exception
AN ERROR OCCURRED, PLEASE ASK ABOUT IT ON STACKOVERFLOW!
I mean every exception, including the built-in ones. How can I do this?
I am not sure if its possible to elegantly change all exception messages.
Here's the next best thing I could come up with.
We are going to use decorators.
In general, decorators are like wrappers for functions.
There is a good explanation for how they work here: https://youtu.be/7lmCu8wz8ro?t=2720
This is the one I came up with:
def except_message(message=''):
def inner(f):
def wrapper(*args, **kwargs):
try:
return f(*args, **kwargs)
except Exception as e:
raise type(e)(str(e) + "\n" + message).with_traceback(sys.exc_info()[2])
return wrapper
return inner
Atop a function that you want to use this decorator on, write #except_message(message='My_message') where 'My_message' is whatever you want the message to be. (It will add it to the end of the exception message)
Example:
#except_message(message='FOUND AN EXCEPTION')
def foo():
raise Exception()
After it is run, the following is returned by the console:
Traceback (most recent call last):
File "main.py", line 7, in wrapper
return f(*args, **kwargs)
File "main.py", line 15, in foo
raise Exception()
Exception
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 17, in <module>
foo()
File "main.py", line 9, in wrapper
raise type(e)(str(e) + "\n" + message).with_traceback(sys.exc_info()[2])
File "main.py", line 7, in wrapper
return f(*args, **kwargs)
File "main.py", line 15, in foo
raise Exception()
Exception:
FOUND AN EXCEPTION
If you only want the message you chose to appear, change in the decorator's function str(e) + "\n" + message to message.
Also, to change all exceptions to this message, you could wrap you code in a function (either by calling it inside a function in a different file or by simply changing the indentation) and then using the decorator.
Credits:
https://stackoverflow.com/a/6062799/5323429
https://stackoverflow.com/a/13898994/5323429
Python 3 features the raises... from construct:
def f():
assert False
def g():
try:
f()
except Exception as e:
raise Exception('this is a wrapper') from e
g()
The above prints out:
Traceback (most recent call last):
File "raise_from.py", line 6, in g
f()
File "raise_from.py", line 2, in f
assert False
AssertionError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "raise_from.py", line 10, in <module>
g()
File "raise_from.py", line 8, in g
raise Exception('this is a wrapper') from e
Exception: this is a wrapper
Which is very nice and neat.
However, when I try debugging the program, either with command-line pdb or PyCharm, I cannot find any way to jump onto the stack of the wrapped exception. How do I retrieve the local variables of f() at the moment it raised the original AssertionError?
When initializer throw Error like below, script won't stop.
I would like to abort before starting main process(do not run 'do_something').
from multiprocessing import Pool
import contextlib
def initializer():
raise Exception("init failed")
def do_something(args):
# main process
pass
pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
try:
pool.map_async(do_something, [1]).get(100)
except:
pool.terminate()
The never stopping stacktrace on console is below
...
Exception: init failed
Process ForkPoolWorker-18:
Traceback (most recent call last):
File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/pool.py", line 103, in worker
initializer(*initargs)
File "hoge.py", line 5, in initializer
raise Exception("init failed")
Exception: init failed
...
My workaround is suppressing initializer error and return at the beginning of the main process by using global flag like below.
But I would like to learn better one.
def initializer():
try:
raise Exception("init failed")
except:
global failed
failed = True
def do_something(args):
global failed
if failed:
# skip when initializer failed
return
# main process
After navigating through the implementation of multiprocessing using PyCharm, I'm convinced that there is no better solution, because Pool started a thread to _maintain_pool() by _repopulate_pool() if any worker process exists--either accidentally or failed to initialize.
Check this out: Lib/multiprocessing/pool.py line 244
I just came across the same woe. My first solution was to catch the exception and raise it in the worker function (see below). But on second thought it really means that initializer support of multiprocessing.Pool is broken and sould not be used. So I now prefer to do the initialization stuff directly in the worker.
from multiprocessing import Pool
import contextlib, sys
_already_inited = False
def initializer():
global _already_inited
if _already_inited:
return
_already_inited = True
raise Exception("init failed")
def do_something(args):
initializer()
# main process
pool = Pool(1)
with contextlib.closing(pool):
pool.map_async(do_something, [1]).get(100)
Both the code and the stacktrace are simpler.
Off course all your worker function need to call initializer().
My initial solution was to defer the exception to the worker function.
from multiprocessing import Pool
import contextlib, sys
failed = None
def initializer():
try:
raise Exception("init failed")
except:
global failed
failed = sys.exc_info()[1]
def do_something(args):
global failed
if failed is not None:
raise RuntimeError(failed) from failed
# main process
pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
pool.map_async(do_something, [1]).get(100)
That way the caller still gets access to the exception.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/tmp/try.py", line 15, in do_something
raise RuntimeError(failed)
RuntimeError: init failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/try.py", line 20, in <module>
pool.map_async(do_something, [1]).get(100)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
RuntimeError: init failed
(venv) kmkaplan#dev1:~/src/options$ python3 /tmp/try.py
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/tmp/try.py", line 7, in initializer
raise Exception("init failed")
Exception: init failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/tmp/try.py", line 15, in do_something
raise RuntimeError(failed) from failed
RuntimeError: init failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/try.py", line 20, in <module>
pool.map_async(do_something, [1]).get(100)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
RuntimeError: init failed
I'm using the Pool function of the multiprocessing module in order to run the same code in parallel on different data.
It turns out that on some data my code raises an exception, but the precise line in which this happens is not given:
Traceback (most recent call last):
File "my_wrapper_script.py", line 366, in <module>
main()
File "my_wrapper_script.py", line 343, in main
results = pool.map(process_function, folders)
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 148, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 422, in get
raise self._value
KeyError: 'some_key'
I am aware of multiprocessing.log_to_stderr() , but it seems that it is useful when concurrency issues arise, which is not my case.
Any ideas?
If you're using a new enough version of Python, you'll actually see the real exception get printed prior to that one. For example, here's a sample that fails:
import multiprocessing
def inner():
raise Exception("FAIL")
def f():
print("HI")
inner()
p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()
Here's the exception when running this with python 3.4:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "test.py", line 9, in f
inner()
File "test.py", line 4, in inner
raise Exception("FAIL")
Exception: FAIL
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test.py", line 13, in <module>
p.apply(f)
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 253, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
Exception: FAIL
If using a newer version isn't an option, the easiest thing to do is to wrap your worker function in a try/except block that will print the exception prior to re-raising it:
import multiprocessing
import traceback
def inner():
raise Exception("FAIL")
def f():
try:
print("HI")
inner()
except Exception:
print("Exception in worker:")
traceback.print_exc()
raise
p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()
Output:
HI
Exception in worker:
Traceback (most recent call last):
File "test.py", line 11, in f
inner()
File "test.py", line 5, in inner
raise Exception("FAIL")
Exception: FAIL
Traceback (most recent call last):
File "test.py", line 18, in <module>
p.apply(f)
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 244, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
Exception: FAIL
You need to implement your own try/except block in the worker. Depending on how you want to organize your code, you could log to stderr as you mention above, log to some other place like a file, return some sort of error code or even tag the exception with the current traceback and re-raise. Here's an example of the last technique:
import traceback
import multiprocessing as mp
class MyError(Exception):
pass
def worker():
try:
# your real code here
raise MyError("boom")
except Exception, e:
e.traceback = traceback.format_exc()
raise
def main():
pool = mp.Pool()
try:
print "run worker"
result = pool.apply_async(worker)
result.get()
# handle exceptions you expect
except MyError, e:
print e.traceback
# re-raise the rest
except Exception, e:
print e.traceback
raise
if __name__=="__main__":
main()
It returns
run worker
Traceback (most recent call last):
File "doit.py", line 10, in worker
raise MyError("boom")
MyError: boom
I have a traceback print and want to customize the last part of it.
What: The error occurred in another process and traceback lies there (as is the case in multiprocessing).
Problem: I want to have the full traceback and error report.
Similar to this code:
>>> def f():
g()
>>> def g():
raise Exception, Exception(), None ## my traceback here
>>> f()
Traceback (most recent call last):
File "<pyshell#14>", line 1, in <module>
f()
File "<pyshell#8>", line 2, in f
g()
File "<pyshell#11>", line 2, in g
raise Exception, Exception(), None ## my traceback starts here
my traceback appears here
my traceback appears here
Exception
Impossible "Solutions": subclass and mock-object
>>> from types import *
>>> class CostomTB(TracebackType):
pass
Traceback (most recent call last):
File "<pyshell#125>", line 1, in <module>
class CostomTB(TracebackType):
TypeError: Error when calling the metaclass bases
type 'traceback' is not an acceptable base type
>>> class CostomTB(object):
pass
>>> try: zzzzzzzzz
except NameError:
import sys
ty, err, tb = sys.exc_info()
raise ty, err, CostomTB()
Traceback (most recent call last):
File "<pyshell#133>", line 5, in <module>
raise ty, err, CostomTB()
TypeError: raise: arg 3 must be a traceback or None
I am using python 2.7.
I guess you want the full traceback stack. See this which is having very good examples python logging module.
If some confusion comes See the logging documentation.
You mentioned a separate process: if your problem is to capture the traceback in process A and show it in process B, as if the exception was actually raised in the latter, then I'm afraid there is no clean way to do it.
I would suggest to serialize the traceback in process A, send it to process B and from there raise a new exception that includes the former in its description. The result is a somewhat longer output, but it carries information about both processes stacks.
In the following example there aren't really two separate processes, but I hope it makes my point clearer:
import traceback, StringI
def functionInProcessA():
raise Exception('Something happened in A')
class RemoteException(Exception):
def __init__(self, tb):
Exception.__init__(self, "Remote traceback:\n\n%s" % tb)
def controlProcessB():
try:
functionInProcessA()
except:
fd = StringIO.StringIO()
traceback.print_exc(file=fd)
tb = fd.getvalue()
raise RemoteException(tb)
if __name__ == '__main__':
controlProcessB()
Output:
Traceback (most recent call last):
File "a.py", line 20, in <module>
controlProcessB()
File "a.py", line 17, in controlProcessB
raise RemoteException(tb)
__main__.RemoteException: Remote traceback:
Traceback (most recent call last):
File "a.py", line 12, in controlProcessB
functionInProcessA()
File "a.py", line 4, in functionInProcessA
raise Exception('Something happened in A')
Exception: Something happened in A