I'm trying to generate a custom exception message but get the below error -
import time
try:
start_time = time.time()
1/0
except Exception as ex:
elapsed_time = (time.time() - start_time)/60
e = "elapsed time(in mins) - {0}".format(elapsed_time)
print(type(ex))
raise ex(e)
Error:-
1/0
ZeroDivisionError: division by zero
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/lakshmananp2/PycharmProjects/Scratch/exception.py", line 9, in <module>
raise ex(e)
TypeError: 'ZeroDivisionError' object is not callable
ex is an instance of ZeroDivisionError, not the type ZeroDivisionError itself.
raise type(ex)(e)
You've close, but you're calling the instance instead of the type. We can build a new instance by grabbing the type of the exception using type builtin:
import time
try:
start_time = time.time()
1/0
except Exception as ex:
elapsed_time = (time.time() - start_time)/60
e = "elapsed time(in mins) - {0}".format(elapsed_time)
error_constructor = type(ex)
raise error_constructor(e)
If you want to retain the original traceback you can instead do:
import time
try:
start_time = time.time()
1/0
except Exception as ex:
elapsed_time = (time.time() - start_time)/60
e = "elapsed time(in mins) - {0}".format(elapsed_time)
ex.__init__(e)
raise # re-raises ex with the original line number
Related
Let's say I have a special exception that does some neat and wonderful things - Solving world hunger, good will toward men, logging, etc:
class SpecialException(Exception):
# Does something really neat here.
pass
Now let's say that an exception may be raised, but we don't know what type of exception we'll encounter:
def crashAndBurn():
try:
import random
choice = random.choice([1,2])
if choice == 1:
int('asdf') # Can't cast string to int: ValueError.
else:
x # Variable `x` doesn't exist: NameError.
except Exception as e:
# Code to wrap `e` into `SpecialException` class
raise e
When that unknown type of exception is raised, we want to catch it, wrap it in our SpecialException class, and raise it so it can be caught either by the original type of exception thrown, or by catching SpecialException:
try:
crashAndBurn()
except ValueError as e:
print('This time it was a ValueError, and we want to handle this differently for some reason')
except SpecialException as e:
print('Handle this more generically')
Any recommendations on how to (reasonably) solve this?
In summary, we need:
Ability to catch exception originally raised, and ability to catch SpecialException
Retain traceback for easy debugging
Contain the original exception's error message
What I've Tried:
Tried using raise SpecialException from e. While we are able to view the error message and traceback from the originally raised exception, we are no longer able to catch it by the type of exception originally thrown... Eg: We can catch SpecialException, but can't catch ValueError:
def crashAndBurn():
try:
int('asdf') # ValueError
except Exception as e:
raise SpecialException('Special Exception Encountered').with_traceback(e.__traceback__) from e
try:
crashAndBurn()
except ValueError as e:
print('This will never be encountered')
except SpecialException as e:
print('This will be encountered, when we wanted it to be handled above')
The closest we've gotten technically fulfills our needs, however:
While the exception raised can be caught as either SpecialException or ValueError, it's actually raised as another, one-time use class: DynamicSpecialException
It's really gross and seemingly very un-pythonic
def crashAndBurn():
try:
int('asdf') # ValueError
except Exception as e:
class DynamicSpecialException(SpecialException, e.__class__):
pass # I feel dirty
raise DynamicSpecialException('Special Exception Encountered').with_traceback(e.__traceback__)
try:
crashAndBurn()
except ValueError as e:
print('Caught as a ValueError!')
try:
crashAndBurn()
except SpecialException as e:
print('Caught as SpecialException!')
What I was really expecting to find was something similar to raise e.extend(SpecialException) or raise SpecialException.from(e) - rather than this rabbit hole I've seemingly wiggled my way down today! :)
Here's a bit of a stab at it. It seems to do most of what you want, except that it appends the stack trace of the specialfactory handling.
The thing I learned is that you can't swap the exception class, e.__class__ = <dynamic class>, you have to create a new one and raise it.
import pdb
from traceback import print_exc as xp
import sys
def cpdb():
""" put `pdb` on commmand line to halt execution in debugger """
return "pdb" in sys.argv
class SpecialException(Exception):
def solve_world_hunger(self):
print(f"eat more 🦄")
def specialfactory(e):
""" creates a class dynamically and keeps the original as a base"""
cls_ = type("SpecialException", (SpecialException, e.__class__),{})
e2 = cls_(str(e))
e2.ori = e
e2.__dict__.update(**e.__dict__)
# 👇 you can try different flavors to see what changes:
# basically, I haven't found a way to suppress `The above exception was the direct cause of the following exception:`
# see also https://stackoverflow.com/questions/33809864/disable-exception-chaining-in-python-3
# return e2
# raise e2. raise specialfactory(e).with_traceback(e.__traceback__) from e
# raise e2 from e
raise e2.with_traceback(e.__traceback__) from e
def crashAndBurn(no_special=False, custom_message=None):
try:
if custom_message:
exc = ValueError(custom_message)
exc.foo = "bar"
raise exc
int('asdf') # ValueError
except Exception as e:
if no_special:
#just to investigate what things look like for a plain ValueError
raise
# raise specialfactory(e).with_traceback(e.__traceback__) from e
raise specialfactory(e) from e
#################################################################
# check what a regular unchanged ValueError looks like
#################################################################
try:
print("\n\n\n🔬regular ValueError, unchanged")
crashAndBurn(no_special=1)
except ValueError as e:
if cpdb(): pdb.set_trace()
print(f' plain ValueError: {e}')
xp()
except SpecialException as e:
if cpdb(): pdb.set_trace()
print(f' Caught as a SpecialException!: {e}')
xp()
#################################################################
# catch a Special as a ValueError
#################################################################
try:
print("\n\n\n🔬ValueError ")
crashAndBurn()
except ValueError as e:
if cpdb(): pdb.set_trace()
print(f' Caught as a ValueError! {e}')
xp()
except SpecialException as e:
if cpdb(): pdb.set_trace()
print(f' Caught as a SpecialException! {e}')
xp()
#################################################################
# catch a Special
#################################################################
try:
print("\n\n\n🔬SpecialException handling")
crashAndBurn()
except SpecialException as e:
if cpdb(): pdb.set_trace()
print(f' Caught as a SpecialException! {e} {e.solve_world_hunger()}')
xp()
except ValueError as e:
if cpdb(): pdb.set_trace()
print(f' Caught as a ValueError! {e}')
xp()
#################################################################
# custom variables are still available
#################################################################
try:
print("\n\n\n🔬ValueError with custom_message/content ")
crashAndBurn(custom_message="my custom_message")
except SpecialException as e:
if cpdb(): pdb.set_trace()
print(f' Caught as a SpecialException! {e} {e.foo=} {e.solve_world_hunger()}')
xp()
except ValueError as e:
if cpdb(): pdb.set_trace()
print(f' Caught as a ValueError! {e}')
xp()
output:
Traceback (most recent call last):
File "test_183.py", line 57, in <module>
crashAndBurn(no_special=1)
File "test_183.py", line 41, in crashAndBurn
int('asdf') # ValueError
ValueError: invalid literal for int() with base 10: 'asdf'
Traceback (most recent call last):
File "test_183.py", line 41, in crashAndBurn
int('asdf') # ValueError
ValueError: invalid literal for int() with base 10: 'asdf'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test_183.py", line 74, in <module>
crashAndBurn()
File "test_183.py", line 47, in crashAndBurn
raise specialfactory(e) from e
File "test_183.py", line 30, in specialfactory
raise e2.with_traceback(e.__traceback__) from e
File "test_183.py", line 41, in crashAndBurn
int('asdf') # ValueError
SpecialException: invalid literal for int() with base 10: 'asdf'
Traceback (most recent call last):
File "test_183.py", line 41, in crashAndBurn
int('asdf') # ValueError
ValueError: invalid literal for int() with base 10: 'asdf'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test_183.py", line 92, in <module>
crashAndBurn()
File "test_183.py", line 47, in crashAndBurn
raise specialfactory(e) from e
File "test_183.py", line 30, in specialfactory
raise e2.with_traceback(e.__traceback__) from e
File "test_183.py", line 41, in crashAndBurn
int('asdf') # ValueError
SpecialException: invalid literal for int() with base 10: 'asdf'
Traceback (most recent call last):
File "test_183.py", line 39, in crashAndBurn
raise exc
ValueError: my custom_message
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test_183.py", line 108, in <module>
crashAndBurn(custom_message="my custom_message")
File "test_183.py", line 47, in crashAndBurn
raise specialfactory(e) from e
File "test_183.py", line 30, in specialfactory
raise e2.with_traceback(e.__traceback__) from e
File "test_183.py", line 39, in crashAndBurn
raise exc
SpecialException: my custom_message
🔬regular ValueError, unchanged
plain ValueError: invalid literal for int() with base 10: 'asdf'
🔬ValueError
Caught as a ValueError! invalid literal for int() with base 10: 'asdf'
🔬SpecialException handling
eat more 🦄
Caught as a SpecialException! invalid literal for int() with base 10: 'asdf' None
🔬ValueError with custom_message/content
eat more 🦄
Caught as a SpecialException! my custom_message e.foo='bar' None
I got a Flask app calling different modules. One of them eventually raises a NoPrice exception but the except NoPrice doesn't catch it as a NoPrice but except Exceptioncatches it...
class NoPrice(Exception):
pass
somewhere in a module
raise NoPrice('error')
in my flask app
try:
raise NoPrice('main')
except NoPrice as e:
print('main')
except Exception as e:
print('main2')
try:
resp = alicia.create_predictions(select)
except NoPrice as e:
print('main3')
except Exception as e:
print('main4')
print(repr(e))
print(e)
print(traceback.format_exc())
The output of this is
main
main4
NoPrice('No price found for material_reference 0148054b-e681-4fb6-9722-946f3cfa8529 the weight 1.7471266917293233', 'occurred at index 0')
('No price found for material_reference 0148054b-e681-4fb6-9722-946f3cfa8529 the weight 1.7471266917293233', 'occurred at index 0')
Traceback (most recent call last):
File "c/main.py", line 71, in create_predictions_api
resp = alicia.create_predictions(select)
File "/absolutepath/app/alicia.py", line 703, in create_predictions
self.set_machines_and_format()
File "/absolutepath/app/alicia.py", line 574, in set_machines_and_format
x["is_screenprinting_option"]), axis = 1)
File "/absolutepath/venv/lib/python3.7/site-packages/pandas/core/frame.py", line 6913, in apply
return op.get_result()
File "/absolutepath/venv/lib/python3.7/site-packages/pandas/core/apply.py", line 186, in get_result
return self.apply_standard()
File "/absolutepath/venv/lib/python3.7/site-packages/pandas/core/apply.py", line 292, in apply_standard
self.apply_series_generator()
File "/absolutepath/venv/lib/python3.7/site-packages/pandas/core/apply.py", line 321, in apply_series_generator
results[i] = self.f(v)
File "/absolutepath/app/alicia.py", line 574, in <lambda>
x["is_screenprinting_option"]), axis = 1)
File "/absolutepath/app/alicia.py", line 359, in predict_price_relying_on_format
output = Jerome(self.product, self.hipe_client_config).run(self.quote_input)
File "/absolutepath/app/jerome/jerome.py", line 235, in run
pliant = pliant_obj.price(quote_input, self.material_reference)
File "/absolutepath/app/jerome/options_construction/pliant.py", line 66, in price
quote_input['matter_margin_percentage'])
File "/absolutepath/app/jerome/options_construction/pliant.py", line 52, in pliant
raise NoPrice(f"No price found for material_reference {material_reference.id} the weight {x1}")
exceptions.NoPrice: ('No price found for material_reference 0148054b-e681-4fb6-9722-946f3cfa8529 the weight 1.7471266917293233', 'occurred at index 0')
Why doesn't the except NoPrice catch my exception ?
Shouldn't the ouput be
main
main3
My goal is ultimately to handle only my NoPrice exception
my_function must not be raising the NoPrice exception, and therefore it isn't caught in the 'main 3' section.
I'm trying to create a python decorator which takes a function with args and kwargs, executes it in a new process, shuts it down and returns whatever the function returned, including raising the same exception, if any.
For now, my decorator handles functions okay, if they raise no exceptions, but fails to provide the traceback. How do I pass it back to the parent process?
from functools import wraps
from multiprocessing import Process, Queue
import sys
def process_wrapper(func):
#wraps(func)
def wrapper(*args, **kwargs):
# queue for communicating between parent and child processes
q = Queue()
def func_to_q(_q: Queue, *_args, **_kwargs):
# do the same as func, but put result into the queue. Also put
# there an exception if any.
try:
_res = func(*_args, **_kwargs)
_q.put(_res)
except:
_q.put(sys.exc_info())
# start another process and wait for it to join
p = Process(target=func_to_q, args=(q, )+args, kwargs=kwargs)
p.start()
p.join()
# get result from the queue and return it, or raise if it's an exception
res = q.get(False)
if isinstance(res, tuple) and isinstance(res[0], Exception):
raise res[1].with_traceback(res[2])
else:
return res
return wrapper
if __name__ == '__main__':
#process_wrapper
def ok():
return 'ok'
#process_wrapper
def trouble():
def inside():
raise UserWarning
inside()
print(ok())
print(trouble())
I expect result to be something like:
ok
Traceback (most recent call last):
File "/temp.py", line 47, in <module>
print(trouble())
File "/temp.py", line 44, in trouble
inside()
File "/temp.py", line 43, in inside
raise UserWarning
UserWarning
Process finished with exit code 1
But it seems like the child process cannot put stacktrace into the queue and I get the following:
ok
Traceback (most recent call last):
File "/temp.py", line 47, in <module>
print(trouble())
File "/temp.py", line 26, in wrapper
res = q.get(False)
File "/usr/lib/python3.6/multiprocessing/queues.py", line 107, in get
raise Empty
queue.Empty
Process finished with exit code 1
Also, if the child puts into the queue only the exception itself _q.put(sys.exc_info()[1]), parent gets it from there and raises but with new stacktrace (note missing call to inside()):
ok
Traceback (most recent call last):
File "/temp.py", line 47, in <module>
print(trouble())
File "/temp.py", line 28, in wrapper
raise res
UserWarning
Process finished with exit code 1
Take a look at multiprocessing.pool.py and the stringification-hack for sending Exceptions to the parent. You can use multiprocessing.pool.ExceptionWithTraceback from there.
That's just enough code for demonstrating the basic principle:
from multiprocessing import Process, Queue
from multiprocessing.pool import ExceptionWithTraceback
def worker(outqueue):
try:
result = (True, 1 / 0) # will raise ZeroDivisionError
except Exception as e:
e = ExceptionWithTraceback(e, e.__traceback__)
result = (False, e)
outqueue.put(result)
if __name__ == '__main__':
q = Queue()
p = Process(target=worker, args=(q,))
p.start()
success, value = q.get()
p.join()
if success:
print(value)
else:
raise value # raise again
Output:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/...", line 7, in worker
result = (True, 1 / 0) # will raise ZeroDivisionError
ZeroDivisionError: division by zero
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/...", line 23, in <module>
raise value
ZeroDivisionError: division by zero
Process finished with exit code 1
I have the following code inwhich I try to call a function compute_cluster which do some computations and write the results in a txt file (each process write its results in different txt files independently), however, when I run the following code:
def main():
p = Pool(19)
p.map(compute_cluster, [(l, r) for l in range(6, 25) for r in range(1, 4)])
p.close()
if __name__ == "__main__":
main()
it crashes with the following errors:
File "RMSD_calc.py", line 124, in <module>
main()
File "RMSD_calc.py", line 120, in main
p.map(compute_cluster, [(l, r) for l in range(6, 25) for r in range(1, 4)])
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 225, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 522, in get
raise self._value
KeyError: 0
and when I searched online for the meaning of "KeyError: 0" i didn't find anything helpful so any suggestions why this error happens is highly appreciated
KeyError happens in compute_cluster() in a child process and p.map() reraises it for you in the parent:
from multiprocessing import Pool
def f(args):
d = {}
d[0] # <-- raises KeyError
if __name__=="__main__":
p = Pool()
p.map(f, [None])
Output
Traceback (most recent call last):
File "raise-exception-in-child.py", line 9, in <module>
p.map(f, [None])
File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
raise self._value
KeyError: 0
To see the full traceback, catch the exception in the child process:
import logging
from multiprocessing import Pool
def f(args):
d = {}
d[0] # <-- raises KeyError
def f_mp(args):
try:
return f(args)
except Exception:
logging.exception("f(%r) failed" % (args,))
if __name__=="__main__":
p = Pool()
p.map(f_mp, [None])
Output
ERROR:root:f(None) failed
Traceback (most recent call last):
File "raise-exception-in-child.py", line 10, in f_mp
return f(args)
File "raise-exception-in-child.py", line 6, in f
d[0] # <-- raises KeyError
KeyError: 0
It shows that d[0] caused the exception.
This error occurs on pynsq 0.4.2
STACKTRACE
2013-07-30 15:03:43,205 ERROR [ip-10-114-195-89:4150:nsq_msg_handler] failed to handle_message()
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/nsq/Reader.py", line 367, in _data_callback
self._handle_message(conn, message)
File "/usr/local/lib/python2.7/dist-packages/nsq/Reader.py", line 291, in _handle_message
return message.requeue()
File "/usr/local/lib/python2.7/dist-packages/nsq/nsq.py", line 47, in requeue
self.respond(REQ, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/nsq/Reader.py", line 211, in _message_responder
self._requeue(conn, message, time_ms=kwargs.get('time_ms', -1))
File "/usr/local/lib/python2.7/dist-packages/nsq/Reader.py", line 222, in _requeue
return self.finish(conn, message)
AttributeError: 'Reader' object has no attribute 'finish'
CODE
def connect_nsq(self):
r = nsq.Reader(message_handler=self.nsq_msg_handler, lookupd_http_addresses=["127.0.0.1:4161"], topic="test_topic", channel="test_channel", max_in_flight=500)
nsq.run()
# callback
def nsq_msg_handler(self, message):
try:
before_ts = get_utc_now_ts()
json_data = json.loads(message.body)
my_data = json_data["key1"]
my_data = json_data["key2"]
my_data = json_data["key3"]
after_ts = get_utc_now_ts()
delta = after_ts - before_ts
logger.debug("took %f seconds for json_data _id: %s" % (delta, json_data["_id"]))
except Exception as reason:
print reason, traceback.format_exc()
return False
return true
This issue was fixed in the latest release of pynsq which is 0.5.0
Here's the bug report: https://github.com/bitly/pynsq/issues/44