We Keep Coding

Hiding the raise call from traceback

I've been trying to get a specific (and most likely useless) effect on the traceback when I raise an error.
I would like to hide the raise Exception("Message") of the traceback inside my function and instead have my trace indicate the function call that triggered the raise inside the function
def crash(something):
if something > 200:
raise ValueError("Value too large !")
if __name__ == '__main__':
crash(500)
Normally generates :
Traceback (most recent call last):
File "file.py", line 7, in <module>
crash(500)
File "file.py", line 3, in crash
raise ValueError("Value too large !")
ValueError: Value too large !
Instead I would like to have this :
Traceback (most recent call last):
File "file.py", line 7, in <module>
crash(500)
ValueError: Value too large !
I find it clearer for the user trying to use the module because it clearly shows where the problem is as otherwise he could potentially think the module itself is at fault
I tried to find solution that would "remove" the last call from the traceback but it always had "side effects" like showing part of the code trying to remove the last trace call INSIDE the traceback itself, making it even more confusing. Sometimes it would also repeat the the traceback over what I really want.
For example using this :
def crash():
frame = sys._getframe(1)
tb = types.TracebackType(None, frame, frame.f_lasti, frame.f_lineno)
raise ValueError("Wrong value").with_traceback(tb)
Prints this :
Traceback (most recent call last):
File "file.py", line 34, in <module>
crash()
File "file.py", line 9, in crash
raise ValueError("Wrong value").with_traceback(tb)
File "file.py", line 34, in <module>
crash()
ValueError: Wrong value
I also tried using another function to create the trace myself but ended up with a weird behavior
def crash():
raise exception_no_raise(ValueError("Wrong value"))
def exception_no_raise(exc: Exception):
tb = None
depth = 0
while True:
try:
sys._getframe(depth)
depth += 1
except ValueError:
break
# for i in range(depth-1, 1, -1):
# frame = sys._getframe(i)
# tb = types.TracebackType(tb, frame, frame.f_lasti, frame.f_lineno)
# traceback.print_tb(tb)
# print(file=sys.stderr)
frame = sys._getframe(depth-1)
tb = types.TracebackType(tb, frame, frame.f_lasti, frame.f_lineno)
return exc.with_traceback(tb)
Prints :
Traceback (most recent call last):
File "file.py", line 32, in <module>
crash()
File "file.py", line 7, in crash
raise exception_no_raise(ValueError("Wrong value"))
File "file.py", line 32, in <module>
crash()
ValueError: Wrong value
Even tho when you use traceback.print_tb(tb) (before the return) you get the exact trace I want even tho using it to raise the exception doesn't print it :
File "file.py", line 34, in <module>
crash()
The last solution I found was to "strip" the last traceback from the tb.tb_next chain of traces after catching the exception and then re-raise it (but it showed the raise code anyway so ...)
I have difficulties to grasp how exactly traceback works and it seems that it changes drastically from version to version as I found code from Python 2 and 3 and also code the works in Python 3.8 and not before (related to the fact that you couldn't write to the tb.next of a trace before
Thanks for your help and clarifications !

How to catch TimeExhausted exception in web3

wondering if anyone can help me figure out why I am not catching the TimeExhausted exception when using
wait_for_transaction_receipt()
from web3.py
I want the error to be thrown after waiting 45 seconds, I want to catch the error, then go do some other stuff, and wait again. I want to repeat this cycle until the transaction is mined & then exit / break. I think it's a simple syntax issue but not sure -- last time the exception was raised, my code didn't catch it, and instead it lead to an AttributeError
here's my code:
from web3 import Web3
while True:
try:
tx_receipt = w3.eth.wait_for_transaction_receipt(tx_hash, timeout=45)
print(f"{str(dt.utcnow())[:-7]} | Transaction receipt: {tx_receipt}")
break
except Web3.TimeExhausted as e:
print(f"{str(dt.utcnow())[:-7]} | A Inner exception caught: {e}.")
print(
f"{str(dt.utcnow())[:-7]} | Timed out waiting for transaction. "
+ f"Will update SMA & wait again..."
)
update_sma_data()
pass
Finally here is my error text:
Traceback (most recent call last):
File "/home/flim/.local/lib/python3.8/site-packages/web3/eth.py", line 783, in wait_for_transaction_receipt
_timeout.sleep(poll_latency)
File "/home/flim/.local/lib/python3.8/site-packages/web3/_utils/threads.py", line 89, in sleep
self.check()
File "/home/flim/.local/lib/python3.8/site-packages/web3/_utils/threads.py", line 82, in check
raise self
web3._utils.threads.Timeout: 45 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "money-maker-mike.py", line 301, in sign_and_send_tx
tx_receipt = w3.eth.wait_for_transaction_receipt(tx_hash, timeout=45)
File "/home/flim/.local/lib/python3.8/site-packages/web3/eth.py", line 787, in wait_for_transaction_receipt
raise TimeExhausted(
web3.exceptions.TimeExhausted: Transaction HexBytes('0xd60bc63a96b5b81b98ca39308fc9ab709d2f2ccfd1b03c064db285f954633d34') is not in the chain after 45 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "money-maker-mike.py", line 529, in <module>
main()
File "money-maker-mike.py", line 522, in main
trade()
File "money-maker-mike.py", line 373, in trade
zero_ex_trade(sell_token, buy_token, sell_amount)
File "money-maker-mike.py", line 278, in zero_ex_trade
sign_and_send_tx(tx)
File "money-maker-mike.py", line 306, in sign_and_send_tx
except Web3.TimeExhausted as e:
AttributeError: type object 'Web3' has no attribute 'TimeExhausted'
Thanks in advance :D

from web3.exceptions import TimeExhausted
try:
...
except TimeExhausted as e:
...
Source: web3.py/test_transactions.py at master · eric-s321/web3.py

How to handle initializer error in multiprocessing.Pool?

When initializer throw Error like below, script won't stop.
I would like to abort before starting main process(do not run 'do_something').
from multiprocessing import Pool
import contextlib
def initializer():
raise Exception("init failed")
def do_something(args):
# main process
pass
pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
try:
pool.map_async(do_something, [1]).get(100)
except:
pool.terminate()
The never stopping stacktrace on console is below
...
Exception: init failed
Process ForkPoolWorker-18:
Traceback (most recent call last):
File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/pool.py", line 103, in worker
initializer(*initargs)
File "hoge.py", line 5, in initializer
raise Exception("init failed")
Exception: init failed
...
My workaround is suppressing initializer error and return at the beginning of the main process by using global flag like below.
But I would like to learn better one.
def initializer():
try:
raise Exception("init failed")
except:
global failed
failed = True
def do_something(args):
global failed
if failed:
# skip when initializer failed
return
# main process

After navigating through the implementation of multiprocessing using PyCharm, I'm convinced that there is no better solution, because Pool started a thread to _maintain_pool() by _repopulate_pool() if any worker process exists--either accidentally or failed to initialize.
Check this out: Lib/multiprocessing/pool.py line 244

I just came across the same woe. My first solution was to catch the exception and raise it in the worker function (see below). But on second thought it really means that initializer support of multiprocessing.Pool is broken and sould not be used. So I now prefer to do the initialization stuff directly in the worker.
from multiprocessing import Pool
import contextlib, sys
_already_inited = False
def initializer():
global _already_inited
if _already_inited:
return
_already_inited = True
raise Exception("init failed")
def do_something(args):
initializer()
# main process
pool = Pool(1)
with contextlib.closing(pool):
pool.map_async(do_something, [1]).get(100)
Both the code and the stacktrace are simpler.
Off course all your worker function need to call initializer().
My initial solution was to defer the exception to the worker function.
from multiprocessing import Pool
import contextlib, sys
failed = None
def initializer():
try:
raise Exception("init failed")
except:
global failed
failed = sys.exc_info()[1]
def do_something(args):
global failed
if failed is not None:
raise RuntimeError(failed) from failed
# main process
pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
pool.map_async(do_something, [1]).get(100)
That way the caller still gets access to the exception.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/tmp/try.py", line 15, in do_something
raise RuntimeError(failed)
RuntimeError: init failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/try.py", line 20, in <module>
pool.map_async(do_something, [1]).get(100)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
RuntimeError: init failed
(venv) kmkaplan#dev1:~/src/options$ python3 /tmp/try.py
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/tmp/try.py", line 7, in initializer
raise Exception("init failed")
Exception: init failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/tmp/try.py", line 15, in do_something
raise RuntimeError(failed) from failed
RuntimeError: init failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/try.py", line 20, in <module>
pool.map_async(do_something, [1]).get(100)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
RuntimeError: init failed

debugging errors in python multiprocessing

I'm using the Pool function of the multiprocessing module in order to run the same code in parallel on different data.
It turns out that on some data my code raises an exception, but the precise line in which this happens is not given:
Traceback (most recent call last):
File "my_wrapper_script.py", line 366, in <module>
main()
File "my_wrapper_script.py", line 343, in main
results = pool.map(process_function, folders)
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 148, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 422, in get
raise self._value
KeyError: 'some_key'
I am aware of multiprocessing.log_to_stderr() , but it seems that it is useful when concurrency issues arise, which is not my case.
Any ideas?

If you're using a new enough version of Python, you'll actually see the real exception get printed prior to that one. For example, here's a sample that fails:
import multiprocessing
def inner():
raise Exception("FAIL")
def f():
print("HI")
inner()
p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()
Here's the exception when running this with python 3.4:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "test.py", line 9, in f
inner()
File "test.py", line 4, in inner
raise Exception("FAIL")
Exception: FAIL
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test.py", line 13, in <module>
p.apply(f)
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 253, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
Exception: FAIL
If using a newer version isn't an option, the easiest thing to do is to wrap your worker function in a try/except block that will print the exception prior to re-raising it:
import multiprocessing
import traceback
def inner():
raise Exception("FAIL")
def f():
try:
print("HI")
inner()
except Exception:
print("Exception in worker:")
traceback.print_exc()
raise
p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()
Output:
HI
Exception in worker:
Traceback (most recent call last):
File "test.py", line 11, in f
inner()
File "test.py", line 5, in inner
raise Exception("FAIL")
Exception: FAIL
Traceback (most recent call last):
File "test.py", line 18, in <module>
p.apply(f)
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 244, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
Exception: FAIL

You need to implement your own try/except block in the worker. Depending on how you want to organize your code, you could log to stderr as you mention above, log to some other place like a file, return some sort of error code or even tag the exception with the current traceback and re-raise. Here's an example of the last technique:
import traceback
import multiprocessing as mp
class MyError(Exception):
pass
def worker():
try:
# your real code here
raise MyError("boom")
except Exception, e:
e.traceback = traceback.format_exc()
raise
def main():
pool = mp.Pool()
try:
print "run worker"
result = pool.apply_async(worker)
result.get()
# handle exceptions you expect
except MyError, e:
print e.traceback
# re-raise the rest
except Exception, e:
print e.traceback
raise
if __name__=="__main__":
main()
It returns
run worker
Traceback (most recent call last):
File "doit.py", line 10, in worker
raise MyError("boom")
MyError: boom

SystemExit and NameError issues with exiting

def main():
try:
print "hardfart"
return 0
except:
return 1
if __name__ == '__main__':
exit(main())
Can one kind programmer tell me why this spits out the following error on exit?
Traceback (most recent call last):
File "C:/Apps/exp_exit.py", line 9, in ,module.
exit(main())
File "C:\Apps\python2.7.2\lib\site.py", line 372 in __call__
raise SystemExit(code)
SystemExit: 0
This is causing an error on exit in a project that's set up similarly. For that project, after using gui2exe to compile an exe, when closing the program I get this related error:
Traceback (most recent call last):
File "checkHDBox.py", line 303, in <module>
NameError: name 'exit' is not defined
So if exit is generating this error, how do I exit then? And if I create an exception handler for exit, doesn't that replace the default action that python takes with the exit function?
Thanks.
Edit:
I think this answers my own question.
The traceback here is from IDLE, I think it's a default behavior from other sources I've read.
Traceback (most recent call last):
File "C:/Apps/exp_exit.py", line 9, in ,module.
exit(main())
File "C:\Apps\python2.7.2\lib\site.py", line 372 in __call__
raise SystemExit(code)
SystemExit: 0
The traceback here was fixed by using sys.exit() instead of exit(0)
Traceback (most recent call last):
File "checkHDBox.py", line 303, in <module>
NameError: name 'exit' is not defined

You exit a program by raising SystemExit. This is what exit() does. Someone has incorrectly written an exception handler that catches all exceptions. This is why you only catch the exceptions you can handle.