I'm trying to catch the exceptions I raise in a function called callback.
Here's a simplification of the code.
try:
logger.info("This message gets logged!!!")
with Pool() as executor:
results = executor.map(callback, made_up_list)
except Exception as e:
pass
This doesn't seem to work, since any exception raised by callback shows on the terminal.
One interesting thing is if we use the debug mode in an IDE (my case vscode), and put a breakpoint on the except line, we never get there, since we remain stuck due to the raised exception. However, if I do CTRL+c, I stop the process, and then I pass to the except line.
It may be worth knowing the callback itself already uses multithreading when doing some computations which will result in raising an exception.
If I change the line order a bit, to:
with Pool() as executor::
logger.info("This message gets logged!!!")
try:
results = executor.map(callback, made_up_list)
except Exception as e:
pass
I get a similar undesired result of not catching the exception.
I would suggest you to wrap your callback inside decorator to catch exception and then pass your decorated function inside executor like this
import logging
from multiprocessing import Pool
from functools import update_wrapper
class my_decorator(object):
def __init__(self, target):
self.target = target
try:
update_wrapper(self, target)
except:
pass
def __call__(self, args):
self.target(args)
def func(a):
print(int(a))
callback = my_decorator(func)
if __name__ == "__main__":
with Pool(4) as executor:
logging.info("This message gets logged!!!")
results = executor.map(callback,[1,2,3, '4', 'a'])
I am not sure why you are not catching an exception (assuming one occurs in your worker function, callback. Perhaps this is due to some peculiarity of your IDE. The following code when run from a command line (not an IDE) works as expected:
def callback(x):
if x == 2:
raise ValueError(f'Invalid value {x}')
return x ** 2
if __name__ == '__main__':
from multiprocessing import Pool
made_up_list = [1, 2, 3]
with Pool() as executor:
try:
results = executor.map(callback, made_up_list)
print(results)
except Exception as e:
print('Got exception:', e)
Prints:
Got exception: Invalid value 2
You might see if the following works, which has the additional benefit of getting a results list for those tasks that have completed normally, i.e. if you want to catch individual exceptions as well as the results of successful completions that occur in your worker function callback in your main process, then you could use the Pool.imap method instead of Pool.map:
def callback(x):
if x == 2:
raise ValueError(f'Invalid value {x}')
return x ** 2
if __name__ == '__main__':
from multiprocessing import Pool
made_up_list = [1, 2, 3]
with Pool() as executor:
results = []
it = executor.imap(callback, made_up_list)
while True:
try:
result = next(it)
except StopIteration:
break
except Exception as e:
result = e
print('Got exception:', e)
results.append(result) # Include Exception instances
print(results)
Prints:
Got exception: Invalid value 2
[1, ValueError('Invalid value 2'), 9]
If this makes no difference, then you need to post a minimal, reproducible example.
I have the following code where I am running a check periodically in the background for when 5 is hit by a random int generator.
Where I'm getting stuck is trying to:
When it hits: Pass up the exception from the thread running random_error when it hits 5 so then it can be caught by the test() function and test() ends.
When it doesn't hit: The thread running random_error stops trying and ends when test() ends.
from time import sleep
import threading
import random
def random_error():
while True:
sleep(1)
x = random.randint(1,30)
print(x)
if x == 5:
raise ValueError("5 Hit")
def test():
try:
countdown_thread = threading.Thread(target = random_error)
countdown_thread.start()
y = 0
while True:
sleep(1)
print('still going')
y += 1
if y == 10:
print('didnt hit in time')
break
except:
print("Error caught!")
test()
random_error is a completely different thread from test. There is no way to just pass the error.
That said, you still have many options. test can look to see if countdown_thread.is_alive(). You can have random_thread do something like set a variable or put a value into a queue before it throws the exception and have test check periodically for that.
But there is no way that test will get the exception thrown by a different thread.
Here is what I want to do:
def bfunc():
try:
do_someting
except Exception as e:
return
def afunc():
bfunc()
xxx
def cfunc():
xxx
def main():
afunc()
cfunc()
in bfunc(),I catch the exception.Now in my main(), I want to stop the afunc() execution when an exception occurs but proceed to execute cfunc().
How can I do this or is there any other way to catch the exception without too many nested try statements?
Tx
Because bfunc() is a function, therefore, to stop the execution of bfunc you can simply use return to stop bfunc. This won't affect cfunc because return only affect bfunc.
def bfunc():
try:
# do_someting
except Exception as e:
return # Exit the bfunc() immediately
You can use below code to see whether print will work or not
def bfunc():
try:
raise IndexError
except Exception as e:
return
def main():
bfunc()
print("Hello world")
if __name__ == "__main__":
main()
Just move the try exception block to afunc. It should give the effect you want.
def bfunc():
do_someting
def afunc():
try:
bfunc()
except Exception as e:
return
xxx #you can move it to try block in order to catch exceptions here too, but I don't know if it's what you like to do
I want to create a timer so that a particular line of code only has 60 seconds to process.
def foo():
for i in xrange(10):
output = action(arg1, arg2)
print i
If this action takes more than 60 seconds, then don't print i, but just continue to next one. How can I do this?
action is not a loop, so timer will not work. I also tried signal but that also does not work because I have these nested inside loops and I get an error saying: ValueError: signal only works in main thread // Werkzeug Debugger.
Here's what I have tried with signals so far and the error I get:
fileName.py
class TimeoutException(Exception):
pass
def timeout_handler(signum, frame):
raise TimeoutException
signal.signal(signal.SIGALRM, timeout_handler)
def foo():
signal.alarm(60)
try:
output = foo2(arg1, arg2)
except TimeoutException:
continue
else:
# Reset the alarm
signal.alarm(0)
I do this and instead of raising the exception which should just pass and then eventually continue with my operation, I get this and the script terminates.
line X in timeout_handler
raise TimeoutException
fileName.TimeoutException
import time
for i in xrange(10):
time_start = time.time()
output = action(arg1, arg2)
if time.time() - time_start <= 60:
print i
I am defining a context manager class and I would like to be able to skip the block of code without raising an exception if certain conditions are met during instantiation. For example,
class My_Context(object):
def __init__(self,mode=0):
"""
if mode = 0, proceed as normal
if mode = 1, do not execute block
"""
self.mode=mode
def __enter__(self):
if self.mode==1:
print 'Exiting...'
CODE TO EXIT PREMATURELY
def __exit__(self, type, value, traceback):
print 'Exiting...'
with My_Context(mode=1):
print 'Executing block of codes...'
According to PEP-343, a with statement translates from:
with EXPR as VAR:
BLOCK
to:
mgr = (EXPR)
exit = type(mgr).__exit__ # Not calling it yet
value = type(mgr).__enter__(mgr)
exc = True
try:
try:
VAR = value # Only if "as VAR" is present
BLOCK
except:
# The exceptional case is handled here
exc = False
if not exit(mgr, *sys.exc_info()):
raise
# The exception is swallowed if exit() returns true
finally:
# The normal and non-local-goto cases are handled here
if exc:
exit(mgr, None, None, None)
As you can see, there is nothing obvious you can do from the call to the __enter__() method of the context manager that can skip the body ("BLOCK") of the with statement.
People have done Python-implementation-specific things, such as manipulating the call stack inside of the __enter__(), in projects such as withhacks. I recall Alex Martelli posting a very interesting with-hack on stackoverflow a year or two back (don't recall enough of the post off-hand to search and find it).
But the simple answer to your question / problem is that you cannot do what you're asking, skipping the body of the with statement, without resorting to so-called "deep magic" (which is not necessarily portable between python implementations). With deep magic, you might be able to do it, but I recommend only doing such things as an exercise in seeing how it might be done, never in "production code".
If you want an ad-hoc solution that uses the ideas from withhacks (specifically from AnonymousBlocksInPython), this will work:
import sys
import inspect
class My_Context(object):
def __init__(self,mode=0):
"""
if mode = 0, proceed as normal
if mode = 1, do not execute block
"""
self.mode=mode
def __enter__(self):
if self.mode==1:
print 'Met block-skipping criterion ...'
# Do some magic
sys.settrace(lambda *args, **keys: None)
frame = inspect.currentframe(1)
frame.f_trace = self.trace
def trace(self, frame, event, arg):
raise
def __exit__(self, type, value, traceback):
print 'Exiting context ...'
return True
Compare the following:
with My_Context(mode=1):
print 'Executing block of code ...'
with
with My_Context(mode=0):
print 'Executing block of code ... '
A python 3 update to the hack mentioned by other answers from
withhacks (specifically from AnonymousBlocksInPython):
class SkipWithBlock(Exception):
pass
class SkipContextManager:
def __init__(self, skip):
self.skip = skip
def __enter__(self):
if self.skip:
sys.settrace(lambda *args, **keys: None)
frame = sys._getframe(1)
frame.f_trace = self.trace
def trace(self, frame, event, arg):
raise SkipWithBlock()
def __exit__(self, type, value, traceback):
if type is None:
return # No exception
if issubclass(type, SkipWithBlock):
return True # Suppress special SkipWithBlock exception
with SkipContextManager(skip=True):
print('In the with block') # Won't be called
print('Out of the with block')
As mentioned before by joe, this is a hack that should be avoided:
The method trace() is called when a new local scope is entered, i.e. right when the code in your with block begins. When an exception is raised here it gets caught by exit(). That's how this hack works. I should add that this is very much a hack and should not be relied upon. The magical sys.settrace() is not actually a part of the language definition, it just happens to be in CPython. Also, debuggers rely on sys.settrace() to do their job, so using it yourself interferes with that. There are many reasons why you shouldn't use this code. Just FYI.
Based on #Peter's answer, here's a version that uses no string manipulations but should work the same way otherwise:
from contextlib import contextmanager
#contextmanager
def skippable_context(skip):
skip_error = ValueError("Skipping Context Exception")
prev_entered = getattr(skippable_context, "entered", False)
skippable_context.entered = False
def command():
skippable_context.entered = True
if skip:
raise skip_error
try:
yield command
except ValueError as err:
if err != skip_error:
raise
finally:
assert skippable_context.entered, "Need to call returned command at least once."
skippable_context.entered = prev_entered
print("=== Running with skip disabled ===")
with skippable_context(skip=False) as command:
command()
print("Entering this block")
print("... Done")
print("=== Running with skip enabled ===")
with skippable_context(skip=True) as command:
command()
raise NotImplementedError("... But this will never be printed")
print("... Done")
What you're trying to do isn't possible, unfortunately. If __enter__ raises an exception, that exception is raised at the with statement (__exit__ isn't called). If it doesn't raise an exception, then the return value is fed to the block and the block executes.
Closest thing I could think of is a flag checked explicitly by the block:
class Break(Exception):
pass
class MyContext(object):
def __init__(self,mode=0):
"""
if mode = 0, proceed as normal
if mode = 1, do not execute block
"""
self.mode=mode
def __enter__(self):
if self.mode==1:
print 'Exiting...'
return self.mode
def __exit__(self, type, value, traceback):
if type is None:
print 'Normal exit...'
return # no exception
if issubclass(type, Break):
return True # suppress exception
print 'Exception exit...'
with MyContext(mode=1) as skip:
if skip: raise Break()
print 'Executing block of codes...'
This also lets you raise Break() in the middle of a with block to simulate a normal break statement.
Context managers are not the right construct for this. You're asking for the body to be executed n times, in this case zero or one. If you look at the general case, n where n >= 0, you end up with a for loop:
def do_squares(n):
for i in range(n):
yield i ** 2
for x in do_squares(3):
print('square: ', x)
for x in do_squares(0):
print('this does not print')
In your case, which is more special purpose, and doesn't require binding to the loop variable:
def should_execute(mode=0):
if mode == 0:
yield
for _ in should_execute(0):
print('this prints')
for _ in should_execute(1):
print('this does not')
Another slightly hacky option makes use of exec. This is handy because it can be modified to do arbitrary things (e.g. memoization of context-blocks):
from contextlib import contextmanager
#contextmanager
def skippable_context_exec(skip):
SKIP_STRING = 'Skipping Context Exception'
old_value = skippable_context_exec.is_execed if hasattr(skippable_context_exec, 'is_execed') else False
skippable_context_exec.is_execed=False
command = "skippable_context_exec.is_execed=True; "+("raise ValueError('{}')".format(SKIP_STRING) if skip else '')
try:
yield command
except ValueError as err:
if SKIP_STRING not in str(err):
raise
finally:
assert skippable_context_exec.is_execed, "You never called exec in your context block."
skippable_context_exec.is_execed = old_value
print('=== Running with skip disabled ===')
with skippable_context_exec(skip=False) as command:
exec(command)
print('Entering this block')
print('... Done')
print('=== Running with skip enabled ===')
with skippable_context_exec(skip=True) as command:
exec(command)
print('... But this will never be printed')
print('... Done')
Would be nice to have something that gets rid of the exec without weird side effects, so if you can think of a way I'm all ears. The current lead answer to this question appears to do that but has some issues.