Suppose we have the two files namely mymanger.py and mysub.py.
mymanager.py
import time
from multiprocessing import Process
import mysub # the process file
def main():
xprocess = Process(
target=mysub.main,
)
xprocess.start()
xprocess.join()
time.sleep(1)
print(f"== Done, errorcode is {xprocess.exitcode} ==")
if __name__ == '__main__':
main()
mysub.py
import sys
def myexception(exc_type, exc_value, exc_traceback):
print("I want this to be printed!")
print("Uncaught exception", exc_type, exc_value, exc_traceback)
def main():
sys.excepthook = myexception # !!!
raise ValueError()
if __name__ == "__main__":
sys.exit()
When executing mymanager.py the resulting output is:
Process Process-1:
Traceback (most recent call last):
File "c:\program files\python\3.9\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
File "c:\program files\python\3.9\lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\lx\mysub.py", line 11, in main
raise ValueError()
ValueError
== Done, errorcode is 1 ==
When the output i expected would be something like:
I want this to be printed!
Uncaught exception <class 'ValueError'> <traceback object at 0x0000027B6F952780>
which is what i get if i execute main from mysub.py without the multiprocessing.Process.
I've checked the underlying cpython (reference) and the problem seems to be that the try-except in the _boostrap function takes precedence over my child processes sys.excepthook but from my understanding, shouldn't the excepthook from the childs process fire first and then trigger the except from the _boostrap?
I need the child process to handle the exception using the sys.excepthook function.
How can i achieve that?
sys.excepthook is invoked when an exception goes uncaught (bubbling all the way out of the running program). But Process objects run their target function in a special bootstrap function (BaseProcess._bootstrap if it matters to you) that intentionally catches all exceptions, prints information about the failing process plus the traceback, then returns an exit code to the caller (a launcher that varies by start method).
When using the fork start method, the caller of _bootstrap then exits the worker with os._exit(code) (a "hard exit" command which bypasses the normal exception handling system, though since your exception was already caught and handled this hardly matters). When using 'spawn', it uses plain sys.exit over os._exit, but AFAICT the SystemExit exception that sys.exit is implemented in terms of is special cased in the interpreter so it doesn't pass through sys.excepthook when uncaught (presumably because it being implemented via exceptions is considered an implementation detail; when you ask to exit the program it's not the same as dying with an unexpected exception).
Summarizing: No matter the start method, there is no possible way for an exception raised by your code to be "unhandled" (for the purposes of reaching sys.excepthook), because multiprocessing handles all exceptions your function can throw on its own. It's theoretically possible to have an excepthook you set in the worker execute for exceptions raised after your target completes if the multiprocessing wrapper code itself raises an exception, but only if you do pathological things like replace the definition of os._exit or sys.exit (and it would only report the horrible things that happened because you replaced them, your own exception was already swallowed by that point, so don't do that).
If you really want to do this, the closest you could get would be to explicitly catch exceptions and manually call your handler. A simple wrapper function would allow this for instance:
def handle_exceptions_with(excepthook, target, /, *args, **kwargs)
try:
target(*args, **kwargs)
except:
excepthook(*sys.exc_info())
raise # Or maybe convert to sys.exit(1) if you don't want multiprocessing to print it again
changing your Process launch to:
xprocess = Process(
target=handle_exceptions_with,
args=(mysub.myexception, mysub.main)
)
Or for one-off use, just be lazy and only rewrite mysub.main as:
def main():
try:
raise ValueError()
except:
myexception(*sys.exc_info())
raise # Or maybe convert to sys.exit(1) if you don't want multiprocessing to print it again
and leave everything else untouched. You could still set your handler in sys.excepthook and/or threading.excepthook() (to handle cases where a thread launched in the worker process might die with an unhandled exception), but it won't apply to the main thread of the worker process (or more precisely, there's no way for an exception to reach it).
Related
Hitting ctrl+c while the dump operation is saving data, the interrupt results in the file being corrupted (i.e. only partially written, so it cannot be loaded again.
Is there a way to make dump, or in general any block of code, uninterruptable?
My current workaround looks something like this:
try:
file = open(path, 'w')
dump(obj, file)
file.close()
except KeyboardInterrupt:
file.close()
file.open(path,'w')
dump(obj, file)
file.close()
raise
It seems silly to restart the operation if it is interrupted, so how can the interrupt be deferred?
The following is a context manager that attaches a signal handler for SIGINT. If the context manager's signal handler is called, the signal is delayed by only passing the signal to the original handler when the context manager exits.
import signal
import logging
class DelayedKeyboardInterrupt:
def __enter__(self):
self.signal_received = False
self.old_handler = signal.signal(signal.SIGINT, self.handler)
def handler(self, sig, frame):
self.signal_received = (sig, frame)
logging.debug('SIGINT received. Delaying KeyboardInterrupt.')
def __exit__(self, type, value, traceback):
signal.signal(signal.SIGINT, self.old_handler)
if self.signal_received:
self.old_handler(*self.signal_received)
with DelayedKeyboardInterrupt():
# stuff here will not be interrupted by SIGINT
critical_code()
Put the function in a thread, and wait for the thread to finish.
Python threads cannot be interrupted except with a special C api.
import time
from threading import Thread
def noInterrupt():
for i in xrange(4):
print i
time.sleep(1)
a = Thread(target=noInterrupt)
a.start()
a.join()
print "done"
0
1
2
3
Traceback (most recent call last):
File "C:\Users\Admin\Desktop\test.py", line 11, in <module>
a.join()
File "C:\Python26\lib\threading.py", line 634, in join
self.__block.wait()
File "C:\Python26\lib\threading.py", line 237, in wait
waiter.acquire()
KeyboardInterrupt
See how the interrupt was deferred until the thread finished?
Here it is adapted to your use:
import time
from threading import Thread
def noInterrupt(path, obj):
try:
file = open(path, 'w')
dump(obj, file)
finally:
file.close()
a = Thread(target=noInterrupt, args=(path,obj))
a.start()
a.join()
Use the signal module to disable SIGINT for the duration of the process:
s = signal.signal(signal.SIGINT, signal.SIG_IGN)
do_important_stuff()
signal.signal(signal.SIGINT, s)
In my opinion using threads for this is an overkill. You can make sure the file is being saved correctly by simply doing it in a loop until a successful write was done:
def saveToFile(obj, filename):
file = open(filename, 'w')
cPickle.dump(obj, file)
file.close()
return True
done = False
while not done:
try:
done = saveToFile(obj, 'file')
except KeyboardInterrupt:
print 'retry'
continue
This question is about blocking the KeyboardInterrupt, but for this situation I find atomic file writing to be cleaner and provide additional protection.
With atomic writes either the entire file gets written correctly, or nothing does. Stackoverflow has a variety of solutions, but personally I like just using atomicwrites library.
After running pip install atomicwrites, just use it like this:
from atomicwrites import atomic_write
with atomic_write(path, overwrite=True) as file:
dump(obj, file)
I've been thinking a lot about the criticisms of the answers to this question, and I believe I have implemented a better solution, which is used like so:
with signal_fence(signal.SIGINT):
file = open(path, 'w')
dump(obj, file)
file.close()
The signal_fence context manager is below, followed by an explanation of its improvements on the previous answers. The docstring of this function documents its interface and guarantees.
import os
import signal
from contextlib import contextmanager
from types import FrameType
from typing import Callable, Iterator, Optional, Tuple
from typing_extensions import assert_never
#contextmanager
def signal_fence(
signum: signal.Signals,
*,
on_deferred_signal: Callable[[int, Optional[FrameType]], None] = None,
) -> Iterator[None]:
"""
A `signal_fence` creates an uninterruptible "fence" around a block of code. The
fence defers a specific signal received inside of the fence until the fence is
destroyed, at which point the original signal handler is called with the deferred
signal. Multiple deferred signals will result in a single call to the original
handler. An optional callback `on_deferred_signal` may be specified which will be
called each time a signal is handled while the fence is active, and can be used
to print a message or record the signal.
A `signal_fence` guarantees the following with regards to exception-safety:
1. If an exception occurs prior to creating the fence (installing a custom signal
handler), the exception will bubble up as normal. The code inside of the fence will
not run.
2. If an exception occurs after creating the fence, including in the fenced code,
the original signal handler will always be restored before the exception bubbles up.
3. If an exception occurs while the fence is calling the original signal handler on
destruction, the original handler may not be called, but the original handler will
be restored. The exception will bubble up and can be detected by calling code.
4. If an exception occurs while the fence is restoring the original signal handler
(exceedingly rare), the original signal handler will be restored regardless.
5. No guarantees about the fence's behavior are made if exceptions occur while
exceptions are being handled.
A `signal_fence` can only be used on the main thread, or else a `ValueError` will
raise when entering the fence.
"""
handled: Optional[Tuple[int, Optional[FrameType]]] = None
def handler(signum: int, frame: Optional[FrameType]) -> None:
nonlocal handled
if handled is None:
handled = (signum, frame)
if on_deferred_signal is not None:
try:
on_deferred_signal(signum, frame)
except:
pass
# https://docs.python.org/3/library/signal.html#signal.getsignal
original_handler = signal.getsignal(signum)
if original_handler is None:
raise TypeError(
"signal_fence cannot be used with signal handlers that were not installed"
" from Python"
)
if isinstance(original_handler, int) and not isinstance(
original_handler, signal.Handlers
):
raise NotImplementedError(
"Your Python interpreter's signal module is using raw integers to"
" represent SIG_IGN and SIG_DFL, which shouldn't be possible!"
)
# N.B. to best guarantee the original handler is restored, the #contextmanager
# decorator is used rather than a class with __enter__/__exit__ methods so
# that the installation of the new handler can be done inside of a try block,
# whereas per [PEP 343](https://www.python.org/dev/peps/pep-0343/) the
# __enter__ call is not guaranteed to have a corresponding __exit__ call if an
# exception interleaves
try:
try:
signal.signal(signum, handler)
yield
finally:
if handled is not None:
if isinstance(original_handler, signal.Handlers):
if original_handler is signal.Handlers.SIG_IGN:
pass
elif original_handler is signal.Handlers.SIG_DFL:
signal.signal(signum, signal.SIG_DFL)
os.kill(os.getpid(), signum)
else:
assert_never(original_handler)
elif callable(original_handler):
original_handler(*handled)
else:
assert_never(original_handler)
signal.signal(signum, original_handler)
except:
signal.signal(signum, original_handler)
raise
First, why not use a thread (accepted answer)?
Running code in a non-daemon thread does guarantee that the thread will be joined on interpreter shutdown, but any exception on the main thread (e.g. KeyboardInterrupt) will not prevent the main thread from continuing to execute.
Consider what would happen if the thread method is using some data that the main thread mutates in a finally block after the KeyboardInterrupt.
Second, to address #benrg's feedback on the most upvoted answer using a context manager:
if an exception is raised after signal is called but before __enter__ returns, the signal will be permanently blocked;
My solution avoids this bug by using a generator context manager with the aid of the #contextmanager decorator. See the full comment in the code above for more details.
this code may call third-party exception handlers in threads other than the main thread, which CPython never does;
I don't think this bug is real. signal.signal is required to be called from the main thread, and raises ValueError otherwise. These context managers can only run on the main thread, and thus will only call third-party exception handlers from the main thread.
if signal returns a non-callable value, __exit__ will crash
My solution handles all possible values of the signal handler and calls them appropriately. Additionally I use assert_never to benefit from exhaustiveness checking in static analyzers.
Do note that signal_fence is designed to handle one interruption on the main thread such as a KeyboardInterrupt. If your user is spamming ctrl+c while the signal handler is being restored, not much can save you. This is unlikely given the relatively few opcodes that need to execute to restore the handler, but it's possible. (For maximum robustness, this solution would need to be rewritten in C)
A generic approach would be to use a context manager that accepts a set of signal to suspend:
import signal
from contextlib import contextmanager
#contextmanager
def suspended_signals(*signals):
"""
Suspends signal handling execution
"""
signal.pthread_sigmask(signal.SIG_BLOCK, set(signals))
try:
yield None
finally:
signal.pthread_sigmask(signal.SIG_UNBLOCK, set(signals))
This is not interruptible (try it), but also maintains a nice interface, so your functions can work the way you expect.
import concurrent.futures
import time
def do_task(func):
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as run:
fut = run.submit(func)
return fut.result()
def task():
print("danger will robinson")
time.sleep(5)
print("all ok")
do_task(task)
and here's an easy way to create an uninterruptible sleep with no signal handling needed:
def uninterruptible_sleep(secs):
fut = concurrent.futures.Future()
with contextlib.suppress(concurrent.futures.TimeoutError):
fut.result(secs)
How can I catch exceptions from a process that was executed using multiprocessing.Process()?
Consider the following python script that executes a simple failFunction() (which immediately throws a runtime error) inside of a child process using mulitprocessing.Process()
#!/usr/bin/env python3
import multiprocessing, time
# this function will be executed in a child process asynchronously
def failFunction():
raise RuntimeError('trust fall, catch me!')
# execute the helloWorld() function in a child process in the background
process = multiprocessing.Process(
target = failFunction,
)
process.start()
# <this is where async stuff would happen>
time.sleep(1)
# try (and fail) to catch the exception
try:
process.join()
except Exception as e:
print( "This won't catch the exception" )
As you can see from the following execution, attempting to wrap the .join() does not actually catch the exception
user#host:~$ python3 example.py
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "example4.py", line 6, in failFunction
raise RuntimeError('trust fall, catch me!')
RuntimeError: trust fall, catch me!
user#host:~$
How can I update the above script to actually catch the exception from the function that was executed inside of a child process using multiprocessing.Process()?
This can be achieved by overloading the run() method in the multiprocessing.Proccess() class with a try..except statement and setting up a Pipe() to get and store any raised exceptions from the child process into an instance field for named exception:
#!/usr/bin/env python3
import multiprocessing, traceback, time
class Process(multiprocessing.Process):
def __init__(self, *args, **kwargs):
multiprocessing.Process.__init__(self, *args, **kwargs)
self._pconn, self._cconn = multiprocessing.Pipe()
self._exception = None
def run(self):
try:
multiprocessing.Process.run(self)
self._cconn.send(None)
except Exception as e:
tb = traceback.format_exc()
self._cconn.send((e, tb))
#raise e # You can still rise this exception if you need to
#property
def exception(self):
if self._pconn.poll():
self._exception = self._pconn.recv()
return self._exception
# this function will be executed in a child process asynchronously
def failFunction():
raise RuntimeError('trust fall, catch me!')
# execute the helloWorld() function in a child process in the background
process = Process(
target = failFunction,
)
process.start()
# <this is where async stuff would happen>
time.sleep(1)
# catch the child process' exception
try:
process.join()
if process.exception:
raise process.exception
except Exception as e:
print( "Exception caught!" )
Example execution:
user#host:~$ python3 example.py
Exception caught!
user#host:~$
Solution taken from this answer:
https://stackoverflow.com/a/33599967/1174102
This solution does not require the target function having to catch its own exceptions.
It may seem like overkill, but you can use class ProcessPoolExecutor in module concurrent.futures to create a process pool of size 1, which is all you that is required for your needs. When you submit a "job" to the executor a Future instance is created representing the state of execution of the process. When you call result() on the Future instance, you block until the process terminates and returns a results (the target function returns). If the target function throws an exception, you can catch it when you call result():
import concurrent.futures
def failFunction():
raise RuntimeError('trust fall, catch me!')
def main():
with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
future = executor.submit(failFunction)
try:
result = future.result()
except Exception as e:
print('exception = ', e)
else:
print('result = ', result)
if __name__ == '__main__':
main()
Prints:
exception = trust fall, catch me!
The bonus of using a process pool is you have a ready-made process already created if you have additional functions you need to invoke in a sub-process.
I want to run a task when my Python program finishes, but only if it finishes successfully. As far as I know, using the atexit module means that my registered function will always be run at program termination, regardless of success. Is there a similar functionality to register a function so that it runs only on successful exit? Alternatively, is there a way for my exit function to detect whether the exit was normal or exceptional?
Here is some code that demonstrates the problem. It will print that the program succeeded, even when it has failed.
import atexit
def myexitfunc():
print "Program succeeded!"
atexit.register(myexitfunc)
raise Exception("Program failed!")
Output:
$ python atexittest.py
Traceback (most recent call last):
File "atexittest.py", line 8, in <module>
raise Exception("Program failed!")
Exception: Program failed!
Program succeeded!
Out of the box, atexit is not quite suited for what you want to do: it's primarily used for resource cleanup at the very last moment, as things are shutting down and exiting. By analogy, it's the "finally" of a try/except, whereas what you want is the "else" of a try/except.
The simplest way I can think of is continuing to create a global flag which you set only when your script "succeeds"... and then have all the functions you attach to atexit check that flag, and do nothing unless it's been set.
Eg:
_success = False
def atsuccess(func, *args, **kwds):
def wrapper():
if _success:
func(*args,**kwds)
atexit(wrapper)
def set_success():
global _success
_success = True
# then call atsuccess() to attach your callbacks,
# and call set_success() before your script returns
One limitation is if you have any code which calls sys.exit(0) before setting the success flag. Such code should (probably) be refactored to return to the main function first, so that you call set_success and sys.exit in only one place. Failing that, you'll need add something like the following wrapper around the main entry point in your script:
try:
main()
except SystemExit, err:
if err.code == 0:
set_success()
raise
Wrap the body of your program in a with statement and define a corresponding context object that only performs your action when no exceptions have been raised. Something like:
class AtExit(object):
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
if exc_value is None:
print "Success!"
else:
print "Failure!"
if __name__ == "__main__":
with AtExit():
print "Running"
# raise Exception("Error")
How can you have a function or something that will be executed before your program quits? I have a script that will be constantly running in the background, and I need it to save some data to a file before it exits. Is there a standard way of doing this?
Check out the atexit module:
http://docs.python.org/library/atexit.html
For example, if I wanted to print a message when my application was terminating:
import atexit
def exit_handler():
print 'My application is ending!'
atexit.register(exit_handler)
Just be aware that this works great for normal termination of the script, but it won't get called in all cases (e.g. fatal internal errors).
If you want something to always run, even on errors, use try: finally: like this -
def main():
try:
execute_app()
finally:
handle_cleanup()
if __name__=='__main__':
main()
If you want to also handle exceptions you can insert an except: before the finally:
If you stop the script by raising a KeyboardInterrupt (e.g. by pressing Ctrl-C), you can catch that just as a standard exception. You can also catch SystemExit in the same way.
try:
...
except KeyboardInterrupt:
# clean up
raise
I mention this just so that you know about it; the 'right' way to do this is the atexit module mentioned above.
If you have class objects, that exists during the whole lifetime of the program, you can also execute commands from the classes with the __del__(self) method:
class x:
def __init__(self):
while True:
print ("running")
sleep(1)
def __del__(self):
print("destructuring")
a = x()
this works on normal program end as well if the execution is aborted, for sure there will be some exceptions:
running
running
running
running
running
Traceback (most recent call last):
File "x.py", line 14, in <module>
a = x()
File "x.py", line 8, in __init__
sleep(1)
KeyboardInterrupt
destructuring
This is a version adapted from other answers.
It should work (not fully tested) with graceful exits, kills, and PyCharm stop button (the last one I can confirm).
import signal
import atexit
def handle_exit(*args):
try:
... do computation ...
except BaseException as exception:
... handle the exception ...
atexit.register(handle_exit)
signal.signal(signal.SIGTERM, handle_exit)
signal.signal(signal.SIGINT, handle_exit)
Hitting ctrl+c while the dump operation is saving data, the interrupt results in the file being corrupted (i.e. only partially written, so it cannot be loaded again.
Is there a way to make dump, or in general any block of code, uninterruptable?
My current workaround looks something like this:
try:
file = open(path, 'w')
dump(obj, file)
file.close()
except KeyboardInterrupt:
file.close()
file.open(path,'w')
dump(obj, file)
file.close()
raise
It seems silly to restart the operation if it is interrupted, so how can the interrupt be deferred?
The following is a context manager that attaches a signal handler for SIGINT. If the context manager's signal handler is called, the signal is delayed by only passing the signal to the original handler when the context manager exits.
import signal
import logging
class DelayedKeyboardInterrupt:
def __enter__(self):
self.signal_received = False
self.old_handler = signal.signal(signal.SIGINT, self.handler)
def handler(self, sig, frame):
self.signal_received = (sig, frame)
logging.debug('SIGINT received. Delaying KeyboardInterrupt.')
def __exit__(self, type, value, traceback):
signal.signal(signal.SIGINT, self.old_handler)
if self.signal_received:
self.old_handler(*self.signal_received)
with DelayedKeyboardInterrupt():
# stuff here will not be interrupted by SIGINT
critical_code()
Put the function in a thread, and wait for the thread to finish.
Python threads cannot be interrupted except with a special C api.
import time
from threading import Thread
def noInterrupt():
for i in xrange(4):
print i
time.sleep(1)
a = Thread(target=noInterrupt)
a.start()
a.join()
print "done"
0
1
2
3
Traceback (most recent call last):
File "C:\Users\Admin\Desktop\test.py", line 11, in <module>
a.join()
File "C:\Python26\lib\threading.py", line 634, in join
self.__block.wait()
File "C:\Python26\lib\threading.py", line 237, in wait
waiter.acquire()
KeyboardInterrupt
See how the interrupt was deferred until the thread finished?
Here it is adapted to your use:
import time
from threading import Thread
def noInterrupt(path, obj):
try:
file = open(path, 'w')
dump(obj, file)
finally:
file.close()
a = Thread(target=noInterrupt, args=(path,obj))
a.start()
a.join()
Use the signal module to disable SIGINT for the duration of the process:
s = signal.signal(signal.SIGINT, signal.SIG_IGN)
do_important_stuff()
signal.signal(signal.SIGINT, s)
In my opinion using threads for this is an overkill. You can make sure the file is being saved correctly by simply doing it in a loop until a successful write was done:
def saveToFile(obj, filename):
file = open(filename, 'w')
cPickle.dump(obj, file)
file.close()
return True
done = False
while not done:
try:
done = saveToFile(obj, 'file')
except KeyboardInterrupt:
print 'retry'
continue
This question is about blocking the KeyboardInterrupt, but for this situation I find atomic file writing to be cleaner and provide additional protection.
With atomic writes either the entire file gets written correctly, or nothing does. Stackoverflow has a variety of solutions, but personally I like just using atomicwrites library.
After running pip install atomicwrites, just use it like this:
from atomicwrites import atomic_write
with atomic_write(path, overwrite=True) as file:
dump(obj, file)
I've been thinking a lot about the criticisms of the answers to this question, and I believe I have implemented a better solution, which is used like so:
with signal_fence(signal.SIGINT):
file = open(path, 'w')
dump(obj, file)
file.close()
The signal_fence context manager is below, followed by an explanation of its improvements on the previous answers. The docstring of this function documents its interface and guarantees.
import os
import signal
from contextlib import contextmanager
from types import FrameType
from typing import Callable, Iterator, Optional, Tuple
from typing_extensions import assert_never
#contextmanager
def signal_fence(
signum: signal.Signals,
*,
on_deferred_signal: Callable[[int, Optional[FrameType]], None] = None,
) -> Iterator[None]:
"""
A `signal_fence` creates an uninterruptible "fence" around a block of code. The
fence defers a specific signal received inside of the fence until the fence is
destroyed, at which point the original signal handler is called with the deferred
signal. Multiple deferred signals will result in a single call to the original
handler. An optional callback `on_deferred_signal` may be specified which will be
called each time a signal is handled while the fence is active, and can be used
to print a message or record the signal.
A `signal_fence` guarantees the following with regards to exception-safety:
1. If an exception occurs prior to creating the fence (installing a custom signal
handler), the exception will bubble up as normal. The code inside of the fence will
not run.
2. If an exception occurs after creating the fence, including in the fenced code,
the original signal handler will always be restored before the exception bubbles up.
3. If an exception occurs while the fence is calling the original signal handler on
destruction, the original handler may not be called, but the original handler will
be restored. The exception will bubble up and can be detected by calling code.
4. If an exception occurs while the fence is restoring the original signal handler
(exceedingly rare), the original signal handler will be restored regardless.
5. No guarantees about the fence's behavior are made if exceptions occur while
exceptions are being handled.
A `signal_fence` can only be used on the main thread, or else a `ValueError` will
raise when entering the fence.
"""
handled: Optional[Tuple[int, Optional[FrameType]]] = None
def handler(signum: int, frame: Optional[FrameType]) -> None:
nonlocal handled
if handled is None:
handled = (signum, frame)
if on_deferred_signal is not None:
try:
on_deferred_signal(signum, frame)
except:
pass
# https://docs.python.org/3/library/signal.html#signal.getsignal
original_handler = signal.getsignal(signum)
if original_handler is None:
raise TypeError(
"signal_fence cannot be used with signal handlers that were not installed"
" from Python"
)
if isinstance(original_handler, int) and not isinstance(
original_handler, signal.Handlers
):
raise NotImplementedError(
"Your Python interpreter's signal module is using raw integers to"
" represent SIG_IGN and SIG_DFL, which shouldn't be possible!"
)
# N.B. to best guarantee the original handler is restored, the #contextmanager
# decorator is used rather than a class with __enter__/__exit__ methods so
# that the installation of the new handler can be done inside of a try block,
# whereas per [PEP 343](https://www.python.org/dev/peps/pep-0343/) the
# __enter__ call is not guaranteed to have a corresponding __exit__ call if an
# exception interleaves
try:
try:
signal.signal(signum, handler)
yield
finally:
if handled is not None:
if isinstance(original_handler, signal.Handlers):
if original_handler is signal.Handlers.SIG_IGN:
pass
elif original_handler is signal.Handlers.SIG_DFL:
signal.signal(signum, signal.SIG_DFL)
os.kill(os.getpid(), signum)
else:
assert_never(original_handler)
elif callable(original_handler):
original_handler(*handled)
else:
assert_never(original_handler)
signal.signal(signum, original_handler)
except:
signal.signal(signum, original_handler)
raise
First, why not use a thread (accepted answer)?
Running code in a non-daemon thread does guarantee that the thread will be joined on interpreter shutdown, but any exception on the main thread (e.g. KeyboardInterrupt) will not prevent the main thread from continuing to execute.
Consider what would happen if the thread method is using some data that the main thread mutates in a finally block after the KeyboardInterrupt.
Second, to address #benrg's feedback on the most upvoted answer using a context manager:
if an exception is raised after signal is called but before __enter__ returns, the signal will be permanently blocked;
My solution avoids this bug by using a generator context manager with the aid of the #contextmanager decorator. See the full comment in the code above for more details.
this code may call third-party exception handlers in threads other than the main thread, which CPython never does;
I don't think this bug is real. signal.signal is required to be called from the main thread, and raises ValueError otherwise. These context managers can only run on the main thread, and thus will only call third-party exception handlers from the main thread.
if signal returns a non-callable value, __exit__ will crash
My solution handles all possible values of the signal handler and calls them appropriately. Additionally I use assert_never to benefit from exhaustiveness checking in static analyzers.
Do note that signal_fence is designed to handle one interruption on the main thread such as a KeyboardInterrupt. If your user is spamming ctrl+c while the signal handler is being restored, not much can save you. This is unlikely given the relatively few opcodes that need to execute to restore the handler, but it's possible. (For maximum robustness, this solution would need to be rewritten in C)
A generic approach would be to use a context manager that accepts a set of signal to suspend:
import signal
from contextlib import contextmanager
#contextmanager
def suspended_signals(*signals):
"""
Suspends signal handling execution
"""
signal.pthread_sigmask(signal.SIG_BLOCK, set(signals))
try:
yield None
finally:
signal.pthread_sigmask(signal.SIG_UNBLOCK, set(signals))
This is not interruptible (try it), but also maintains a nice interface, so your functions can work the way you expect.
import concurrent.futures
import time
def do_task(func):
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as run:
fut = run.submit(func)
return fut.result()
def task():
print("danger will robinson")
time.sleep(5)
print("all ok")
do_task(task)
and here's an easy way to create an uninterruptible sleep with no signal handling needed:
def uninterruptible_sleep(secs):
fut = concurrent.futures.Future()
with contextlib.suppress(concurrent.futures.TimeoutError):
fut.result(secs)