I have a long-running Python process that I want to be able to terminate in the event it gets hung-up and stops reporting progress. But I want to signal it in a way that allows it to safely cleanup, in case it hasn't completely hung-up and there's still something running that can respond to signals gracefully. What's the best order of signals to send before outright killing it?
I'm currently doing something like:
def safe_kill(pid):
for sig in [SIGTERM, SIGABRT, SIGINT, SIGKILL]:
os.kill(pid, sig)
time.sleep(1)
if not pid_exists(pid):
return
Is there a better order? I know SIGKILL bypasses the process entirely, but is there any significant difference between SIGTERM/SIGABRT/SIGINT or do they all have the same effect as far as Python is concerned?
I believe the proper way for stopping a process is SIGTERM followed by SIGKILL after a small timeout.
I don't think that SIGINT and SIGABRT are necessary if that process handles signals in a standard way. SIGINT is usually handled the same way as SIGTERM and SIGABRT is usually used by process itself on abort() (wikipedia).
Anything more complex than a small script usually implements custom SIGTERM handling to shutdown gracefully (cleaning up all the resources, etc).
For example, take a look at Upstart. It is an init daemon - it starts and stops most of processes in Ubuntu and some other distributions. The default Upstart behavior for stopping a process is to send SIGTERM, wait 5 seconds and send SIGKILL (source - upstart cookbook).
You probably should do some testing to determine the best timeout for your process.
You need to register a signal handler, as you would do in C.
import signal
import sys
def clean_termination(signal):
# perform your cleanup
sys.exit(1)
# register the signal handler for the signals specified in the question
signal.signal(signal.SIGTERM, clean_termination)
signal.signal(signal.SIGABRT, clean_termination)
Note that Python maps the SIGINT signal to a KeyboardInterrupt exception, that you can catch with a regular except statement.
Related
So I have a Flask API running through a Systemd service running on a piece of hardware that's battery powered (to control other hardware). I have a bunch of state that I need to save and in case something goes wrong, like a power outage, I need to be able to restore that state.
Right now I save the state as JSON files so I can load them (if they exist) on startup. But I'd also need to be able to remove them again in case it gets the shutdown signal.
I saw somewhere I could set KillSignal to SIGINT and handle the shutdown as a keyboard interrupt. Or something about ExecStop. Would that be enough, or is there a better way to handle such a scenario?
If you look at the shutdown logs of a linux system you'll see 'sending sigterm to all processes... sending sigkill to all processes'. In a normal shutdown processes get a few second's grace before being killed. So if you trap sigterm you can run your shutdown code, but it had better be over before the untrappable sigkill comes along. Since sigterm is always sent to kill a running process, trapping it is indeed the Right Way (TM) to cleanup on exit. But since you are using systemd services you could also cleanup in the service.
I am trying to salvage test data from an aborted Jenkins build. The documentation (https://wiki.jenkins-ci.org/display/JENKINS/Aborting+a+build) appears to throw an interrupt and so I attempted to catch it with a custom handler:
signal.signal(signal.SIGINT, signal_handler)
Running my program through CMD and sending crtl-c stimulates the signal_handler function. However, when running this through Jenkins, the interrupt signal is not captured. Are there any plugins that alter Jenkins abort signal, or is there a way to handle it gracefully?
Well, I know this a bit late, but jenkins sends a TERM signal when aborting. Basically you just gotta register that signal
signal.signal(signal.SIGTERM, signal_handler)
Info obtained from:
https://gist.github.com/datagrok/dfe9604cb907523f4a2f
where it says:
Jenkins
When a Jenkins job is cancelled, it sends TERM to the process group of
the process it spawns, and immediately disconnects, reporting "Finished:
ABORTED," regardless of the state of the job.
This causes the spawned process and all its subprocesses (unless they
are spawned in new process groups) to receive TERM.
This is the same effect as running the job in your terminal and pressing
CTRL+C, except in the latter situation INT is sent, not TERM.
Since Jenkins disconnects immediately from the child process and reports
no further output:
it can misleadingly appear that signal handlers are not being
invoked. (Which has mislead
many to
think that Jenkins has KILLed the process.)
it can misleadingly appear that spawned processes have completed or
exited, while they continue to run.
I know that you can detect SIGHUP, SIGTERM, SIGQUIT etc, but is it possible to detect when the system receives a halt / poweroff / shutdown signal ?
Signals like SIGHUP, SIGTERM, SIGQUIT are sent to a specific process and can be handled there. Powerr off and shutdown are handled by the init process of your system. They depend on the implementation of init you are using (Upstart, SysV init), and there is no general way to detect and handle them from another process, regardless of whether this process is written in Python or any other language.
All processes are sent the SIGTERM signal, and then the SIGKILL signal. To my knowledge, there is no way to know if those signals are sent specifically because of a shutdown or not. See this relevant question for more information.
I have been using Pyro 3 for a little while now, with great success, but occasionally I have noticed, that when a signal such as SIGHUP or SIGINT arrives while Pyro is doing some remote communications, the process hangs, hence the question, is Pyro signal safe?
Thanks in advance.
Seems the issue here is by default Python sets up a handlers for SIGINT and SIGTERM which raise exceptions. If you therfore receive a signal while doing some Pyro comms, the exception is raised, and off it goes to look for an appropriate except clause, not finishing what it was doing, if you then try and use Pyro again, for example in the except/finally clause, you can get issues. In my case it was sending some messages from finally to a log via a queue which was proxied to another process using Pyro.
What are the rules surrounding Python threads and how Unix signals are handled?
Is KeyboardInterrupt, which is triggered by SIGINT but handled internally by the Python runtime, handled differently?
First, when setting up signal handlers using the signal module, you must create them in the main thread. You will receive an exception if you try to create them in a separate thread.
Signal handlers registered via the signal.signal() function will always be called in the main thread. On architectures which support sending signals to threads, at the C level I believe the Python runtime ignores all signals on threads and has a signal handler on the main thread, which it uses to dispatch to your Python-code signal handler.
The documentation for the thread module states that the KeyboardInterrupt exception (which is ordinarily triggered by SIGINT) can be delivered to an arbitrary thread unless you have the signal module available to you, which all Unix systems should have. In that case, it's delivered to the main thread. If you're on a system without signal, you'll have to catch KeyboardInterrupt in your thread and call thread.interrupt_main() to re-raise it in the main thread.
More information can be found in the Python docs for the thread and signal modules.
From the signal documentation:
Some care must be taken if both signals and threads are used in the same program. The fundamental thing to remember in using signals and threads simultaneously is: always perform signal() operations in the main thread of execution. Any thread can perform an alarm(), getsignal(), pause(), setitimer() or getitimer(); only the main thread can set a new signal handler, and the main thread will be the only one to receive signals (this is enforced by the Python signal module, even if the underlying thread implementation supports sending signals to individual threads). This means that signals can’t be used as a means of inter-thread communication. Use locks instead.