I am trying to salvage test data from an aborted Jenkins build. The documentation (https://wiki.jenkins-ci.org/display/JENKINS/Aborting+a+build) appears to throw an interrupt and so I attempted to catch it with a custom handler:
signal.signal(signal.SIGINT, signal_handler)
Running my program through CMD and sending crtl-c stimulates the signal_handler function. However, when running this through Jenkins, the interrupt signal is not captured. Are there any plugins that alter Jenkins abort signal, or is there a way to handle it gracefully?
Well, I know this a bit late, but jenkins sends a TERM signal when aborting. Basically you just gotta register that signal
signal.signal(signal.SIGTERM, signal_handler)
Info obtained from:
https://gist.github.com/datagrok/dfe9604cb907523f4a2f
where it says:
Jenkins
When a Jenkins job is cancelled, it sends TERM to the process group of
the process it spawns, and immediately disconnects, reporting "Finished:
ABORTED," regardless of the state of the job.
This causes the spawned process and all its subprocesses (unless they
are spawned in new process groups) to receive TERM.
This is the same effect as running the job in your terminal and pressing
CTRL+C, except in the latter situation INT is sent, not TERM.
Since Jenkins disconnects immediately from the child process and reports
no further output:
it can misleadingly appear that signal handlers are not being
invoked. (Which has mislead
many to
think that Jenkins has KILLed the process.)
it can misleadingly appear that spawned processes have completed or
exited, while they continue to run.
Related
So I have a Flask API running through a Systemd service running on a piece of hardware that's battery powered (to control other hardware). I have a bunch of state that I need to save and in case something goes wrong, like a power outage, I need to be able to restore that state.
Right now I save the state as JSON files so I can load them (if they exist) on startup. But I'd also need to be able to remove them again in case it gets the shutdown signal.
I saw somewhere I could set KillSignal to SIGINT and handle the shutdown as a keyboard interrupt. Or something about ExecStop. Would that be enough, or is there a better way to handle such a scenario?
If you look at the shutdown logs of a linux system you'll see 'sending sigterm to all processes... sending sigkill to all processes'. In a normal shutdown processes get a few second's grace before being killed. So if you trap sigterm you can run your shutdown code, but it had better be over before the untrappable sigkill comes along. Since sigterm is always sent to kill a running process, trapping it is indeed the Right Way (TM) to cleanup on exit. But since you are using systemd services you could also cleanup in the service.
I have a long-running Python process that I want to be able to terminate in the event it gets hung-up and stops reporting progress. But I want to signal it in a way that allows it to safely cleanup, in case it hasn't completely hung-up and there's still something running that can respond to signals gracefully. What's the best order of signals to send before outright killing it?
I'm currently doing something like:
def safe_kill(pid):
for sig in [SIGTERM, SIGABRT, SIGINT, SIGKILL]:
os.kill(pid, sig)
time.sleep(1)
if not pid_exists(pid):
return
Is there a better order? I know SIGKILL bypasses the process entirely, but is there any significant difference between SIGTERM/SIGABRT/SIGINT or do they all have the same effect as far as Python is concerned?
I believe the proper way for stopping a process is SIGTERM followed by SIGKILL after a small timeout.
I don't think that SIGINT and SIGABRT are necessary if that process handles signals in a standard way. SIGINT is usually handled the same way as SIGTERM and SIGABRT is usually used by process itself on abort() (wikipedia).
Anything more complex than a small script usually implements custom SIGTERM handling to shutdown gracefully (cleaning up all the resources, etc).
For example, take a look at Upstart. It is an init daemon - it starts and stops most of processes in Ubuntu and some other distributions. The default Upstart behavior for stopping a process is to send SIGTERM, wait 5 seconds and send SIGKILL (source - upstart cookbook).
You probably should do some testing to determine the best timeout for your process.
You need to register a signal handler, as you would do in C.
import signal
import sys
def clean_termination(signal):
# perform your cleanup
sys.exit(1)
# register the signal handler for the signals specified in the question
signal.signal(signal.SIGTERM, clean_termination)
signal.signal(signal.SIGABRT, clean_termination)
Note that Python maps the SIGINT signal to a KeyboardInterrupt exception, that you can catch with a regular except statement.
I have an application which is stuck in a file.read call. Before it goes into a loop where this call is made it forks a child which starts a gevent WSGI server. The purpose of this setup is that I want to wait for a keystroke, send this keystroke to the child websocket server which spreads the message among other connected websocket-clients. My problem is that I don't know how to stop this thing.
If I Ctrl+C the child server process gets the sigint and stops. But my parent only responds if it can read something out of his file. Isn't there something like an asynchronous handler? I also tried registering for SIGINT via signal.signal and manually sending the signal, but the signal handler was only called if something was written to the file.
BTW: I'm runnning Linux.
What is a good way to Reduce the number of workers on a machine in Python-RQ?
According to the documentation, I need to send a SIGINT or SIGTERM command to one of the worker processes on the machine:
Taking down workers
If, at any time, the worker receives SIGINT (via Ctrl+C) or SIGTERM (via kill), the worker wait until the currently running task is finished, stop the work loop and gracefully register its own death.
If, during this takedown phase, SIGINT or SIGTERM is received again,the worker will forcefully terminate the child process (sending it SIGKILL), but will still try to register its own death.
This seems to imply a lot of coding overhead:
Would need to keep track of the PID for the worker process
Would need to have a way to send a SIGINT command from a remote machine
Do I really need to custom build this, or is there a way to do this easily using the Python-RQ library or some other existing library?
Get all running workers using rq.Worker.all()
Select the worker you want to kill
Use os.kill(worker.pid, signal.SIGINT)
I wonder if I stop the twistd process using
kill `cat twistd.pid`
What will happen if there are exactly some sql execution committing?
Will it waiting for the execution done? or just unknown, it could be done, or abandon?
I know if I put the execution in the stopFactory method, the factory will do such things like waiting for the execution done. But if I don't, I mean the execution out the stopFactory method, will it waiting for the execution done before the factory stopping?
Thanks.
kill sends SIGTERM by default. Twisted installs a SIGTERM handler which calls reactor.stop(). Anything that would happen when you call reactor.stop() will happen when you use that kill command.
More specifically, any shutdown triggers will run. This means any services attached to an Application will have their stopService method called (and if a Deferred is returned, it will be allowed to finish before shutdown proceeds). It also means worker threads in the reactor threadpool will be shutdown in an orderly manner - ie, allowed to complete whatever job they have in progress.
If you're using adbapi, then the ConnectionPool uses its own ThreadPool and also registers a shutdown trigger to shut that pool down in a similar orderly manner.
So, when you use kill to stop a Twisted-based process, any SQL mid-execution will be allowed to complete before shutdown takes place.