I have a bash shell script which calls a Python script in an endless loop:
while true; do python testomat.py ; done
The Python script testomat.py:
try:
do_something()
except KeyboardInterrupt:
print 'exit'
finally:
raise SystemExit
The problem I'm having is that I cannot stop this by Ctrl-C. My only chance: Ctrl-z and then kill -TERM %1 assuming that I have no other jobs running.
Would Python somehow have to propagate Ctrl-C to the parent or how would I be able to stop this?
I know I could run the endless loop inside Python, but I actually do more in both the bash shell script and the Python script which both must continue to exist.
Set an exit status in the Python script and catch it in the Bash script. I would use 130 since that's what coreutils like grep and find use.
except KeyboardInterrupt:
sys.exit(130)
while true; do
python testomat.py
case $? in
130) break ;;
esac
done
You'll notice that you generally can interrupt shell loops. For example, this loop will exit just fine when you hit Ctrl-C:
while true; do sleep 10; done
A parent process -- here the shell -- can tell whether a child exited normally, or whether it was killed by a signal. If it was killed by a signal, the shell will consider the signal unhandled and will stop the loop and/or script. If the child receives a signal but exits normally, the shell will consider it handled and continue.
When a process wants to do some cleanup, it necessarily handles the signal and therefore isn't killed. The canonical Unix behavior is therefore, when cleanup is done, to uninstall the signal handler and re-kill yourself:
import time
import os
import signal
try:
time.sleep(10);
except KeyboardInterrupt:
print('exit')
signal.signal(signal.SIGINT, signal.SIG_DFL)
os.kill(os.getpid(), signal.SIGINT)
Since the cause of death is being correctly relayed back to the shell, you will now be able to Ctrl-C out of your while true; do python yourfile.py; done loop.
Related
I am developing a wrapper around docker compose with python.
However, I struggle with Popen.
Here is how I launch launch it :
import subprocess as sp
argList=['docker-compose', 'up']
env={'HOME': '/home/me/somewhere'}
p = sp.Popen(argList, env=env)
def handler(signum, frame):
p.send_signal(signum)
for s in (signal.SIGINT,):
signal.signal(s, handler) # to redirect Ctrl+C
p.wait()
Everything works fine, when I hit Ctrl+C, docker-compose kills gracelly the container, however, p.wait() never returns...
Any hint ?
NOTE : While writing the question, I though I needed to check if p.wait() does actually return and if the block is after (it's the last instruction in the script). Adding a print after it end in the process exiting normally, any further hints on this behavior ?
When I run your code as written, it works as intended in that it causes docker-compose to exit and then p.wait() returns. However, I occasionally see this behavior:
Killing example_service_1 ... done
ERROR: 2
I think that your code may end up delivering SIGINT twice to docker-compose. That is, I think docker-compose receives an initial SIGINT when you type CTRL-C, because it has the same controlling terminal as your Python script, and then you explicitly deliver another SIGINT in your handler function.
I don't always see this behavior, so it's possible my explanation is incorrect.
In any case, I think the correct solution here is imply to ignore SIGINT in your Python code:
import signal
import subprocess
argList = ["docker-compose", "up"]
p = subprocess.Popen(argList)
signal.signal(signal.SIGINT, signal.SIG_IGN) # to redirect Ctrl+C
p.wait()
With this implementation, your Python code ignores the SIGINT generated by CTRL-C, but it is received and processed normally by docker-compose.
I wrote a simple python script ./vader-shell which uses subprocess.Popen to launch a spark-shell and I have to deal with KeyboardInterrupt, since otherwise the child process would not die
command = ['/opt/spark/current23/bin/spark-shell']
command.extend(params)
p = subprocess.Popen(command)
try:
p.communicate()
except KeyboardInterrupt:
p.terminate()
This is what I see with ps f
When I actually interrupt with ctrl-C, I see the processes dying (most of the time). However the terminal starts acting weird: I don't see any cursor, and all the lines starts to appear randomly
I am really lost in what is the best way to run a subprocess with this library and how to handle killing of the child processes. What I want to achieve is basic: whenever my python process is killed with a ctrl-C, I want all the family of process being killed. I googled several solutions os.kill, p.wait() after termination, calling subprocess.Popen(['reset']) after termination but none of them worked.
Do you know what is the best way to kill when KeyboardInterrupt happens? Or do you know any other more reliable library to use to spin-up processes?
There is nothing blatantly wrong with your code, the problem is that the command you are launching tries to do stuff with the current terminal, and does not correctly restore the settings where shutting down. Replacing your command with a "sleep" like below will run just fine and stop on Ctrl+C without problems:
import subprocess
command = ['/bin/bash']
command.extend(['-c', 'sleep 600'])
p = subprocess.Popen(command)
try:
p.communicate()
except KeyboardInterrupt:
p.terminate()
I don't know what you're trying to do with spark-shell, but if you don't need it's output you could try to redirect it to /dev/null so that it's doesn't mess up the terminal display:
p = subprocess.Popen(command, stdout=subprocess.DEVNULL)
I have this simple python script called myMain.py to execute another python program automatically with incremental number, and I'm running it on CentOS 7:
#!/usr/bin/python
import os
import sys
import time
def main():
step_indicator = ""
arrow = ">"
step = 2
try:
for i in range(0,360, step):
step_percentage = float(i)/360.0 * 100
if i % 10 == 0:
step_indicator += "="
os.system("python myParsePDB.py -i BP1.pdb -c 1 -s %s" % step)
print("step_percentage%s%s%.2f" % (step_indicator,arrow,step_percentage)+"%")
except KeyboardInterrupt:
print("Stop me!")
sys.exit(0)
if __name__ == "__main__":
main()
For now I only know this script is single thread safe, but I can't terminate it with Ctrl+C keyboard interruption.
I have read some relative questions: such as Cannot kill Python script with Ctrl-C and Stopping python using ctrl+c I realized that Ctrl+Z does not kill the process, it only pauses the process and keep the process in background. Ctrl+Break does work for my case either, I think it only terminates my main thread but keeps the child process.
I also noticed that calling os.system() will spawn a child process from the current executing process. At the same time, I also have os file I/O functions and os.system("rm -rf legacy/*") will be invoked in myParsePDB.py which means this myParsePDB.py child process will spawn child process as well. Then, if I want to catch Ctrl+C in myMain.py, should I daemon only myMain.py or should I daemon each process when they spawn?
This is a general problem that could raise when dealing with signal handling. Python signal is not an exception, it's a wrapper of operating system signal. Therefore, signal processing in python depends on operating system, hardware and many conditions. However, how to deal with these problem is similar.
According to this tutorial, I'll quote the following paragraphs: signal – Receive notification of asynchronous system events
Signals are an operating system feature that provide a means of
notifying your program of an event, and having it handled
asynchronously. They can be generated by the system itself, or sent
from one process to another. Since signals interrupt the regular flow
of your program, it is possible that some operations (especially I/O)
may produce error if a signal is received in the middle.
Signals are identified by integers and are defined in the operating
system C headers. Python exposes the signals appropriate for the
platform as symbols in the signal module. For the examples below, I
will use SIGINT and SIGUSR1. Both are typically defined for all Unix
and Unix-like systems.
In my code:
os.system("python myParsePDB.py -i BP1.pdb -c 1 -s %s" % step) inside the for loop will be executed for a bit of time and will spend some time on I/O files. If the keyboard interrupt is passing too fast and do not catch asynchronously after writing files, the signal might be blocked in operating system, so my execution will still remain the try clause for loop. (Errors detected during execution are called exceptions and are not unconditionally fatal: Python Errors and Exceptions).
Therefore the simplest way to make them asynchonous is wait:
try:
for i in range(0,360, step):
os.system("python myParsePDB.py -i BP1.pdb -c 1 -s %s" % step)
time.sleep(0.2)
except KeyboardInterrupt:
print("Stop me!")
sys.exit(0)
It might hurt performance but it guaranteed that the signal can be caught after waiting the execution of os.system(). You might also want to use other sync/async functions to solve the problem if better performance is required.
For more unix signal reference, please also look at: Linux Signal Manpage
I am executing a Ruby script from within a Python script. Here is what my Python script "script007.py" looks like:
.
.
.
os.system("ruby script.rb") #executing ctrl+c here
print "should not be here"
.
.
.
I execute CTRL+C when the Ruby script is running but it just stops "script.rb" and continues with the rest of "script007.py". I know this because it prints "should not be here" when the Ruby script is stopped.
Is there a way that I can catch the CTRL+C in my Python script even though it happens in Ruby script? Let me know if further explanation is required.
In Python, SIGINT raises a special exception which you could catch. If, however, the child consumes the SIGINT signal and responds to it, it does not arrive at the parent process. Then you need to find a different way to communicate from child to parent about why the child exited. This usually is the exit code.
In any case, you should start replacing os.system() with tools from the subprocess module (this is documented, just go and read about this in the subprocess docs). You could emit a certain exit code in the child when it exits after retrieving SIGINT, and analyze the exit code in the parent. You can then exit the parent conditionally right after the child process has terminated, depending on what the exit code of the child was.
Example: your child (Ruby program) exits with code 15 after retrieving SIGINT. In the parent (Python program) you would do something in the lines of:
p = subprocess.Popen(...)
out, err = p.communicate(...)
if p.returncode == 15:
sys.exit(1)
print "should not be here"
In your Ruby script:
trap('SIGINT') { exit 1 }
In your Python script, os.system() should return the value passed to exit. You can use that redirect the flow of control however needed, e.g. call sys.exit().
I am executing a bash script in Python using the tempfile and subprocess like so:
with tempfile.NamedTemporaryFile() as scriptfile:
scriptfile.write(teststr)
scriptfile.flush()
subprocess.call(['/bin/bash', scriptfile.name])
Here, teststr has the entire bash script within it.
My question is, once it starts to execute, it doesn't capture keyboard interrupts like Ctrl+c and ctrl+z.
Is there anyway to interrupt the execution of the script once it has begun?
I assume that the problem is that Python parent process receives SIGINT from Ctrl+C and quits with unhandled exception, but the child ignores signal and keeps running. That is the only scenario I was able to reproduce. Actual problem may differ. Catching exception and killing subprocess explicitly with SIGKILL may work.
Instead of subprocess.call:
proc = subprocess.Popen(['/bin/bash', scriptfile.name])
try:
proc.wait()
except:
proc.kill()
raise