We Keep Coding

Popen.wait never returning with docker-compose

I am developing a wrapper around docker compose with python.
However, I struggle with Popen.
Here is how I launch launch it :
import subprocess as sp
argList=['docker-compose', 'up']
env={'HOME': '/home/me/somewhere'}
p = sp.Popen(argList, env=env)
def handler(signum, frame):
p.send_signal(signum)
for s in (signal.SIGINT,):
signal.signal(s, handler) # to redirect Ctrl+C
p.wait()
Everything works fine, when I hit Ctrl+C, docker-compose kills gracelly the container, however, p.wait() never returns...
Any hint ?
NOTE : While writing the question, I though I needed to check if p.wait() does actually return and if the block is after (it's the last instruction in the script). Adding a print after it end in the process exiting normally, any further hints on this behavior ?

When I run your code as written, it works as intended in that it causes docker-compose to exit and then p.wait() returns. However, I occasionally see this behavior:
Killing example_service_1 ... done
ERROR: 2
I think that your code may end up delivering SIGINT twice to docker-compose. That is, I think docker-compose receives an initial SIGINT when you type CTRL-C, because it has the same controlling terminal as your Python script, and then you explicitly deliver another SIGINT in your handler function.
I don't always see this behavior, so it's possible my explanation is incorrect.
In any case, I think the correct solution here is imply to ignore SIGINT in your Python code:
import signal
import subprocess
argList = ["docker-compose", "up"]
p = subprocess.Popen(argList)
signal.signal(signal.SIGINT, signal.SIG_IGN) # to redirect Ctrl+C
p.wait()
With this implementation, your Python code ignores the SIGINT generated by CTRL-C, but it is received and processed normally by docker-compose.

Ctrl-C from python to bash

I have a bash shell script which calls a Python script in an endless loop:
while true; do python testomat.py ; done
The Python script testomat.py:
try:
do_something()
except KeyboardInterrupt:
print 'exit'
finally:
raise SystemExit
The problem I'm having is that I cannot stop this by Ctrl-C. My only chance: Ctrl-z and then kill -TERM %1 assuming that I have no other jobs running.
Would Python somehow have to propagate Ctrl-C to the parent or how would I be able to stop this?
I know I could run the endless loop inside Python, but I actually do more in both the bash shell script and the Python script which both must continue to exist.

Set an exit status in the Python script and catch it in the Bash script. I would use 130 since that's what coreutils like grep and find use.
except KeyboardInterrupt:
sys.exit(130)
while true; do
python testomat.py
case $? in
130) break ;;
esac
done

You'll notice that you generally can interrupt shell loops. For example, this loop will exit just fine when you hit Ctrl-C:
while true; do sleep 10; done
A parent process -- here the shell -- can tell whether a child exited normally, or whether it was killed by a signal. If it was killed by a signal, the shell will consider the signal unhandled and will stop the loop and/or script. If the child receives a signal but exits normally, the shell will consider it handled and continue.
When a process wants to do some cleanup, it necessarily handles the signal and therefore isn't killed. The canonical Unix behavior is therefore, when cleanup is done, to uninstall the signal handler and re-kill yourself:
import time
import os
import signal
try:
time.sleep(10);
except KeyboardInterrupt:
print('exit')
signal.signal(signal.SIGINT, signal.SIG_DFL)
os.kill(os.getpid(), signal.SIGINT)
Since the cause of death is being correctly relayed back to the shell, you will now be able to Ctrl-C out of your while true; do python yourfile.py; done loop.

subprocess.Popen makes terminal crash after KeyboardInterrupt

I wrote a simple python script ./vader-shell which uses subprocess.Popen to launch a spark-shell and I have to deal with KeyboardInterrupt, since otherwise the child process would not die
command = ['/opt/spark/current23/bin/spark-shell']
command.extend(params)
p = subprocess.Popen(command)
try:
p.communicate()
except KeyboardInterrupt:
p.terminate()
This is what I see with ps f
When I actually interrupt with ctrl-C, I see the processes dying (most of the time). However the terminal starts acting weird: I don't see any cursor, and all the lines starts to appear randomly
I am really lost in what is the best way to run a subprocess with this library and how to handle killing of the child processes. What I want to achieve is basic: whenever my python process is killed with a ctrl-C, I want all the family of process being killed. I googled several solutions os.kill, p.wait() after termination, calling subprocess.Popen(['reset']) after termination but none of them worked.
Do you know what is the best way to kill when KeyboardInterrupt happens? Or do you know any other more reliable library to use to spin-up processes?

There is nothing blatantly wrong with your code, the problem is that the command you are launching tries to do stuff with the current terminal, and does not correctly restore the settings where shutting down. Replacing your command with a "sleep" like below will run just fine and stop on Ctrl+C without problems:
import subprocess
command = ['/bin/bash']
command.extend(['-c', 'sleep 600'])
p = subprocess.Popen(command)
try:
p.communicate()
except KeyboardInterrupt:
p.terminate()
I don't know what you're trying to do with spark-shell, but if you don't need it's output you could try to redirect it to /dev/null so that it's doesn't mess up the terminal display:
p = subprocess.Popen(command, stdout=subprocess.DEVNULL)

Reliably terminating subprocess on Python 2.5

I have a script that repeatedly runs an Ant buildfile and scrapes output into a parsable format. When I create the subprocess using Popen, there is a small time window where hitting Ctrl+C will kill the script, but will not kill the subprocess running Ant, leaving a zombie that is printing output to the console that can only be killed using Task Manager. Once Ant has started printing output, hitting Ctrl+C will always kill my script as well as Ant. Is there a way to make it so that hitting Ctrl+C will always kill the subprocess running Ant without leaving a zombie behind?
Also of note: I have a handler for SIGINT that performs a few cleanup operations before calling exit(0). If I manually kill the subprocess in the handler using os.kill(p.pid, signal.SIGTERM) (not SIGINT), then I can successfully kill the subprocess in situations where it would normally zombify. However, when you hit Ctrl+C once Ant has started producing output, you get a stacktrace from subprocess where it is unable to kill the subprocess itself as I have already killed it.
EDIT: My code looked something like:
p = Popen('ls')
def handle_sig_int(signum, stack_frame):
# perform cleanup
os.kill(p.pid, signal.SIGTERM)
exit(0)
signal.signal(signal.SIGINT, handle_sig_int)
p.wait()
Which would produce the following stacktrace when triggered incorrectly:
File "****.py", line ***, in run_test
p.wait()
File "/usr/lib/python2.5/subprocess.py", line 1122, in wait
pid, sts = os.waitpid(self.pid, 0)
File "****.py", line ***, in handle_sig_int
os.kill(p.pid, signal.SIGTERM)
I fixed it by catching the OSError raised by p.wait and exiting:
try:
p.wait()
except OSError:
exit('The operation was interrupted by the user')
This seems to work in the vast majority of my test runs. I occasionally get a uname: write error: Broken pipe, though I don't know what causes it. It seems to happen if I time the Ctrl+C just right before the child process can start displaying output.

Call p.terminate() in your SIGTERM handler:
if p.poll() is None: # Child still around?
p.terminate() # kill it
[EDIT] Since you're stuck with Python 2.5, use os.kill(p.pid, signal.SIGTERM) instead of p.terminate(). The check should make sure you don't get an exception (or reduce the number of times you get one).
To make it even better, you can catch the exception and check the message. If it means "child process not found", then ignore the exception. Otherwise, rethrow it with raise (no arguments).

Python run system command and then exit... won't exit

I have the following python code:
os.system("C:/Python27/python.exe C:/GUI/TestGUI.py")
sys.exit(0)
It runs the command fine, and a window pops up. However, it doesn't exit the first script. It just stays there, and I eventually have to force kill the process. No errors are produced. What's going on?

instead of os.system use subprocess.Popen
this runs a command and doesn't wait for it and then exits:
import subprocess
import sys
subprocess.Popen(["mupdf", "/home/dan/Desktop/Sieve-JFP.pdf"])
sys.exit(0)
note that os.system(command) like:
p = subprocess.Popen(command)
p.wait()

KeyboardInterrupts and signals are only seen by the process (ie the main thread). If your nested command hangs due to some kind of file read or write block, you won't be able to quit the program using any keyboard commands.
Why does a read-only open of a named pipe block?
If you can't eliminate the source of the disk block, then one way is to wrap the process in the thread so you can force kill it. But if you do this, you leave opportunity for half-written and corrupted files on disk.

I suggest using os._exit instead of sys.exit, as sys.exit doesnt quit a program but raises exception level, or exits a thread. os._exit(-1) quits the entire program

import sys ,subprocess
subprocess.Popen(["C:/Python27/python.exe", "C:/GUI/TestGUI.py"])
sys.exit(0)
Popen from subprocess module what you are looking for.