We Keep Coding

python create subprocess (newbie)

I'm new to python, so here's what I'm looking to get done.
I would like to use python to manage some of my gameservers and start/stop them. For this I would like to run every gameserver in a own process.
What's the best way to create processes using python, so these processes can continue even if the main application is stopped?
To start a server I only need to execute shell code.
How can I get access after stopping my main application and restarting it to these processes?

I'm not sure if I understand the question completely, but maybe something like this?
Run process:
import subprocess
subprocess.Popen(['/path/gameserver']) #keeps running
And in another script you can use 'ps -A' to find the pid and kill (or restart) it:
import subprocess, signal
p = subprocess.Popen(['ps', '-A'], stdout=subprocess.PIPE)
out, err = p.communicate()
for line in out.splitlines():
if 'gameserver' in line:
pid = int(line.split(None, 1)[0])
os.kill(pid, signal.SIGKILL)

Check the subprocess module. There is a function called call. See here.
You may need to set the process to not be a daemon process.

Python multiprocessing. How do you get the status of the parent from the child?

I'm writing an application (Linux) using the multiprocessing module, which spawns several children. When a child dies, I can detect it from the parent using something like the following:
process = multiprocessing.Process(...)
if process.is_alive():
print "Process died"
However, I'd also like to be able to detect from the children if the parent is still alive, to handle cleanup if someone goes and kill -9's the parent process.
From my example above, I can get the parent id with either:
process._parent_pid
Or:
os.getppid()
But I can't find an easy way to get a status of the process. I'd rather not write something to grep/regex the ps list using subprocess. Is there a cleaner way?

You could compare the parent's process id against 1; if it is 1 then you can deduce the parent process has terminated, because the subprocess now has the init process (pid 1) as parent.
import os
import time
from multiprocessing import Process
def subprocess():
while True:
ppid = os.getppid()
print "Parent process id:", ppid
if ppid == 1:
print "Parent process has terminated"
break
time.sleep(1)
p = Process(target=subprocess)
p.start()

Kill a chain of sub processes on KeyboardInterrupt

I'm having a strange problem I've encountered as I wrote a script to start my local JBoss instance.
My code looks something like this:
with open("/var/run/jboss/jboss.pid", "wb") as f:
process = subprocess.Popen(["/opt/jboss/bin/standalone.sh", "-b=0.0.0.0"])
f.write(str(process.pid))
try:
process.wait()
except KeyboardInterrupt:
process.kill()
Should be fairly simple to understand, write the PID to a file while its running, once I get a KeyboardInterrupt, kill the child process.
The problem is that JBoss keeps running in the background after I send the kill signal, as it seems that the signal doesn't propagate down to the Java process started by standalone.sh.
I like the idea of using Python to write system management scripts, but there are a lot of weird edge cases like this where if I would have written it in Bash, everything would have just worked™.
How can I kill the entire subprocess tree when I get a KeyboardInterrupt?

You can do this using the psutil library:
import psutil
#..
proc = psutil.Process(process.pid)
for child in proc.children(recursive=True):
child.kill()
proc.kill()
As far as I know the subprocess module does not offer any API function to retrieve the children spawned by subprocesses, nor does the os module.
A better way of killing the processes would probably be the following:
proc = psutil.Process(process.pid)
procs = proc.children(recursive=True)
procs.append(proc)
for proc in procs:
proc.terminate()
gone, alive = psutil.wait_procs(procs, timeout=1)
for p in alive:
p.kill()
This would give a chance to the processes to terminate correctly and when the timeout ends the remaining processes will be killed.
Note that psutil also provides a Popen class that has the same interface of subprocess.Popen plus all the extra functionality of psutil.Process. You may want to simply use that instead of subprocess.Popen. It is also safer because psutil checks that PIDs don't get reused if a process terminates, while subprocess doesn't.

kill subprocess spawned by spawnProcess in twisted doesn't work

all
I start a process using spawnProcess and want to kill when my certain Factory stops.
something I wrote like these
p = SomeProtocol(ProcessProtocol)
reactor.spawnProcess(p, 'twistd', ['twistd', '-y', 'anotherMain.py'], {})
class Factory(ServerFactory):
...
def StopFactory(self):
# which is the p above
p.transport.signalProcess("KILL")
I thought the subprocess will be killed which is not.
I tried using p.transport.signalProcess("KILL") some other place, and it works.
What's wrong with my code? Thanks!

This can be because twistd daemonizes anotherMain.py. After anotherMain.py becomes a daemon twistd process exits. So anotherMain.py isn't really a subprocess of your main process.
Try to add -n option:
reactor.spawnProcess(p, 'twistd', ['twistd', '-ny', 'anotherMain.py'], {})

how to kill process and child processes from python?

for example from bash:
kill -9 -PID
os.kill(pid, signal.SIGKILL) kill only parent process.

If the parent process is not a "process group" but you want to kill it with the children, you can use psutil (https://psutil.readthedocs.io/en/latest/#processes). os.killpg cannot identify pid of a non-process-group.
import psutil
parent_pid = 30437 # my example
parent = psutil.Process(parent_pid)
for child in parent.children(recursive=True): # or parent.children() for recursive=False
child.kill()
parent.kill()

When you pass a negative PID to kill, it actually sends the signal to the process group by that (absolute) number. You do the equivalent with os.killpg() in Python.

Another solution if your process is not a process group and you don't want to use psutil, is to run this shell command:
pkill -TERM -P 12345
For instance with
os.system('pkill -TERM -P {pid}'.format(pid=12345))

None of answers can helped me, so I made some research and wrote my answer:
you can easily do it using os module, but it is platform sensitive. This mean that some commands are availiable only on Unix, some - on any platform.
So my project starts one Process, and several Child processes in the different places and times. Some of Child starts Grand-Child Processes :)
So I found this solution:
import os
import signal
import platform
# get the current PID for safe terminate server if needed:
PID = os.getpid()
if platform.system() is not 'Windows':
PGID = os.getpgid(PID)
if platform.system() is not 'Windows':
os.killpg(PGID, signal.SIGKILL)
else:
os.kill(PID, signal.SIGTERM)
I use SIGKILL on Linux, to kill process immediatly, and SIGTERM on Windows, because there is no SIGKILL on it.
Also I used killpg() to kill the whole group of processes on Linux.
P.S. Check on Linux, but still doesn't check on Windows, so maybe we need one more additional command for Windows (for example CTRL_C_EVENT or use another answer.)

you should use signal parameter 9 to kill the process tree.
root#localhost:~$ python
>>> import os
>>> os.kill(pid, 9)
if you should use signal.SIGKILL constant, you should use os.killpg(pgid, signal.SIGKILL) to kill the process tree.

I don't know if this is what you asked for, but if you wish to kill other processes of your application and they were all created using multiprocessing package you can do something like this:
import multiprocessing
from time import sleep
...
def on_shutdown():
for child in multiprocessing.active_children():
print('Terminating', child)
child.terminate()
sleep(0.5)

def kill_children_processes(pid):
# TODO: Find a way to not have to use a kill -9.
processes = os.popen("ps -ej | grep -i 'python' | grep -v 'grep' | awk '{ print $2,$3 }'").read()
processes = [p.split(" ") for p in processes.split("\n")[:-1]]
processes = "\n".join([child for child, parent in processes if parent == str(pid) and child != str(pid)])
if processes:
logger.debug(f"Killing ghost processes {processes}")
os.system(f"kill -9 {processes}")