I am trying to open a subprocess but have it be detached from the parent script that called it. Right now if I call subprocess.popen and the parent script crashes the subprocess dies as well.
I know there are a couple of options for windows but I have not found anything for *nix.
I also don't need to call this using subprocess. All I need is to be able to cal another process detached and get the pid.
With linux, it's no issue at all. Just Popen(). For example, here is a little dying_demon.py
#!/usr/bin/python -u
from time import sleep
from subprocess import Popen
print Popen(["python", "-u", "child.py"]).pid
i = 0
while True:
i += 1
print "demon: %d" % i
sleep(1)
if i == 3:
i = hurz # exception
spinning off a child.py
#!/usr/bin/python -u
from time import sleep
i = 0
while True:
i += 1
print "child: %d" % i
sleep(1)
if i == 20:
break
The child continues to count (to the console), while the demon is dying by exception.
I think this should do the trick: https://www.python.org/dev/peps/pep-3143/#reference-implementation
You can create daemon which will call your subprocess, passing detach_process=True.
This might do what you want:
def cmd_detach(*command, **kwargs) -> subprocess.CompletedProcess:
# https://stackoverflow.com/questions/62521658/python-subprocess-detach-a-process
# if using with ffmpeg remember to run it with `-nostdin`
stdout = os.open(os.devnull, os.O_WRONLY)
stderr = os.open(os.devnull, os.O_WRONLY)
stdin = os.open(os.devnull, os.O_RDONLY)
command = conform(command)
if command[0] in ["fish", "bash"]:
import shlex
command = command[0:2] + [shlex.join(command[2:])]
subprocess.Popen(command, stdin=stdin, stdout=stdout, stderr=stderr, close_fds=True, start_new_session=True, **kwargs)
return subprocess.CompletedProcess(command, 0, "Detached command is async")
On Windows you might need
CREATE_NEW_PROCESS_GROUP = 0x00000200
DETACHED_PROCESS = 0x00000008
creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP
instead of start_new_session=True
I managed to get it working by doing the following using python-daemon:
process = subprocess.Popen(["python", "-u", "Child.py"])
time.sleep(2)
process.kill()
Then in Child.py:
with daemon.DaemonContext():
print("Child Started")
time.sleep(30)
print "Done"
exit()
I do process.kill() because otherwise it creates a defunct python process. The main problem I have now is that the PID that popen returns does not match the final pid of the process. I can get by this by adding a function in Child.py to update a database with the pid.
Let me know if there is something that I am missing or if this is an ok method of doing this.
fork the subprocs using the NOHUP option
Related
I am working on UBUNTU and I have file main.py with a code inside:
#!/usr/bin/env python3
# coding=utf-8
import os
import time
from subprocess import Popen, PIPE, call, signal
base_path = os.path.abspath('')
path_to_file = base_path + '/test_subprocess.py'
p = Popen(['gnome-terminal', "--", path_to_file])
time.sleep(2)
os.kill(p.pid, signal.SIGKILL)
I have test_subprocess.py with code like that:
#!/usr/bin/env python3
# coding=utf-8
import time
def print_message():
while True:
print('I am working!')
time.sleep(0.5)
print_message()
I tried to kill the subprocess but after
os.kill(p.pid, signal.SIGKILL)
subprocess is still working and prints 'I am working!'
How can I finish subprocess and how to close gnome terminal?
If I selected completely wrong way. Can you show me working example?
New version of test_subprocess.py
#!/usr/bin/env python3
# coding=utf-8
import sys
from subprocess import signal
import time
def print_message():
while True:
print('I am working!')
time.sleep(0.5)
if signal.SIGKILL: # it is braking a loop when parent process terminate!
print('I am killing self!')
break
print_message()
Should I do it like above?
You could try the following:
p = Popen(['gnome-terminal', "--", path_to_file])
PIDs = p.pid
os.system("kill {0}".format(PIDs))
Popen.pid The process ID of the child process.
Note that if you set the shell argument to True, this is the process
ID of the spawned shell.
http://docs.python.org/library/subprocess.html
This will at least kill the correct process. Not sure if it will close the terminal.
Edit: to kill the process and close the terminal:
p = Popen(['gnome-terminal', '--disable-factory', '-e', path_to_file], preexec_fn=os.setpgrp)
os.killpg(p.pid, signal.SIGINT)
Credit to https://stackoverflow.com/a/34690644/15793575, whih I modified for your command:
--disable-factory is used to avoid re-using an active terminal so that we can kill newly created terminal via the subprocess handle
os.setpgrp puts gnome-terminal in its own process group so that
os.killpg() could be used to send signal to this group
Popen.pid
The process ID of the child process.
Note that if you set the shell argument to True, this is the process
ID of the spawned shell.
Try setting the shell argument of the Popen constructor to False. (p = Popen(['gnome-terminal', "--", path_to_file]) -> p = Popen(['gnome-terminal', "--", path_to_file], shell=False)). I had a similar issue not long ago - this fixed it for me.
I want code like this:
if True:
run('ABC.PY')
else:
if ScriptRunning('ABC.PY):
stop('ABC.PY')
run('ABC.PY'):
Basically, I want to run a file, let's say abc.py, and based on some conditions. I want to stop it, and run it again from another python script. Is it possible?
I am using Windows.
You can use python Popen objects for running processes in a child process
So run('ABC.PY') would be p = Popen("python 'ABC.PY'")
if ScriptRunning('ABC.PY) would be if p.poll() == None
stop('ABC.PY') would be p.kill()
This is a very basic example for what you are trying to achieve
Please checkout subprocess.Popen docs to fine tune your logic for running the script
import subprocess
import shlex
import time
def run(script):
scriptArgs = shlex.split(script)
commandArgs = ["python"]
commandArgs.extend(scriptArgs)
procHandle = subprocess.Popen(commandArgs, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return procHandle
def isScriptRunning(procHandle):
return procHandle.poll() is None
def stopScript(procHandle):
procHandle.terminate()
time.sleep(5)
# Forcefully terminate the script
if isScriptRunning(procHandle):
procHandle.kill()
def getOutput(procHandle):
# stderr will be redirected to stdout due "stderr=subprocess.STDOUT" argument in Popen call
stdout, _ = procHandle.communicate()
returncode = procHandle.returncode
return returncode, stdout
def main():
procHandle = run("main.py --arg 123")
time.sleep(5)
isScriptRunning(procHandle)
stopScript(procHandle)
print getOutput(procHandle)
if __name__ == "__main__":
main()
One thing that you should be aware about is stdout=subprocess.PIPE.
If your python script has a very large output, the pipes may overflow causing your script to block until .communicate is called over the handle.
To avoid this, pass a file handle to stdout, like this
fileHandle = open("main_output.txt", "w")
subprocess.Popen(..., stdout=fileHandle)
In this way, the output of the python process will be dumped into the file.(You will have to modily the getOutput() function too for this)
import subprocess
process = None
def run_or_rerun(flag):
global process
if flag:
assert(process is None)
process = subprocess.Popen(['python', 'ABC.PY'])
process.wait() # must wait or caller will hang
else:
if process.poll() is None: # it is still running
process.terminate() # terminate process
process = subprocess.Popen(['python', 'ABC.PY']) # rerun
process.wait() # must wait or caller will hang
So basically, i want python to run another programm and wait till that program is not visible in the taskmanger and then continue with the script.
Any Ideas?
As #eryksun suggested, the subprocess module can handle the waiting as well:
import subprocess
process = subprocess.Popen(["notepad.exe"], shell=False)
process.wait()
print ("notepad.exe closed")
You could use something like this, tracking the process id of the opened program:
import subprocess, win32com.client, time
wmi=win32com.client.GetObject('winmgmts:')
process = subprocess.Popen(["notepad.exe"], shell=False)
pid = process.pid
flag = True
while flag:
flag = False
for p in wmi.InstancesOf('win32_process'):
if pid == int(p.Properties_('ProcessId')):
flag = True
time.sleep(.1)
print ("notepad.exe closed")
Output when notepad is closed:
notepad.exe closed
>>>
Here's an example of a simple way to see if something is running on Windows that uses its built-in tasklist command:
import os
import subprocess
target = 'notepad.exe'
results = subprocess.check_output(['tasklist'], universal_newlines=True)
if any(line.startswith(target) for line in results.splitlines()):
print(target, 'is running')
else:
print(target, 'is *not* running')
It can be done with pywinauto:
from pywinauto import Application
app = Application().connect(process=pid) # or connect(title_re="") or other options
app.wait_for_process_exit(timeout=50, retry_interval=0.1)
I'm launching a subprocess with the following command:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
However, when I try to kill using:
p.terminate()
or
p.kill()
The command keeps running in the background, so I was wondering how can I actually terminate the process.
Note that when I run the command with:
p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
It does terminate successfully when issuing the p.terminate().
Use a process group so as to enable sending a signal to all the process in the groups. For that, you should attach a session id to the parent process of the spawned/child processes, which is a shell in your case. This will make it the group leader of the processes. So now, when a signal is sent to the process group leader, it's transmitted to all of the child processes of this group.
Here's the code:
import os
import signal
import subprocess
# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
os.killpg(os.getpgid(pro.pid), signal.SIGTERM) # Send the signal to all the process groups
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
p.kill()
p.kill() ends up killing the shell process and cmd is still running.
I found a convenient fix this by:
p = subprocess.Popen("exec " + cmd, stdout=subprocess.PIPE, shell=True)
This will cause cmd to inherit the shell process, instead of having the shell launch a child process, which does not get killed. p.pid will be the id of your cmd process then.
p.kill() should work.
I don't know what effect this will have on your pipe though.
If you can use psutil, then this works perfectly:
import subprocess
import psutil
def kill(proc_pid):
process = psutil.Process(proc_pid)
for proc in process.children(recursive=True):
proc.kill()
process.kill()
proc = subprocess.Popen(["infinite_app", "param"], shell=True)
try:
proc.wait(timeout=3)
except subprocess.TimeoutExpired:
kill(proc.pid)
I could do it using
from subprocess import Popen
process = Popen(command, shell=True)
Popen("TASKKILL /F /PID {pid} /T".format(pid=process.pid))
it killed the cmd.exe and the program that i gave the command for.
(On Windows)
When shell=True the shell is the child process, and the commands are its children. So any SIGTERM or SIGKILL will kill the shell but not its child processes, and I don't remember a good way to do it.
The best way I can think of is to use shell=False, otherwise when you kill the parent shell process, it will leave a defunct shell process.
None of these answers worked for me so Im leaving the code that did work. In my case even after killing the process with .kill() and getting a .poll() return code the process didn't terminate.
Following the subprocess.Popen documentation:
"...in order to cleanup properly a well-behaved application should kill the child process and finish communication..."
proc = subprocess.Popen(...)
try:
outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
In my case I was missing the proc.communicate() after calling proc.kill(). This cleans the process stdin, stdout ... and does terminate the process.
As Sai said, the shell is the child, so signals are intercepted by it -- best way I've found is to use shell=False and use shlex to split the command line:
if isinstance(command, unicode):
cmd = command.encode('utf8')
args = shlex.split(cmd)
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Then p.kill() and p.terminate() should work how you expect.
Send the signal to all the processes in group
self.proc = Popen(commands,
stdout=PIPE,
stderr=STDOUT,
universal_newlines=True,
preexec_fn=os.setsid)
os.killpg(os.getpgid(self.proc.pid), signal.SIGHUP)
os.killpg(os.getpgid(self.proc.pid), signal.SIGTERM)
There is a very simple way for Python 3.5 or + (Actually tested on Python 3.8)
import subprocess, signal, time
p = subprocess.Popen(['cmd'], shell=True)
time.sleep(5) #Wait 5 secs before killing
p.send_signal(signal.CTRL_C_EVENT)
Then, your code may crash at some point if you have a keyboard input detection, or sth like this. In this case, on the line of code/function where the error is given, just use:
try:
FailingCode #here goes the code which is raising KeyboardInterrupt
except KeyboardInterrupt:
pass
What this code is doing is just sending a "CTRL+C" signal to the running process, what will cause the process to get killed.
Solution that worked for me
if os.name == 'nt': # windows
subprocess.Popen("TASKKILL /F /PID {pid} /T".format(pid=process.pid))
else:
os.kill(process.pid, signal.SIGTERM)
Full blown solution that will kill running process (including subtree) on timeout reached or specific conditions via a callback function.
Works both on windows & Linux, from Python 2.7 up to 3.10 as of this writing.
Install with pip install command_runner
Example for timeout:
from command_runner import command_runner
# Kills ping after 2 seconds
exit_code, output = command_runner('ping 127.0.0.1', shell=True, timeout=2)
Example for specific condition:
Here we'll stop ping if current system time seconds digit is > 5
from time import time
from command_runner import command_runner
def my_condition():
# Arbitrary condition for demo
return True if int(str(int(time()))[-1]) > 5
# Calls my_condition() every second (check_interval) and kills ping if my_condition() returns True
exit_code, output = command_runner('ping 127.0.0.1', shell=True, stop_on=my_condition, check_interval=1)
I started with a simple testcase:
cat foo2.py
#!/usr/bin/python
import subprocess, sys, os
def alert():
subprocess.Popen ("xterm &", shell=True, stdin=None, stdout=None, stderr=None, close_fds=True)
if __name__ == "__main__":
print "hello"
alert ()
os._exit (0)
When I run that code on the command line regularly, it works:
./foo2.py
returns the unix prompt, and the xterm is running in the background.
However, when I run that code with a tee
./foo2.py | tee my.log
I do not get the unix prompt until I close the xterm.
How can I get the python script to exit, but keep the xterm running in the background?
I've seen: Python spawn off a child subprocess, detach, and exit and the activestate recipe mentioned there. I've used that code to create a simple testcase which just opens an xterm in the background:
cat foo.py
#!/usr/bin/python
import subprocess, sys, os
def alert():
subprocess.Popen ("xterm &", shell=True, stdin=None, stdout=None, stderr=None)
def createDaemon():
"""Detach a process from the controlling terminal and run it in the
background as a daemon.
"""
try:
pid = os.fork() # Fork a first child.
except OSError, e:
raise Exception, "%s [%d]" % (e.strerror, e.errno)
if (pid == 0): # The first child.
os.setsid()
try:
pid = os.fork() # Fork a second child.
except OSError, e:
raise Exception, "%s [%d]" % (e.strerror, e.errno)
if (pid == 0): # The second child.
alert ()
else:
os._exit(0) # Exit parent (the first child) of the second child.
else:
os._exit(0) # Exit parent of the first child.
# Close all open file descriptors. This prevents the child from keeping
import resource # Resource usage information.
maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
if (maxfd == resource.RLIM_INFINITY):
maxfd = 1024
# Iterate through and close all file descriptors.
for fd in range(0, maxfd):
try:
os.close(fd)
except OSError: # ERROR, fd wasn't open to begin with (ignored)
pass
os.open(REDIRECT_TO, os.O_RDWR) # standard input (0)
os.dup2(0, 1) # standard output (1)
os.dup2(0, 2) # standard error (2)
return(0)
if __name__ == "__main__":
print "hello"
retCode = createDaemon()
sys.exit (0)
When I run that code on the command line regularly, it works:
./foo.py
returns the unix prompt, and the xterm is running in the background.
However, when I run that code with a tee
./foo.py | tee my.log
I do not get the unix prompt until I close the xterm.
How can I get the python script to exit, but keep the xterm running in the background?
Run nohup xterm & (so that the process is decoupled)