I want a script to start a new process, such that the new process continues running after the initial script exits. I expected that I could use multiprocessing.Process to start a new process, and set daemon=True so that the main script may exit while the created process continues running.
But it seems that the second process is silently terminated when the main script exits. Is this expected behavior, or am I doing something wrong?
From the Python docs:
When a process exits, it attempts to
terminate all of its daemonic child
processes.
This is the expected behavior.
If you are on a unix system, you could use os.fork:
import os
import time
pid=os.fork()
if pid:
# parent
while True:
print("I'm the parent")
time.sleep(0.5)
else:
# child
while True:
print("I'm just a child")
time.sleep(0.5)
Running this creates two processes. You can kill the parent without killing the child.
For example, when you run script you'll see something like:
% script.py
I'm the parent
I'm just a child
I'm the parent
I'm just a child
...
Stop the script with ctrl-Z:
^Z
[1]+ Stopped script.py
Find the process ID number for the parent. It will be the smaller of the two process ID numbers since the parent came first:
% ps axuw | grep script.py
unutbu 6826 0.1 0.1 33792 6388 pts/24 T 15:09 0:00 python /home/unutbu/pybin/script.py
unutbu 6827 0.0 0.1 33792 4352 pts/24 T 15:09 0:00 python /home/unutbu/pybin/script.py
unutbu 6832 0.0 0.0 17472 952 pts/24 S+ 15:09 0:00 grep --color=auto script.py
Kill the parent process:
% kill 6826
Restore script.py to the foreground:
% fg
script.py
Terminated
You'll see the child process is still running:
% I'm just a child
I'm just a child
I'm just a child
...
Kill the child (in a new terminal) with
% kill 6827
Simply use the subprocess module:
import subprocess
subprocess.Popen(["sleep", "60"])
Here is a related question on SO, where one of the answers gives a nice solution to this problem:
"spawning process from python"
If you are on a unix system (using docs):
#!/usr/bin/env python3
import os
import sys
import time
import subprocess
import multiprocessing
from multiprocessing import Process
def to_use_in_separate_process(*args):
print(args)
#check args before using them:
if len(args)>1:
subprocess.call((args[0], args[1]))
print('subprocess called')
def main(apathtofile):
print('checking os')
if os.name == 'posix':
print('os is posix')
multiprocessing.get_context('fork')
p = Process(target=to_use_in_separate_process, args=('xdg-open', apathtofile))
p.run()
print('exiting def main')
if __name__ == '__main__':
#parameter [1] must be some file that can be opened by xdg-open that this
#program uses.
if len(sys.argv)>1:
main(sys.argv[1])
print('we can exit now.')
else:
print('no parameters...')
print('mother program will end now!')
sys.exit(0)
In Ubuntu the following commands keep working even though the python app exit.
url = "https://www.youtube.com/watch?v=t3kcqTE6x4A"
cmd = f"mpv '{url}' && zenity --info --text 'you have watched {url}' &"
os.system(cmd)
Related
I am working on UBUNTU and I have file main.py with a code inside:
#!/usr/bin/env python3
# coding=utf-8
import os
import time
from subprocess import Popen, PIPE, call, signal
base_path = os.path.abspath('')
path_to_file = base_path + '/test_subprocess.py'
p = Popen(['gnome-terminal', "--", path_to_file])
time.sleep(2)
os.kill(p.pid, signal.SIGKILL)
I have test_subprocess.py with code like that:
#!/usr/bin/env python3
# coding=utf-8
import time
def print_message():
while True:
print('I am working!')
time.sleep(0.5)
print_message()
I tried to kill the subprocess but after
os.kill(p.pid, signal.SIGKILL)
subprocess is still working and prints 'I am working!'
How can I finish subprocess and how to close gnome terminal?
If I selected completely wrong way. Can you show me working example?
New version of test_subprocess.py
#!/usr/bin/env python3
# coding=utf-8
import sys
from subprocess import signal
import time
def print_message():
while True:
print('I am working!')
time.sleep(0.5)
if signal.SIGKILL: # it is braking a loop when parent process terminate!
print('I am killing self!')
break
print_message()
Should I do it like above?
You could try the following:
p = Popen(['gnome-terminal', "--", path_to_file])
PIDs = p.pid
os.system("kill {0}".format(PIDs))
Popen.pid The process ID of the child process.
Note that if you set the shell argument to True, this is the process
ID of the spawned shell.
http://docs.python.org/library/subprocess.html
This will at least kill the correct process. Not sure if it will close the terminal.
Edit: to kill the process and close the terminal:
p = Popen(['gnome-terminal', '--disable-factory', '-e', path_to_file], preexec_fn=os.setpgrp)
os.killpg(p.pid, signal.SIGINT)
Credit to https://stackoverflow.com/a/34690644/15793575, whih I modified for your command:
--disable-factory is used to avoid re-using an active terminal so that we can kill newly created terminal via the subprocess handle
os.setpgrp puts gnome-terminal in its own process group so that
os.killpg() could be used to send signal to this group
Popen.pid
The process ID of the child process.
Note that if you set the shell argument to True, this is the process
ID of the spawned shell.
Try setting the shell argument of the Popen constructor to False. (p = Popen(['gnome-terminal', "--", path_to_file]) -> p = Popen(['gnome-terminal', "--", path_to_file], shell=False)). I had a similar issue not long ago - this fixed it for me.
I'm using Windows 10 and Python 3.7.
I ran the following command.
import subprocess
exeFilePath = "C:/Users/test/test.exe"
subprocess.run(exeFilePath)
The .exe file launched with this command, I want to force-quit when the button is clicked or when the function is executed.
Looking at a past question, it has been indicated that the way to force quit is to get a PID and do an OS.kill as follows.
import signal
os.kill(self.p.pid, signal.CTRL_C_EVENT)
However, I don't know how to get the PID of the process started in subprocess.run.
What should I do?
Assign a variable to your subprocess
import os
import signal
import subprocess
exeFilePath = "C:/Users/test/test.exe"
p = subprocess.Popen(exeFilePath)
print(p.pid) # the pid
os.kill(p.pid, signal.SIGTERM) #or signal.SIGKILL
In same cases the process has children
processes. You need to kill all processes to terminate it. In that case you can use psutil
#python -m pip install —user psutil
import psutil
#remember to assign subprocess to a variable
def kills(pid):
'''Kills all process'''
parent = psutil.Process(pid)
for child in parent.children(recursive=True):
child.kill()
parent.kill()
#assumes variable p
kills(p.pid)
This will kill all processes in that PID
I have a script "run.py" that must print "Hello", launch another script "run2.py", and then terminate (do not wait for run2.py to return).
run2.py is not in the local directory and is only required to print "Hello again".
How can I do this?
# run_path = "C:/Program Files (x86)/xxx/run.py"
# run2_path = "//network_share/folder/run2.py"
**run.py**
import os
print("Hello")
# What do I do here?
# os.execl("//network_share/folder/run2.py")
exit()
**run2.py**
print("Hello again")
This seems to work for a script I have in the same folder I'm running this one in.
This should verify that the first script finishes and doesn't linger while the second script runs in its own process. It is possible on some systems, due to their configuration, the child process will terminate when the parent does. But not in this case...
I put more time into this post to add code that shows how to check if the parent process is still running. This would be a good way for the child to ensure it's exited. Also shows how to pass parameters to the child process.
# launch.py
import subprocess as sp
import os
if __name__ == '__main__':
sp.Popen(['ps']) # Print out runniing processes.
print("launch.py's process id is %s." % os.getpid())
# Give child process this one's process ID in the parameters.
sp.Popen(['python3', 'runinproc.py', str(os.getpid())])
# ^^^ This line above anwers the main question of how to kick off a
# child Python script.
print("exiting launch.py")
Other script.
# runinproc.py
import time
import subprocess as sp
import sys
import os
def is_launcher_running():
try:
# This only checks the status of the process. It doesn't
# kill it, or otherwise affect it.
os.kill(int(sys.argv[1]), 0)
except OSError:
return False
else:
return True
if __name__ == '__main__':
print("runinproc.py was launched by process ID %s" % sys.argv[1])
for i in range(100):
if is_launcher_running():
# Is launch.py still running?
print("[[ launch.py is still running... ]]")
sp.Popen(['ps']) # Print out the running processes.
print("going to sleep for 2 seconds...")
time.sleep(2)
Bash output:
Todds-iMac:pyexperiments todd$ python3 launch.py
launch.py process id is 40975.
exiting launch.py
Todds-iMac:pyexperiments todd$ runinproc.py was launched by process ID 40975
going to sleep for 2 seconds...
PID TTY TIME CMD
PID TTY TIME CMD
40866 ttys000 0:00.09 -bash
40866 ttys000 0:00.09 -bash
40977 ttys000 0:00.04 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/C
40977 ttys000 0:00.04 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/C
going to sleep for 2 seconds...
PID TTY TIME CMD
40866 ttys000 0:00.09 -bash
40977 ttys000 0:00.04 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/C
going to sleep for 2 seconds...
PID TTY TIME CMD
40866 ttys000 0:00.09 -bash
40977 ttys000 0:00.04 /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/C
going to sleep for 2 seconds...
Note that the first call to the shell, ps from launch.py is executed after launch.py exited. That's why it doesn't show up in the printed process list.
subprocess is your friend, but if you need to not wait, check out the P_NOWAIT--replacing example code in https://docs.python.org/3/library/subprocess.html
EG:
pid = Popen(["/bin/mycmd", "myarg"]).pid
I don't think .communicate is what you need this time around - isn't it more for waiting?
The cleanest way to do this (since both scripts are written in pure Python) is to import the other script as a module and execute its content, placed within a function:
run.py
import os
import sys
sys.path.append("//network_share/folder/")
import run2
print("Hello")
run2.main()
exit()
run2.py
def main():
print("Hello again")
I'm running an etcd process, which stays active until you kill it. (It doesn't provide a daemon mode option.) I want to detach it so I can keep running more python.
What I would do in the shell;
etcd & next_cmd
I'm using python's sh library, at the enthusiastic recommendation of the whole internet. I'd rather not dip into subprocess or Popen, but I haven't found solutions using those either.
What I want;
sh.etcd(detach=True)
sh.next_cmd()
or
sh.etcd("&")
sh.next_cmd()
Unfortunately detach is not a kwarg and sh treats "&" as a flag to etcd.
Am I missing anything here? What's the good way to do this?
To implement sh's &, avoid cargo cult programming and use subprocess module directly:
import subprocess
etcd = subprocess.Popen('etcd') # continue immediately
next_cmd_returncode = subprocess.call('next_cmd') # wait for it
# ... run more python here ...
etcd.terminate()
etcd.wait()
This ignores exception handling and your talk about "daemon mode" (if you want to implement a daemon in Python; use python-daemon. To run a process as a system service, use whatever your OS provides or a supervisor program such as supervisord).
Author of sh here. I believe you want to use the _bg special keyword parameter http://amoffat.github.io/sh/#background-processes
This will fork your command and return immediately. The process will continue to run even after your script exits.
Note in the following two examples there is a call to
time.sleep(...) to give etcd time to finish starting up before we
send it a request. A real solution would probably involving probing
the API endpoint to see if it was available and looping if not.
Option 1 (abusing the multiprocessing module):
import sh
import requests
import time
from multiprocessing import Process
etcd = Process(target=sh.etcd)
try:
# start etcd
etcd.start()
time.sleep(3)
# do other stuff
r = requests.get('http://localhost:4001/v2/keys/')
print r.text
finally:
etcd.terminate()
This uses the multiprocessing module to handle the mechanics of
spawning a background tasks. Using this model, you won't see the
output from etcd.
Option 2 (tried and true):
import os
import signal
import time
import requests
pid = os.fork()
if pid == 0:
# start etcd
os.execvp('etcd', ['etcd'])
try:
# do other stuff
time.sleep(3)
r = requests.get('http://localhost:4001/v2/keys/')
print r.text
finally:
os.kill(pid, signal.SIGTERM)
This uses the traditional fork and exec model, which works just as
well in Python as it does in C. In this model, the output of etcd
will show up on your console, which may or may not be what you want. You can control this by redirecting stdout and stderr in the child process.
subprocess is easy enough to do this too:
This approach works (python3). The key is using "start_new_session=True"
UPDATE: despite Popen docs saying this works, it does not. I found by forking the child and then doing os.setsid() it works as I want
client.py:
#!/usr/bin/env python3
import time
import subprocess
subprocess.Popen("python3 child.py", shell=True, start_new_session=True)
i = 0
while True:
i += 1
print("demon: %d" % i)
time.sleep(1)
child.py:
#!/usr/bin/env python3
import time
import subprocess
import os
pid = os.fork()
if (pid == 0):
os.setsid()
i = 0
while True:
i += 1
print("child: %d" % i)
time.sleep(1)
if i == 10:
print("child exiting")
break
output:
./client.py
demon: 1
child: 1
demon: 2
child: 2
^CTraceback (most recent call last):
File "./client.py", line 9, in <module>
time.sleep(1)
KeyboardInterrupt
$ child: 3
child: 4
child: 5
child: 6
child: 7
child: 8
child: 9
child: 10
child exiting
Posting this if for no other reason than finding it next time I google the same question:
if os.fork() == 0:
os.close(0)
os.close(1)
os.close(2)
subprocess.Popen(('etcd'),close_fds=True)
sys.exit(0)
Popen close_fds closes the file descriptors other than 0,1,2, so the code closes them explicitly.
I am running on a linux machine a python script which creates a child process using subprocess.check_output() as it follows:
subprocess.check_output(["ls", "-l"], stderr=subprocess.STDOUT)
The problem is that even if the parent process dies, the child is still running.
Is there any way I can kill the child process as well when the parent dies?
Yes, you can achieve this by two methods. Both of them require you to use Popen instead of check_output. The first is a simpler method, using try..finally, as follows:
from contextlib import contextmanager
#contextmanager
def run_and_terminate_process(*args, **kwargs):
try:
p = subprocess.Popen(*args, **kwargs)
yield p
finally:
p.terminate() # send sigterm, or ...
p.kill() # send sigkill
def main():
with run_and_terminate_process(args) as running_proc:
# Your code here, such as running_proc.stdout.readline()
This will catch sigint (keyboard interrupt) and sigterm, but not sigkill (if you kill your script with -9).
The other method is a bit more complex, and uses ctypes' prctl PR_SET_PDEATHSIG. The system will send a signal to the child once the parent exits for any reason (even sigkill).
import signal
import ctypes
libc = ctypes.CDLL("libc.so.6")
def set_pdeathsig(sig = signal.SIGTERM):
def callable():
return libc.prctl(1, sig)
return callable
p = subprocess.Popen(args, preexec_fn = set_pdeathsig(signal.SIGTERM))
Your problem is with using subprocess.check_output - you are correct, you can't get the child PID using that interface. Use Popen instead:
proc = subprocess.Popen(["ls", "-l"], stdout=PIPE, stderr=PIPE)
# Here you can get the PID
global child_pid
child_pid = proc.pid
# Now we can wait for the child to complete
(output, error) = proc.communicate()
if error:
print "error:", error
print "output:", output
To make sure you kill the child on exit:
import os
import signal
def kill_child():
if child_pid is None:
pass
else:
os.kill(child_pid, signal.SIGTERM)
import atexit
atexit.register(kill_child)
Don't know the specifics, but the best way is still to catch errors (and perhaps even all errors) with signal and terminate any remaining processes there.
import signal
import sys
import subprocess
import os
def signal_handler(signal, frame):
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
a = subprocess.check_output(["ls", "-l"], stderr=subprocess.STDOUT)
while 1:
pass # Press Ctrl-C (breaks the application and is catched by signal_handler()
This is just a mockup, you'd need to catch more than just SIGINT but the idea might get you started and you'd need to check for spawned process somehow still.
http://docs.python.org/2/library/os.html#os.kill
http://docs.python.org/2/library/subprocess.html#subprocess.Popen.pid
http://docs.python.org/2/library/subprocess.html#subprocess.Popen.kill
I'd recommend rewriting a personalized version of check_output cause as i just realized check_output is really just for simple debugging etc since you can't interact so much with it during executing..
Rewrite check_output:
from subprocess import Popen, PIPE, STDOUT
from time import sleep, time
def checkOutput(cmd):
a = Popen('ls -l', shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
print(a.pid)
start = time()
while a.poll() == None or time()-start <= 30: #30 sec grace period
sleep(0.25)
if a.poll() == None:
print('Still running, killing')
a.kill()
else:
print('exit code:',a.poll())
output = a.stdout.read()
a.stdout.close()
a.stdin.close()
return output
And do whatever you'd like with it, perhaps store the active executions in a temporary variable and kill them upon exit with signal or other means of intecepting errors/shutdowns of the main loop.
In the end, you still need to catch terminations in the main application in order to safely kill any childs, the best way to approach this is with try & except or signal.
As of Python 3.2 there is a ridiculously simple way to do this:
from subprocess import Popen
with Popen(["sleep", "60"]) as process:
print(f"Just launched server with PID {process.pid}")
I think this will be best for most use cases because it's simple and portable, and it avoids any dependence on global state.
If this solution isn't powerful enough, then I would recommend checking out the other answers and discussion on this question or on Python: how to kill child process(es) when parent dies?, as there are a lot of neat ways to approach the problem that provide different trade-offs around portability, resilience, and simplicity. 😊
Manually you could do this:
ps aux | grep <process name>
get the PID(second column) and
kill -9 <PID>
-9 is to force killing it