I'm running an etcd process, which stays active until you kill it. (It doesn't provide a daemon mode option.) I want to detach it so I can keep running more python.
What I would do in the shell;
etcd & next_cmd
I'm using python's sh library, at the enthusiastic recommendation of the whole internet. I'd rather not dip into subprocess or Popen, but I haven't found solutions using those either.
What I want;
sh.etcd(detach=True)
sh.next_cmd()
or
sh.etcd("&")
sh.next_cmd()
Unfortunately detach is not a kwarg and sh treats "&" as a flag to etcd.
Am I missing anything here? What's the good way to do this?
To implement sh's &, avoid cargo cult programming and use subprocess module directly:
import subprocess
etcd = subprocess.Popen('etcd') # continue immediately
next_cmd_returncode = subprocess.call('next_cmd') # wait for it
# ... run more python here ...
etcd.terminate()
etcd.wait()
This ignores exception handling and your talk about "daemon mode" (if you want to implement a daemon in Python; use python-daemon. To run a process as a system service, use whatever your OS provides or a supervisor program such as supervisord).
Author of sh here. I believe you want to use the _bg special keyword parameter http://amoffat.github.io/sh/#background-processes
This will fork your command and return immediately. The process will continue to run even after your script exits.
Note in the following two examples there is a call to
time.sleep(...) to give etcd time to finish starting up before we
send it a request. A real solution would probably involving probing
the API endpoint to see if it was available and looping if not.
Option 1 (abusing the multiprocessing module):
import sh
import requests
import time
from multiprocessing import Process
etcd = Process(target=sh.etcd)
try:
# start etcd
etcd.start()
time.sleep(3)
# do other stuff
r = requests.get('http://localhost:4001/v2/keys/')
print r.text
finally:
etcd.terminate()
This uses the multiprocessing module to handle the mechanics of
spawning a background tasks. Using this model, you won't see the
output from etcd.
Option 2 (tried and true):
import os
import signal
import time
import requests
pid = os.fork()
if pid == 0:
# start etcd
os.execvp('etcd', ['etcd'])
try:
# do other stuff
time.sleep(3)
r = requests.get('http://localhost:4001/v2/keys/')
print r.text
finally:
os.kill(pid, signal.SIGTERM)
This uses the traditional fork and exec model, which works just as
well in Python as it does in C. In this model, the output of etcd
will show up on your console, which may or may not be what you want. You can control this by redirecting stdout and stderr in the child process.
subprocess is easy enough to do this too:
This approach works (python3). The key is using "start_new_session=True"
UPDATE: despite Popen docs saying this works, it does not. I found by forking the child and then doing os.setsid() it works as I want
client.py:
#!/usr/bin/env python3
import time
import subprocess
subprocess.Popen("python3 child.py", shell=True, start_new_session=True)
i = 0
while True:
i += 1
print("demon: %d" % i)
time.sleep(1)
child.py:
#!/usr/bin/env python3
import time
import subprocess
import os
pid = os.fork()
if (pid == 0):
os.setsid()
i = 0
while True:
i += 1
print("child: %d" % i)
time.sleep(1)
if i == 10:
print("child exiting")
break
output:
./client.py
demon: 1
child: 1
demon: 2
child: 2
^CTraceback (most recent call last):
File "./client.py", line 9, in <module>
time.sleep(1)
KeyboardInterrupt
$ child: 3
child: 4
child: 5
child: 6
child: 7
child: 8
child: 9
child: 10
child exiting
Posting this if for no other reason than finding it next time I google the same question:
if os.fork() == 0:
os.close(0)
os.close(1)
os.close(2)
subprocess.Popen(('etcd'),close_fds=True)
sys.exit(0)
Popen close_fds closes the file descriptors other than 0,1,2, so the code closes them explicitly.
Related
So basically, i want python to run another programm and wait till that program is not visible in the taskmanger and then continue with the script.
Any Ideas?
As #eryksun suggested, the subprocess module can handle the waiting as well:
import subprocess
process = subprocess.Popen(["notepad.exe"], shell=False)
process.wait()
print ("notepad.exe closed")
You could use something like this, tracking the process id of the opened program:
import subprocess, win32com.client, time
wmi=win32com.client.GetObject('winmgmts:')
process = subprocess.Popen(["notepad.exe"], shell=False)
pid = process.pid
flag = True
while flag:
flag = False
for p in wmi.InstancesOf('win32_process'):
if pid == int(p.Properties_('ProcessId')):
flag = True
time.sleep(.1)
print ("notepad.exe closed")
Output when notepad is closed:
notepad.exe closed
>>>
Here's an example of a simple way to see if something is running on Windows that uses its built-in tasklist command:
import os
import subprocess
target = 'notepad.exe'
results = subprocess.check_output(['tasklist'], universal_newlines=True)
if any(line.startswith(target) for line in results.splitlines()):
print(target, 'is running')
else:
print(target, 'is *not* running')
It can be done with pywinauto:
from pywinauto import Application
app = Application().connect(process=pid) # or connect(title_re="") or other options
app.wait_for_process_exit(timeout=50, retry_interval=0.1)
This is the current code I have right now and I'm wondering how I can get it to kill the pid.
import commands, signal
stuid = commands.getoutput("pgrep student")
deaid = commands.getoutput("pgrep daemon")
print stuid
os.kill(stuid, signal.SIGKILL)
print deaid
os.kill(deaid, signal.SIGKILL)
Edit:
So, in the end, I just used os.system to get the terminal to run the kill command then place the pid after kill.
import commands, os
stuid = commands.getoutput("pgrep student")
deaid = commands.getoutput("pgrep daemon")
print stuid
os.system("kill "+stuid)
print deaid
os.system("kill "+deaid)
Overall this is my end result. Hope this helps people in the future.
Read this answer.
BTW a more pythonic solution may be this:
import re
import psutil
convicted = re.compile(r'student|daemon')
for p in psutil.process_iter():
if convicted.search(p.name):
p.terminate()
Edit: To be more accurate I changed the line p.kill() to p.terminate(). The common kill in bash is actually the same as p.terminate() (it sends the TERM signal). But the p.kill() corresponds to kill -9 in bash (it sends the KILL signal).
On Windows there is a WinAPI: FindWindow that you can use to get window handle of a existing window and use this handle to send message to it. Is there a python module can do that too? Find a window & communicate with it?
If this module do exist, could the same mechainsm be able applied on Ubuntu too?
Thanks a lot!
You can execute your commands with a subprocess:
import subprocess
import time
process = subprocess.Popen("echo 'start' & sleep 60 & echo 'stop'", shell=True)
time.sleep(60) # Maybe you want a timer...
The you have two options of closing, use terminate or kill methods in the Popen returned object or simulate a Ctrl. + C
import subprocess
import time
process = subprocess.Popen(cmd, shell=True)
time.sleep(5)
process.terminate() # Or kill
Simulate de ctrl + c:
import subprocess
import time
import os
import signal
process = subprocess.Popen(cmd, shell=True)
time.sleep(5)
os.kill(process.pid, signal.SIGINT) # Ctrl. + C
If you want to get the output you can use:
process.communicate()[0].strip()
Which gives you a string.
If you want a console GUI, you can use a command like:
gnome-terminal -x sh yourCommand
Or the equivalent for the terminal you have installed.
I am running on a linux machine a python script which creates a child process using subprocess.check_output() as it follows:
subprocess.check_output(["ls", "-l"], stderr=subprocess.STDOUT)
The problem is that even if the parent process dies, the child is still running.
Is there any way I can kill the child process as well when the parent dies?
Yes, you can achieve this by two methods. Both of them require you to use Popen instead of check_output. The first is a simpler method, using try..finally, as follows:
from contextlib import contextmanager
#contextmanager
def run_and_terminate_process(*args, **kwargs):
try:
p = subprocess.Popen(*args, **kwargs)
yield p
finally:
p.terminate() # send sigterm, or ...
p.kill() # send sigkill
def main():
with run_and_terminate_process(args) as running_proc:
# Your code here, such as running_proc.stdout.readline()
This will catch sigint (keyboard interrupt) and sigterm, but not sigkill (if you kill your script with -9).
The other method is a bit more complex, and uses ctypes' prctl PR_SET_PDEATHSIG. The system will send a signal to the child once the parent exits for any reason (even sigkill).
import signal
import ctypes
libc = ctypes.CDLL("libc.so.6")
def set_pdeathsig(sig = signal.SIGTERM):
def callable():
return libc.prctl(1, sig)
return callable
p = subprocess.Popen(args, preexec_fn = set_pdeathsig(signal.SIGTERM))
Your problem is with using subprocess.check_output - you are correct, you can't get the child PID using that interface. Use Popen instead:
proc = subprocess.Popen(["ls", "-l"], stdout=PIPE, stderr=PIPE)
# Here you can get the PID
global child_pid
child_pid = proc.pid
# Now we can wait for the child to complete
(output, error) = proc.communicate()
if error:
print "error:", error
print "output:", output
To make sure you kill the child on exit:
import os
import signal
def kill_child():
if child_pid is None:
pass
else:
os.kill(child_pid, signal.SIGTERM)
import atexit
atexit.register(kill_child)
Don't know the specifics, but the best way is still to catch errors (and perhaps even all errors) with signal and terminate any remaining processes there.
import signal
import sys
import subprocess
import os
def signal_handler(signal, frame):
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
a = subprocess.check_output(["ls", "-l"], stderr=subprocess.STDOUT)
while 1:
pass # Press Ctrl-C (breaks the application and is catched by signal_handler()
This is just a mockup, you'd need to catch more than just SIGINT but the idea might get you started and you'd need to check for spawned process somehow still.
http://docs.python.org/2/library/os.html#os.kill
http://docs.python.org/2/library/subprocess.html#subprocess.Popen.pid
http://docs.python.org/2/library/subprocess.html#subprocess.Popen.kill
I'd recommend rewriting a personalized version of check_output cause as i just realized check_output is really just for simple debugging etc since you can't interact so much with it during executing..
Rewrite check_output:
from subprocess import Popen, PIPE, STDOUT
from time import sleep, time
def checkOutput(cmd):
a = Popen('ls -l', shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
print(a.pid)
start = time()
while a.poll() == None or time()-start <= 30: #30 sec grace period
sleep(0.25)
if a.poll() == None:
print('Still running, killing')
a.kill()
else:
print('exit code:',a.poll())
output = a.stdout.read()
a.stdout.close()
a.stdin.close()
return output
And do whatever you'd like with it, perhaps store the active executions in a temporary variable and kill them upon exit with signal or other means of intecepting errors/shutdowns of the main loop.
In the end, you still need to catch terminations in the main application in order to safely kill any childs, the best way to approach this is with try & except or signal.
As of Python 3.2 there is a ridiculously simple way to do this:
from subprocess import Popen
with Popen(["sleep", "60"]) as process:
print(f"Just launched server with PID {process.pid}")
I think this will be best for most use cases because it's simple and portable, and it avoids any dependence on global state.
If this solution isn't powerful enough, then I would recommend checking out the other answers and discussion on this question or on Python: how to kill child process(es) when parent dies?, as there are a lot of neat ways to approach the problem that provide different trade-offs around portability, resilience, and simplicity. 😊
Manually you could do this:
ps aux | grep <process name>
get the PID(second column) and
kill -9 <PID>
-9 is to force killing it
I want a script to start a new process, such that the new process continues running after the initial script exits. I expected that I could use multiprocessing.Process to start a new process, and set daemon=True so that the main script may exit while the created process continues running.
But it seems that the second process is silently terminated when the main script exits. Is this expected behavior, or am I doing something wrong?
From the Python docs:
When a process exits, it attempts to
terminate all of its daemonic child
processes.
This is the expected behavior.
If you are on a unix system, you could use os.fork:
import os
import time
pid=os.fork()
if pid:
# parent
while True:
print("I'm the parent")
time.sleep(0.5)
else:
# child
while True:
print("I'm just a child")
time.sleep(0.5)
Running this creates two processes. You can kill the parent without killing the child.
For example, when you run script you'll see something like:
% script.py
I'm the parent
I'm just a child
I'm the parent
I'm just a child
...
Stop the script with ctrl-Z:
^Z
[1]+ Stopped script.py
Find the process ID number for the parent. It will be the smaller of the two process ID numbers since the parent came first:
% ps axuw | grep script.py
unutbu 6826 0.1 0.1 33792 6388 pts/24 T 15:09 0:00 python /home/unutbu/pybin/script.py
unutbu 6827 0.0 0.1 33792 4352 pts/24 T 15:09 0:00 python /home/unutbu/pybin/script.py
unutbu 6832 0.0 0.0 17472 952 pts/24 S+ 15:09 0:00 grep --color=auto script.py
Kill the parent process:
% kill 6826
Restore script.py to the foreground:
% fg
script.py
Terminated
You'll see the child process is still running:
% I'm just a child
I'm just a child
I'm just a child
...
Kill the child (in a new terminal) with
% kill 6827
Simply use the subprocess module:
import subprocess
subprocess.Popen(["sleep", "60"])
Here is a related question on SO, where one of the answers gives a nice solution to this problem:
"spawning process from python"
If you are on a unix system (using docs):
#!/usr/bin/env python3
import os
import sys
import time
import subprocess
import multiprocessing
from multiprocessing import Process
def to_use_in_separate_process(*args):
print(args)
#check args before using them:
if len(args)>1:
subprocess.call((args[0], args[1]))
print('subprocess called')
def main(apathtofile):
print('checking os')
if os.name == 'posix':
print('os is posix')
multiprocessing.get_context('fork')
p = Process(target=to_use_in_separate_process, args=('xdg-open', apathtofile))
p.run()
print('exiting def main')
if __name__ == '__main__':
#parameter [1] must be some file that can be opened by xdg-open that this
#program uses.
if len(sys.argv)>1:
main(sys.argv[1])
print('we can exit now.')
else:
print('no parameters...')
print('mother program will end now!')
sys.exit(0)
In Ubuntu the following commands keep working even though the python app exit.
url = "https://www.youtube.com/watch?v=t3kcqTE6x4A"
cmd = f"mpv '{url}' && zenity --info --text 'you have watched {url}' &"
os.system(cmd)