I have a C++ program, called C, that is designed to shut down when it receives a SIGINT signal. I've written a Python program P that runs C as a subprocess. I want P to stop C. I tried 3 things and I'd like to know why some of them didn't work.
Attempt #1:
import subprocess
import signal
import os
p = subprocess.Popen(...)
...
os.killpg(p.pid, signal.SIGINT)
This code gives me the error
OSError [Errno 3]: No such process`
even though the p.pid matches the pid displayed by ps.
Attempt #2:
import subprocess
import signal
import os
p = subprocess.Popen(...)
...
os.system('kill -SIGINT %u' % p.pid)
This gives me the error
sh: kill: bad signal`
even though kill -SIGINT <pid> works from the terminal.
Attempt #3:
import subprocess
import signal
import os
p = subprocess.Popen(...)
...
os.system('kill -2 %u' % p.pid)
This works.
My question is, why didn't #1 and #2 work?
Edit: my original assumption was that since the documentation for os.kill() says New in version 2.7: Windows support, I thought that os.kill() is (a) first available in 2.7 and (b) works in Windows. After reading the answers below, I ran os.kill() on Solaris, which I should have done in the first place sorry, and it does work in 2.4. Obviously, the documentation means that Windows support is new in 2.7. Opps.
The first fails because os.killpg kills a process group, identified by its leader; you have a simple process, not a process group. Try os.kill instead. The second fails because the shell builtin kill understands symbolic signals, but the external command on Solaris doesn't (whereas on *BSD and Linux it does); use a numeric signal (SIGINT is 2 on Solaris, or use Python's predefined signal constants from the signal module). That said, use Popen's own interface instead as mentioned by someone else; don't reinvent the wheel, you're liable to create some corners.
The Popen object has a kill() method that you can invoke as well as a terminate() method and a generic send_signal() method.
I would use one of these rather than trying any of the out of band stuff you'd use with the os interface. You've already got a handle to the process, you should use it!
Related
I'm trying to port a shell script to the much more readable python version. The original shell script starts several processes (utilities, monitors, etc.) in the background with "&". How can I achieve the same effect in python? I'd like these processes not to die when the python scripts complete. I am sure it's related to the concept of a daemon somehow, but I couldn't find how to do this easily.
While jkp's solution works, the newer way of doing things (and the way the documentation recommends) is to use the subprocess module. For simple commands its equivalent, but it offers more options if you want to do something complicated.
Example for your case:
import subprocess
subprocess.Popen(["rm","-r","some.file"])
This will run rm -r some.file in the background. Note that calling .communicate() on the object returned from Popen will block until it completes, so don't do that if you want it to run in the background:
import subprocess
ls_output=subprocess.Popen(["sleep", "30"])
ls_output.communicate() # Will block for 30 seconds
See the documentation here.
Also, a point of clarification: "Background" as you use it here is purely a shell concept; technically, what you mean is that you want to spawn a process without blocking while you wait for it to complete. However, I've used "background" here to refer to shell-background-like behavior.
Note: This answer is less current than it was when posted in 2009. Using the subprocess module shown in other answers is now recommended in the docs
(Note that the subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using these functions.)
If you want your process to start in the background you can either use system() and call it in the same way your shell script did, or you can spawn it:
import os
os.spawnl(os.P_DETACH, 'some_long_running_command')
(or, alternatively, you may try the less portable os.P_NOWAIT flag).
See the documentation here.
You probably want the answer to "How to call an external command in Python".
The simplest approach is to use the os.system function, e.g.:
import os
os.system("some_command &")
Basically, whatever you pass to the system function will be executed the same as if you'd passed it to the shell in a script.
I found this here:
On windows (win xp), the parent process will not finish until the longtask.py has finished its work. It is not what you want in CGI-script. The problem is not specific to Python, in PHP community the problems are the same.
The solution is to pass DETACHED_PROCESS Process Creation Flag to the underlying CreateProcess function in win API. If you happen to have installed pywin32 you can import the flag from the win32process module, otherwise you should define it yourself:
DETACHED_PROCESS = 0x00000008
pid = subprocess.Popen([sys.executable, "longtask.py"],
creationflags=DETACHED_PROCESS).pid
Use subprocess.Popen() with the close_fds=True parameter, which will allow the spawned subprocess to be detached from the Python process itself and continue running even after Python exits.
https://gist.github.com/yinjimmy/d6ad0742d03d54518e9f
import os, time, sys, subprocess
if len(sys.argv) == 2:
time.sleep(5)
print 'track end'
if sys.platform == 'darwin':
subprocess.Popen(['say', 'hello'])
else:
print 'main begin'
subprocess.Popen(['python', os.path.realpath(__file__), '0'], close_fds=True)
print 'main end'
Both capture output and run on background with threading
As mentioned on this answer, if you capture the output with stdout= and then try to read(), then the process blocks.
However, there are cases where you need this. For example, I wanted to launch two processes that talk over a port between them, and save their stdout to a log file and stdout.
The threading module allows us to do that.
First, have a look at how to do the output redirection part alone in this question: Python Popen: Write to stdout AND log file simultaneously
Then:
main.py
#!/usr/bin/env python3
import os
import subprocess
import sys
import threading
def output_reader(proc, file):
while True:
byte = proc.stdout.read(1)
if byte:
sys.stdout.buffer.write(byte)
sys.stdout.flush()
file.buffer.write(byte)
else:
break
with subprocess.Popen(['./sleep.py', '0'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc1, \
subprocess.Popen(['./sleep.py', '10'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc2, \
open('log1.log', 'w') as file1, \
open('log2.log', 'w') as file2:
t1 = threading.Thread(target=output_reader, args=(proc1, file1))
t2 = threading.Thread(target=output_reader, args=(proc2, file2))
t1.start()
t2.start()
t1.join()
t2.join()
sleep.py
#!/usr/bin/env python3
import sys
import time
for i in range(4):
print(i + int(sys.argv[1]))
sys.stdout.flush()
time.sleep(0.5)
After running:
./main.py
stdout get updated every 0.5 seconds for every two lines to contain:
0
10
1
11
2
12
3
13
and each log file contains the respective log for a given process.
Inspired by: https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/
Tested on Ubuntu 18.04, Python 3.6.7.
You probably want to start investigating the os module for forking different threads (by opening an interactive session and issuing help(os)). The relevant functions are fork and any of the exec ones. To give you an idea on how to start, put something like this in a function that performs the fork (the function needs to take a list or tuple 'args' as an argument that contains the program's name and its parameters; you may also want to define stdin, out and err for the new thread):
try:
pid = os.fork()
except OSError, e:
## some debug output
sys.exit(1)
if pid == 0:
## eventually use os.putenv(..) to set environment variables
## os.execv strips of args[0] for the arguments
os.execv(args[0], args)
You can use
import os
pid = os.fork()
if pid == 0:
Continue to other code ...
This will make the python process run in background.
I haven't tried this yet but using .pyw files instead of .py files should help. pyw files dosen't have a console so in theory it should not appear and work like a background process.
I asked a question related to this several weeks ago on here:
Python, mpg123 and subprocess not properly using stdin.write or communicate
Thanks to help from there I was able to do what I needed at the time. (Didn't call q, but terminated the subprocess to stop it).## Heading ##
Now though I seem to be in another bit of a mess.
from subprocess import Popen, PIPE, STDOUT
p = Popen(["mpg123", "-C", "test.mp3"], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
#wait a few seconds to enter this, "q" without a newline is how the controls for the player work to quit out if it were ran like "mpg123 -C test.mp3" on the command line
p.communicate(input='q')[0]
much like before, I need this to be able to quit out of mpg123 like it would be with it's standard controls (like press 'q' to quit, or '-' to turn volume down, '+' to turn volume up, etc), now I use the code above, which should theoretically work, and it works with similar programs. Does anyone know of a way I can use the controls built into mpg123 (the one accessible by using "mpg123 -C whatever.mp3") using a subprocess? terminate isn't enough anymore as I will need the controls ^_^
EDIT: Many thanks to abarnert for the amazing answer =)
ok, so the new code is simply a slightly modified version of abarnert's answer, however mpg123 doesn't seem to be accepting the commands
import os
import pty
import sys
import time
pid, fd = os.forkpty()
if pid:
time.sleep(5)
os.write(fd, 'b') #this should've restarted the file
time.sleep(5)
os.write(fd, 'q') #unfortunately doesn't quit here =(
time.sleep(5) # quits after this is finished executing
else:
os.spawnl(os.P_WAIT, '/usr/bin/mpg123', '-C', 'TEST file.mp3')
If you really need the controls, you can't just use Popen.
mpg123 only enables terminal control if its stdin is a tty, not if it's a file or pipe. That's why you get this line in the banner:
Terminal control enabled, press 'h' for listing of keys and functions.
And the whole point of Popen (and subprocess, and the POSIX APIs it's built on) is pipes.
So, what can you do about it?
On linux, you can use the pty module. It may also work on other *nix platforms, but it may not—even if it gets built and included in your stdlib. As the docs say:
Because pseudo-terminal handling is highly platform dependent, there is code to do it only for Linux. (The Linux code is supposed to work on other platforms, but hasn’t been tested yet.)
It definitely runs on *BSD platforms on 2.7 and 3.3, and the example in the docs seem to work on both Mac OS X and FreeBSD… but that's as far as I've checked.
Meanwhile, most POSIX platforms will at least have os.forkpty, and that's not much harder, so here's a trivial program that plays the first 5 seconds of a song passed as its first arg:
import os
import pty
import sys
import time
pid, fd = os.forkpty()
if pid:
time.sleep(5)
os.write(fd, 'q')
else:
os.spawnl(os.P_WAIT, # mode
'/usr/local/bin/mpg123', # path
'/usr/local/bin/mpg123', '-C', sys.argv[1]) # args
Note that I used os.spawnl above. This is probably not what you want in a real program; it's for pedagogic purposes, to encourage you to read the docs (and the corresponding manpages) and understand this family of functions.
As the docs explain, this does not use the PATH environment variable, so you need to specify the full path to the program. You can just use spawnlp instead of spawnl to fix this.
Also, spawn may (in fact, always does, although the docs aren't entirely clear) do another fork to execute the child. This really isn't necessary, but spawn does things that you would need to do manually if you just called exec. If you know what you're doing, you may well want to use execl (or execlp) instead of spawnl.
You can even use most of the functionality in subprocess as long as you're careful (do not create any pipes, and remember that you'll end up doing two forks, so make sure to set up the parent/child relationship properly).
Also notice that you need to pass the path to mpg123 twice—once as the path, and then once as the child program's argv[0]. You could also just pass mpg123 the second time. Or, ideally, look at what ps says when you run it from the shell, and pass that. At any rate, you have to pass something as the argv[0]; otherwise, -C ends up being the argv[0], which means mpg123 won't think you gave it a -C flag to enable control keys, but rather than you renamed it to -C and ran it with no flags…
Anyway, you really do need to read the docs to understand what each of these functions does, instead of just treating it like magic code that you don't understand. So, I intentionally used the simplest possible solution to encourage that.
On Windows, there is no such thing as a pty, and no way to do this at all with the facilities built in to Python. You will need to use one of the various third-party libraries for controlling a cmd.exe console (aka DOS prompt) instead.
Based on abarnert's idea, we can open a pseudo-terminal and pass it to subprocess.
import os
import pty
import subprocess
import time
master, slave = os.openpty()
p = subprocess.Popen(['mpg123', '-C', 'music.mp3'], stdin=master)
time.sleep(3)
os.write(slave, 's')
time.sleep(3)
os.write(slave, 's')
time.sleep(6)
os.write(slave, 'q')
I am currently working on a wrapper for a dedicated server running in the shell. The wrapper spawns the server process via subprocess and observes and reacts to its output.
The dedicated server must be explicitly given a command to shut down gracefully. Thus, CTRL-C must not reach the server process.
If I capture the KeyboardInterrupt exception or overwrite the SIGINT-handler in python, the server process still receives the CTRL-C and stops immediately.
So my question is:
How to prevent subprocesses from receiving CTRL-C / Control-C / SIGINT?
Somebody in the #python IRC-Channel (Freenode) helped me by pointing out the preexec_fn parameter of subprocess.Popen(...):
If preexec_fn is set to a callable
object, this object will be called in
the child process just before the
child is executed. (Unix only)
Thus, the following code solves the problem (UNIX only):
import subprocess
import signal
def preexec_function():
# Ignore the SIGINT signal by setting the handler to the standard
# signal handler SIG_IGN.
signal.signal(signal.SIGINT, signal.SIG_IGN)
my_process = subprocess.Popen(
["my_executable"],
preexec_fn = preexec_function
)
Note: The signal is actually not prevented from reaching the subprocess. Instead, the preexec_fn above overwrites the signal's default handler so that the signal is ignored. Thus, this solution may not work if the subprocess overwrites the SIGINT handler again.
Another note: This solution works for all sorts of subprocesses, i.e. it is not restricted to subprocesses written in Python, too. For example the dedicated server I am writing my wrapper for is in fact written in Java.
Combining some of other answers that will do the trick - no signal sent to main app will be forwarded to the subprocess.
import os
from subprocess import Popen
def preexec(): # Don't forward signals.
os.setpgrp()
Popen('whatever', preexec_fn = preexec)
you can do something like this to make it work in windows and unix:
import subprocess
import sys
def pre_exec():
# To ignore CTRL+C signal in the new process
signal.signal(signal.SIGINT, signal.SIG_IGN)
if sys.platform.startswith('win'):
#https://msdn.microsoft.com/en-us/library/windows/desktop/ms684863(v=vs.85).aspx
#CREATE_NEW_PROCESS_GROUP=0x00000200 -> If this flag is specified, CTRL+C signals will be disabled
my_sub_process=subprocess.Popen(["executable"], creationflags=0x00000200)
else:
my_sub_process=subprocess.Popen(["executable"], preexec_fn = pre_exec)
After an hour of various attempts, this works for me:
process = subprocess.Popen(["someprocess"], creationflags=subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP)
It's solution for windows.
Try setting SIGINT to be ignored before spawning the subprocess (reset it to default behavior afterward).
If that doesn't work, you'll need to read up on job control and learn how to put a process in its own background process group, so that ^C doesn't even cause the kernel to send the signal to it in the first place. (May not be possible in Python without writing C helpers.)
See also this older question.
I want to get screenshots of a webpage in Python. For this I am using http://github.com/AdamN/python-webkit2png/ .
newArgs = ["xvfb-run", "--server-args=-screen 0, 640x480x24", sys.argv[0]]
for i in range(1, len(sys.argv)):
if sys.argv[i] not in ["-x", "--xvfb"]:
newArgs.append(sys.argv[i])
logging.debug("Executing %s" % " ".join(newArgs))
os.execvp(newArgs[0], newArgs)
Basically calls xvfb-run with the correct args. But man xvfb says:
Note that the demo X clients used in the above examples will not exit on their own, so they will have to be killed before xvfb-run will exit.
So that means that this script will <????> if this whole thing is in a loop, (To get multiple screenshots) unless the X server is killed. How can I do that?
The documentation for os.execvp states:
These functions all execute a new
program, replacing the current
process; they do not return. [..]
So after calling os.execvp no other statement in the program will be executed. You may want to use subprocess.Popen instead:
The subprocess module allows you to
spawn new processes, connect to their
input/output/error pipes, and obtain
their return codes. This module
intends to replace several other,
older modules and functions, such as:
Using subprocess.Popen, the code to run xlogo in the virtual framebuffer X server becomes:
import subprocess
xvfb_args = ['xvfb-run', '--server-args=-screen 0, 640x480x24', 'xlogo']
process = subprocess.Popen(xvfb_args)
Now the problem is that xvfb-run launches Xvfb in a background process. Calling process.kill() will not kill Xvfb (at least not on my machine...). I have been fiddling around with this a bit, and so far the only thing that works for me is:
import os
import signal
import subprocess
SERVER_NUM = 99 # 99 is the default used by xvfb-run; you can leave this out.
xvfb_args = ['xvfb-run', '--server-num=%d' % SERVER_NUM,
'--server-args=-screen 0, 640x480x24', 'xlogo']
subprocess.Popen(xvfb_args)
# ... do whatever you want to do here...
pid = int(open('/tmp/.X%s-lock' % SERVER_NUM).read().strip())
os.kill(pid, signal.SIGINT)
So this code reads the process ID of Xvfb from /tmp/.X99-lock and sends the process an interrupt. It works, but does yield an error message every now and then (I suppose you can ignore it, though). Hopefully somebody else can provide a more elegant solution. Cheers.
I'm trying to start up a child process and get its output on Linux from Python using the subprocess module:
#!/usr/bin/python2.4
import subprocess
p = subprocess.Popen(['ls', '-l', '/etc'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = p.communicate()
However, I experience some flakiness: sometimes, p.communicate() would throw
OSError: [Errno 10] No child processes
What can cause this exception? Is there any non-determinism or race condition here that can cause flakiness?
Are you intercepting SIGCHLD in the script? If you are then Popen will not run as expected since it relies on it's own handler for that signal.
You can check for SIGCHLD handlers by commenting out the Popen call and then running:
strace python <your_script.py> | grep SIGCHLD
if you see something similar to:
rt_sigaction(SIGCHLD, ...)
then, you are in trouble. You need to disable the handler prior to calling Popen and then resetting it after communicate is done (this might introduce a race conditions so beware).
signal.signal(SIGCHLD, handler)
...
signal.signal(SIGCHLD, signal.SIG_DFL)
'''
now you can go wild with Popen.
WARNING!!! during this time no signals will be delivered to handler
'''
...
signal.signal(SIGCHLD, handler)
There is a python bug reported on this and as far as I see it hasn't been resolved yet:
http://bugs.python.org/issue9127
Hope that helps.
You might be running into the bug mentioned here: http://bugs.python.org/issue1731717
I'm not able to reproduce this on my Python (2.4.6-1ubuntu3). How are you running your script? How often does this occur?
I ran into this problem using Python 2.6.4 which I built into my home directory (because I don't want to upgrade the "built-in" Python on the machine).
I worked around it by replacing subprocess.Popen() with (the deprecated) os.popen3().