I asked a question related to this several weeks ago on here:
Python, mpg123 and subprocess not properly using stdin.write or communicate
Thanks to help from there I was able to do what I needed at the time. (Didn't call q, but terminated the subprocess to stop it).## Heading ##
Now though I seem to be in another bit of a mess.
from subprocess import Popen, PIPE, STDOUT
p = Popen(["mpg123", "-C", "test.mp3"], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
#wait a few seconds to enter this, "q" without a newline is how the controls for the player work to quit out if it were ran like "mpg123 -C test.mp3" on the command line
p.communicate(input='q')[0]
much like before, I need this to be able to quit out of mpg123 like it would be with it's standard controls (like press 'q' to quit, or '-' to turn volume down, '+' to turn volume up, etc), now I use the code above, which should theoretically work, and it works with similar programs. Does anyone know of a way I can use the controls built into mpg123 (the one accessible by using "mpg123 -C whatever.mp3") using a subprocess? terminate isn't enough anymore as I will need the controls ^_^
EDIT: Many thanks to abarnert for the amazing answer =)
ok, so the new code is simply a slightly modified version of abarnert's answer, however mpg123 doesn't seem to be accepting the commands
import os
import pty
import sys
import time
pid, fd = os.forkpty()
if pid:
time.sleep(5)
os.write(fd, 'b') #this should've restarted the file
time.sleep(5)
os.write(fd, 'q') #unfortunately doesn't quit here =(
time.sleep(5) # quits after this is finished executing
else:
os.spawnl(os.P_WAIT, '/usr/bin/mpg123', '-C', 'TEST file.mp3')
If you really need the controls, you can't just use Popen.
mpg123 only enables terminal control if its stdin is a tty, not if it's a file or pipe. That's why you get this line in the banner:
Terminal control enabled, press 'h' for listing of keys and functions.
And the whole point of Popen (and subprocess, and the POSIX APIs it's built on) is pipes.
So, what can you do about it?
On linux, you can use the pty module. It may also work on other *nix platforms, but it may not—even if it gets built and included in your stdlib. As the docs say:
Because pseudo-terminal handling is highly platform dependent, there is code to do it only for Linux. (The Linux code is supposed to work on other platforms, but hasn’t been tested yet.)
It definitely runs on *BSD platforms on 2.7 and 3.3, and the example in the docs seem to work on both Mac OS X and FreeBSD… but that's as far as I've checked.
Meanwhile, most POSIX platforms will at least have os.forkpty, and that's not much harder, so here's a trivial program that plays the first 5 seconds of a song passed as its first arg:
import os
import pty
import sys
import time
pid, fd = os.forkpty()
if pid:
time.sleep(5)
os.write(fd, 'q')
else:
os.spawnl(os.P_WAIT, # mode
'/usr/local/bin/mpg123', # path
'/usr/local/bin/mpg123', '-C', sys.argv[1]) # args
Note that I used os.spawnl above. This is probably not what you want in a real program; it's for pedagogic purposes, to encourage you to read the docs (and the corresponding manpages) and understand this family of functions.
As the docs explain, this does not use the PATH environment variable, so you need to specify the full path to the program. You can just use spawnlp instead of spawnl to fix this.
Also, spawn may (in fact, always does, although the docs aren't entirely clear) do another fork to execute the child. This really isn't necessary, but spawn does things that you would need to do manually if you just called exec. If you know what you're doing, you may well want to use execl (or execlp) instead of spawnl.
You can even use most of the functionality in subprocess as long as you're careful (do not create any pipes, and remember that you'll end up doing two forks, so make sure to set up the parent/child relationship properly).
Also notice that you need to pass the path to mpg123 twice—once as the path, and then once as the child program's argv[0]. You could also just pass mpg123 the second time. Or, ideally, look at what ps says when you run it from the shell, and pass that. At any rate, you have to pass something as the argv[0]; otherwise, -C ends up being the argv[0], which means mpg123 won't think you gave it a -C flag to enable control keys, but rather than you renamed it to -C and ran it with no flags…
Anyway, you really do need to read the docs to understand what each of these functions does, instead of just treating it like magic code that you don't understand. So, I intentionally used the simplest possible solution to encourage that.
On Windows, there is no such thing as a pty, and no way to do this at all with the facilities built in to Python. You will need to use one of the various third-party libraries for controlling a cmd.exe console (aka DOS prompt) instead.
Based on abarnert's idea, we can open a pseudo-terminal and pass it to subprocess.
import os
import pty
import subprocess
import time
master, slave = os.openpty()
p = subprocess.Popen(['mpg123', '-C', 'music.mp3'], stdin=master)
time.sleep(3)
os.write(slave, 's')
time.sleep(3)
os.write(slave, 's')
time.sleep(6)
os.write(slave, 'q')
Related
I am dealing with a Python script which does, after some preparation work, launch ssh. My script is actually a small CLI tool. On Unix-like systems, at the end of its life, the Python script replaces itself with the ssh client, so the user can now interact with ssh directly (i.e. run arbitrary commands on the remote machine etc):
os.execvpe('ssh', ['ssh', '-o', 'foo', 'user#host'], os.environ)
Positive surprise & side-note in case you are wondering: Windows 10 actually has a native version of OpenSSH now built-in, so there is a ssh command on this platform.
os.execvpe is present in the Python standard library on Windows, but it does not replace the original (Python) process. The situation is ... somewhat complicated: 1, 2, 3. Bottom line: Windows does not implement the corresponding POSIX semantics for replacing a running process.
The common wisdom is to use subprocess.Popen instead, ok, effectively creating a child process. I can launch the child so that the parent keeps running OR I can launch the child while the parent dies (I think that Windows does support the latter just like Unix-like systems). Either way, the user can not interact with the child in the command line.
Assuming that I keep the parent alive, I now have to write a ton of code to pass user I/O to/from the child through the parent, like so for instance. The latter involves managing streams and even threads, depending on how well it is supposed to behave - a lot of places for potential issues and breakages down the road. I do not like to do this (if I can avoid it).
How can I efficiently replace os.execvpe on Windows in the described scenario?
EDIT (1): Bits and pieces, which may be relevant ...
Handle Inheritance I
Handle Inheritance II
STARTUPINFO in Windows
STARTUPINFO in Windows - for Python
I guess it depends on figuring out how to correctly configure a STARTUPINFO object before passing it into Popen. A command line can in fact be inherited in Windows.
EDIT (2): A partial solution via pywin32 - ssh opens into a second, new cmd window and can be interacted with. The original shell with Python remains open, Python itself quits:
from win32.Demos.winprocess import Process
from shlex import join
Process(join(['ssh', '-o', 'foo', 'user#host']))
A partial and incomplete solution looks about as follows, see TODO comments:
import win32api, win32process, win32con
from shlex import join
si = win32process.STARTUPINFO()
# TODO fix flags
si.dwFlags = win32con.STARTF_USESTDHANDLES ^ win32con.STARTF_USESHOWWINDOW
# inherit stdin, stdout and stderr
si.hStdInput = win32api.GetStdHandle(win32api.STD_INPUT_HANDLE)
si.hStdOutput = win32api.GetStdHandle(win32api.STD_OUTPUT_HANDLE)
si.hStdError = win32api.GetStdHandle(win32api.STD_ERROR_HANDLE)
# TODO fix value?
si.wShowWindow = 1
# TODO set values?
# si.dwX, si.dwY = ...
# si.dwXSize, si.dwYSize = ...
# si.lpDesktop = ...
procArgs = (
None, # appName
join(['ssh', '-o', 'foo', 'user#host']), # commandLine
None, # processAttributes
None, # threadAttributes
1, # bInheritHandles TODO ?
win32process.CREATE_NEW_CONSOLE, # dwCreationFlags
None, # newEnvironment
None, # currentDirectory
si, # startupinfo
)
procHandles = win32process.CreateProcess(*procArgs) # run ...
ssh opens into a second, new cmd.exe window and can be interacted with. The original cmd.exe window with Python in it remains open, Python itself quits, returning control to cmd.exe itself. It is usable, although inconsistent and ugly.
I guess it comes down to configuring win32process.STARTUPINFO correctly, but even after heaving read tons of documentation on it, I am somehow failing to make sense of it ...
You can use subprocess.Popen or or subprocess.call function instead of os.execvpe. They have flag shell which ensures that child process can get stdin.
I have tried in windows using following code:
import os
import subprocess
subprocess.Popen('ssh -o foo user#host', shell=True, env=os.environ)
And it works.
I'm trying to port a shell script to the much more readable python version. The original shell script starts several processes (utilities, monitors, etc.) in the background with "&". How can I achieve the same effect in python? I'd like these processes not to die when the python scripts complete. I am sure it's related to the concept of a daemon somehow, but I couldn't find how to do this easily.
While jkp's solution works, the newer way of doing things (and the way the documentation recommends) is to use the subprocess module. For simple commands its equivalent, but it offers more options if you want to do something complicated.
Example for your case:
import subprocess
subprocess.Popen(["rm","-r","some.file"])
This will run rm -r some.file in the background. Note that calling .communicate() on the object returned from Popen will block until it completes, so don't do that if you want it to run in the background:
import subprocess
ls_output=subprocess.Popen(["sleep", "30"])
ls_output.communicate() # Will block for 30 seconds
See the documentation here.
Also, a point of clarification: "Background" as you use it here is purely a shell concept; technically, what you mean is that you want to spawn a process without blocking while you wait for it to complete. However, I've used "background" here to refer to shell-background-like behavior.
Note: This answer is less current than it was when posted in 2009. Using the subprocess module shown in other answers is now recommended in the docs
(Note that the subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using these functions.)
If you want your process to start in the background you can either use system() and call it in the same way your shell script did, or you can spawn it:
import os
os.spawnl(os.P_DETACH, 'some_long_running_command')
(or, alternatively, you may try the less portable os.P_NOWAIT flag).
See the documentation here.
You probably want the answer to "How to call an external command in Python".
The simplest approach is to use the os.system function, e.g.:
import os
os.system("some_command &")
Basically, whatever you pass to the system function will be executed the same as if you'd passed it to the shell in a script.
I found this here:
On windows (win xp), the parent process will not finish until the longtask.py has finished its work. It is not what you want in CGI-script. The problem is not specific to Python, in PHP community the problems are the same.
The solution is to pass DETACHED_PROCESS Process Creation Flag to the underlying CreateProcess function in win API. If you happen to have installed pywin32 you can import the flag from the win32process module, otherwise you should define it yourself:
DETACHED_PROCESS = 0x00000008
pid = subprocess.Popen([sys.executable, "longtask.py"],
creationflags=DETACHED_PROCESS).pid
Use subprocess.Popen() with the close_fds=True parameter, which will allow the spawned subprocess to be detached from the Python process itself and continue running even after Python exits.
https://gist.github.com/yinjimmy/d6ad0742d03d54518e9f
import os, time, sys, subprocess
if len(sys.argv) == 2:
time.sleep(5)
print 'track end'
if sys.platform == 'darwin':
subprocess.Popen(['say', 'hello'])
else:
print 'main begin'
subprocess.Popen(['python', os.path.realpath(__file__), '0'], close_fds=True)
print 'main end'
Both capture output and run on background with threading
As mentioned on this answer, if you capture the output with stdout= and then try to read(), then the process blocks.
However, there are cases where you need this. For example, I wanted to launch two processes that talk over a port between them, and save their stdout to a log file and stdout.
The threading module allows us to do that.
First, have a look at how to do the output redirection part alone in this question: Python Popen: Write to stdout AND log file simultaneously
Then:
main.py
#!/usr/bin/env python3
import os
import subprocess
import sys
import threading
def output_reader(proc, file):
while True:
byte = proc.stdout.read(1)
if byte:
sys.stdout.buffer.write(byte)
sys.stdout.flush()
file.buffer.write(byte)
else:
break
with subprocess.Popen(['./sleep.py', '0'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc1, \
subprocess.Popen(['./sleep.py', '10'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc2, \
open('log1.log', 'w') as file1, \
open('log2.log', 'w') as file2:
t1 = threading.Thread(target=output_reader, args=(proc1, file1))
t2 = threading.Thread(target=output_reader, args=(proc2, file2))
t1.start()
t2.start()
t1.join()
t2.join()
sleep.py
#!/usr/bin/env python3
import sys
import time
for i in range(4):
print(i + int(sys.argv[1]))
sys.stdout.flush()
time.sleep(0.5)
After running:
./main.py
stdout get updated every 0.5 seconds for every two lines to contain:
0
10
1
11
2
12
3
13
and each log file contains the respective log for a given process.
Inspired by: https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/
Tested on Ubuntu 18.04, Python 3.6.7.
You probably want to start investigating the os module for forking different threads (by opening an interactive session and issuing help(os)). The relevant functions are fork and any of the exec ones. To give you an idea on how to start, put something like this in a function that performs the fork (the function needs to take a list or tuple 'args' as an argument that contains the program's name and its parameters; you may also want to define stdin, out and err for the new thread):
try:
pid = os.fork()
except OSError, e:
## some debug output
sys.exit(1)
if pid == 0:
## eventually use os.putenv(..) to set environment variables
## os.execv strips of args[0] for the arguments
os.execv(args[0], args)
You can use
import os
pid = os.fork()
if pid == 0:
Continue to other code ...
This will make the python process run in background.
I haven't tried this yet but using .pyw files instead of .py files should help. pyw files dosen't have a console so in theory it should not appear and work like a background process.
I have a ROS code rostopic pub toggle_led std_msgs/Empty that basically starts once and keeps running until CTRL+C is pressed.
Now, I would like to automate this command from Python. I checked Calling an external command in Python but it only shows how to start the command.
How would I start and stop running this process as and when I want?
How would I start and stop running this process as and when I want?
Well, you already know how to start it, as you said in the previous sentence.
How do you stop it? If you want to stop it exactly like a Ctrl-C,* you do that by calling send_signal on it, using CTRL_C_EVENT on Windows, or SIGTERM on Unix.** So:
import signal
import subprocess
try:
sig = signal.CTRL_C_EVENT
except NameError:
sig = signal.SIGTERM
p = subprocess.Popen(['/path/to/prog', '-opt', '42', 'arg'])
# ... later
p.send_signal(sig)
If you only care about Linux (or *nix in general), you can make this even simpler: terminate is guaranteed to do the same thing as send_signal(SIGTERM). So:
import subprocess
p = subprocess.Popen(['/path/to/prog', '-opt', '42', 'arg'])
# ... later
p.terminate()
Since you asked in a comment "Could you please explain the various parameters to subprocess.Popen()": Well, there are a whole lot of them (see Popen Constructor and Frequently Used Arguments in the docs, but I'm only using one, the args parameter.
Normally, you pass a list to args, with the name of the program as the first element in the list, and each separate command-line argument as a separate element. But if you want to use the shell, you pass a string for args, and add a shell=True as another argument.
* Note that "exactly like a Ctrl-C" may not actually be what you want on Windows, unless the program has a console and is a process group owner. This may mean you'll need to add creationflags=subprocess.CREATE_NEW_PROCESS_GROUP to the Popen call. Or it may not—e.g.., if you use shell=True.
** In Python, you can usually ignore the platform differences between CTRL_C_EVENT and SIGTERM and always use the latter, but subprocess.send_signal is one of the few places you can't. On Windows, send_signal(SIGTERM) will call terminate instead of sending a Ctrl-C. If you don't actually care exactly how the process gets stopped, just that it gets stopped somehow, then of course you can use SIGTERM… but in that case, you might as well just call terminate.
I have a C++ program, called C, that is designed to shut down when it receives a SIGINT signal. I've written a Python program P that runs C as a subprocess. I want P to stop C. I tried 3 things and I'd like to know why some of them didn't work.
Attempt #1:
import subprocess
import signal
import os
p = subprocess.Popen(...)
...
os.killpg(p.pid, signal.SIGINT)
This code gives me the error
OSError [Errno 3]: No such process`
even though the p.pid matches the pid displayed by ps.
Attempt #2:
import subprocess
import signal
import os
p = subprocess.Popen(...)
...
os.system('kill -SIGINT %u' % p.pid)
This gives me the error
sh: kill: bad signal`
even though kill -SIGINT <pid> works from the terminal.
Attempt #3:
import subprocess
import signal
import os
p = subprocess.Popen(...)
...
os.system('kill -2 %u' % p.pid)
This works.
My question is, why didn't #1 and #2 work?
Edit: my original assumption was that since the documentation for os.kill() says New in version 2.7: Windows support, I thought that os.kill() is (a) first available in 2.7 and (b) works in Windows. After reading the answers below, I ran os.kill() on Solaris, which I should have done in the first place sorry, and it does work in 2.4. Obviously, the documentation means that Windows support is new in 2.7. Opps.
The first fails because os.killpg kills a process group, identified by its leader; you have a simple process, not a process group. Try os.kill instead. The second fails because the shell builtin kill understands symbolic signals, but the external command on Solaris doesn't (whereas on *BSD and Linux it does); use a numeric signal (SIGINT is 2 on Solaris, or use Python's predefined signal constants from the signal module). That said, use Popen's own interface instead as mentioned by someone else; don't reinvent the wheel, you're liable to create some corners.
The Popen object has a kill() method that you can invoke as well as a terminate() method and a generic send_signal() method.
I would use one of these rather than trying any of the out of band stuff you'd use with the os interface. You've already got a handle to the process, you should use it!
I guess I'm not clear on what what the function of the getty/agetty/mgetty programs are on a linux/unix machine. I can start a shell on a tty with something like this:
TTY = '/dev/tty3'
cpid = os.fork()
if cpid == 0:
os.closerange(0, 4)
sys.stdin = open(TTY, 'r')
sys.stdout = open(TTY, 'w')
sys.stderr = open(TTY, 'w')
os.execv(('/bin/bash',), ('bash',))
..and if i switch over to tty3, there is a shell running- but some keystrokes are ignored / are never being sent to the shell. the shell knows the TTY settings are not correct because bash will say something like 'unable to open tty, job control disabled'
I know the 'termios' module has functions to change the settings on the TTY, which is what the 'tty' module uses, but i am unable to find an example of python setting the TTY correctly and starting a shell. I feel like it should be something simple, but i don't know where to look.
looking at the source for the *etty programs didn't help me- C looks like greek to me :-/
Maybe im just not looking for the right terms? Anyone replaced the *etty programs with Python in the past and have an explanation they would care to share?
Thanks for entertaining my basic question :)
I can see at least two things you're missing - there may be more:
Firstly, you need to call setsid() in the child process after closing the old standard input/standard output, and before opening the new TTY. This does two important things - it makes your new process the leader of a new session, and it disassociates it from its previous controlling terminal (merely closing that terminal is not sufficient). This will mean that when you open the new tty, it will become the controlling terminal, which is what you want.
Secondly, you need to set the TERM environment variable to match the new tty.
You should have a look at the source of the *tty* programs, to see what they do.
My guess is that they mostly issue a bunch of ioctl commands to initialise the terminal into the mode that programs normally expect (e.g. for login etc). However some of them may also prompt for a username (not password; login does that).