I need to call a shellscript from python.
The problem is that the shellscript will ask a couple of questions along the way until it is finished.
I can't find a way to do so using subprocess! (using pexpect seems a bit over-kill since I only need to start it and send a couple of YES to it)
PLEASE don't suggest ways that requires modification to the shell-script!
Using the subprocess library, you can tell the Popen class that you want to manage the standard input of the process like this:
import subprocess
shellscript = subprocess.Popen(["shellscript.sh"], stdin=subprocess.PIPE)
Now shellscript.stdin is a file-like object on which you can call write:
shellscript.stdin.write("yes\n")
shellscript.stdin.close()
returncode = shellscript.wait() # blocks until shellscript is done
You can also get standard out and standard error from a process by setting stdout=subprocess.PIPE and stderr=subprocess.PIPE, but you shouldn't use PIPEs for both standard input and standard output, because deadlock could result. (See the documentation.) If you need to pipe in and pipe out, use the communicate method instead of the file-like objects:
shellscript = subprocess.Popen(["shellscript.sh"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = shellscript.communicate("yes\n") # blocks until shellscript is done
returncode = shellscript.returncode
Related
I want to use Python's subprocess.Popen to execute a child process, and collect all its output.
The problem is that the process in question writes not only to stdout and stderr, but sometimes also to stdin 1.
I want to capture all three of these output streams separately.
Naively, I tried this:
import subprocess
...
proc = subprocess.Popen(('/usr/bin/modulecmd', 'python', 'help', 'null'),
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,)
stdout = proc.stdout.read()
stderr = proc.stderr.read()
stdin = proc.stdin.read()
proc.wait()
...but when the expression proc.stdin.read() is evaluated, an exception ensues (io.UnsupportedOperation: read).
Does anyone know how to use subprocess.Popen to handle this situation?
(BTW, I did try to implement a subclass of subprocess.Popen to support this use-case, but I kept running into errors similar to the one mentioned above. I just don't understands subprocess.Popen's internals well enough.)
1 See https://unix.stackexchange.com/a/667660/10618, as well as the comments therein, for a good discussion of this unusual behavior.
Use the redirection function of the shell, like this.
import shlex
...
args = ('/usr/bin/modulecmd', 'python', 'help', 'null')
proc = subprocess.Popen(shlex.join(args) + ' 0>&1',
shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
...
If you don't want to depend on an external program such as a shell, you have to implement on your own using os.fork(), os.exec(), os.pipe(), etc. See subprocess.py.
I'm trying to write a Python script that starts a subprocess, and writes to the subprocess stdin. I'd also like to be able to determine an action to be taken if the subprocess crashes.
The process I'm trying to start is a program called nuke which has its own built-in version of Python which I'd like to be able to submit commands to, and then tell it to quit after the commands execute. So far I've worked out that if I start Python on the command prompt like and then start nuke as a subprocess then I can type in commands to nuke, but I'd like to be able to put this all in a script so that the master Python program can start nuke and then write to its standard input (and thus into its built-in version of Python) and tell it to do snazzy things, so I wrote a script that starts nuke like this:
subprocess.call(["C:/Program Files/Nuke6.3v5/Nuke6.3", "-t", "E:/NukeTest/test.nk"])
Then nothing happens because nuke is waiting for user input. How would I now write to standard input?
I'm doing this because I'm running a plugin with nuke that causes it to crash intermittently when rendering multiple frames. So I'd like this script to be able to start nuke, tell it to do something and then if it crashes, try again. So if there is a way to catch a crash and still be OK then that'd be great.
It might be better to use communicate:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['myapp'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
stdout_data = p.communicate(input='data_to_write')[0]
"Better", because of this warning:
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
To clarify some points:
As jro has mentioned, the right way is to use subprocess.communicate.
Yet, when feeding the stdin using subprocess.communicate with input, you need to initiate the subprocess with stdin=subprocess.PIPE according to the docs.
Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
Also qed has mentioned in the comments that for Python 3.4 you need to encode the string, meaning you need to pass Bytes to the input rather than a string. This is not entirely true. According to the docs, if the streams were opened in text mode, the input should be a string (source is the same page).
If streams were opened in text mode, input must be a string. Otherwise, it must be bytes.
So, if the streams were not opened explicitly in text mode, then something like below should work:
import subprocess
command = ['myapp', '--arg1', 'value_for_arg1']
p = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate(input='some data'.encode())[0]
I've left the stderr value above deliberately as STDOUT as an example.
That being said, sometimes you might want the output of another process rather than building it up from scratch. Let's say you want to run the equivalent of echo -n 'CATCH\nme' | grep -i catch | wc -m. This should normally return the number characters in 'CATCH' plus a newline character, which results in 6. The point of the echo here is to feed the CATCH\nme data to grep. So we can feed the data to grep with stdin in the Python subprocess chain as a variable, and then pass the stdout as a PIPE to the wc process' stdin (in the meantime, get rid of the extra newline character):
import subprocess
what_to_catch = 'catch'
what_to_feed = 'CATCH\nme'
# We create the first subprocess, note that we need stdin=PIPE and stdout=PIPE
p1 = subprocess.Popen(['grep', '-i', what_to_catch], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We immediately run the first subprocess and get the result
# Note that we encode the data, otherwise we'd get a TypeError
p1_out = p1.communicate(input=what_to_feed.encode())[0]
# Well the result includes an '\n' at the end,
# if we want to get rid of it in a VERY hacky way
p1_out = p1_out.decode().strip().encode()
# We create the second subprocess, note that we need stdin=PIPE
p2 = subprocess.Popen(['wc', '-m'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We run the second subprocess feeding it with the first subprocess' output.
# We decode the output to convert to a string
# We still have a '\n', so we strip that out
output = p2.communicate(input=p1_out)[0].decode().strip()
This is somewhat different than the response here, where you pipe two processes directly without adding data directly in Python.
Hope that helps someone out.
Since subprocess 3.5, there is the subprocess.run() function, which provides a convenient way to initialize and interact with Popen() objects. run() takes an optional input argument, through which you can pass things to stdin (like you would using Popen.communicate(), but all in one go).
Adapting jro's example to use run() would look like:
import subprocess
p = subprocess.run(['myapp'], input='data_to_write', capture_output=True, text=True)
After execution, p will be a CompletedProcess object. By setting capture_output to True, we make available a p.stdout attribute which gives us access to the output, if we care about it. text=True tells it to work with regular strings rather than bytes. If you want, you might also add the argument check=True to make it throw an error if the exit status (accessible regardless via p.returncode) isn't 0.
This is the "modern"/quick and easy way to do to this.
One can write data to the subprocess object on-the-fly, instead of collecting all the input in a string beforehand to pass through the communicate() method.
This example sends a list of animals names to the Unix utility sort, and sends the output to standard output.
import sys, subprocess
p = subprocess.Popen('sort', stdin=subprocess.PIPE, stdout=sys.stdout)
for v in ('dog','cat','mouse','cow','mule','chicken','bear','robin'):
p.stdin.write( v.encode() + b'\n' )
p.communicate()
Note that writing to the process is done via p.stdin.write(v.encode()). I tried using
print(v.encode(), file=p.stdin), but that failed with the message TypeError: a bytes-like object is required, not 'str'. I haven't figured out how to get print() to work with this.
You can provide a file-like object to the stdin argument of subprocess.call().
The documentation for the Popen object applies here.
To capture the output, you should instead use subprocess.check_output(), which takes similar arguments. From the documentation:
>>> subprocess.check_output(
... "ls non_existent_file; exit 0",
... stderr=subprocess.STDOUT,
... shell=True)
'ls: non_existent_file: No such file or directory\n'
Basically I'm trying to automate some Linux installers (and other tasks) using the subprocess library (Popen).
In the past I've been able to open processes like this:
self.process = subprocess.Popen( self.executable,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True, shell=True)
output, cmdError = self.process.communicate()
I can then print output or cmdError for error messages and this works well for single processes or commands.
But when I need to interact with a subprocess and examine the output, it is very difficult, here is my code for doing this:
def ExecProcessWithAnswers(self):
self.process = subprocess.Popen( self.executable,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
stdin=subprocess.PIPE,
universal_newlines=True, shell=True)
while self.process.poll() is None:
print self.process.stdout.readline()
Basically the idea is that I would poll the output (stdout pipe) and then send commands when a certain input is requested from the installer (stdin).
I've tried flushing the buffer, using 3 different ways to read/iterate the output from stdout, but all of them just block or only give me a small fraction of the output text. On the other hand, if I use the communicate method, I get all of the text I expect, but it terminates the process. I've also had EOF errors and other random things.
I've read around: some guides say this is a bug in 2.6.x but it is still in 2.7.x - apparently the stdout is buffered and cannot be changed. I've tried many different ways of parsing the output from various threads here but I still can't get this to work on 2.7.X.
Surely someone must know how to interact with a subprocess? Is my only option here to use pexcept?
I can't really switch to Python 3.x.x within my environment. I was hoping this would be fairly straight forward :(
Cheers
Edit: I've also tried removing the different Pipes, writing to files, changing the buffer size on popen, disabling the shell and universal newlines, etc.
I'm trying to write a Python script that starts a subprocess, and writes to the subprocess stdin. I'd also like to be able to determine an action to be taken if the subprocess crashes.
The process I'm trying to start is a program called nuke which has its own built-in version of Python which I'd like to be able to submit commands to, and then tell it to quit after the commands execute. So far I've worked out that if I start Python on the command prompt like and then start nuke as a subprocess then I can type in commands to nuke, but I'd like to be able to put this all in a script so that the master Python program can start nuke and then write to its standard input (and thus into its built-in version of Python) and tell it to do snazzy things, so I wrote a script that starts nuke like this:
subprocess.call(["C:/Program Files/Nuke6.3v5/Nuke6.3", "-t", "E:/NukeTest/test.nk"])
Then nothing happens because nuke is waiting for user input. How would I now write to standard input?
I'm doing this because I'm running a plugin with nuke that causes it to crash intermittently when rendering multiple frames. So I'd like this script to be able to start nuke, tell it to do something and then if it crashes, try again. So if there is a way to catch a crash and still be OK then that'd be great.
It might be better to use communicate:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['myapp'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
stdout_data = p.communicate(input='data_to_write')[0]
"Better", because of this warning:
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
To clarify some points:
As jro has mentioned, the right way is to use subprocess.communicate.
Yet, when feeding the stdin using subprocess.communicate with input, you need to initiate the subprocess with stdin=subprocess.PIPE according to the docs.
Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
Also qed has mentioned in the comments that for Python 3.4 you need to encode the string, meaning you need to pass Bytes to the input rather than a string. This is not entirely true. According to the docs, if the streams were opened in text mode, the input should be a string (source is the same page).
If streams were opened in text mode, input must be a string. Otherwise, it must be bytes.
So, if the streams were not opened explicitly in text mode, then something like below should work:
import subprocess
command = ['myapp', '--arg1', 'value_for_arg1']
p = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate(input='some data'.encode())[0]
I've left the stderr value above deliberately as STDOUT as an example.
That being said, sometimes you might want the output of another process rather than building it up from scratch. Let's say you want to run the equivalent of echo -n 'CATCH\nme' | grep -i catch | wc -m. This should normally return the number characters in 'CATCH' plus a newline character, which results in 6. The point of the echo here is to feed the CATCH\nme data to grep. So we can feed the data to grep with stdin in the Python subprocess chain as a variable, and then pass the stdout as a PIPE to the wc process' stdin (in the meantime, get rid of the extra newline character):
import subprocess
what_to_catch = 'catch'
what_to_feed = 'CATCH\nme'
# We create the first subprocess, note that we need stdin=PIPE and stdout=PIPE
p1 = subprocess.Popen(['grep', '-i', what_to_catch], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We immediately run the first subprocess and get the result
# Note that we encode the data, otherwise we'd get a TypeError
p1_out = p1.communicate(input=what_to_feed.encode())[0]
# Well the result includes an '\n' at the end,
# if we want to get rid of it in a VERY hacky way
p1_out = p1_out.decode().strip().encode()
# We create the second subprocess, note that we need stdin=PIPE
p2 = subprocess.Popen(['wc', '-m'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We run the second subprocess feeding it with the first subprocess' output.
# We decode the output to convert to a string
# We still have a '\n', so we strip that out
output = p2.communicate(input=p1_out)[0].decode().strip()
This is somewhat different than the response here, where you pipe two processes directly without adding data directly in Python.
Hope that helps someone out.
Since subprocess 3.5, there is the subprocess.run() function, which provides a convenient way to initialize and interact with Popen() objects. run() takes an optional input argument, through which you can pass things to stdin (like you would using Popen.communicate(), but all in one go).
Adapting jro's example to use run() would look like:
import subprocess
p = subprocess.run(['myapp'], input='data_to_write', capture_output=True, text=True)
After execution, p will be a CompletedProcess object. By setting capture_output to True, we make available a p.stdout attribute which gives us access to the output, if we care about it. text=True tells it to work with regular strings rather than bytes. If you want, you might also add the argument check=True to make it throw an error if the exit status (accessible regardless via p.returncode) isn't 0.
This is the "modern"/quick and easy way to do to this.
One can write data to the subprocess object on-the-fly, instead of collecting all the input in a string beforehand to pass through the communicate() method.
This example sends a list of animals names to the Unix utility sort, and sends the output to standard output.
import sys, subprocess
p = subprocess.Popen('sort', stdin=subprocess.PIPE, stdout=sys.stdout)
for v in ('dog','cat','mouse','cow','mule','chicken','bear','robin'):
p.stdin.write( v.encode() + b'\n' )
p.communicate()
Note that writing to the process is done via p.stdin.write(v.encode()). I tried using
print(v.encode(), file=p.stdin), but that failed with the message TypeError: a bytes-like object is required, not 'str'. I haven't figured out how to get print() to work with this.
You can provide a file-like object to the stdin argument of subprocess.call().
The documentation for the Popen object applies here.
To capture the output, you should instead use subprocess.check_output(), which takes similar arguments. From the documentation:
>>> subprocess.check_output(
... "ls non_existent_file; exit 0",
... stderr=subprocess.STDOUT,
... shell=True)
'ls: non_existent_file: No such file or directory\n'
I have successfully managed to pipe a variable to a command which is not piped further, in this way:
from subprocess import Popen, PIPE
def exec_command(command, some_input):
proc = Popen(command, stdout=PIPE, stderr=PIPE, stdin=PIPE)
(stdout, stderr) = proc.communicate(input=some_input)
... but when trying to pipe it to a further piped command (for example piping to tar, and then further piping to split), it does not seem to work:
from subprocess import Popen, PIPE
def exec_piped_command(command1, command2, some_input):
proc1 = Popen(command1, stdout=PIPE, stdin=PIPE)
proc2 = Popen(command1, stdin=proc1.stdout, stdout=PIPE)
(stdout, stderr) = proc2.communicate(input=some_input)[0]
So, how is the correct way to do this second variant? It seems like the problem with the above code is that the input in the "proc2.communicate()" command does not reach the stdin pipe of proc1? (Not sure though ... unfortunatly I'm a bit confused about the subprocess syntax ...).
One possibility would be to set up the entire command to be executed by a shell (shell=True to among the keyword args of your Popen() call ... and only .communicate() with the ends of the whole pipeline (your input going to command1's stdin, and your stdout/stderr coming from command2's stdout/stderr).
A more complicated, but fine grained, approach would be to use the os.fork() along with os.pipe() and os.dup2() and possibly some os.fcntl() calls to set up your own subprocesses, with your own custom plumping, and any of the desired blocking/non-blocking characteristics you need on your file descriptors and then, finally, using your own os.exec* functions in each of these.
This latter approach, obviously, would be duplicating quite a bit of the code that's already in the subprocess module. However, it provides you the option of doing some of these steps differently in ways that are not exposed via the subprocess.Popen class.
The middle road would be to make a sub class that inherits from subprocess.Popen
Of course it may be preferable to perform some of the parts of this pipeline through Python code and modules (such as tar, gzip, and split operations).