I'm trying to get the standard output of a bash command as a string in Python. Following Popen documentation, I've tried:
import subprocess
p = subprocess.Popen(["echo", "hello"])
stdoutdata, stderrdata = p.communicate()
print stdoutdata
Running this script yields the following output:
hello
None
[Finished in 0.0s]
So although the output is getting printed by Python, the stdoutdata variable is None, and not "hello" as I would like. How can I make it so?
You're not providing any stdout to the Popen constructor, the default functionality simply writes the output to parent's stdout handle. Hence you're seeing it being printed in your shell.
Quoting from Popen's docs:
With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent.
To populate stdout in resulting tuple use subprocess.PIPE as stdout.
Quoting from docs:
To get anything other than None in the result tuple, you need to give
stdout=PIPE and/or stderr=PIPE too.
>>> import subprocess
>>> p = subprocess.Popen(["echo", "hello"], stdout=subprocess.PIPE)
>>> p.communicate()
('hello\n', None)
You need to pass stdout, stderr flags to Popen constructor.
Per default they are set to None, resulting in Popen is not capturing them.
cmd = subprocess.Popen(["echo", "hello"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = cmd.communicate()
# retCode = cmd.returncode
# retCode != 0, indicates an error occured in execution.
print (out)
>>> b'hello\n'
It seems like the subprocess.check_output method is what I need:
import subprocess
output = subprocess.check_output(["echo", "hello"])
The output is now 'hello\n' (including a newline character) as I would expect.
I have this python code
input()
print('spam')
saved as ex1.py
in interactive shell
>>>from subprocess import Popen ,PIPE
>>>a=Popen(['python.exe','ex1.py'],stdout=PIPE,stdin=PIPE)
>>> a.communicate()
(b'', None)
>>>
why it is not printing the spam
Input expects a whole line, but your input is empty. So there is only an exception written to stderr and nothing to stdout. At least provide a newline as input:
>>> a = Popen(['python3', 'ex1.py'], stdout=PIPE, stdin=PIPE)
>>> a.communicate(b'\n')
(b'spam\n', None)
>>>
You are missing stderr piping:
from subprocess import Popen ,PIPE
proc = Popen(['python.exe','ex1.py'], stdout=PIPE, stderr=PIPE)
out, err = proc.communicate()
print(out, err)
What you're looking for is subprocess.check_output
If I do the following:
import subprocess
from cStringIO import StringIO
subprocess.Popen(['grep','f'],stdout=subprocess.PIPE,stdin=StringIO('one\ntwo\nthree\nfour\nfive\nsix\n')).communicate()[0]
I get:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/build/toolchain/mac32/python-2.4.3/lib/python2.4/subprocess.py", line 533, in __init__
(p2cread, p2cwrite,
File "/build/toolchain/mac32/python-2.4.3/lib/python2.4/subprocess.py", line 830, in _get_handles
p2cread = stdin.fileno()
AttributeError: 'cStringIO.StringI' object has no attribute 'fileno'
Apparently a cStringIO.StringIO object doesn't quack close enough to a file duck to suit subprocess.Popen. How do I work around this?
Popen.communicate() documentation:
Note that if you want to send data to
the process’s stdin, you need to
create the Popen object with
stdin=PIPE. Similarly, to get anything
other than None in the result tuple,
you need to give stdout=PIPE and/or
stderr=PIPE too.
Replacing os.popen*
pipe = os.popen(cmd, 'w', bufsize)
# ==>
pipe = Popen(cmd, shell=True, bufsize=bufsize, stdin=PIPE).stdin
Warning Use communicate() rather than
stdin.write(), stdout.read() or
stderr.read() to avoid deadlocks due
to any of the other OS pipe buffers
filling up and blocking the child
process.
So your example could be written as follows:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
grep_stdout = p.communicate(input=b'one\ntwo\nthree\nfour\nfive\nsix\n')[0]
print(grep_stdout.decode())
# -> four
# -> five
# ->
On Python 3.5+ (3.6+ for encoding), you could use subprocess.run, to pass input as a string to an external command and get its exit status, and its output as a string back in one call:
#!/usr/bin/env python3
from subprocess import run, PIPE
p = run(['grep', 'f'], stdout=PIPE,
input='one\ntwo\nthree\nfour\nfive\nsix\n', encoding='ascii')
print(p.returncode)
# -> 0
print(p.stdout)
# -> four
# -> five
# ->
I figured out this workaround:
>>> p = subprocess.Popen(['grep','f'],stdout=subprocess.PIPE,stdin=subprocess.PIPE)
>>> p.stdin.write(b'one\ntwo\nthree\nfour\nfive\nsix\n') #expects a bytes type object
>>> p.communicate()[0]
'four\nfive\n'
>>> p.stdin.close()
Is there a better one?
There's a beautiful solution if you're using Python 3.4 or better. Use the input argument instead of the stdin argument, which accepts a bytes argument:
output_bytes = subprocess.check_output(
["sed", "s/foo/bar/"],
input=b"foo",
)
This works for check_output and run, but not call or check_call for some reason.
In Python 3.7+, you can also add text=True to make check_output take a string as input and return a string (instead of bytes):
output_string = subprocess.check_output(
["sed", "s/foo/bar/"],
input="foo",
text=True,
)
I'm a bit surprised nobody suggested creating a pipe, which is in my opinion the far simplest way to pass a string to stdin of a subprocess:
read, write = os.pipe()
os.write(write, "stdin input here")
os.close(write)
subprocess.check_call(['your-command'], stdin=read)
I am using python3 and found out that you need to encode your string before you can pass it into stdin:
p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
out, err = p.communicate(input='one\ntwo\nthree\nfour\nfive\nsix\n'.encode())
print(out)
Apparently a cStringIO.StringIO object doesn't quack close enough to
a file duck to suit subprocess.Popen
I'm afraid not. The pipe is a low-level OS concept, so it absolutely requires a file object that is represented by an OS-level file descriptor. Your workaround is the right one.
from subprocess import Popen, PIPE
from tempfile import SpooledTemporaryFile as tempfile
f = tempfile()
f.write('one\ntwo\nthree\nfour\nfive\nsix\n')
f.seek(0)
print Popen(['/bin/grep','f'],stdout=PIPE,stdin=f).stdout.read()
f.close()
"""
Ex: Dialog (2-way) with a Popen()
"""
p = subprocess.Popen('Your Command Here',
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
stdin=PIPE,
shell=True,
bufsize=0)
p.stdin.write('START\n')
out = p.stdout.readline()
while out:
line = out
line = line.rstrip("\n")
if "WHATEVER1" in line:
pr = 1
p.stdin.write('DO 1\n')
out = p.stdout.readline()
continue
if "WHATEVER2" in line:
pr = 2
p.stdin.write('DO 2\n')
out = p.stdout.readline()
continue
"""
..........
"""
out = p.stdout.readline()
p.wait()
On Python 3.7+ do this:
my_data = "whatever you want\nshould match this f"
subprocess.run(["grep", "f"], text=True, input=my_data)
and you'll probably want to add capture_output=True to get the output of running the command as a string.
On older versions of Python, replace text=True with universal_newlines=True:
subprocess.run(["grep", "f"], universal_newlines=True, input=my_data)
Beware that Popen.communicate(input=s)may give you trouble ifsis too big, because apparently the parent process will buffer it before forking the child subprocess, meaning it needs "twice as much" used memory at that point (at least according to the "under the hood" explanation and linked documentation found here). In my particular case,swas a generator that was first fully expanded and only then written tostdin so the parent process was huge right before the child was spawned,
and no memory was left to fork it:
File "/opt/local/stow/python-2.7.2/lib/python2.7/subprocess.py", line 1130, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
This is overkill for grep, but through my journeys I've learned about the Linux command expect, and the python library pexpect
expect: dialogue with interactive programs
pexpect: Python module for spawning child applications; controlling them; and responding to expected patterns in their output.
import pexpect
child = pexpect.spawn('grep f', timeout=10)
child.sendline('text to match')
print(child.before)
Working with interactive shell applications like ftp is trivial with pexpect
import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('noah#example.com')
child.expect ('ftp> ')
child.sendline ('ls /pub/OpenBSD/')
child.expect ('ftp> ')
print child.before # Print the result of the ls command.
child.interact() # Give control of the child to the user.
p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
p.stdin.write('one\n')
time.sleep(0.5)
p.stdin.write('two\n')
time.sleep(0.5)
p.stdin.write('three\n')
time.sleep(0.5)
testresult = p.communicate()[0]
time.sleep(0.5)
print(testresult)
Have been trying to get something like this to work for a while, the below doesn't seem to be sending the correct arg to the c program arg_count, which outputs argc = 1. When I'm pretty sure I would like it to be 2. ./arg_count -arg from the shell outputs 2...
I have tried with another arg (so it would output 3 in the shell) and it still outputs 1 when calling via subprocess.
import subprocess
pipe = subprocess.Popen(["./args/Release/arg_count", "-arg"], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = pipe.communicate()
result = out.decode()
print "Result : ",result
print "Error : ",err
Any idea where im falling over? I'm running linux btw.
From the documentation:
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence.
Thus,
pipe = subprocess.Popen("./args/Release/arg_count -arg", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
should give you what you want.
If shell=True then your call is equivalent to:
from subprocess import Popen, PIPE
proc = Popen(['/bin/sh', '-c', "./args/Release/arg_count", "-arg"],
stdout=PIPE, stderr=PIPE)
i.e., -arg is passed to the shell itself and not your program. Drop shell=True to pass -arg to the program:
proc = Popen(["./args/Release/arg_count", "-arg"],
stdout=PIPE, stderr=PIPE)
If you don't need to capture stderr separately from stdout then you could use check_output():
from subprocess import check_output, STDOUT
output = check_output(["./args/Release/arg_count", "-arg"]) # or
output_and_errors = check_output(["./args/Release/arg_count", "-arg"],
stderr=STDOUT)
I am trying to spawn a process using Popen and send it a particular string to its stdin.
I have:
pipe = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
pipe.communicate( my_stdin_str.encode(encoding='ascii') )
pipe.stdin.close()
However, the second line actually escapes the whitespace in my_stdin_str. For example, if I have:
my_stdin_str="This is a string"
The process will see:
This\ is\ a\ string
How can I prevent this behaviour?
I can't reproduce it on Ubuntu:
from subprocess import Popen, PIPE
shell_cmd = "perl -pE's/.\K/-/g'"
p = Popen(shell_cmd, shell=True, stdin=PIPE)
p.communicate("This $PATH is a string".encode('ascii'))
In this case shell=True is unnecessary:
from subprocess import Popen, PIPE
cmd = ["perl", "-pE" , "s/.\K/-/g"]
p = Popen(cmd, stdin=PIPE)
p.communicate("This $PATH is a string".encode('ascii'))
Both produce the same output:
T-h-i-s- -$-P-A-T-H- -i-s- -a- -s-t-r-i-n-g-
Unless you know you need it for some reason, don't run with "shell=True" in general (which, without testing, sounds like what's going on here).