We Keep Coding

Unexpected output of Popen

I am using the Popen constructor from subprocess to capture the output of the command that I am running in my python script:
import os
from subprocess import Popen, PIPE
p = Popen(["my-cli", "ls", "/mypics/"], stdin=PIPE, stdout=PIPE,stderr=PIPE)
output, err = p.communicate()
print(output)
print(output.count('jpg'))
My objective is to save the output file names as an array of strings.
However, when I print the output, I notice that instead of saving file names as strings, the script saves each byte (letter) of each file as a string. Therefore, the printed output looks like this
f
i
l
e
.
j
p
g
1
So instead of printing one filename file.jpg I am getting a printout of the 8 separate characters that make up the filename. But running the ls command in the terminal directly will just list the filenames row by row as it should.
What am I doing wrong in this script and what is the workaround here? I am running Python 2.7 Any suggestions would be appreciated

What was that my-cli inside your Popen array. I think some new line character where appending after each char output. Just remove that my-cli and this could fork for you.
p = Popen(["ls", "/mypics/"], stdin=PIPE, stdout=PIPE,stderr=PIPE)
I hope this will work for you.

Python 2 to 3 conversion: iterating over lines in subprocess stdout

I have the following Python 2 example code that I want to make compatible with Python 3:
call = 'for i in {1..5}; do sleep 1; echo "Hello $i"; done'
p = subprocess.Popen(call, stdout=subprocess.PIPE, shell=True)
for line in iter(p.stdout.readline, ''):
print(line, end='')
This works well in Python 2 but in Python 3 p.stdout does not allow me to specify an encoding and reading it will return byte strings, rather than Unicode, so the comparison with '' will always return false and iter won't stop. This issue seems to imply that in Python 3.6 there'll be a way to define this encoding.
For now, I have changed the iter call to stop when it finds an empty bytes string iter(p.stdout.readline, b''), which seems to work in 2 and 3. My questions are: Is this safe in both 2 and 3? Is there a better way of ensuring compatibility?
Note: I'm not using for line in p.stdout: because I need each line to be printed as it's generated and according to this answer p.stdout has a too large a buffer.

You can add unversal_newlines=True.
p = subprocess.Popen(call, stdout=subprocess.PIPE, shell=True, universal_newlines=True)
for line in iter(p.stdout.readline, ''):
print(line, end='')
Instead of bytes, str will be returned so '' will work in both situations.
Here is what the docs have to say about the option:
If universal_newlines is False the file objects stdin, stdout and
stderr will be opened as binary streams, and no line ending conversion
is done.
If universal_newlines is True, these file objects will be opened as
text streams in universal newlines mode using the encoding returned by
locale.getpreferredencoding(False). For stdin, line ending characters
'\n' in the input will be converted to the default line separator
os.linesep. For stdout and stderr, all line endings in the output will
be converted to '\n'. For more information see the documentation of
the io.TextIOWrapper class when the newline argument to its
constructor is None.
It's not explicitly called out about the bytes versus str difference, but it is implied by stating that False returns a binary stream and True returns a text stream.

You can use p.communicate() and then decode it if it is a bytes object:
from __future__ import print_function
import subprocess
def b(t):
if isinstance(t, bytes):
return t.decode("utf8")
return t
call = 'for i in {1..5}; do sleep 1; echo "Hello $i"; done'
p = subprocess.Popen(call, stdout=subprocess.PIPE, shell=True)
stdout, stderr = p.communicate()
for line in iter(b(stdout).splitlines(), ''):
print(line, end='')
This would work in both Python 2 and Python 3

How to print part of output from a subprocess

How can I print some part of a returned stdout of a subprocess? E.g. From the example below, how can I print only the "Number of features" and the "Number of predicates" ?
command = "java -cp \"C:\mallet-2.0.7\class;C:\mallet-2.0.7\lib\mallet-deps.jar\" cc.mallet.fst.SimpleTagger --train true --model-file model train
p = Popen(command, stdout=PIPE, stderr=STDOUT, shell = True)
for line in p.stdout:
print line
>>>
Number of features in training data: 6
Number of predicates: 6
Labels: O noun non-noun
Preparing O
O->O(O) O,O
.........

Well the easiest way I can see is to change your last piece of code to:
for line in p.stdout:
if "Number of" in line:
print line
It's only data, you just need to select the bits you want (or need).
EDIT: Note this approach is somewhat simplistic. If the child process were to create so much output on stderr that buffers filled up it would stall unaccountably. Sending stderr to a file, even /dev/null, avoids this possibility.

Getting output of a process at runtime

I am using a python script to run a process using subprocess.Popen and simultaneously store the output in a text file as well as print it on the console. This is my code:
result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for line in result.stdout.readlines(): #read and store result in log file
openfile.write("%s\n" %line)
print("%s" %line)
Above code works fine, but what it does is it first completes the process and stores the output in result variable. After that for loop stores the output as well as print it.
But i want the output at runtime (as my process can take hours to complete, i don't get any output for all these hours).
So is there any other function that gives me the output dynamically (at runtime), means as soon as the process gives first line, it should get printed.

The problem here is that .readlines() gets the entire output before returning, as it constructs a full list. Just iterate directly:
for line in result.stdout:
print(line)

.readlines() returns a list of all the lines the process will return while open, i.e., it doesn't return anything until all output from the subprocess is received. To read line by line in "real time":
import sys
from subprocess import Popen, PIPE
proc = Popen(cmd, shell=True, bufsize=1, stdout=PIPE)
for line in proc.stdout:
openfile.write(line)
sys.stdout.buffer.write(line)
sys.stdout.buffer.flush()
proc.stdout.close()
proc.wait()
Note: if the subprocess uses block-buffering when it is run in non-interactive mode; you might need pexpect, pty modules or stdbuf, unbuffer, script commands.
Note: on Python 2, you might also need to use iter(), to get "real time" output:
for line in iter(proc.stdout.readline, ""):
openfile.write(line)
print line,

You can iterate over the lines one by one by using readline on the pipe:
while True:
line = result.stdout.readline()
print line.strip()
if not line:
break
The lines contain a trailing \n which I stripped for printing.
When the process terminates, readline returns an empty string, so you know when to stop.

Reading/writing to a Popen() subprocess

I'm trying to talk to a child process using the python subprocess.Popen() call. In my real code, I'm implementing a type of IPC, so I want to write some data, read the response, write some more data, read the response, and so on. Because of this, I cannot use Popen.communicate(), which otherwise works well for the simple case.
This code shows my problem. It never even gets the first response, hangs at the first "Reading result". Why? How can I make this work as I expect?
import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
stdout = subprocess.PIPE,
stdin = subprocess.PIPE)
p.stdin.write("abc\n")
print "Reading result:"
print p.stdout.readline()
p.stdin.write("cat\n")
print "Reading result:"
print p.stdout.readline()

sed's output is buffered and only outputs its data until enough has been cumulated or the input stream is exhausted and closed.
Try this:
import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
stdout = subprocess.PIPE,
stdin = subprocess.PIPE)
p.stdin.write("abc\n")
p.stdin.write("cat\n")
p.stdin.close()
print "Reading result 1:"
print p.stdout.readline()
print "Reading result 2:"
print p.stdout.readline()
Be aware that this cannot be done reliably which huge data as wriring to stdin blocks once the buffer is full. The best way to do is using communicate().

I would try to use Popen().communicate() if you can as it does a lot of nice things for you, but if you need to use Popen() exactly as you described, you'll need to set sed to flush its buffer after newlines with the -l option:
p = subprocess.Popen(['sed', '-l', 's/a/x/g'],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE)
and your code should work fine