I am using a python script to run a process using subprocess.Popen and simultaneously store the output in a text file as well as print it on the console. This is my code:
result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for line in result.stdout.readlines(): #read and store result in log file
openfile.write("%s\n" %line)
print("%s" %line)
Above code works fine, but what it does is it first completes the process and stores the output in result variable. After that for loop stores the output as well as print it.
But i want the output at runtime (as my process can take hours to complete, i don't get any output for all these hours).
So is there any other function that gives me the output dynamically (at runtime), means as soon as the process gives first line, it should get printed.
The problem here is that .readlines() gets the entire output before returning, as it constructs a full list. Just iterate directly:
for line in result.stdout:
print(line)
.readlines() returns a list of all the lines the process will return while open, i.e., it doesn't return anything until all output from the subprocess is received. To read line by line in "real time":
import sys
from subprocess import Popen, PIPE
proc = Popen(cmd, shell=True, bufsize=1, stdout=PIPE)
for line in proc.stdout:
openfile.write(line)
sys.stdout.buffer.write(line)
sys.stdout.buffer.flush()
proc.stdout.close()
proc.wait()
Note: if the subprocess uses block-buffering when it is run in non-interactive mode; you might need pexpect, pty modules or stdbuf, unbuffer, script commands.
Note: on Python 2, you might also need to use iter(), to get "real time" output:
for line in iter(proc.stdout.readline, ""):
openfile.write(line)
print line,
You can iterate over the lines one by one by using readline on the pipe:
while True:
line = result.stdout.readline()
print line.strip()
if not line:
break
The lines contain a trailing \n which I stripped for printing.
When the process terminates, readline returns an empty string, so you know when to stop.
Related
I am trying to run following code
process = subprocess.Popen(args=cmd, shell=True, stdout=subprocess.PIPE)
while process.poll() is None:
stdoutput = process.stdout.readline()
print(stdoutput.decode())
if '(Y/N)' in stdoutput.decode():
process.communicate(input=b'Y\n')
this cmd argument runs for a few minutes after which it prompts for a confirmation, but the process.communicate is not working, neither is process.stdin.write()
How do I send input string 'Y' to this running process when it prompts for confirmation
Per the doc on Popen.communicate(input=None, timeout=None):
Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE.
Please try that, and if it's not sufficient, do indicate what the symptom is.
On top of the answer from #Jerry101, if the subprocess that you are calling is a python script that uses the input(), be aware that as documented:
If the prompt argument is present, it is written to standard output without a trailing newline.
Thus if you perform readline() as in process.stdout.readline(), it would hang there waiting for the new line \n character as documented:
f.readline() reads a single line from the file; a newline character (\n) is left at the end of the string
A quick fix is append the newline \n when requesting the input() e.g. input("(Y/N)\n") instead of just input("(Y/N)").
Related question:
Python subprocess stdout doesn't capture input prompt
I am using the Popen constructor from subprocess to capture the output of the command that I am running in my python script:
import os
from subprocess import Popen, PIPE
p = Popen(["my-cli", "ls", "/mypics/"], stdin=PIPE, stdout=PIPE,stderr=PIPE)
output, err = p.communicate()
print(output)
print(output.count('jpg'))
My objective is to save the output file names as an array of strings.
However, when I print the output, I notice that instead of saving file names as strings, the script saves each byte (letter) of each file as a string. Therefore, the printed output looks like this
f
i
l
e
.
j
p
g
1
So instead of printing one filename file.jpg I am getting a printout of the 8 separate characters that make up the filename. But running the ls command in the terminal directly will just list the filenames row by row as it should.
What am I doing wrong in this script and what is the workaround here? I am running Python 2.7 Any suggestions would be appreciated
What was that my-cli inside your Popen array. I think some new line character where appending after each char output. Just remove that my-cli and this could fork for you.
p = Popen(["ls", "/mypics/"], stdin=PIPE, stdout=PIPE,stderr=PIPE)
I hope this will work for you.
So this is my problem,
I'm trying to do a simple program that runs another process using Python's subprocess module, and I want to catch real-time output of the process.
I know this can be done as such:
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in iter(proc.stdout.readline, ""):
line = line.rstrip()
if line != "":
print(line)
The issue is, the process might generate output with a carriage return \r, and I want to simulate that behavior in my program.
If I use the universal_newlines flag in Popen, then I could catch the output that is generated with a carriage return, but I wouldn't know it was as such, and I could only print it "regularly" with a newline. I want to avoid that, as this could be a lot of output.
My question is basically if I could catch the \r output like it is a \n but differentiate it from actual \n output
EDIT
Here is some simplified code of what I tried:
File download.py:
import subprocess
try:
subprocess.check_call(
[
"aws",
"s3",
"cp",
"S3_LINK",
"TARGET",
]
)
except subprocess.CalledProcessError as err:
print(err)
raise SystemExit(1)
File process_runner.py:
import os
import sys
import subprocess
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for char in iter(lambda: proc.stdout.read(1), ""):
sys.stdout.write(char)
The code in download uses aws s3 cp, which gives carriage returns of the download progress. I want to simulate this behavior of output in my program process_runner which receives download's output.
At first I tried to iter readline instead of read(1). That did not work due to the CR being overlooked.
A possible way is to use the binary interface of Popen by specifying neither encoding nor error and of course not universal_newline. And then, we can use a TextIOWrapper around the binary stream, with newline=''. Because the documentation for TextIOWrapper says:
... if newline is None... If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated
(which is conformant with PEP 3116)
You original code could be changed to:
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
out = io.TextIOWrapper(proc.stdout, newline='')
for line in out:
# line is delimited with the universal newline convention and actually contains
# the original end of line, be it a raw \r, \n of the pair \r\n
...
In general this questions has a lot of answers but they are all limited to line by line reading. For example this code:
def execute(cmd):
popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
for stdout_line in iter(popen.stdout.readline, ""):
yield stdout_line
popen.stdout.close()
return_code = popen.wait()
if return_code:
raise subprocess.CalledProcessError(return_code, cmd)
But there are output lines for example like this (where dots are added once in ~10s):
............................
They show progress of a task that runs. I don't want to stop output until the line is finished and only then print the whole dots line.
So I need to yield string when:
I can read a block of 1024 symbols of output (or just the whole output)
there are ANY symbols of output and more then 1s passed (no matter line is finished or not)
But I don't know how to do this.
p.s. Maybe a dup. Didn't find.
If you're on Linux you can use stdbuf -o0 in front of the command you're executing to make its stdout become unbuffered (i.e. instantaneous).
I have a program which opens a subprocess and communicates with it by writing to its stdin and reading from its stdout.
proc = subprocess.Popen(['foo'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
proc.stdin.write('stuff\n')
proc.stdin.flush()
The problem is that when reading, it always blocks if I call proc.stdout.read(), and when I try to read line by line using the following:
output = str()
while proc.stdout in select.select([proc.stdout], [], [])[0]:
output += proc.stdout.readline()
it still blocks because select.select returns proc.stdout even after all the output has been read already. What can I do?
note that I am not using proc.communicate because I would like to communicate with the process multiple times