Let's say you have the following:
command = shlex.split("mcf -o -q -e -w %s %s" % (SOLFILE, NETFILE))
task = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = task.communicate()
print "stdout: %s" % stdout #debugging
print "stderr: %s" % stderr #debugging
if stderr:
sys.exit("MCF crashed on %s" % NETFILE)
It's not necessary to know what mcf is, except that it's a C program which will overflow if it's not given a satisfiable netfile. (Why can't I just ensure that all the netfiles are satisfiable? Well, because the easiest way to check that is to feed it to mcf and see if it overflows...)
Anyway, when I run this in an executable script, task.communicate() doesn't seem to store anything in stdout and stderr. (To be precise, I get stdout == stderr == ''.) Instead, the stderr stream from mcf seems to be "leaking" to the terminal rather than getting captured by the subprocess pipe. Here's some sample output to illustrate:
Netfile: facility3cat_nat5000_wholesaler_capacitation_test_.net
Solfile: facility3cat_nat5000_wholesaler_capacitation_test_.sol
*** buffer overflow detected ***: mcf terminated
======= Backtrace: =========
...
...[fifty lines of Linda Blair-esque output]...
...
stdout: None
stderr:
...[program continues, since stderr did not evaluate to True]...
This only fails when running the script from the command line. When I step through it line by line in the interpreter, stdout and stderr are correctly assigned:
>>> task = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> stdout, stderr = task.communicate()
>>> stderr
'*** buffer overflow detected ***: mcf terminated\n======= Backtrace: =========\n'
...[more headspinning and vomit]...
Could anyone help me to understand why this works in the interpreter, but not when executed? Thanks in advance!
I wrote a little test script to test the subprocess module with.
#!/bin/bash
echo echo to stderr 1>&2
echo echo to stdout
Then I wrote a small Python script that calls it:
#!/usr/bin/python
import subprocess
command = ('./joe.sh',)
task = subprocess.Popen(command, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = task.communicate()
print 'stdout == %r\nstderr == %r' % (stdout, stderr)
The output of running it looks just like this:
$ python joe.py
stdout == 'echo to stdout\n'
stderr == 'echo to stderr\n'
The output of running that same sequence in ipython is the same.
So the subprocess module is behaving in the manner you expect, and not how it's behaving for you in your question. I think something other than the subprocess module must be at fault here because what you're doing works for me.
I'm running Python 2.7, so another possibility is that maybe there is some kind of weird bug in older versions of the subprocess module.
Related
I tried the below code to capture the output from screen using the sub-process, but its not doing what I intended to do.
#!/tools/bin/python
import subprocess
result = subprocess.check_output("echo $USERNAME", shell=True)
print result
expected output is:
vimo
vimo
i.e. one for the echo process and one for printing the result output.
But what I see is
vimo
But when I try to print the result output, its always empty.
What am I missing in the above puzzle !! Help out !!
Here you goes some greatly stripped (and altered for privacy reasons) raw dummy piece of code, grabbing both stdin and stdout from external script output.
from subprocess import Popen, PIPE
cmd = ['echo', 'foo']
proc = Popen(cmd, stdout=PIPE, stderr=PIPE)
comm = proc.communicate()
if proc.returncode != 0:
# code to handle / parse stderr (comm[1])
raise RuntimeError(
'\'%s\': command has failed (%d):\n%s'
% ('some value', proc.returncode, comm[1]))
for line in comm[0].split('\n'):
if line.find('Wrote:') == 0:
# some code to parse stdout
pass
Python >= 3.7 (tested with Python 3.9)
r=subprocess.run(['echo','$XDG_DATA_HOME'],capture_output=True,shell=True)
assert r.stdout.find(b'share')>0 # ERROR
r=subprocess.run('echo $XDG_DATA_HOME',capture_output=True,shell=True)
assert r.stdout.find(b'share')>0 # OK
I am trying to run an LSF command , 'bjobs' inside a python code using subprocess and I am unable to get the output into a variable
Ways I have already tried and failed are as follows:
proc = subprocess.Popen(['bjobs'],stdout=subprocess.PIPE)
print proc.stdout.read() ## Not working
stdout,stderr = subprocess.Popen(['bjobs'],stdout=subprocess.PIPE).communicate()
print stdout # prints empty line
I do not want to redirect that to a physical file.
So please help me to find a way to directly capture them to a variable
As pointed out by a comment above, the "No unfinished job found" message is printed to stderr:
[~]$ bjobs > /dev/null
No unfinished job found
[~]$ bjobs >& /dev/null
[~]$
If you want all bjobs output you should redirect subprocess stderr to stdout:
proc = subprocess.Popen(["bjobs"],stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
print proc.stdout.read()
I'm trying to process both stdout and stderr from a subprocess.Popen call that captures both via subprocess.PIPE but would like to handle the output (for example printing them on the terminal) as it comes.
All the current solutions that I've seen will wait for the completion of the Popen call to ensure that all of the stdout and stderr is captured so that then it can be processed.
This is an example Python script with mixed output that I can't seem to replicate the order when processing it in real time (or as real time as I can):
$ cat mix_out.py
import sys
sys.stdout.write('this is an stdout line\n')
sys.stdout.write('this is an stdout line\n')
sys.stderr.write('this is an stderr line\n')
sys.stderr.write('this is an stderr line\n')
sys.stderr.write('this is an stderr line\n')
sys.stdout.write('this is an stdout line\n')
sys.stderr.write('this is an stderr line\n')
sys.stdout.write('this is an stdout line\n')
The one approach that seems that it might work would be using threads, because then the reading would be asynchronous, and could be processed as subprocess is yielding the output.
The current implementation of this just process stdout first and stderr last, which can be deceiving if the output was originally alternating between both:
cmd = ['python', 'mix_out.py']
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=True,
**kw
)
if process.stdout:
while True:
out = process.stdout.readline()
if out == '' and process.poll() is not None:
break
if out != '':
print 'stdout: %s' % out
sys.stdout.flush()
if process.stderr:
while True:
err = process.stderr.readline()
if err == '' and process.poll() is not None:
break
if err != '':
print 'stderr: %s' % err
sys.stderr.flush()
If I run the above (saved as out.py) to handle the mix_out.py example script from above, the streams are (as expected) handled in order:
$ python out.py
stdout: this is an stdout line
stdout: this is an stdout line
stdout: this is an stdout line
stdout: this is an stdout line
stderr: this is an stderr line
stderr: this is an stderr line
stderr: this is an stderr line
stderr: this is an stderr line
I understand that some system calls might buffer, and I am OK with that, the one thing I am looking to solve is respecting the order of the streams as they happened.
Is there a way to be able to process both stdout and stderr as it comes from subprocess without having to use threads? (the code gets executed in restricted remote systems where threading is not possible).
The need to differentiate stdout from stderr is a must (as shown in the example output)
Ideally, no extra libraries would be best (e.g. I know pexpect solves this)
A lot of examples out there mention the use of select but I have failed to come up with something that would preserve the order of the output with it.
If you are looking for a way of having subprocess.Popen` output to stdout/stderr in realtime, you should be able to achieve that with:
import sys, subprocess
p = subprocess.Popen(cmdline,
stdout=sys.stdout,
stderr=sys.stderr)
Maybe using stderr=subprocess.STDOUT may simplify your filtering, IMO.
I found working example here (see listing of capture_together.py). Compiled C++ code that mixes cerr and cout executed as subprocess on both Windows and UNIX OSes. Results are identitical
I was able to solve this by using select.select()
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=True,
**kw
)
while True:
reads, _, _ = select(
[process.stdout.fileno(), process.stderr.fileno()],
[], []
)
for descriptor in reads:
if descriptor == process.stdout.fileno():
read = process.stdout.readline()
if read:
print 'stdout: %s' % read
if descriptor == process.stderr.fileno():
read = process.stderr.readline()
if read:
print 'stderr: %s' % read
sys.stdout.flush()
if process.poll() is not None:
break
By passing in the file descriptors to select() on the reads argument (first argument for select()) and looping over them (as long as process.poll()indicated that the process was still alive).
No need for threads. Code was adapted from this stackoverflow answer
I have a python (v3.3) script that runs other shell scripts. My python script also prints message like "About to run script X" and "Done running script X".
When I run my script I'm getting all the output of the shell scripts separate from my print statements. I see something like this:
All of script X's output
All of script Y's output
All of script Z's output
About to run script X
Done running script X
About to run script Y
Done running script Y
About to run script Z
Done running script Z
My code that runs the shell scripts looks like this:
print( "running command: " + cmnd )
ret_code = subprocess.call( cmnd, shell=True )
print( "done running command")
I wrote a basic test script and do *not* see this behaviour. This code does what I would expect:
print("calling")
ret_code = subprocess.call("/bin/ls -la", shell=True )
print("back")
Any idea on why the output is not interleaved?
Thanks. This works but has one limitation - you can't see any output until after the command completes. I found an answer from another question (here) that uses popen but also lets me see the output in real time. Here's what I ended up with this:
import subprocess
import sys
cmd = ['/media/sf_git/test-automation/src/SalesVision/mswm/shell_test.sh', '4', '2']
print('running command: "{0}"'.format(cmd)) # output the command.
# Here, we join the STDERR of the application with the STDOUT of the application.
process = subprocess.Popen(cmd, bufsize=1, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(process.stdout.readline, ''):
line = line.replace('\n', '')
print(line)
sys.stdout.flush()
process.wait() # Wait for the underlying process to complete.
errcode = process.returncode # Harvest its returncode, if needed.
print( 'Script ended with return code of: ' + str(errcode) )
This uses Popen and allows me to see the progress of the called script.
It has to do with STDOUT and STDERR buffering. You should be using subprocess.Popen to redirect STDOUT and STDERR from your child process into your application. Then, as needed, output them. Example:
import subprocess
cmd = ['ls', '-la']
print('running command: "{0}"'.format(cmd)) # output the command.
# Here, we join the STDERR of the application with the STDOUT of the application.
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process.wait() # Wait for the underlying process to complete.
out, err = process.communicate() # Capture what it outputted on STDOUT and STDERR
errcode = process.returncode # Harvest its returncode, if needed.
print(out)
print('done running command')
Additionally, I wouldn't use shell = True unless it's really required. It forces subprocess to fire up a whole shell environment just to run a command. It's usually better to inject directly into the env parameter of Popen.
So I am trying to store the output of a command into a variable. I do not want it to display output while running the command though...
The code I have right now is as follows...
def getoutput(*args):
myargs=args
listargs=[l.split(' ',1) for l in myargs]
import subprocess
output=subprocess.Popen(listargs[0], shell=False ,stdout=subprocess.PIPE)
out, error = output.communicate()
return(out,error)
def main():
a,b=getoutput("httpd -S")
if __name__ == '__main__':
main()
If I put this in a file and execute it on the command line. I get the following output even though I do not have a print statement in the code. How can I prevent this, while still storing the output?
#python ./apache.py
httpd: Could not reliably determine the server's fully qualified domain name, using xxx.xxx.xxx.xx for ServerName
Syntax OK
What you are seeing is standard-error output, not standard-output output. Stderr redirection is controlled by the stderr constructor argument. It defaults to None, which means no redirection occurs, which is why you see this output.
Usually it's a good idea to keep stderr output since it aids debugging and doesn't affect normal redirection (e.g. | and > shell redirection won't capture stderr by default). However you can redirect it somewhere else the same way you do stdout:
sp = subprocess.Popen(listargs[0], shell=False,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = sp.communicate()
Or you can just drop stderr:
devnull = open(os.devnull, 'wb') #python >= 2.4
sp = subprocess.Popen(listargs[0], shell=False,
stdout=subprocess.PIPE, stderr=devnull)
#python 3.x:
sp = subprocess.Popen(listargs[0], shell=False
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
You're catching stdout, but you're not catching stderr(standard error) which I think is where that message is coming from.
output=subprocess.Popen(listargs[0], shell=False ,stdout=subprocess.PIPE, stderr=STDOUT)
That will put anything from stderr into the same place as stdout.