allocating I want to launch a process and retrieve the stdout and stderr.
I don t really care about getting this in real time.
I wanted to use subprocess.check_ouput(), but the process might fail.
After reading StackOverflow and the Python docs I added a try .. catch block:
def execute(cmd,timeinsec=60):
print("execute ",cmd, " with time out ",timeinsec)
try:
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, timeout=timeinsec,universal_newlines=True)
except subprocess.TimeoutExpired:
print("timeout expired")
return "",2
except subprocess.CalledProcessError:
print("called process failed")
return "",1
print ('The command returned the following back to python:'+output)
return output,0
But when I print the output with output.decode('utf-8')
I just get the first line of the output.
Note : I'm running this in the MSys environment distributed with Msysgit 1.8 on windows.
Do you have any idea of what can be wrong?
Do you know any better way to do this?
You must be using python-3.x. Please tag your question accordingly. Also, I am not sure why you are calling read() method of output. The output is a byte string and does not have a read() method. The following code works for me:
#! /usr/bin/env python3
import subprocess
try :
retcode = 0
cmd = ["/bin/ls", "/usr/local"]
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
output = e.output
retcode = e.returncode
print(output.decode('utf-8'))
print(retcode)
The output is:
bin
etc
games
include
lib
man
sbin
share
src
0
If I trigger an error by replacing /usr/local with /usr/localfoo (which does not exist), then the output is:
/bin/ls: cannot access /usr/localfoo: No such file or directory
2
Finally, you can add universal_newlines=True to check_output() call and not have to worry about calling decode() on the output:
...
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT,universal_newlines=True)
...
...
print(output)
Please take the above example and see if you can make it reproduce your problem. If you can reproduce the problem, please post your code, its output, and all error messages (copy, paste, and reformat for SO).
Solution
The problem was that the application in windows launched in the subprocess was allocating a console (built using Visual) and the stdout of the process was already redirected outside, and only one print was done in the original cout before this redirection
Related
Is there a doc somewhere which indicates what the different return codes of python's subprocess check_output() command means? I'm seeing the returncode equal 3, and I have no idea what is setting that value and/or what it means.
Example code:
try:
output = subprocess.check_output(cmd,
stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
print e.returncode
This code is printing 3.
The Python subprocess.check_output() doesn't itself return a code, it returns the output of the command being run. The doco can be found here.
If you're seeing an error code in that string, it's almost certainly specific to whatever you're trying to run, not a result of subprocess itself.
If, however, you're capturing the CalledProcessError exception caused by a non-zero return code (which can be extracted from the returncode attribute), that's still specific to whatever you're running.
In both cases, the documentation you need to check is that of whatever tool subprocess is running.
I am creating and using the QGIS tool PlugIn myself.
In conclusion, the plugin needs logic to ensure that the user has Java installed.
So I try to run java -version and pass the output when it comes out.
However, the Java version is not printed.
It is my source.
try:
check_process = subprocess.Popen(["java", "-version", "2>&1"], stderr=subprocess.PIPE)
check_process = check_process.communicate()
# this is print func
QgsMessageLog.logMessage(str(check_process), tag="Validating", level=QgsMessageLog.INFO)
except Exception as e:
QgsMessageLog.logMessage(str(e), tag="Validating", level=QgsMessageLog.INFO)
return
and result is
2018-09-21T09:36:21 0 (None, '')
If you have any idea, I would appreciate your advice. Thank you.
Question: subprocess is not working
You are using 2>&1, this is as Shell command and will not work until you use shell=True.
You are right to, redirect stderr to stdout, as java -version will write to stderr.
Do this for example: (Note the diffs, no list and stdout=, to yours!)
check_process = subprocess.Popen("java -version 2>&1", shell=True, stdout=subprocess.PIPE)
As this will get the expected output for me, you get (None, '') using:
check_process = subprocess.Popen(["java", "-version", "2>&1"], stderr=subprocess.PIPE)
The first tuple is the output of stdout which is not used in Popen.
The second tuple is the output of stderr which is empty string.
For testing purpose try inside QGIS:
result = subprocess.check_output(["echo", "Hello World!"])
print(result)
I'm running some terminal commands from within Python using the subprocess.check_output() call. This works all fine when it returns correct results. When interacting with the bitcoin daemon I can get several responses however. For example, the raw output (as seen on a normal bash command line) can be:
: Cannot obtain a lock on data directory /home/kramer65/.bitcoin. Bitcoin is probably already running.
or
error: couldn't connect to server
Both these responses give an error code 1 however. I tried printing out the following attributes of the subprocess.CalledProcessError as e, which in both cases results in the same outputs (except for the cmd attribute of course:
print e.args, e.cmd, e.message, e.output, e.returncode
# () 'BTC' 1
# () 'BTC getinfo' 1
I guess the only thing that distincts the two errors, is the raw string that is outputted on the command line which I listed above. So my question is: how can I get the raw string that is shown on the command line from within Python?
It is likely coming on the stderr.
You could setup your subprocess with
stderr=subprocess.STDOUT
as a kwarg to merge the stdout and stderr together.
Alternatively, if you need them separately, do something like
proc = subprocess.Popen(..., stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()
Note: unlike check_output, this method will not raise exception if the return code was nonzero, so you will have to do that manually if that's the behaviour you wanted.
I have a Python script which executes a system call (the call is actually a java program).
I am using Popen.communicate() to attempt to capture the stdout and stderr. However, it only works for some Java applications (of which, ALL the java applications I'm calling do display something to the screen when executed directly in the shell).
The code I am using is as follows:
# Perform the java call
try:
p = subprocess.Popen(args,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
except OSError as e:
error_msg = ' Error reading file while executing system command: {0}'.format(e.message)
except ValueError as e:
error_msg = ' Error in arugments while executing system command: {0}'.format(e.message)
else:
# Display the results in real time
for line in iter(p.stdout.readline, ''):
sys.stdout.write(line)
# Grab the stdout and stderr from the process.
output, err = p.communicate()
finally:
# Check to see if an internal error occurred. If not, flip the flag
if not err and not error_msg:
java_completed = True
else:
java_completed = False
In some cases, the line sys.stdout.write(line) is never called, and in other's it is.
Does anyone have any experience with this?
The first thing I notice is that you're reading everything from the process stdout, and then trying to do communicate afterwards: you've already read everything from the pipe, so the output result from p.communicate() will always be empty.
But that doesn't explain why sometimes the sys.stdout.write is not being called.
One thing to keep in mind is that there are multiple ways to display things in a console (i.e., in a command window). Printing to the processes stdout stream is only one way, and that's the way that you'll be able to capture with the above code. If I recall from Java days, there is some type of Console class that is sometimes used for displaying to the console, this may not use stdout. Also, advanced console display libraries, like curses for instance, don't generally write to stdout, they interact with the console object directly. I don't know how, or if, you can capture output from these types of processes, but if it is possible it would probably be very system-specific.
I have some Python code that works correctly when I use python.exe to run it, but fails if I use pythonw.exe.
def runStuff(commandLine):
outputFileName = 'somefile.txt'
outputFile = open(outputFileName, "w")
try:
result = subprocess.call(commandLine, shell=True, stdout=outputFile)
except:
print 'Exception thrown:', str(sys.exc_info()[1])
myThread = threading.Thread(None, target=runStuff, commandLine=['whatever...'])
myThread.start()
The message I get is:
Exception thrown: [Error 6] The handle is invalid
However, if I don't specify the 'stdout' parameter, subprocess.call() starts okay.
I can see that pythonw.exe might be redirecting output itself, but I can't see why I'm blocked from specifying stdout for a new thread.
sys.stdin and sys.stdout handles are invalid because pythonw does not provide console support as it runs as a deamon, so default arguments of subprocess.call() are failing.
Deamon programs close stdin/stdout/stderr purposedly and use logging instead, so that you have to manage this yourself: I would suggest to use subprocess.PIPE.
If you really don't care about what the sub process says for errors and all, you could use os.devnull (I'm not really sure how portable it is?) but I wouldn't recommend that.
For the record, my code now looks like this:
def runStuff(commandLine):
outputFileName = 'somefile.txt'
outputFile = open(outputFileName, "w")
if guiMode:
result = subprocess.call(commandLine, shell=True, stdout=outputFile, stderr=subprocess.STDOUT)
else:
proc = subprocess.Popen(commandLine, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
proc.stdin.close()
proc.wait()
result = proc.returncode
outputFile.write(proc.stdout.read())
Note that, due to an apparent bug in the subprocess module, the call to Popen() has to specify a pipe for stdin as well, which we close immediately afterwards.
This is an old question, but the same problem happened with pyInstaller too.
In the truth, this will happen with any framework that converts code in python for exe without console.
In my tests, I observed that if I use the flag "console=True" into my spec file (pyInstaller) the error no longer occurs. .
The solution was follow the tip of Piotr Lesnicki.