I have a Python function that I made with the subprocess package:
def run_sh(command):
"""Print output of bash command"""
try:
process = Popen(shlex.split(command), stdout=PIPE)
for line in TextIOWrapper(process.stdout, newline=""):
print(line)
except CalledProcessError as e:
raise RuntimeError(
"command '{}' return with error (code {}): {}".format(
e.cmd, e.returncode, e.output
)
)
Let's say I want to run the following from within my Python script:
run_sh(newman run MY_COLLECTION.json "--env-var 'current_branch'=`git branch --show-current`")
Currently, it does not evaluate it as git branch --show-current but just treats it like test - how do I get it to evaluate it from my shell, and then run it?
Thanks!
Here's a code snippet that might help
import subprocess
def run_command(cmd):
try:
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
if out:
print(out.decode())
if err:
print(err.decode())
except Exception as e:
print(e)
if __name__ == '__main__':
run_command('ls -l')
run_command('ls -l /tmp')
run_command('ls -l /tmp1')
Related
In my file 'wrapper.py' I call a subprocess and print its output to stdout at realtime . This works just fine if I call the python script from the console. However, when calling it from a jupyter notebook the code hangs at the line proc.stdout.readline(). All previous print statements work fine..
proc = subprocess.Popen(["calc", "input.txt"], cwd=__dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
out = proc.stdout.readline().decode("utf-8")
err = proc.stderr.readline().decode("utf-8")
if out == '' and err == '' and proc.poll() is not None:
break
if out:
print("::::%s"%(out), end='')
if err:
print("::::%s"%(err), end='', file=sys.stderr)
rc = proc.poll()
print("---> returns %s"%(rc))
Does anyone have an idea on how to fix this?
I use this custom function to execute a command in Notebooks.
import subprocess
def run_cmd(cmd: str, stderr=subprocess.STDOUT) -> None:
"""Run a command in terminal
Args:
cmd (str): command to run in terminal
stderr (subprocess, optional): Where the error has to go. Defaults to subprocess.STDOUT.
Raises:
e: Excetion of the CalledProcessError
"""
out = None
try:
out = subprocess.check_output(
[cmd],
shell=True,
stderr=stderr,
universal_newlines=True,
)
except subprocess.CalledProcessError as e:
print(f'ERROR {e.returncode}: {cmd}\n\t{e.output}',
flush=True, file=sys.stderr)
raise e
print(out)
Usecase:
run_cmd("pip install emoji")
I am converting bash code to python.
I use mkdir of bash through subprocess.run() in python.
In the following example, subprocess.run() raise an exception.
However I could not check what the error is because I could not
get an resultant object returned by subprocess.run().
Are there any smart ways to know what the error was?
Or should not I use 'try exception' here?
import sys
import subprocess
directory = '/tmp/test_dir'
options = ''
try:
result=subprocess.run(['mkdir', options, directory], check=True)
except subprocess.CalledProcessError as ex:
print("In this example, subprocess.run() above raise an exception CalledProcessError.")
# print("I would like to check result.returncode = {0}. But it failed because object 'result' is not defined.".format(result.returncode))
except Exception as ex:
sys.stderr.write("This must not happen.")
sys.exit(1)
Thank you very much.
you can always do
import subprocess
# make the subprocess
pr = subprocess.Popen(['your', 'command', 'here'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# blocks until process finishes
out, err = pr.communicate()
# check the return code
if pr.returncode != 0:
sys.stderr.write("oh no")
When running bash command using subprocess, I might run into situation where the command is not valid. In this case, bash would return an error messsage. How can we catch this message? I would like to save this message to a log file.
The following is an example, where I try to list files in a non-existed directory.
try:
subprocess.check_call(["ls", "/home/non"])
df = subprocess.Popen(["ls", "/home/non"], stdout=subprocess.PIPE)
output, err = df.communicate()
# process outputs
except Exception as error:
print error
sys.exit(1)
Bash would prints "ls: cannot access /home/non: No such file or directory". How can I get this error message? The error caught by the except line is clearly different, it says "Command '['ls', '/home/non']' returned non-zero exit status 2".
"ls: cannot access /home/non: No such file or directory" is generated by ls command, not bash here.
If you want to handle non-existing files using exception handling then use subprocess.check_output():
#!/usr/bin/env python
from subprocess import check_output, STDOUT, CalledProcessError
try:
output = check_output(['ls', 'nonexistent'], stderr=STDOUT)
except CalledProcessError as exc:
print(exc.output)
else:
assert 0
Output
ls: cannot access nonexistent: No such file or directory
You can redirect stderr to a file object:
from subprocess import PIPE, CalledProcessError, check_call, Popen
with open("log.txt", "w") as f:
try:
check_call(["ls", "/home/non"], stderr=f)
df = Popen(["ls", "/home/non"], stdout=PIPE)
output, err = df.communicate()
except CalledProcessError as e:
print(e)
exit(1)
Output to log.txt:
ls: cannot access /home/non: No such file or directory
If you want the message in the except:
try:
check_call(["ls", "/home/non"])
df = Popen(["ls", "/home/non"], stdout=PIPE)
output, err = df.communicate()
except CalledProcessError as e:
print(e.message)
For python 2.6 the e.message won't work. You can use a similar version of python 2.7's check_output that will work with python 2.6:
from subprocess import PIPE, CalledProcessError, Popen
def check_output(*args, **kwargs):
process = Popen(stdout=PIPE, *args, **kwargs)
out, err = process.communicate()
ret = process.poll()
if ret:
cmd = kwargs.get("args")
if cmd is None:
cmd = args[0]
error = CalledProcessError(ret, cmd)
error.out = out
error.message = err
raise error
return out
try:
out = check_output(["ls", "/home"], stderr=PIPE)
df = Popen(["ls", "/home/non"], stdout=PIPE)
output, err = df.communicate()
except CalledProcessError as e:
print(e.message)
else:
print(out)
Per the subprocess docs, the command run is now prefered.
Example:
import logging
import subprocess
logger = logging.getLogger(__name__)
try:
subprocess.run(["ls", "-l"], shell=True, check=True, capture_output=True)
except subprocess.CalledProcessError as err:
logger.error(f"{err} {err.stderr.decode('utf8')}")
As others have mentioned, if you want to save to a file, you cand use the stdout param to run; however, you may as well use logging to do that, and then just log the error in your method.
I am writing a script which can take a file name as input, compile it and run it.
I am taking the name of a file as input(input_file_name). I first compile the file from within python:
self.process = subprocess.Popen(['gcc', input_file_name, '-o', 'auto_gen'], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT, shell=False)
Next, I'm executing the executable using the same(Popen) call:
subprocess.Popen('./auto_gen', stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT, shell=False)
In both cases, I'm catching the stdout(and stderr) contents using
(output, _) = self.process.communicate()
Now, if there is an error during compilation, I am able to catch the error because the returncode is 1 and I can get the details of the error because gcc sends them on stderr.
However, the program itself can return a random value even on executing successfully(because there might not be a "return 0" at the end). So I can't catch runtime errors using the returncode. Moreover, the executable does not send the error details on stderr. So I can't use the trick I used for catching compile-time errors.
What is the best way to catch a runtime error OR to print the details of the error? That is, if ./auto_gen throws a segmentation fault, I should be able to print either one of:
'Runtime error'
'Segmentation Fault'
'Program threw a SIGSEGV'
Try this. The code runs a subprocess which fails and prints to stderr. The except block captures the specific error exit code and stdout/stderr, and displays it.
#!/usr/bin/env python
import subprocess
try:
out = subprocess.check_output(
"ls non_existent_file",
stderr=subprocess.STDOUT,
shell=True)
print 'okay:',out
except subprocess.CalledProcessError as exc:
print 'error: code={}, out="{}"'.format(
exc.returncode, exc.output,
)
Example output:
$ python ./subproc.py
error: code=2, out="ls: cannot access non_existent_file: No such file or directory
"
If ./autogen is killed by a signal then self.process.returncode (after .wait() or .communicate()) is less than zero and its absolute value reports the signal e.g., returncode == -11 for SIGSERV.
please check following link for runtime errors or output of subprocess
https://www.endpoint.com/blog/2015/01/28/getting-realtime-output-using-python
def run_command(command):
process = subprocess.Popen(shlex.split(command),
stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print output.strip()
rc = process.poll()
return rc
I am Using Python 2.7.1 on a Windows Server 2008 R2 x64 box.
I'm trying to get the output of a command line process which gives a nonzero exit status after outputting the information I need.
I was initially using subprocess.check_output, and catching the CalledProcessError which occurs with nonzero exit status, but while the returncode was stored in the error, no output revealed this.
Running this against cases which give output but have an exit status of 0 works properly and I can get the output using subprocess.check_output.
My assumption was that the output was being written to STDOUT but the exception pulls its 'output' from STDERR. I've tried to re implement the functionality of check_output, but I still get nothing on the output when I believe I should be seeing output to STDOUT and STDERR. My current code is below (where 'command' is the full text, including parameters, of command I am running:
process = subprocess.Popen(command, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, universal_newlines=True)
output = process.communicate()
retcode = process.poll()
if retcode:
raise subprocess.CalledProcessError(retcode, image_check, output=output)
return output
This gives me the following in the variable output: [('', None)]
Is my subprocess.Popen code correct?
You code works fine. Turns out that the process that you are calling is probably outputing to CON. See the following example
import subprocess
def check_output(command):
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
output = process.communicate()
retcode = process.poll()
if retcode:
raise subprocess.CalledProcessError(retcode, command, output=output[0])
return output
command = "echo this>CON"
print "subprocess -> " + subprocess.check_output(command, shell=True)
print "native -> " + str(check_output(command))
try:
subprocess.check_output("python output.py", shell=True)
except subprocess.CalledProcessError, e:
print "subproces CalledProcessError.output = " + e.output
try:
check_output("python output.py")
except subprocess.CalledProcessError, e:
print "native CalledProcessError.output = " + e.output
Output
subprocess ->
native -> ('', None)
stderr subproces CalledProcessError.output = stdout
native CalledProcessError.output = stderr stdout
Sadly, I do not know how to resolve the issue. Notice that subprocess.check_output results contains only the output from stdout. Your check_output replacement would output both stderr and stdout.
After inspecting subprocess.check_output, it does indeed generate a CalledProcessError with the output containing only stdout.
Have you tried stderr=subprocess.STDOUT as mentioned in the python doc page:
To also capture standard error in the result, use
stderr=subprocess.STDOUT:
Here is a test code:
import subprocess
try:
subprocess.check_output('>&2 echo "errrrr"; exit 1', shell=True)
except subprocess.CalledProcessError as e:
print 'e.output: ', e.output
try:
subprocess.check_output('>&2 echo "errrrr"; exit 1', shell=True, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
print 'e.output: ', e.output
output:
errrrr
e.output:
e.output: errrrr
There is an issue here that might be hitting you-
http://bugs.python.org/issue9905