When running bash command using subprocess, I might run into situation where the command is not valid. In this case, bash would return an error messsage. How can we catch this message? I would like to save this message to a log file.
The following is an example, where I try to list files in a non-existed directory.
try:
subprocess.check_call(["ls", "/home/non"])
df = subprocess.Popen(["ls", "/home/non"], stdout=subprocess.PIPE)
output, err = df.communicate()
# process outputs
except Exception as error:
print error
sys.exit(1)
Bash would prints "ls: cannot access /home/non: No such file or directory". How can I get this error message? The error caught by the except line is clearly different, it says "Command '['ls', '/home/non']' returned non-zero exit status 2".
"ls: cannot access /home/non: No such file or directory" is generated by ls command, not bash here.
If you want to handle non-existing files using exception handling then use subprocess.check_output():
#!/usr/bin/env python
from subprocess import check_output, STDOUT, CalledProcessError
try:
output = check_output(['ls', 'nonexistent'], stderr=STDOUT)
except CalledProcessError as exc:
print(exc.output)
else:
assert 0
Output
ls: cannot access nonexistent: No such file or directory
You can redirect stderr to a file object:
from subprocess import PIPE, CalledProcessError, check_call, Popen
with open("log.txt", "w") as f:
try:
check_call(["ls", "/home/non"], stderr=f)
df = Popen(["ls", "/home/non"], stdout=PIPE)
output, err = df.communicate()
except CalledProcessError as e:
print(e)
exit(1)
Output to log.txt:
ls: cannot access /home/non: No such file or directory
If you want the message in the except:
try:
check_call(["ls", "/home/non"])
df = Popen(["ls", "/home/non"], stdout=PIPE)
output, err = df.communicate()
except CalledProcessError as e:
print(e.message)
For python 2.6 the e.message won't work. You can use a similar version of python 2.7's check_output that will work with python 2.6:
from subprocess import PIPE, CalledProcessError, Popen
def check_output(*args, **kwargs):
process = Popen(stdout=PIPE, *args, **kwargs)
out, err = process.communicate()
ret = process.poll()
if ret:
cmd = kwargs.get("args")
if cmd is None:
cmd = args[0]
error = CalledProcessError(ret, cmd)
error.out = out
error.message = err
raise error
return out
try:
out = check_output(["ls", "/home"], stderr=PIPE)
df = Popen(["ls", "/home/non"], stdout=PIPE)
output, err = df.communicate()
except CalledProcessError as e:
print(e.message)
else:
print(out)
Per the subprocess docs, the command run is now prefered.
Example:
import logging
import subprocess
logger = logging.getLogger(__name__)
try:
subprocess.run(["ls", "-l"], shell=True, check=True, capture_output=True)
except subprocess.CalledProcessError as err:
logger.error(f"{err} {err.stderr.decode('utf8')}")
As others have mentioned, if you want to save to a file, you cand use the stdout param to run; however, you may as well use logging to do that, and then just log the error in your method.
Related
I have a Python function that I made with the subprocess package:
def run_sh(command):
"""Print output of bash command"""
try:
process = Popen(shlex.split(command), stdout=PIPE)
for line in TextIOWrapper(process.stdout, newline=""):
print(line)
except CalledProcessError as e:
raise RuntimeError(
"command '{}' return with error (code {}): {}".format(
e.cmd, e.returncode, e.output
)
)
Let's say I want to run the following from within my Python script:
run_sh(newman run MY_COLLECTION.json "--env-var 'current_branch'=`git branch --show-current`")
Currently, it does not evaluate it as git branch --show-current but just treats it like test - how do I get it to evaluate it from my shell, and then run it?
Thanks!
Here's a code snippet that might help
import subprocess
def run_command(cmd):
try:
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
if out:
print(out.decode())
if err:
print(err.decode())
except Exception as e:
print(e)
if __name__ == '__main__':
run_command('ls -l')
run_command('ls -l /tmp')
run_command('ls -l /tmp1')
I call an external binary using the subprocess module:
try:
subprocess.check_output([param1, param2], stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
print(e.output)
This outputs an error both when the check_output runs and in the except statement.
What I really want to have is to print the output only if I catch an exception. So that means I want to have stderr in the exception object.
The problem is that if I suppress the stdout:
subprocess.check_output(..., stdout=open(sys.os.devnull, 'wb'))
Then e.output sure becomes empty. Also I tried setting stderr to None - the same effect.
How to add stderr to the subprocess.CalledProcessError instance without redirecting stderr to stdout because I do not need stdout at all.
check_output is a convenience function with limited functionality. If it doesn't do what you want, roll your own:
proc = subprocess.Popen([param1, param2],
stderr=subprocess.PIPE)
out, err = proc.communicate()
if proc.returncode != 0:
print(err)
In my file 'wrapper.py' I call a subprocess and print its output to stdout at realtime . This works just fine if I call the python script from the console. However, when calling it from a jupyter notebook the code hangs at the line proc.stdout.readline(). All previous print statements work fine..
proc = subprocess.Popen(["calc", "input.txt"], cwd=__dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
out = proc.stdout.readline().decode("utf-8")
err = proc.stderr.readline().decode("utf-8")
if out == '' and err == '' and proc.poll() is not None:
break
if out:
print("::::%s"%(out), end='')
if err:
print("::::%s"%(err), end='', file=sys.stderr)
rc = proc.poll()
print("---> returns %s"%(rc))
Does anyone have an idea on how to fix this?
I use this custom function to execute a command in Notebooks.
import subprocess
def run_cmd(cmd: str, stderr=subprocess.STDOUT) -> None:
"""Run a command in terminal
Args:
cmd (str): command to run in terminal
stderr (subprocess, optional): Where the error has to go. Defaults to subprocess.STDOUT.
Raises:
e: Excetion of the CalledProcessError
"""
out = None
try:
out = subprocess.check_output(
[cmd],
shell=True,
stderr=stderr,
universal_newlines=True,
)
except subprocess.CalledProcessError as e:
print(f'ERROR {e.returncode}: {cmd}\n\t{e.output}',
flush=True, file=sys.stderr)
raise e
print(out)
Usecase:
run_cmd("pip install emoji")
Error message:
Failed to open output_file_path/**.txt
Code:
cmd = 'showTxt "%s" > "%s"' % (file_path, output_file_path)
LoggerInstance.log('[cmd] '+cmd)
#os.system(cmd)
splited_cmd=shlex.split(cmd)
p = subprocess.Popen(splited_cmd, stderr=subprocess.PIPE)
#p.wait()
output = p.stderr.read()
print output
LoggerInstance.log('[console std error]'+ output)
How to redirect stdout to a file in a cmd?
You can provide a file-handler as stdout parameter to Popen, i.e:
p = subprocess.Popen(splited_cmd,
stderr=subprocess.PIPE,
stdout=open(output_file_path, "w"))
Of course, be ready to catch the exception that it can throw.
I am Using Python 2.7.1 on a Windows Server 2008 R2 x64 box.
I'm trying to get the output of a command line process which gives a nonzero exit status after outputting the information I need.
I was initially using subprocess.check_output, and catching the CalledProcessError which occurs with nonzero exit status, but while the returncode was stored in the error, no output revealed this.
Running this against cases which give output but have an exit status of 0 works properly and I can get the output using subprocess.check_output.
My assumption was that the output was being written to STDOUT but the exception pulls its 'output' from STDERR. I've tried to re implement the functionality of check_output, but I still get nothing on the output when I believe I should be seeing output to STDOUT and STDERR. My current code is below (where 'command' is the full text, including parameters, of command I am running:
process = subprocess.Popen(command, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, universal_newlines=True)
output = process.communicate()
retcode = process.poll()
if retcode:
raise subprocess.CalledProcessError(retcode, image_check, output=output)
return output
This gives me the following in the variable output: [('', None)]
Is my subprocess.Popen code correct?
You code works fine. Turns out that the process that you are calling is probably outputing to CON. See the following example
import subprocess
def check_output(command):
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
output = process.communicate()
retcode = process.poll()
if retcode:
raise subprocess.CalledProcessError(retcode, command, output=output[0])
return output
command = "echo this>CON"
print "subprocess -> " + subprocess.check_output(command, shell=True)
print "native -> " + str(check_output(command))
try:
subprocess.check_output("python output.py", shell=True)
except subprocess.CalledProcessError, e:
print "subproces CalledProcessError.output = " + e.output
try:
check_output("python output.py")
except subprocess.CalledProcessError, e:
print "native CalledProcessError.output = " + e.output
Output
subprocess ->
native -> ('', None)
stderr subproces CalledProcessError.output = stdout
native CalledProcessError.output = stderr stdout
Sadly, I do not know how to resolve the issue. Notice that subprocess.check_output results contains only the output from stdout. Your check_output replacement would output both stderr and stdout.
After inspecting subprocess.check_output, it does indeed generate a CalledProcessError with the output containing only stdout.
Have you tried stderr=subprocess.STDOUT as mentioned in the python doc page:
To also capture standard error in the result, use
stderr=subprocess.STDOUT:
Here is a test code:
import subprocess
try:
subprocess.check_output('>&2 echo "errrrr"; exit 1', shell=True)
except subprocess.CalledProcessError as e:
print 'e.output: ', e.output
try:
subprocess.check_output('>&2 echo "errrrr"; exit 1', shell=True, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
print 'e.output: ', e.output
output:
errrrr
e.output:
e.output: errrrr
There is an issue here that might be hitting you-
http://bugs.python.org/issue9905