In my file 'wrapper.py' I call a subprocess and print its output to stdout at realtime . This works just fine if I call the python script from the console. However, when calling it from a jupyter notebook the code hangs at the line proc.stdout.readline(). All previous print statements work fine..
proc = subprocess.Popen(["calc", "input.txt"], cwd=__dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
out = proc.stdout.readline().decode("utf-8")
err = proc.stderr.readline().decode("utf-8")
if out == '' and err == '' and proc.poll() is not None:
break
if out:
print("::::%s"%(out), end='')
if err:
print("::::%s"%(err), end='', file=sys.stderr)
rc = proc.poll()
print("---> returns %s"%(rc))
Does anyone have an idea on how to fix this?
I use this custom function to execute a command in Notebooks.
import subprocess
def run_cmd(cmd: str, stderr=subprocess.STDOUT) -> None:
"""Run a command in terminal
Args:
cmd (str): command to run in terminal
stderr (subprocess, optional): Where the error has to go. Defaults to subprocess.STDOUT.
Raises:
e: Excetion of the CalledProcessError
"""
out = None
try:
out = subprocess.check_output(
[cmd],
shell=True,
stderr=stderr,
universal_newlines=True,
)
except subprocess.CalledProcessError as e:
print(f'ERROR {e.returncode}: {cmd}\n\t{e.output}',
flush=True, file=sys.stderr)
raise e
print(out)
Usecase:
run_cmd("pip install emoji")
Related
I have a Python function that I made with the subprocess package:
def run_sh(command):
"""Print output of bash command"""
try:
process = Popen(shlex.split(command), stdout=PIPE)
for line in TextIOWrapper(process.stdout, newline=""):
print(line)
except CalledProcessError as e:
raise RuntimeError(
"command '{}' return with error (code {}): {}".format(
e.cmd, e.returncode, e.output
)
)
Let's say I want to run the following from within my Python script:
run_sh(newman run MY_COLLECTION.json "--env-var 'current_branch'=`git branch --show-current`")
Currently, it does not evaluate it as git branch --show-current but just treats it like test - how do I get it to evaluate it from my shell, and then run it?
Thanks!
Here's a code snippet that might help
import subprocess
def run_command(cmd):
try:
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
if out:
print(out.decode())
if err:
print(err.decode())
except Exception as e:
print(e)
if __name__ == '__main__':
run_command('ls -l')
run_command('ls -l /tmp')
run_command('ls -l /tmp1')
I am trying to make a terminal with python subprocess.I have this code but i don't know how to give it the inputs.
import subprocess
while True:
command = input("cmd>")
process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
while True:
line = process.stdout.readline()
if process.poll() is not None:
break
if line:
print(line.decode(), end="")
Example:
if run this python file with it how can type the message?
Command:
cmd>python test.py
#test.py
message = input("message:")
print(message)
I am running the diff command on MAC as follows and it works fine but when I run through python the --exclude option doesn't work meaning the command output still lists files under /Users/username/FWIntegration/branch_4355c1/.git, can anyone suggest how to debug this or how to fix this
/usr/bin/diff -x '.*' -x 'tech' -rq /Users/username/FWIntegration/repo2mirror /Users/username/FWIntegration/branch_4355c1 --exclude=/Users/username/FWIntegration/branch_4355c1/.git/
Running from python
cmd = "/usr/bin/diff -x '.*' -x 'tech' -rq /Users/username/FWIntegration/repo2mirror /Users/username/FWIntegration/branch_4355c1 --exclude=/Users/username/FWIntegration/branch_4355c1/.git/"
output,error = runCmd(cmd)
def runCmd(cmd):
out = ""
err = ""
logger.info("Running command %s"%cmd)
proc = Popen(cmd.split(' '), stdout=PIPE, stderr=PIPE)
try:
with proc.stdout as stdout:
for line in stdout:
print line,
out = out + line
#android_loader_output+=line
#if 'ERROR:' in line:
#print line
except:
print "%s failed"%cmd
print traceback.format_exc()
try:
with proc.stderr as stderr:
for line in stderr:
print line,
err = err + line
#android_loader_output+=line
#if 'ERROR:' in line:
#print line
except:
print "%s failed"%cmd
print traceback.format_exc()
#print out
#print err
return out,err
it lists like
Only in /Users/username/FWIntegration/branch_4355c1/.git/refs/tags: DIN2944T146R6_REL_9_74_5
Only in /Users/username/FWIntegration/branch_4355c1/.git/refs/tags: DIN2944T18R2_REL_9_48_1
Only in /Users/username/FWIntegration/branch_4355c1/.git/refs/tags: DIN2944T51R2_REL_9_55_2
The problem is due to the quotes you have in the cmd string. You're not processing the command with a shell, you're using cmd.split() to parse it, so those quotes are being sent literally to the program.
Use a shell so everything will be parsed properly:
proc = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
When running bash command using subprocess, I might run into situation where the command is not valid. In this case, bash would return an error messsage. How can we catch this message? I would like to save this message to a log file.
The following is an example, where I try to list files in a non-existed directory.
try:
subprocess.check_call(["ls", "/home/non"])
df = subprocess.Popen(["ls", "/home/non"], stdout=subprocess.PIPE)
output, err = df.communicate()
# process outputs
except Exception as error:
print error
sys.exit(1)
Bash would prints "ls: cannot access /home/non: No such file or directory". How can I get this error message? The error caught by the except line is clearly different, it says "Command '['ls', '/home/non']' returned non-zero exit status 2".
"ls: cannot access /home/non: No such file or directory" is generated by ls command, not bash here.
If you want to handle non-existing files using exception handling then use subprocess.check_output():
#!/usr/bin/env python
from subprocess import check_output, STDOUT, CalledProcessError
try:
output = check_output(['ls', 'nonexistent'], stderr=STDOUT)
except CalledProcessError as exc:
print(exc.output)
else:
assert 0
Output
ls: cannot access nonexistent: No such file or directory
You can redirect stderr to a file object:
from subprocess import PIPE, CalledProcessError, check_call, Popen
with open("log.txt", "w") as f:
try:
check_call(["ls", "/home/non"], stderr=f)
df = Popen(["ls", "/home/non"], stdout=PIPE)
output, err = df.communicate()
except CalledProcessError as e:
print(e)
exit(1)
Output to log.txt:
ls: cannot access /home/non: No such file or directory
If you want the message in the except:
try:
check_call(["ls", "/home/non"])
df = Popen(["ls", "/home/non"], stdout=PIPE)
output, err = df.communicate()
except CalledProcessError as e:
print(e.message)
For python 2.6 the e.message won't work. You can use a similar version of python 2.7's check_output that will work with python 2.6:
from subprocess import PIPE, CalledProcessError, Popen
def check_output(*args, **kwargs):
process = Popen(stdout=PIPE, *args, **kwargs)
out, err = process.communicate()
ret = process.poll()
if ret:
cmd = kwargs.get("args")
if cmd is None:
cmd = args[0]
error = CalledProcessError(ret, cmd)
error.out = out
error.message = err
raise error
return out
try:
out = check_output(["ls", "/home"], stderr=PIPE)
df = Popen(["ls", "/home/non"], stdout=PIPE)
output, err = df.communicate()
except CalledProcessError as e:
print(e.message)
else:
print(out)
Per the subprocess docs, the command run is now prefered.
Example:
import logging
import subprocess
logger = logging.getLogger(__name__)
try:
subprocess.run(["ls", "-l"], shell=True, check=True, capture_output=True)
except subprocess.CalledProcessError as err:
logger.error(f"{err} {err.stderr.decode('utf8')}")
As others have mentioned, if you want to save to a file, you cand use the stdout param to run; however, you may as well use logging to do that, and then just log the error in your method.
I am Using Python 2.7.1 on a Windows Server 2008 R2 x64 box.
I'm trying to get the output of a command line process which gives a nonzero exit status after outputting the information I need.
I was initially using subprocess.check_output, and catching the CalledProcessError which occurs with nonzero exit status, but while the returncode was stored in the error, no output revealed this.
Running this against cases which give output but have an exit status of 0 works properly and I can get the output using subprocess.check_output.
My assumption was that the output was being written to STDOUT but the exception pulls its 'output' from STDERR. I've tried to re implement the functionality of check_output, but I still get nothing on the output when I believe I should be seeing output to STDOUT and STDERR. My current code is below (where 'command' is the full text, including parameters, of command I am running:
process = subprocess.Popen(command, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, universal_newlines=True)
output = process.communicate()
retcode = process.poll()
if retcode:
raise subprocess.CalledProcessError(retcode, image_check, output=output)
return output
This gives me the following in the variable output: [('', None)]
Is my subprocess.Popen code correct?
You code works fine. Turns out that the process that you are calling is probably outputing to CON. See the following example
import subprocess
def check_output(command):
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
output = process.communicate()
retcode = process.poll()
if retcode:
raise subprocess.CalledProcessError(retcode, command, output=output[0])
return output
command = "echo this>CON"
print "subprocess -> " + subprocess.check_output(command, shell=True)
print "native -> " + str(check_output(command))
try:
subprocess.check_output("python output.py", shell=True)
except subprocess.CalledProcessError, e:
print "subproces CalledProcessError.output = " + e.output
try:
check_output("python output.py")
except subprocess.CalledProcessError, e:
print "native CalledProcessError.output = " + e.output
Output
subprocess ->
native -> ('', None)
stderr subproces CalledProcessError.output = stdout
native CalledProcessError.output = stderr stdout
Sadly, I do not know how to resolve the issue. Notice that subprocess.check_output results contains only the output from stdout. Your check_output replacement would output both stderr and stdout.
After inspecting subprocess.check_output, it does indeed generate a CalledProcessError with the output containing only stdout.
Have you tried stderr=subprocess.STDOUT as mentioned in the python doc page:
To also capture standard error in the result, use
stderr=subprocess.STDOUT:
Here is a test code:
import subprocess
try:
subprocess.check_output('>&2 echo "errrrr"; exit 1', shell=True)
except subprocess.CalledProcessError as e:
print 'e.output: ', e.output
try:
subprocess.check_output('>&2 echo "errrrr"; exit 1', shell=True, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
print 'e.output: ', e.output
output:
errrrr
e.output:
e.output: errrrr
There is an issue here that might be hitting you-
http://bugs.python.org/issue9905