I am trying to send two simple commands using subprocess.run & trying to store results in a variable then print it but for one arg the output is coming for subprocess.run & for other its empty
Arg are "help" & "adb devices"
command I am sending which returns the output
result = subprocess.run("help", capture_output=True, text=True, universal_newlines=True)
print(result.stdout)
but this command with a different arg is not returning
result = subprocess.run("adb devices", capture_output=True, text=True, universal_newlines=True)
print(result.stdout)
If I try the same command with subprocess.checkoutput it returns the output can anyone explain what exactly is going on here
Is there any specific usage scenario's for these command's like when to use which one ?
c = subprocess.check_output(
"adb devices", shell=True, stderr=subprocess.STDOUT)
print(c)
output - b'List of devices attached\r\n\r\n'
It is because from the python documentation here:
run method
run method accepts the first parameter as arguments and not string.
So you can try passing the arguments in a list as:
result = subprocess.run(['abd', 'devices'], capture_output=True, text=True, universal_newlines=True)
Also,
check_output method accepts args but it has a parameter call "shell = True" Therefore, it works for multi-word args.
If you want to use the run method without a list, add shell=True in the run method parameter. (I tried for "man ls" command and it worked).
Related
I am new to the subprocess module, and wondering why the first subprocess failed while the second one worked. I am on py3.7 and macOS.
>>> from subprocess import PIPE, Popen, STDOUT
>>> Popen(['touch', '/Users/me/fail.txt'], stdout=PIPE, stderr=STDOUT, shell=True)
>>> Popen(['touch /Users/me/ok.txt'], stdout=PIPE, stderr=STDOUT, shell=True)
According to the docs:
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence.
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
So in the first case the 2nd element of the list is passed as an argument to /bin/sh itself, not the touch command. So you are basically running:
user#name ~$ touch
Which produces the following error:
touch: missing file operand
Try 'touch --help' for more information.
And if you read the stdout of your first command, you will find the same:
>>> Popen(['touch', '/Users/me/fail.txt'], stdout=PIPE, stderr=STDOUT, shell=True).stdout.read()
b"touch: missing file operand\nTry 'touch --help' for more information.\n"
So while shell=True, it is better to pass string.
In subprocess.run which is a high-level function, you need to pass the arguments as a list but for Popen, which is a low-level function needs a direct command hence the first one failed but the second one worked.
On Windows I successfully run this:
cmd = ["gfortran", "test.f90", "-o", "test.exe", "&&", "test.exe"]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
print(p.stdout.read())
On Ubuntu I change cmd to:
cmd = ["gfortran", "test.f90", "-o", "test", "&&", "./test"]
and get:
gfortran: fatal error: no input files
compilation terminated.
I want to retain cmd as a list, instead making it a string.
How can I make this work?
It's impossible. When cmd is a list, it has different meaning when shell is True. Quoting docs:
If args is a sequence, the first item specifies the command string,
and any additional items will be treated as additional arguments to
the shell itself.
Use ' '.join(shlex.quote(arg) for arg in cmd) (pipes.quote in Python2) when passing list of arguments to Popen with shell=True for expected behavior. Original list won't be mutated, string will be built before passing to function and garbage collected as soon as it's possible.
This is what the docs state:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself.
If you want to retain the arguments as a list, you can join them while calling Popen:
p = subprocess.Popen(' '.join(cmd), stdout=subprocess.PIPE, shell=True)
It appears that you are using shell=True to allow you to do && on the command line. This is not strictly necessary if you are already using subprocess anyway:
cmd1 = ["gfortran", "test.f90", "-o", "test.exe"]
cmd2 = ["test.exe"]
if not subprocess.call(cmd1):
p = subprocess.Popen(cmd2, ...)
out, err = p.communicate()
...
I am using subprocess.Popen to execute an OS command. Here is what I am trying to emulate in my code:
ssh -T myhost < /path/to/some/file
It works fine like this:
def runWorkerCode(filer, filename):
command = "/usr/bin/ssh -T " + filer + " < /devel/myscript"
try:
p = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
out, _ = p.communicate()
except Exception:
print "Error: %s" % Exception
sys.exit(1)
return out.rstrip().split('\n')
But the following calls to Popen do not work:
p = subprocess.Popen(["/usr/bin/ssh", "-T", filer, "<", "/devel/myscript"], stdout=subprocess.PIPE, shell=True)
p = subprocess.Popen(["/usr/bin/ssh -T", filer, "< /devel/myscript"], stdout=subprocess.PIPE, shell=True)
I tried a few other combinations but only method I can get to work is defining the command variable and only providing it to Popen(). I've also tried shell=False.
The first method works but the latter approach seems "cleaner" to me.
Why doesn't Popen allow me to specify the arguments in a list?
When you use shell=True on UNIX, you should provide your arguments as a string. When you provide a list, subprocess interprets the first item in the list as your entire command string, and the rest of the items in the list as arguments passed to the shell itself, rather than your command. So in your example above, you're ending up with something like this:
/bin/sh -T filer < /dev/myscript -c "/usr/sbin/ssh"
Definitely not what you meant!
Conversely, when you use shell=False, you can only pass a string if you're running a single command with no arguments. If you do have arguments, have to pass the comamnd as a sequence. You also can't use shell redirection via the < character, because there is no shell involved.
If you want to use shell=False, you can use the stdin keyword argument to pass a file handle to /dev/myscript:
f = open("/dev/myscript")
p = subprocess.Popen(["/usr/bin/ssh", "-T", filer], stdout=subprocess.PIPE, stdin=f, shell=False)
The rules for when to pass a string vs. when to pass a sequence are pretty confusing, especially when you bring Windows into the mix as well. I would read the documentation carefully to try to understand it all. Check out both the section on args and the section on shell.
So I noticed subprocess.call while it waits for the command to finish before proceeding with the python script, I have no way of getting the stdout, except with subprocess.Popen. Are there any alternative function calls that would wait until it finishes? (I also tried Popen.wait)
NOTE: I'm trying to avoid os.system call
result = subprocess.Popen([commands...,
self.tmpfile.path()], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = result.communicate()
print out+"HIHIHI"
my output:
HIHIHI
NOTE: I am trying to run wine with this.
I am using the following construct, although you might want to avoid shell=True. This gives you the output and error message for any command, and the error code as well:
process = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# wait for the process to terminate
out, err = process.communicate()
errcode = process.returncode
subprocess.check_output(...)
calls the process, raises if its error code is nonzero, and otherwise returns its stdout. It's just a quick shorthand so you don't have to worry about PIPEs and things.
If your process gives a huge stdout and no stderr, communicate() might be the wrong way to go due to memory restrictions.
Instead,
process = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# wait for the process to terminate
for line in process.stdout: do_something(line)
errcode = process.returncode
might be the way to go.
process.stdout is a file-like object which you can treat as any other such object, mainly:
you can read() from it
you can readline() from it and
you can iterate over it.
The latter is what I do above in order to get its contents line by line.
I'd try something like:
#!/usr/bin/python
from __future__ import print_function
import shlex
from subprocess import Popen, PIPE
def shlep(cmd):
'''shlex split and popen
'''
parsed_cmd = shlex.split(cmd)
## if parsed_cmd[0] not in approved_commands:
## raise ValueError, "Bad User! No output for you!"
proc = Popen(parsed_command, stdout=PIPE, stderr=PIPE)
out, err = proc.communicate()
return (proc.returncode, out, err)
... In other words let shlex.split() do most of the work. I would NOT attempt to parse the shell's command line, find pipe operators and set up your own pipeline. If you're going to do that then you'll basically have to write a complete shell syntax parser and you'll end up doing an awful lot of plumbing.
Of course this raises the question, why not just use Popen with the shell=True (keyword) option? This will let you pass a string (no splitting nor parsing) to the shell and still gather up the results to handle as you wish. My example here won't process any pipelines, backticks, file descriptor redirection, etc that might be in the command, they'll all appear as literal arguments to the command. Thus it is still safer then running with shell=True ... I've given a silly example of checking the command against some sort of "approved command" dictionary or set --- through it would make more sense to normalize that into an absolute path unless you intend to require that the arguments be normalized prior to passing the command string to this function.
With Python 3.8 this workes for me. For instance to execute a python script within the venv:
import subprocess
import sys
res = subprocess.run(
[
sys.executable, # venv3.8/bin/python
'main.py',
'--help',
],
stdout=subprocess.PIPE,
text=True
)
print(res.stdout)
When i did something in Python as:
ping = subprocess.Popen("ping -n 1 %s" %ip, stdout=subprocess.PIPE)
it is always to print out to screen:
(subprocess.Popen object at 0x.... )
It's a bit annoying for me. Do you know how to avoid that std output ?
It looks like you're trying to get the stdout from the process by printing the ping variable in your example. That is incorrect usage of subprocess.Popen objects. In order to correctly use subprocess.Popen this is what you would write:
ping_process = subprocess.Popen(['ping', '-c', '1', ip], stdout=subprocess.PIPE)
# This will block until ping exits
stdout = ping_process.stdout.read()
print stdout
I also changed the arguments you used for ping because -n is an invalid argument on my machines implementation.