On Windows I successfully run this:
cmd = ["gfortran", "test.f90", "-o", "test.exe", "&&", "test.exe"]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
print(p.stdout.read())
On Ubuntu I change cmd to:
cmd = ["gfortran", "test.f90", "-o", "test", "&&", "./test"]
and get:
gfortran: fatal error: no input files
compilation terminated.
I want to retain cmd as a list, instead making it a string.
How can I make this work?
It's impossible. When cmd is a list, it has different meaning when shell is True. Quoting docs:
If args is a sequence, the first item specifies the command string,
and any additional items will be treated as additional arguments to
the shell itself.
Use ' '.join(shlex.quote(arg) for arg in cmd) (pipes.quote in Python2) when passing list of arguments to Popen with shell=True for expected behavior. Original list won't be mutated, string will be built before passing to function and garbage collected as soon as it's possible.
This is what the docs state:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself.
If you want to retain the arguments as a list, you can join them while calling Popen:
p = subprocess.Popen(' '.join(cmd), stdout=subprocess.PIPE, shell=True)
It appears that you are using shell=True to allow you to do && on the command line. This is not strictly necessary if you are already using subprocess anyway:
cmd1 = ["gfortran", "test.f90", "-o", "test.exe"]
cmd2 = ["test.exe"]
if not subprocess.call(cmd1):
p = subprocess.Popen(cmd2, ...)
out, err = p.communicate()
...
Related
I am new to the subprocess module, and wondering why the first subprocess failed while the second one worked. I am on py3.7 and macOS.
>>> from subprocess import PIPE, Popen, STDOUT
>>> Popen(['touch', '/Users/me/fail.txt'], stdout=PIPE, stderr=STDOUT, shell=True)
>>> Popen(['touch /Users/me/ok.txt'], stdout=PIPE, stderr=STDOUT, shell=True)
According to the docs:
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence.
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
So in the first case the 2nd element of the list is passed as an argument to /bin/sh itself, not the touch command. So you are basically running:
user#name ~$ touch
Which produces the following error:
touch: missing file operand
Try 'touch --help' for more information.
And if you read the stdout of your first command, you will find the same:
>>> Popen(['touch', '/Users/me/fail.txt'], stdout=PIPE, stderr=STDOUT, shell=True).stdout.read()
b"touch: missing file operand\nTry 'touch --help' for more information.\n"
So while shell=True, it is better to pass string.
In subprocess.run which is a high-level function, you need to pass the arguments as a list but for Popen, which is a low-level function needs a direct command hence the first one failed but the second one worked.
Why does Python's subprocess module expect the arguments as a list by default? Why isn't a string with spaces (similar to what you type into a terminal when running the command normally) the default input? There are plenty of sources explaining how to pass in the space delimited string of the command into subprocess, but it's less clear as to why the default isn't the other way around.
TL;DR Using the list bypasses the shell so that you don't need to worry about the shell interpreting a dynamically constructed command line in ways you did not intend.
Suppose you have a really simple command: echo foo. Here it is, using both a string and a list:
Popen("echo foo", shell=True)
Popen(["echo", "foo"])
Not much difference yet. Now suppose the argument contains quotes to protect whitespace and/or a shell pattern, echo "foo * bar":
Popen("echo \"foo * bar\"", shell=True)
Popen(["echo", "foo * bar"])
Yes, I could have used single quotes to avoid needing to escape the double quotes, but you can see the list form is starting to have an advantage. Now imagine I don't have a literal argument for the command, but that it is stored in a variable. Now which do you want to use...
This?
Popen('echo "%s"' % (x,), shell=True)
or this?
Popen(["echo", x])
If you answered "the first one", here's the value of x:
x = "\";rm -rf \""
The command you just executed was echo ""; rm -rf/"". You needed to make sure any special characters in the value of x were first escaped before incorporating it into the string you are building to pass to the shell.
Or you just use a list and avoid the shell altogether.
Forget all that I wrote - just read the relevant PEP yourself
https://www.python.org/dev/peps/pep-0324/
===============
My short guess - the no-shell list version is closer to the format that is eventually passed to the POSIX forking commands. It requires less manipulation. The shell string approach is something of a Windows legacy.
=====================
So you are asking why the shell=False case is the default?
On POSIX, with shell=False (default): In this case, the Popen class
uses os.execvp() to execute the child program. args should normally
be a sequence. A string will be treated as a sequence with the string
as the only item (the program to execute).
On POSIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a sequence,
the first item specifies the command string, and any additional items
will be treated as additional shell arguments.
'why' questions tend to be closed because they rarely have definitive answers, or they involve opinions, or history.
I'd suggest studying the subprocess.py code. I see for example a lot of calls to:
Popen(*popenargs, **kwargs)
It's init is:
def __init__(self, args, bufsize=-1, executable=None,
stdin=None, stdout=None, stderr=None,
preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS,
shell=False, cwd=None, env=None, universal_newlines=False,
startupinfo=None, creationflags=0,
restore_signals=True, start_new_session=False,
pass_fds=()):
As a keyword arg, shell has to have some default value; why not False?
I suspect that in the shell case it passes a whole string to some code that calls the shell. In the no-shell case it must pass a list. But we have to find that code.
There are 2 methods of call the subprocess, one for POSIX and the other Windows. In the POSIX case it appears to convert the string list, regardless whether shell is True or not It may be more nuanced than that, but this is the relevant code:
"""Execute program (POSIX version)"""
if isinstance(args, (str, bytes)):
args = [args]
else:
args = list(args)
if shell:
args = ["/bin/sh", "-c"] + args
if executable:
args[0] = executable
....
self.pid = _posixsubprocess.fork_exec(
args, executable_list,...
In the windows shell case the args string is combined with cmd info:
if shell:
....
comspec = os.environ.get("COMSPEC", "cmd.exe")
args = '{} /c "{}"'.format (comspec, args)
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
# no special security
....
I am using subprocess.Popen to execute an OS command. Here is what I am trying to emulate in my code:
ssh -T myhost < /path/to/some/file
It works fine like this:
def runWorkerCode(filer, filename):
command = "/usr/bin/ssh -T " + filer + " < /devel/myscript"
try:
p = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
out, _ = p.communicate()
except Exception:
print "Error: %s" % Exception
sys.exit(1)
return out.rstrip().split('\n')
But the following calls to Popen do not work:
p = subprocess.Popen(["/usr/bin/ssh", "-T", filer, "<", "/devel/myscript"], stdout=subprocess.PIPE, shell=True)
p = subprocess.Popen(["/usr/bin/ssh -T", filer, "< /devel/myscript"], stdout=subprocess.PIPE, shell=True)
I tried a few other combinations but only method I can get to work is defining the command variable and only providing it to Popen(). I've also tried shell=False.
The first method works but the latter approach seems "cleaner" to me.
Why doesn't Popen allow me to specify the arguments in a list?
When you use shell=True on UNIX, you should provide your arguments as a string. When you provide a list, subprocess interprets the first item in the list as your entire command string, and the rest of the items in the list as arguments passed to the shell itself, rather than your command. So in your example above, you're ending up with something like this:
/bin/sh -T filer < /dev/myscript -c "/usr/sbin/ssh"
Definitely not what you meant!
Conversely, when you use shell=False, you can only pass a string if you're running a single command with no arguments. If you do have arguments, have to pass the comamnd as a sequence. You also can't use shell redirection via the < character, because there is no shell involved.
If you want to use shell=False, you can use the stdin keyword argument to pass a file handle to /dev/myscript:
f = open("/dev/myscript")
p = subprocess.Popen(["/usr/bin/ssh", "-T", filer], stdout=subprocess.PIPE, stdin=f, shell=False)
The rules for when to pass a string vs. when to pass a sequence are pretty confusing, especially when you bring Windows into the mix as well. I would read the documentation carefully to try to understand it all. Check out both the section on args and the section on shell.
Hi all I am learning python and shell script and for GUI i am using wxpython.
I have said to make a automated tool which does some operation , deb creation is also one of that.
for deb creation , command is:
./myfile -u username
I have tried os.Popen, os.system, subprocess.Popen , subprocess.call.But no use everytime "-u" wont take effect, "-u " is must. I have tried by storing "-u" in variable and the passed it but still no use.
Please suggest me the exact way or where i am doing wrong
No error message but "myfile" output shows that "-u" has not given in command.
Code is:
1. cmd = ['./myfile', '-u', 'username']
Popen(cmd,shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
2. user = "-u"
name = "username"
sbprocess.call("./myfile %s %s" %(str(user),str(name)), shell=True)
same kind using "os" command also
The first code example in your question passes incorrect command because shell=True changes the meaning of the first parameter, from the subprocess docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
..[snip]..
If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
The second code example from you question should work if the following command works in a shell:
$ /bin/sh -c './myfile -u username'
To fix your command, you could omit possibly unnecessary shell=True and use check_call():
import subprocess
subprocess.check_call(["./myfile", "-u", "username"])
Try 'bash' instead of './'. If its bash script or else use respective. Its work only if your myfile is shell script.
subprocess.call("bash myfile -u %s" % str(name)), shell=True)
I tried executing a server daemon with gnu screen from subprocess call but it didn't even start
subprocess.call(["screen", "-dmS test ./server"])
I was told that running screen requires terminal, hence the reason why I can't simply execute it with call. Can you show me some piece of codes to do this?
Try
subprocess.call( ["screen", "-d", "-m", "-S", "test", "./server"] )
You need to break the argument string into separate arguments, one per string.
Here's the relevant quote from the subprocess docs:
On UNIX, with shell=False (default): In this case, the Popen class
uses os.execvp() to execute the child program. args should normally
be a sequence. A string will be treated as a sequence with the string
as the only item (the program to execute).
On UNIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a sequence,
the first item specifies the command string, and any additional items
will be treated as additional shell arguments.
So by default, the arguments are used exactly as you give them; it doesn't try to parse a string into multiple arguments. If you set shell to true, you could try the following:
subprocess.call("screen -dmS test ./server", shell=True)
and the string would be parsed exactly like a command line.