Struggling with subprocess.Popen() - why is First and Third working as expected, while the second does not find any of multiple files or directories? Error message is:
>ls: Zugriff auf * nicht möglich: Datei oder Verzeichnis nicht gefunden
English translation:
File not found or directory: access to * not possible
Here is the code.
#!/usr/bin/python
# -*- coding: UTF-8 -*-
import subprocess
args = []
args.append ('ls')
args.append ('-al')
# First does work
cmd1 = subprocess.Popen(args)
cmd1.wait()
# Second does NOT work
args.append ('*')
cmd2 = subprocess.Popen(args)
cmd2.wait()
# Third does work
shellcmd = "ls -al *"
cmd3 = subprocess.Popen(shellcmd, shell=True )
cmd3.wait()
This is because by default subprocess.Popen() doesn't have the shell interpret the commands, so the "*" isn't being expanded into the required list of files. Try adding shell=True as a final argument to the call.
Also note the warning in the documentation about not trusting user input to be processed in this way.
This is happening because of shell globbing.
Basically, the * in ls -al * is expanded by your shell, to match all available files.
When you run the subprocess without the shell=True flag, python is not able to parse the * on its own, and hence, the error message ls: cannot access *: No such file or directory is displayed.
When you run the command with shell=True, python actually passes the control to the shell, and hence the correct output is displayed.
As an aside, executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution, so it should be used with caution (see warning here).
EDIT 1
Both shell globbing and the way Popen consumes args is causing the issue here
From subprocess module,
class subprocess.Popen
args should be a sequence of program arguments or else a single string.
If shell is True, it is recommended to pass argsas astring` rather than as a sequence.
To understand that shell globbing and the manner in which Popen consumes args is the issue here, compare the output of the following. Note that in 2 cases, when shell=True, only ls is executed since the input passed is a list and not a string, against the recommendation
subprocess.Popen(['ls']) #works
subprocess.Popen('ls') #works
subprocess.Popen(['ls', '-al']) #works
subprocess.Popen(['ls -al']) #doesn't work raises OSError since not a single command
subprocess.Popen('ls -al') #doesn't work raises OSError since not a single command
subprocess.Popen(['ls -al'], shell=True) #works since in shell mode
subprocess.Popen('ls -al', shell=True) #works since in shell mode & string is single command
subprocess.Popen(['ls', '-al'], shell=True) #output corresponds to ls only, list passed instead of string, against recommendation
subprocess.Popen(['ls', '-al', '*']) #doesn't work because of shell globbing for *
subprocess.Popen(['ls -al *']) #doesn't work raises OSError since not a single commandfor *
subprocess.Popen('ls -al *') #doesn't work raises OSError since not a single commandvalid arg
subprocess.Popen(['ls', '-al', '*'], shell=True) #output corresponds to ls only, list passed instead of string, against recommendation
subprocess.Popen(['ls -al *'], shell=True) #works
subprocess.Popen('ls -al *', shell=True) #works
Not a direct answer to your question, but you can also try using the python library sh
example:
from sh import ls
print ls("-al")
link to more examples
Related
When using subprocess.Popen(args, shell=True) to run "gcc --version" (just as an example), on Windows we get this:
>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc (GCC) 3.4.5 (mingw-vista special r3) ...
So it's nicely printing out the version as I expect. But on Linux we get this:
>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc: no input files
Because gcc hasn't received the --version option.
The docs don't specify exactly what should happen to the args under Windows, but it does say, on Unix, "If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments." IMHO the Windows way is better, because it allows you to treat Popen(arglist) calls the same as Popen(arglist, shell=True) ones.
Why the difference between Windows and Linux here?
Actually on Windows, it does use cmd.exe when shell=True - it prepends cmd.exe /c (it actually looks up the COMSPEC environment variable but defaults to cmd.exe if not present) to the shell arguments. (On Windows 95/98 it uses the intermediate w9xpopen program to actually launch the command).
So the strange implementation is actually the UNIX one, which does the following (where each space separates a different argument):
/bin/sh -c gcc --version
It looks like the correct implementation (at least on Linux) would be:
/bin/sh -c "gcc --version" gcc --version
Since this would set the command string from the quoted parameters, and pass the other parameters successfully.
From the sh man page section for -c:
Read commands from the command_string operand instead of from the standard input. Special parameter 0 will be set from the command_name operand and the positional parameters ($1, $2, etc.) set from the remaining argument operands.
This patch seems to fairly simply do the trick:
--- subprocess.py.orig 2009-04-19 04:43:42.000000000 +0200
+++ subprocess.py 2009-08-10 13:08:48.000000000 +0200
## -990,7 +990,7 ##
args = list(args)
if shell:
- args = ["/bin/sh", "-c"] + args
+ args = ["/bin/sh", "-c"] + [" ".join(args)] + args
if executable is None:
executable = args[0]
From the subprocess.py source:
On UNIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a sequence,
the first item specifies the command string, and any additional items
will be treated as additional shell arguments.
On Windows: the Popen class uses CreateProcess() to execute the child
program, which operates on strings. If args is a sequence, it will be
converted to a string using the list2cmdline method. Please note that
not all MS Windows applications interpret the command line the same
way: The list2cmdline is designed for applications using the same
rules as the MS C runtime.
That doesn't answer why, just clarifies that you are seeing the expected behavior.
The "why" is probably that on UNIX-like systems, command arguments are actually passed through to applications (using the exec* family of calls) as an array of strings. In other words, the calling process decides what goes into EACH command line argument. Whereas when you tell it to use a shell, the calling process actually only gets the chance to pass a single command line argument to the shell to execute: The entire command line that you want executed, executable name and arguments, as a single string.
But on Windows, the entire command line (according to the above documentation) is passed as a single string to the child process. If you look at the CreateProcess API documentation, you will notice that it expects all of the command line arguments to be concatenated together into a big string (hence the call to list2cmdline).
Plus there is the fact that on UNIX-like systems there actually is a shell that can do useful things, so I suspect that the other reason for the difference is that on Windows, shell=True does nothing, which is why it is working the way you are seeing. The only way to make the two systems act identically would be for it to simply drop all of the command line arguments when shell=True on Windows.
The reason for the UNIX behaviour of shell=True is to do with quoting. When we write a shell command, it will be split at spaces, so we have to quote some arguments:
cp "My File" "New Location"
This leads to problems when our arguments contain quotes, which requires escaping:
grep -r "\"hello\"" .
Sometimes we can get awful situations where \ must be escaped too!
Of course, the real problem is that we're trying to use one string to specify multiple strings. When calling system commands, most programming languages avoid this by allowing us to send multiple strings in the first place, hence:
Popen(['cp', 'My File', 'New Location'])
Popen(['grep', '-r', '"hello"'])
Sometimes it can be nice to run "raw" shell commands; for example, if we're copy-pasting something from a shell script or a Web site, and we don't want to convert all of the horrible escaping manually. That's why the shell=True option exists:
Popen(['cp "My File" "New Location"'], shell=True)
Popen(['grep -r "\"hello\"" .'], shell=True)
I'm not familiar with Windows so I don't know how or why it behaves differently.
I'm having a really strange error with the python subprocess.check_call() function. Here are two tests that should both fail because of permission problems, but the first one only returns a 'usage' (the "unexpected behaviour"):
# Test #1
import subprocess
subprocess.check_call(['git', 'clone', 'https://github.com/achedeuzot/project',
'/var/vhosts/project'], shell=True)
# Shell output
usage: git [--version] [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
[-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
[--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
[-c name=value] [--help]
<command> [<args>]
The most commonly used git commands are:
[...]
Now for the second test (the "expected behaviour" one):
# Test #2
import subprocess
subprocess.check_call(' '.join(['git', 'clone', 'https://github.com/achedeuzot/project',
'/var/vhosts/project']), shell=True)
# Here, we're making it into a string, but the call should be *exactly* the same.
# Shell output
fatal: could not create work tree dir '/var/vhosts/project'.: Permission denied
This second error is the correct one. I don't have the permissions indeed. But why is there a difference between the two calls ? I thought that using a single string or a list is the same with the check_call() function. I have read the python documentation and various usage examples and both look correct.
Did someone have the same strange error ? Or does someone know why is there a difference in output when the commands should be exactly the same ?
Side notes: Python 3.4
Remove shell=True from the first one. If you carefully reread the subprocess module documentation you will see. If shell=False (default) the first argument is a list of the command line with arguments and all (or a string with only the command, no arguments supplied at all). If shell=True, then the first argument is a string representing a shell command line, a shell is executed, which in turn parses the command line for you and splits by white space (+ much more dangerous things you might not want it to do). If shell=True and the first argument is a list, then the first list item is the shell command line, and the rest are passed as arguments to the shell, not the command.
Unless you know you really, really need to, always let shell=False.
Here's the relevant bit from the documentation:
If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
I am using Python script to invoke a Java virtual machine. The following command works:
subprocess.call(["./rvm"], shell=False) # works
subprocess.call(["./rvm xyz"], shell=True) # works
But,
subprocess.call(["./rvm xyz"], shell=False) # not working
does not work. Python documentation advices to avoid shell=True.
You need to split the commands into separate strings:
subprocess.call(["./rvm", "xyz"], shell=False)
A string will work when shell=True but you need a list of args when shell=False
The shlex module is useful more so for more complicated commands and dealing with input but good to learn about:
import shlex
cmd = "python foo.py"
subprocess.call(shlex.split(cmd), shell=False)
shlex tut
If you want to use shell=True, this is legit, otherwise it would have been removed from the standard library. The documentation doesn't say to avoid it, it says:
Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input.
But in your case you are not constructing the command from user input, your command is constant, so your code doesn't present the shell injection issue. You are in control of what the shell will execute, and if your code is not malicious per se, you are safe.
Example of shell injection
To explain why the shell injection is so bad, this is the example used in the documentation:
>>> from subprocess import call
>>> filename = input("What file would you like to display?\n")
What file would you like to display?
non_existent; rm -rf / #
>>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
Edit
With the additional information you have provided editing the question, stick to Padraic's answer. You should use shell=True only when necessary.
In addition to Enrico.bacis' answer, there are two ways to call programs. With shell=True, give it a full command string. With shell=False, give it a list.
If you do shell tricks like *.jpg or 2> /dev/null, use shell=True; but in general I suggest shell=False -- it's more durable as Enrico said.
source
import subprocess
subprocess.check_call(['/bin/echo', 'beer'], shell=False)
subprocess.check_call('/bin/echo beer', shell=True)
output
beer
beer
Instead of using the filename directory, add the word python in front of it, provided that you've added python path to your environmental variables. If you're not sure, you can always rerun the python installer, once again, provided that you have a new version of python.
Here's what I mean:
import subprocess
subprocess.Popen('python "C:/Path/To/File/Here.py"')
I can not get the subprocess.call() to work properly:
>>> from subprocess import call
>>> call(['adduser', '--home=/var/www/myusername/', '--gecos', 'GECOS', '--disabled-login', 'myusername'], shell=True)
adduser: Only one or two names allowed.
1
But without shell=True:
>>> call(['adduser', '--home=/var/www/myusername/', '--gecos', 'GECOS', '--disabled-login', 'myusername'])
Adding user `myusername' ...
Adding new group `myusername' (1001) ...
Adding new user `myusername' (1001) with group `myusername' ...
Creating home directory `/var/www/myusername/' ...
Copying files from `/etc/skel' ...
0
Or the same directly in shell:
root#www1:~# adduser --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername
Adding user `myusername' ...
Adding new group `myusername' (1001) ...
Adding new user `myusername' (1001) with group `myusername' ...
Creating home directory `/var/www/myusername/' ...
Copying files from `/etc/skel' ...
I miss some logic in the shell=True behavior. Can somebody explain me why? What is wrong with the first example? From the adduser command error message it seems that arguments are somehow crippled.
Thanks!
When you specify shell=True you switch to quite different behaviour. From the docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
So you are running the equivalent of
/bin/sh -c "adduser" --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername
The error message you are getting is what happens when you try and run adduser without any arguments as all the arguments are being passed to sh.
If you want to set shell=True then you would need to call it like this:
call('adduser --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername', shell=True)
OR like this:
call(['adduser --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername'], shell=True)
But mostly you just want to use call without the shell=True and use a list of arguments. As per your second, working, example.
I am not 100% sure about this but I think that it you specify Shell=True, you should be passing the command line as a single string which the shell itself will interpret:
>>> call('adduser --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername', shell=True)
It seems that with shell=True you need to pass string into args rather than list of arguments.
A simple test:
In [4]: subprocess.call(['echo', 'foo', 'bar'], shell=True)
Out[4]: 0
In [5]: subprocess.call('echo foo bar', shell=True)
foo bar
Out[5]: 0
I.e. echo got the right arguments only when I used string, not list.
Python version 2.7.3
If shell is True the specified command will be executed through the shell, that is the shell takes care of filename wildcards, environment variable expansion etc. When you use shell=True the cmd is a single string, it must be formatted exactly as it would be typed in the shell. If shell=True and cmd is a sequence, the first argument specifies the command and the additional arguments are treated as arguments to the shell itself (by the -c switch).
If shell=False, and a sequence of arguments is provided the module will take care of properly escaping and quoting the arguments and for example ~ won't be expanded as the home directory etc.
Read more about it in the subprocess documentation, and mind the security hazard related to shell=True.
import os
import subprocess
proc = subprocess.Popen(['ls','*.bc'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out,err = proc.communicate()
print out
This script should print all the files with .bc suffix however it returns an empty list. If I do ls *.bc manually in the command line it works. Doing ['ls','test.bc'] inside the script works as well but for some reason the star symbol doesnt work.. Any ideas ?
You need to supply shell=True to execute the command through a shell interpreter.
If you do that however, you can no longer supply a list as the first argument, because the arguments will get quoted then. Instead, specify the raw commandline as you want it to be passed to the shell:
proc = subprocess.Popen('ls *.bc', shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
Expanding the * glob is part of the shell, but by default subprocess does not send your commands via a shell, so the command (first argument, ls) is executed, then a literal * is used as an argument.
This is a good thing, see the warning block in the "Frequently Used Arguments" section, of the subprocess docs. It mainly discusses security implications, but can also helps avoid silly programming errors (as there are no magic shell characters to worry about)
My main complaint with shell=True is it usually implies there is a better way to go about the problem - with your example, you should use the glob module:
import glob
files = glob.glob("*.bc")
print files # ['file1.bc', 'file2.bc']
This will be quicker (no process startup overhead), more reliable and cross platform (not dependent on the platform having an ls command)
Besides doing shell=True, also make sure that your path is not quoted. Otherwise it will not be expanded by shell.
If your path may have special characters, you will have to escape them manually.