We Keep Coding

Python Subprocess call() does not execute shell command

I am trying to make a python program(python 3.6) that writes commands to terminal to download a specific youtube video(using youtube-dl).
If I go on terminal and execute the following command:
cd; cd Desktop; youtube-dl "https://www.youtube.com/watch?v=b91ovTKCZGU"
It will download the video to my desktop. However, if I execute the below code, which should be doing the same command on terminal, it does not throw an error but also does not download that video.
import subprocess
cmd = ["cd;", "cd", "Desktop;", "youtube-dl", "\"https://www.youtube.com/watch?v=b91ovTKCZGU\""]
print(subprocess.call(cmd, stderr=subprocess.STDOUT,shell=True))
It seems that this just outputs 0. I do not think there is any kind of error 0 that exists(there are error 126 and 127). So if it is not throwing an error, why does it also not download the video?
Update:
I have fixed the above code by passing in a string, and have checked that youtube-dl is installed in my default python and is also in the folder where I want to download the videos, but its still throwing error 127, meaning command "youtube-dl" is not found.

cd; cd Desktop; youtube-dl "https://www.youtube.com/watch?v=b91ovTKCZGU" is not a single command; it's a list (delimited by ;) of three separate commands.
subprocess.call(cmd, ..., shell=True) is effectively the same as
subprocess.call(['sh', '-c'] + cmd)
which is almost never what you want. Instead, just pass a single string and let the shell parse it.
subprocess.call('cd; cd Desktop; youtube-dl "https://www.youtube.com/watch?v=b91ovTKCZGU"', shell=True)
If you really want to use the list form (which is always a good idea), use the cwd parameter instead of running cd.
subprocess.call(['youtube-dl', 'https://www.youtube.com/watch?v=b91ovTKCZGU'],
cwd=os.path.expanduser("~/Desktop"))

I'll answer this with an example:
>>> subprocess.call(["echo $0 $2", "foo", "skipped", "bar"], shell=True)
foo bar
0
The first element of the list is the shell command (echo $0 $2), and the remaining elements are the positional parameters that the command can optionally use ($0, $1, ...).
In your example, you are creating a subshell that only runs the cd; command. The positional parameters are ignored. See the Popen and bash docs for details.
As noted in the comments, you should make the command a string (not a list).

scp with Python3 subprocess [duplicate]

When using subprocess.Popen(args, shell=True) to run "gcc --version" (just as an example), on Windows we get this:
>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc (GCC) 3.4.5 (mingw-vista special r3) ...
So it's nicely printing out the version as I expect. But on Linux we get this:
>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc: no input files
Because gcc hasn't received the --version option.
The docs don't specify exactly what should happen to the args under Windows, but it does say, on Unix, "If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments." IMHO the Windows way is better, because it allows you to treat Popen(arglist) calls the same as Popen(arglist, shell=True) ones.
Why the difference between Windows and Linux here?

Actually on Windows, it does use cmd.exe when shell=True - it prepends cmd.exe /c (it actually looks up the COMSPEC environment variable but defaults to cmd.exe if not present) to the shell arguments. (On Windows 95/98 it uses the intermediate w9xpopen program to actually launch the command).
So the strange implementation is actually the UNIX one, which does the following (where each space separates a different argument):
/bin/sh -c gcc --version
It looks like the correct implementation (at least on Linux) would be:
/bin/sh -c "gcc --version" gcc --version
Since this would set the command string from the quoted parameters, and pass the other parameters successfully.
From the sh man page section for -c:
Read commands from the command_string operand instead of from the standard input. Special parameter 0 will be set from the command_name operand and the positional parameters ($1, $2, etc.) set from the remaining argument operands.
This patch seems to fairly simply do the trick:
--- subprocess.py.orig 2009-04-19 04:43:42.000000000 +0200
+++ subprocess.py 2009-08-10 13:08:48.000000000 +0200
## -990,7 +990,7 ##
args = list(args)
if shell:
- args = ["/bin/sh", "-c"] + args
+ args = ["/bin/sh", "-c"] + [" ".join(args)] + args
if executable is None:
executable = args[0]

From the subprocess.py source:
On UNIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a sequence,
the first item specifies the command string, and any additional items
will be treated as additional shell arguments.
On Windows: the Popen class uses CreateProcess() to execute the child
program, which operates on strings. If args is a sequence, it will be
converted to a string using the list2cmdline method. Please note that
not all MS Windows applications interpret the command line the same
way: The list2cmdline is designed for applications using the same
rules as the MS C runtime.
That doesn't answer why, just clarifies that you are seeing the expected behavior.
The "why" is probably that on UNIX-like systems, command arguments are actually passed through to applications (using the exec* family of calls) as an array of strings. In other words, the calling process decides what goes into EACH command line argument. Whereas when you tell it to use a shell, the calling process actually only gets the chance to pass a single command line argument to the shell to execute: The entire command line that you want executed, executable name and arguments, as a single string.
But on Windows, the entire command line (according to the above documentation) is passed as a single string to the child process. If you look at the CreateProcess API documentation, you will notice that it expects all of the command line arguments to be concatenated together into a big string (hence the call to list2cmdline).
Plus there is the fact that on UNIX-like systems there actually is a shell that can do useful things, so I suspect that the other reason for the difference is that on Windows, shell=True does nothing, which is why it is working the way you are seeing. The only way to make the two systems act identically would be for it to simply drop all of the command line arguments when shell=True on Windows.

The reason for the UNIX behaviour of shell=True is to do with quoting. When we write a shell command, it will be split at spaces, so we have to quote some arguments:
cp "My File" "New Location"
This leads to problems when our arguments contain quotes, which requires escaping:
grep -r "\"hello\"" .
Sometimes we can get awful situations where \ must be escaped too!
Of course, the real problem is that we're trying to use one string to specify multiple strings. When calling system commands, most programming languages avoid this by allowing us to send multiple strings in the first place, hence:
Popen(['cp', 'My File', 'New Location'])
Popen(['grep', '-r', '"hello"'])
Sometimes it can be nice to run "raw" shell commands; for example, if we're copy-pasting something from a shell script or a Web site, and we don't want to convert all of the horrible escaping manually. That's why the shell=True option exists:
Popen(['cp "My File" "New Location"'], shell=True)
Popen(['grep -r "\"hello\"" .'], shell=True)
I'm not familiar with Windows so I don't know how or why it behaves differently.

Strange python error with subprocess.check_call

I'm having a really strange error with the python subprocess.check_call() function. Here are two tests that should both fail because of permission problems, but the first one only returns a 'usage' (the "unexpected behaviour"):
# Test #1
import subprocess
subprocess.check_call(['git', 'clone', 'https://github.com/achedeuzot/project',
'/var/vhosts/project'], shell=True)
# Shell output
usage: git [--version] [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
[-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
[--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
[-c name=value] [--help]
<command> [<args>]
The most commonly used git commands are:
[...]
Now for the second test (the "expected behaviour" one):
# Test #2
import subprocess
subprocess.check_call(' '.join(['git', 'clone', 'https://github.com/achedeuzot/project',
'/var/vhosts/project']), shell=True)
# Here, we're making it into a string, but the call should be *exactly* the same.
# Shell output
fatal: could not create work tree dir '/var/vhosts/project'.: Permission denied
This second error is the correct one. I don't have the permissions indeed. But why is there a difference between the two calls ? I thought that using a single string or a list is the same with the check_call() function. I have read the python documentation and various usage examples and both look correct.
Did someone have the same strange error ? Or does someone know why is there a difference in output when the commands should be exactly the same ?
Side notes: Python 3.4

Remove shell=True from the first one. If you carefully reread the subprocess module documentation you will see. If shell=False (default) the first argument is a list of the command line with arguments and all (or a string with only the command, no arguments supplied at all). If shell=True, then the first argument is a string representing a shell command line, a shell is executed, which in turn parses the command line for you and splits by white space (+ much more dangerous things you might not want it to do). If shell=True and the first argument is a list, then the first list item is the shell command line, and the rest are passed as arguments to the shell, not the command.
Unless you know you really, really need to, always let shell=False.

Here's the relevant bit from the documentation:
If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])

Python: subprocess call with shell=False not working

I am using Python script to invoke a Java virtual machine. The following command works:
subprocess.call(["./rvm"], shell=False) # works
subprocess.call(["./rvm xyz"], shell=True) # works
But,
subprocess.call(["./rvm xyz"], shell=False) # not working
does not work. Python documentation advices to avoid shell=True.

You need to split the commands into separate strings:
subprocess.call(["./rvm", "xyz"], shell=False)
A string will work when shell=True but you need a list of args when shell=False
The shlex module is useful more so for more complicated commands and dealing with input but good to learn about:
import shlex
cmd = "python foo.py"
subprocess.call(shlex.split(cmd), shell=False)
shlex tut

If you want to use shell=True, this is legit, otherwise it would have been removed from the standard library. The documentation doesn't say to avoid it, it says:
Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input.
But in your case you are not constructing the command from user input, your command is constant, so your code doesn't present the shell injection issue. You are in control of what the shell will execute, and if your code is not malicious per se, you are safe.
Example of shell injection
To explain why the shell injection is so bad, this is the example used in the documentation:
>>> from subprocess import call
>>> filename = input("What file would you like to display?\n")
What file would you like to display?
non_existent; rm -rf / #
>>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
Edit
With the additional information you have provided editing the question, stick to Padraic's answer. You should use shell=True only when necessary.

In addition to Enrico.bacis' answer, there are two ways to call programs. With shell=True, give it a full command string. With shell=False, give it a list.
If you do shell tricks like *.jpg or 2> /dev/null, use shell=True; but in general I suggest shell=False -- it's more durable as Enrico said.
source
import subprocess
subprocess.check_call(['/bin/echo', 'beer'], shell=False)
subprocess.check_call('/bin/echo beer', shell=True)
output
beer
beer

Instead of using the filename directory, add the word python in front of it, provided that you've added python path to your environmental variables. If you're not sure, you can always rerun the python installer, once again, provided that you have a new version of python.
Here's what I mean:
import subprocess
subprocess.Popen('python "C:/Path/To/File/Here.py"')

Python subproces.call not working as expected

I can not get the subprocess.call() to work properly:
>>> from subprocess import call
>>> call(['adduser', '--home=/var/www/myusername/', '--gecos', 'GECOS', '--disabled-login', 'myusername'], shell=True)
adduser: Only one or two names allowed.
1
But without shell=True:
>>> call(['adduser', '--home=/var/www/myusername/', '--gecos', 'GECOS', '--disabled-login', 'myusername'])
Adding user `myusername' ...
Adding new group `myusername' (1001) ...
Adding new user `myusername' (1001) with group `myusername' ...
Creating home directory `/var/www/myusername/' ...
Copying files from `/etc/skel' ...
0
Or the same directly in shell:
root#www1:~# adduser --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername
Adding user `myusername' ...
Adding new group `myusername' (1001) ...
Adding new user `myusername' (1001) with group `myusername' ...
Creating home directory `/var/www/myusername/' ...
Copying files from `/etc/skel' ...
I miss some logic in the shell=True behavior. Can somebody explain me why? What is wrong with the first example? From the adduser command error message it seems that arguments are somehow crippled.
Thanks!

When you specify shell=True you switch to quite different behaviour. From the docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
So you are running the equivalent of
/bin/sh -c "adduser" --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername
The error message you are getting is what happens when you try and run adduser without any arguments as all the arguments are being passed to sh.
If you want to set shell=True then you would need to call it like this:
call('adduser --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername', shell=True)
OR like this:
call(['adduser --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername'], shell=True)
But mostly you just want to use call without the shell=True and use a list of arguments. As per your second, working, example.

I am not 100% sure about this but I think that it you specify Shell=True, you should be passing the command line as a single string which the shell itself will interpret:
>>> call('adduser --home=/var/www/myusername/ --gecos GECOS --disabled-login myusername', shell=True)

It seems that with shell=True you need to pass string into args rather than list of arguments.
A simple test:
In [4]: subprocess.call(['echo', 'foo', 'bar'], shell=True)
Out[4]: 0
In [5]: subprocess.call('echo foo bar', shell=True)
foo bar
Out[5]: 0
I.e. echo got the right arguments only when I used string, not list.
Python version 2.7.3

If shell is True the specified command will be executed through the shell, that is the shell takes care of filename wildcards, environment variable expansion etc. When you use shell=True the cmd is a single string, it must be formatted exactly as it would be typed in the shell. If shell=True and cmd is a sequence, the first argument specifies the command and the additional arguments are treated as arguments to the shell itself (by the -c switch).
If shell=False, and a sequence of arguments is provided the module will take care of properly escaping and quoting the arguments and for example ~ won't be expanded as the home directory etc.
Read more about it in the subprocess documentation, and mind the security hazard related to shell=True.