We Keep Coding

Python subprocess.run shell kwarg behavior

There has been a lot of questions regarding shell keyword argument. But I still don't really get it especially if we use the sequence argument instead of the string.
My understanding is that if shell=False, the subprocess module will run the executable in args[0] and pass the rest as arguments to the executable. But if we run it with shell=True, it will be ran as something like "sh -c {}".format(format_escaping(args)).
But why does this happen?
# Ran in OSX
subprocess.run(["touch", "12; touch 34"]) # successfuly make the file '12; touch 34'
subprocess.run(["touch", "56; touch 78"], shell=True) # does not work:
# usage:
# touch [-A [-][[hh]mm]SS] [-acfhm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]] file ...
# CompletedProcess(args=['touch', '123; touch 456'], returncode=1)
What actually happen in subprocess.run(["touch", "56; touch 79"], shell=True)?

I think with shell=True the supbrocess runs first only the first parameter which is touch therefore you become the help mesage from the command , try it like this instead:
subprocess.run("touch 56; touch 78", shell=True)

The behavior when you pass in a list with shell=True is platform-dependent. On Unix-like platforms, Python simply passes in the first argument of the list. On Windows, it happens to work, though it probably shouldn't.

scp with Python3 subprocess [duplicate]

When using subprocess.Popen(args, shell=True) to run "gcc --version" (just as an example), on Windows we get this:
>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc (GCC) 3.4.5 (mingw-vista special r3) ...
So it's nicely printing out the version as I expect. But on Linux we get this:
>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc: no input files
Because gcc hasn't received the --version option.
The docs don't specify exactly what should happen to the args under Windows, but it does say, on Unix, "If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments." IMHO the Windows way is better, because it allows you to treat Popen(arglist) calls the same as Popen(arglist, shell=True) ones.
Why the difference between Windows and Linux here?

Actually on Windows, it does use cmd.exe when shell=True - it prepends cmd.exe /c (it actually looks up the COMSPEC environment variable but defaults to cmd.exe if not present) to the shell arguments. (On Windows 95/98 it uses the intermediate w9xpopen program to actually launch the command).
So the strange implementation is actually the UNIX one, which does the following (where each space separates a different argument):
/bin/sh -c gcc --version
It looks like the correct implementation (at least on Linux) would be:
/bin/sh -c "gcc --version" gcc --version
Since this would set the command string from the quoted parameters, and pass the other parameters successfully.
From the sh man page section for -c:
Read commands from the command_string operand instead of from the standard input. Special parameter 0 will be set from the command_name operand and the positional parameters ($1, $2, etc.) set from the remaining argument operands.
This patch seems to fairly simply do the trick:
--- subprocess.py.orig 2009-04-19 04:43:42.000000000 +0200
+++ subprocess.py 2009-08-10 13:08:48.000000000 +0200
## -990,7 +990,7 ##
args = list(args)
if shell:
- args = ["/bin/sh", "-c"] + args
+ args = ["/bin/sh", "-c"] + [" ".join(args)] + args
if executable is None:
executable = args[0]

From the subprocess.py source:
On UNIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a sequence,
the first item specifies the command string, and any additional items
will be treated as additional shell arguments.
On Windows: the Popen class uses CreateProcess() to execute the child
program, which operates on strings. If args is a sequence, it will be
converted to a string using the list2cmdline method. Please note that
not all MS Windows applications interpret the command line the same
way: The list2cmdline is designed for applications using the same
rules as the MS C runtime.
That doesn't answer why, just clarifies that you are seeing the expected behavior.
The "why" is probably that on UNIX-like systems, command arguments are actually passed through to applications (using the exec* family of calls) as an array of strings. In other words, the calling process decides what goes into EACH command line argument. Whereas when you tell it to use a shell, the calling process actually only gets the chance to pass a single command line argument to the shell to execute: The entire command line that you want executed, executable name and arguments, as a single string.
But on Windows, the entire command line (according to the above documentation) is passed as a single string to the child process. If you look at the CreateProcess API documentation, you will notice that it expects all of the command line arguments to be concatenated together into a big string (hence the call to list2cmdline).
Plus there is the fact that on UNIX-like systems there actually is a shell that can do useful things, so I suspect that the other reason for the difference is that on Windows, shell=True does nothing, which is why it is working the way you are seeing. The only way to make the two systems act identically would be for it to simply drop all of the command line arguments when shell=True on Windows.

The reason for the UNIX behaviour of shell=True is to do with quoting. When we write a shell command, it will be split at spaces, so we have to quote some arguments:
cp "My File" "New Location"
This leads to problems when our arguments contain quotes, which requires escaping:
grep -r "\"hello\"" .
Sometimes we can get awful situations where \ must be escaped too!
Of course, the real problem is that we're trying to use one string to specify multiple strings. When calling system commands, most programming languages avoid this by allowing us to send multiple strings in the first place, hence:
Popen(['cp', 'My File', 'New Location'])
Popen(['grep', '-r', '"hello"'])
Sometimes it can be nice to run "raw" shell commands; for example, if we're copy-pasting something from a shell script or a Web site, and we don't want to convert all of the horrible escaping manually. That's why the shell=True option exists:
Popen(['cp "My File" "New Location"'], shell=True)
Popen(['grep -r "\"hello\"" .'], shell=True)
I'm not familiar with Windows so I don't know how or why it behaves differently.

Passing variable to shell from python

I want to pass variable from python to shell script. My code looks like below.
Object_Id=12
Class_Name='My_Class'
Folder_Path='/My_Folder'
output=subprocess.check_output(['./execute.sh, 'Object_Id', 'Class_Name', "shell=True"])
print(output)
The execute.sh script is just simple echo statement
echo $1
echo $2
the output is
Object_Id
Class_Name
The script is getting the args as literal strings but I want to get the value of the variable instead.

Your check_output call needs to pass the command and its arguments as a list or str separate from arguments to the check_output API itself. You shouldn't pass the names of your variables (your command has no idea what Object_Id means after all, and couldn't reach back into the Python process to identify if even if it understood), but rather the variables themselves (and your quotes are mismatched and need fixing). Really, you don't need shell=True here at all, so you can just do:
output = subprocess.check_output(['./execute.sh', str(Object_Id), Class_Name])
If shell=True is important for some reason, you can do it by passing the string representing the shell command:
import shlex
output = subprocess.check_output('./execute.sh {} {}'.format(shlex.quote(Object_Id), shlex.quote(Class_Name)),
shell=True)
Note that shell=True is a separate argument to check_output, not part of the command itself (the first positional argument).

it looks like it's taking the args as strings instead of the variable because you are enclosing it with '' removing them should fix it

Strange python error with subprocess.check_call

I'm having a really strange error with the python subprocess.check_call() function. Here are two tests that should both fail because of permission problems, but the first one only returns a 'usage' (the "unexpected behaviour"):
# Test #1
import subprocess
subprocess.check_call(['git', 'clone', 'https://github.com/achedeuzot/project',
'/var/vhosts/project'], shell=True)
# Shell output
usage: git [--version] [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
[-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
[--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
[-c name=value] [--help]
<command> [<args>]
The most commonly used git commands are:
[...]
Now for the second test (the "expected behaviour" one):
# Test #2
import subprocess
subprocess.check_call(' '.join(['git', 'clone', 'https://github.com/achedeuzot/project',
'/var/vhosts/project']), shell=True)
# Here, we're making it into a string, but the call should be *exactly* the same.
# Shell output
fatal: could not create work tree dir '/var/vhosts/project'.: Permission denied
This second error is the correct one. I don't have the permissions indeed. But why is there a difference between the two calls ? I thought that using a single string or a list is the same with the check_call() function. I have read the python documentation and various usage examples and both look correct.
Did someone have the same strange error ? Or does someone know why is there a difference in output when the commands should be exactly the same ?
Side notes: Python 3.4

Remove shell=True from the first one. If you carefully reread the subprocess module documentation you will see. If shell=False (default) the first argument is a list of the command line with arguments and all (or a string with only the command, no arguments supplied at all). If shell=True, then the first argument is a string representing a shell command line, a shell is executed, which in turn parses the command line for you and splits by white space (+ much more dangerous things you might not want it to do). If shell=True and the first argument is a list, then the first list item is the shell command line, and the rest are passed as arguments to the shell, not the command.
Unless you know you really, really need to, always let shell=False.

Here's the relevant bit from the documentation:
If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])

How to control process environment variable in Python?

I'm trying to change the environment of my Python execution process. It seems that the correct way to do that should be to interact with os.environ. However, I the following assertion fails:
import os, subprocess
os.environ['ATESTVARIABLE'] = 'value'
value = subprocess.check_output(['echo', '$ATESTVARIABLE'], shell=True)
assert 'value' in value
Is there something else that I should be doing to change the current environment? What flaw in my understanding of Python is revealed by the above code :)?
(Note that within the current Python interpreter, os.environ['ATESTVARIABLE'] contains the expected value. I am setting up to run some code which requires a specific environment variable, and which may launch external processes. Obviously, if I wanted to control the environment of a specific subprocess, I'd use the env keyword.)

Looking through the source code for the subprocess module, it's because using a list of arguments with shell=True will do the equivalent of...
/bin/sh -c 'echo' '$ATESTVARIABLE'
...when what you want is...
/bin/sh -c 'echo $ATESTVARIABLE'
The following works for me...
import os, subprocess
os.environ['ATESTVARIABLE'] = 'value'
value = subprocess.check_output('echo $ATESTVARIABLE', shell=True)
assert 'value' in value
Update
FWIW, the difference between the two is that the first form...
/bin/sh -c 'echo' '$ATESTVARIABLE'
...will just call the shell's built-in echo with no parameters, and set $0 to the literal string '$ATESTVARIABLE', for example...
$ /bin/sh -c 'echo $0'
/bin/sh
$ /bin/sh -c 'echo $0' '$ATESTVARIABLE'
$ATESTVARIABLE
...whereas the second form...
/bin/sh -c 'echo $ATESTVARIABLE'
...will call the shell's built-in echo with a single parameter equal to the value of the environment variable ATESTVARIABLE.

Actually, what's wrong in the following code:
import os, subprocess
os.environ['ATESTVARIABLE'] = 'value'
value = subprocess.check_output(['echo', '$ATESTVARIABLE'], shell=True)
assert 'value' in value
is that you didn't read thoroughly the help page of subprocess:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
This means that if you call subprocess.check_out() with an array as a first parameter, you won't get the expected result. You shall retry with the following code:
import os, subprocess
os.environ['ATESTVARIABLE'] = 'value'
value = subprocess.check_output('echo $ATESTVARIABLE', shell=True)
assert 'value' in value
and it should work as you expect!
Otherwise, your understanding of environment variables is correct. When you modify the environment, that environment is given to every forked child of your current process.