We Keep Coding

What's the difference between subprocess.Popen("echo $HOME"... and subprocess.Popen(["echo", "$HOME"]

I cannot get it it's bash related or python subprocess, but results are different:
>>> subprocess.Popen("echo $HOME", shell=True, stdout=subprocess.PIPE).communicate()
(b'/Users/mac\n', None)
>>> subprocess.Popen(["echo", "$HOME"], shell=True, stdout=subprocess.PIPE).communicate()
(b'\n', None)
Why in second time it's just newline? Where argument are falling off?

The first argument to subprocess.Popen() tells the system what to run.
When it is a list, you need to use shell=False. It coincidentally happens to work as you hope in Windows; but on Unix-like platforms, you are simply passing in a number of arguments which will typically get ignored. Effectively,
/bin/sh -c 'echo' '$HOME'
which simply causes the second argument to not be used for anything (where I use single quotes to emphasize that these are just static strings).
In my humble opinion, Python should throw an error in this case. On Windows, too. This is an error which should be caught and reported.
(In the opposite case, where shell=False is specified but the string you pass in is not the name of a valid command, you will get an error eventually anyway, and it makes sense if you have even a vague idea of what's going on.)
If you really know what you are doing, you could cause the first argument to access subsequent arguments; for example
/bin/sh -c 'printf "%s\n" "$#"' 'ick' 'foo' 'bar' 'baz'
would print foo, bar, and baz on separate lines. (The "zeroth" argument - here, 'ick' - is used to populate $0.) But this is just an obscure corollary; don't try to use this for anything.
As a further aside, you should not use subprocess.Popen() if you just want a command to run. The subprocess.run() documentation tells you this in some more detail. With text=True you get a string instead of bytes.
result = subprocess.run('echo "$HOME"', shell=True,
text=True, capture_output=True, check=True)
print(result.stdout, result.stderr)
And of course, os.environ['HOME'] lets you access the value of $HOME from within Python. This also allows you to avoid shell=True which you usually should if you can.

In the documentation found on https://docs.python.org/2/library/subprocess.html#popen-constructor, if you look at the shell argument you will find
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence.
Which means that when you execute the second command it runs as echo and hence you get just a new line.

When you have shell=True, actual process that runs is the shell process i.e., think of it running /bin/sh -c on unix. The arguments you pass to Popen are passed as arguments to this shell process. So /bin/sh -c 'echo' '$HOME' prints newline and the second argument is ignored. So usually you should only use string arguments with shell=True.

Why does subprocess use a list instead of a string with spaces by default?

Why does Python's subprocess module expect the arguments as a list by default? Why isn't a string with spaces (similar to what you type into a terminal when running the command normally) the default input? There are plenty of sources explaining how to pass in the space delimited string of the command into subprocess, but it's less clear as to why the default isn't the other way around.

TL;DR Using the list bypasses the shell so that you don't need to worry about the shell interpreting a dynamically constructed command line in ways you did not intend.
Suppose you have a really simple command: echo foo. Here it is, using both a string and a list:
Popen("echo foo", shell=True)
Popen(["echo", "foo"])
Not much difference yet. Now suppose the argument contains quotes to protect whitespace and/or a shell pattern, echo "foo * bar":
Popen("echo \"foo * bar\"", shell=True)
Popen(["echo", "foo * bar"])
Yes, I could have used single quotes to avoid needing to escape the double quotes, but you can see the list form is starting to have an advantage. Now imagine I don't have a literal argument for the command, but that it is stored in a variable. Now which do you want to use...
This?
Popen('echo "%s"' % (x,), shell=True)
or this?
Popen(["echo", x])
If you answered "the first one", here's the value of x:
x = "\";rm -rf \""
The command you just executed was echo ""; rm -rf/"". You needed to make sure any special characters in the value of x were first escaped before incorporating it into the string you are building to pass to the shell.
Or you just use a list and avoid the shell altogether.

Forget all that I wrote - just read the relevant PEP yourself
https://www.python.org/dev/peps/pep-0324/
===============
My short guess - the no-shell list version is closer to the format that is eventually passed to the POSIX forking commands. It requires less manipulation. The shell string approach is something of a Windows legacy.
=====================
So you are asking why the shell=False case is the default?
On POSIX, with shell=False (default): In this case, the Popen class
uses os.execvp() to execute the child program. args should normally
be a sequence. A string will be treated as a sequence with the string
as the only item (the program to execute).
On POSIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a sequence,
the first item specifies the command string, and any additional items
will be treated as additional shell arguments.
'why' questions tend to be closed because they rarely have definitive answers, or they involve opinions, or history.
I'd suggest studying the subprocess.py code. I see for example a lot of calls to:
Popen(*popenargs, **kwargs)
It's init is:
def __init__(self, args, bufsize=-1, executable=None,
stdin=None, stdout=None, stderr=None,
preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS,
shell=False, cwd=None, env=None, universal_newlines=False,
startupinfo=None, creationflags=0,
restore_signals=True, start_new_session=False,
pass_fds=()):
As a keyword arg, shell has to have some default value; why not False?
I suspect that in the shell case it passes a whole string to some code that calls the shell. In the no-shell case it must pass a list. But we have to find that code.
There are 2 methods of call the subprocess, one for POSIX and the other Windows. In the POSIX case it appears to convert the string list, regardless whether shell is True or not It may be more nuanced than that, but this is the relevant code:
"""Execute program (POSIX version)"""
if isinstance(args, (str, bytes)):
args = [args]
else:
args = list(args)
if shell:
args = ["/bin/sh", "-c"] + args
if executable:
args[0] = executable
....
self.pid = _posixsubprocess.fork_exec(
args, executable_list,...
In the windows shell case the args string is combined with cmd info:
if shell:
....
comspec = os.environ.get("COMSPEC", "cmd.exe")
args = '{} /c "{}"'.format (comspec, args)
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
# no special security
....

Calling psexec from a python script doesn't show the whole output

I want to use a python script to show up all local administrators in our domain.
My Code :
for line in open(anfangsrechner,"r"):
zeile = line.strip()
command ='\\\\' +zeile+ ' -i' ' net' ' localgroup' ' Administratoren'
abfrage = subprocess.Popen(['PsExec.exe ',command,],stdin=subprocess.PIPE,
stdout=subprocess.PIPE, )
# print (abfrage)
while True:
line = abfrage.communicate()
if not line:
break
print (line)
But I only get this from the psexec command:
PsExec v2.1 - Execute processes remotely Copyright (C) 2001-2013 Mark
Russinovich Sysinternals - www.sysinternals.com
Process finished with exit code 0
I don't get the whole output. Does someone know how I can fix it?

You are passing the arguments as a long string, rather than a list.
The quick fix would be using shell=True:
abfrage = subprocess.Popen('PsExec.exe '+command,
stdout=subprocess.PIPE,
shell=True)
The right way to do this would be creating a list of arguments and passing it.
Quoting the documentation:
args is required for all calls and should be a string, or a sequence
of program arguments. Providing a sequence of arguments is generally
preferred, as it allows the module to take care of any required
escaping and quoting of arguments (e.g. to permit spaces in file
names). If passing a single string, either shell must be True (see
below) or else the string must simply name the program to be executed
without specifying any arguments.

Launch gnu screen from python?

I tried executing a server daemon with gnu screen from subprocess call but it didn't even start
subprocess.call(["screen", "-dmS test ./server"])
I was told that running screen requires terminal, hence the reason why I can't simply execute it with call. Can you show me some piece of codes to do this?

Try
subprocess.call( ["screen", "-d", "-m", "-S", "test", "./server"] )
You need to break the argument string into separate arguments, one per string.
Here's the relevant quote from the subprocess docs:
On UNIX, with shell=False (default): In this case, the Popen class
uses os.execvp() to execute the child program. args should normally
be a sequence. A string will be treated as a sequence with the string
as the only item (the program to execute).
On UNIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a sequence,
the first item specifies the command string, and any additional items
will be treated as additional shell arguments.
So by default, the arguments are used exactly as you give them; it doesn't try to parse a string into multiple arguments. If you set shell to true, you could try the following:
subprocess.call("screen -dmS test ./server", shell=True)
and the string would be parsed exactly like a command line.

Forwarding command line arguments to a process in Python

I'm using a crude IDE (Microchip MPLAB) with C30 toolchain on Windows XP.
The C compiler has a very noisy output that I'm unable to control, and it's very hard to spot actual warnings and errors in output window.
I want to write a python script that would receive arguments for compiler, call the compiler with same arguments, filter results and output them to stdout. Then I can replace the compiler executable with my script in toolchain settings. The IDE calls my script and receives filtered compiler output.
My code for executing the compiler looks like this:
arguments = ' '.join(sys.argv[1:])
cmd = '%s %s' % (compiler_path, arguments)
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
The problem is that quotes from arguments are consumed on script execution, so if IDE calls my script with following arguments:
main.c -o"main.o"
the value of arguments is
main.c -omain.o
The most obvious solution is to put whole argument list in quotes, but this would require modification in compiler calling code in IDE. I also tried using batch file, but it can only accept nine parameters (%1 to %9), and compiler is called with 15+ parameters.
Is there a way to forward exactly the same arguments to a process from script?

Give the command arguments to Popen as a list:
arguments = sys.argv[1:]
cmd = [compiler_path] + arguments
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

As ChristopheD said the shell removes the quotes.
But you don't need to create the string yourself when using Popen: it can handle that for you automatically. You can do this instead:
import sys, subprocess
process = subprocess.Popen(sys.argv[1:], executable=compiler_path, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
The subprocess module hopefully will pass the arguments correctly for you.

Your shell is eating the quotes (the python script never even receives them) so I suppose it's not very easy to get them 'unaltered'.