We Keep Coding

Python subprocess.Popen : Running multiple commands with shell=True, but it doesn't seem to be waiting for all commands to complete

I'm trying to use subprocess.Popen to run a check on Kafka consumer groups and log their state, but it doesn't appear to be waiting for all the commands to run. It isn't giving me any stdout, but its also returning an exit code of 0.
prompt = ["cd", "~/path/to/kafka_2.11-2.1.0;", "pwd;", "./bin/kafka-consumer-groups.sh",
"--bootstrap-server", "localhost:9092", "--describe", "--group", "groupname"]
response = subprocess.run(prompt, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True, check=True)
print(response)
Prints:
CompletedProcess(args=['cd', '~/path/to/kafka_2.11-2.1.0;', 'pwd;', './bin/kafka-consumer-groups.sh', '--bootstrap-server', 'localhost:9092', '--describe', '--group', 'groupname'], returncode=0, stdout=b'', stderr=b'')
The pwd command was to primarily test if it would return any kind of stout, it won't be kept.
I've looked through the docs for subprocess, and I haven't see anything that suggests that it is unable to capture multiple stdouts. Also, according to the logs, the CompletedProcess is returned in less than 10ms, while running cd ~path/to/kafka_2.11-2.1.0; pwd; ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group groupname takes about 10-15s on my machine.
Please note that I'm using python3.5.2

Your error is more fundamental than you think. Your code runs
sh -c 'cd'
with $0 set to the directory, $1 set to pwd;, etc; so very far from what you want. (Just cd simply switches to your home directory; then the shell exits, without doing anything with all those arguments you passed in, and Python continues back in whichever directory was current before you launched the subprocess.)
Generally, pass a single string as the first argument with shell=True, and a list of strings when you don't have a shell.
subprocess.run(r"cd foo; pwd; use shell commands to your heart\'s content; run as many processes as you like as subprocesses of your shell subprocess", shell=True)
subprocess.run(['/bin/echo', 'one', 'single', 'process', 'with', 'arguments])

I found my error. cd doesn't work with subprocess, but subprocess offers the cwd named argument that accepts the path you need to run arguments. The reason it was returning a CompletedProcess so quickly is that it was successfully changing directories and then exiting the subprocess.
Sorry, I wasn't thinking about cd being the culprit until I came across this question which answered my problem too.

How to run a shell script in python from memory?

The application I'm writing retrieves a shell script through HTTP from Network, I want to run this script in python however I don't want to physically save it to the hard drive because I have its content already in memory, and I would like to just execute it. I have tried something like this:
import subprocess
script = retrieve_script()
popen = subprocess.Popen(scrpit, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
stdOut, stdErr = popen.communicate()
def retrieve_script_content():
# in reality I retrieve a shell script content from network,
# but for testing purposes I will just hardcode some test content here
return "echo command1" + "\n" + "echo command2" + " \n" + "echo command3"
This snippet will not work because subprocess.Popen expects you to provide only one command at a time.
Are there any alternatives to run a shell script from memory?

This snippet will not work because subprocess.Popen expects you to provide only one command at a time.
That is not the case. Instead, the reason why it doesn't work is:
The declaration of retrieve_script has to come before the call
You call it retrieve_script_content instead of retrieve_script
You misspelled script as scrpit
Just fix those and it's fine:
import subprocess
def retrieve_script():
return "echo command1" + "\n" + "echo command2" + " \n" + "echo command3"
script = retrieve_script()
popen = subprocess.Popen(script, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
stdOut, stdErr = popen.communicate()
print(stdOut);
Result:
$ python foo.py
command1
command2
command3
However, note that this will ignore the shebang (if any) and run the script with the system's sh every time.

Are you using a Unix-like OS? If so, you should be able to use a virtual filesystem to make an in-memory file-like object at which you could point subprocess.Popen:
import subprocess
import tempfile
import os
import stat
def retrieve_script_content():
# in reality I retrieve a shell script content from network,
# but for testing purposes I will just hardcode some test content here
return "echo command1" + "\n" + "echo command2" + " \n" + "echo command3"
content = retrieve_script_content()
with tempfile.NamedTemporaryFile(mode='w', delete=False, dir='/dev/shm') as f:
f.write(content)
os.chmod(f.name, stat.S_IRUSR | stat.S_IXUSR)
# print(f.name)
popen = subprocess.Popen(f.name, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True)
stdOut, stdErr = popen.communicate()
print(stdOut.decode('ascii'))
# os.unlink(f.name)
prints
command1
command2
command3
Above I used /dev/shm as the virtual filesystem since Linux systems based on Glibc always have a tmpfs mounted on /dev/shm.
If security is a concern you may wish to setup a ramfs.
One reason why you might want to use a virtual file instead of passing the script contents directly to subprocess.Popen is that the maximum size for a single string argument is limited to 131071 bytes.

You can execute multi command script with Popen. Popen only restricts you to one-command string when shell flag is False, yet it is possible to pass a list of commands. Popen's flag shell=True allows for multi-command scripts (it is considered unsecure, though what you are doing - executing scripts from the web - is already very risky).

python issue with popen and mysql

I'm new to Python, and haven't used Linux in years, so I'm not sure where I'm getting tangled up. I'm trying to use Popen to run sql files in MySQL on Ubuntu.
Here is the relevant code:
command = ['mysql', '-uUSER', '-pPWD','-h192.168.1.132', '--database=dbName', '<', './1477597236_foo.sql' ]
print("command is: "+subprocess.list2cmdline(command))
proc = subprocess.Popen(
command, stderr=subprocess.PIPE, stdout=subprocess.PIPE, cwd='.'
)
the output from this is the same as if had run 'mysql --help'. The puzzling thing to me is that if i take the command output by subprocess.list2cmdline and run it directly, it runs perfectly. Also, if i replace '< file.sql' with '-e select * from foo', it runs. So, the '<' and file are causing my problem. I know WHAT is causing the problem, but nothing I've tried so far has fixed it.
tia, Craig

When a redirection or pipe or built-in command is present in the command line, shell=True is required. However, in simple cases like this, shell=True is overkill. There's a much cleaner way in order to avoid shell=True which gives better control on the input file.
if the input file doesn't exist, you get an exception before reaching the subprocess, which is easier to handle
the process runs without the shell: better portability & performance
the code:
command = ['mysql', '-uUSER', '-pPWD','-h192.168.1.132', '--database=dbName' ]
with open('./1477597236_foo.sql') as input_file:
proc = subprocess.Popen(
command, stdin = input_file, stderr=subprocess.PIPE, stdout=subprocess.PIPE )
output,error = proc.communicate()
(I added the next line which should be a communicate call: since both stdout & stderr are redirected, it's the only simple way to avoid deadlocks between both output streams)

So you need to add shell=True to your Popen call. < is a part of the shell and you can't use shell features without that parameter.
proc = subprocess.Popen( command, stderr=subprocess.PIPE, stdout=subprocess.PIPE, cwd='.',shell=True )

python - subprocess.Popen() synax errors

I am trying to mirror the following shell command using subprocess.Popen():
echo "SELECT employeeid FROM Users WHERE samaccountname=${1};" | bsqldb -S mdw2k8sqlp02.dow.com -D PhoneBookClient -U PortManUser -P plum45\\torts -q
It currently looks like:
stdout = subprocess.Popen(["echo", "\"SELECT", "employeeid", "FROM", "Users", "WHERE", "samaccountname=${1};\"", "|", "bsqldb", "arg1etc"], stdout=subprocess.PIPE)
for line in stdout.stdout.readlines():
print line
It seems that this is wrong, it returns the following standard out:
"SELECT employeeid FROM Users WHERE samaccountname=${1};" | bsqldb arg1etc
Does anyone know where my syntax for subprocess.Popen() has gone wrong?

The problem is that you're trying to run a shell command without the shell. What happens is that you're passing all of those strings—including "|" and everything after is—as arguments to the echo command.
Just add shell=True to your call to fix that.
However, you almost definitely want to pass the command line as a string, instead of trying to guess at the list that will be joined back up into the string to pass to the shell.
Or, even better, don't use the shell, and instead pipe within Python. The docs have a nice section about Replacing shell pipeline (and all kinds of other things) with subprocess code.
But in your case, the thing you're trying to pipe is just echo, which is quite silly, since you already have exactly what echo would return, and can just feed it as the input to the second program.
Also, I'm not sure what you expect that ${1} to get filled in with. Presumably you're porting a shell script that took some arguments on the command line; your Python script may have the same thing in sys.argv[1], but without knowing more about what you're doing, that's little more than a guess.

The analog of echo some string | command arg1 arg2 shell pipeline in Python is:
from subprocess import Popen, PIPE
p = Popen(["command", "arg1", "arg2"], stdin=PIPE)
p.communicate("some string")
In your case, you could write it as:
import shlex
import sys
from subprocess import Popen, PIPE
cmd = shlex.split("bsqldb -S mdw2k8sqlp02.dow.com -D PhoneBookClient "
"-U PortManUser -P plum45\\torts -q")
sql = """SELECT employeeid FROM Users
WHERE samaccountname={name};""".format(name=sql_escape(sys.argv[1]))
p = Popen(cmd, stdin=PIPE)
p.communicate(input=sql)
sys.exit(p.returncode)

Popen not taking the arguments

Hi all I am learning python and shell script and for GUI i am using wxpython.
I have said to make a automated tool which does some operation , deb creation is also one of that.
for deb creation , command is:
./myfile -u username
I have tried os.Popen, os.system, subprocess.Popen , subprocess.call.But no use everytime "-u" wont take effect, "-u " is must. I have tried by storing "-u" in variable and the passed it but still no use.
Please suggest me the exact way or where i am doing wrong
No error message but "myfile" output shows that "-u" has not given in command.
Code is:
1. cmd = ['./myfile', '-u', 'username']
Popen(cmd,shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
2. user = "-u"
name = "username"
sbprocess.call("./myfile %s %s" %(str(user),str(name)), shell=True)
same kind using "os" command also

The first code example in your question passes incorrect command because shell=True changes the meaning of the first parameter, from the subprocess docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
..[snip]..
If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
The second code example from you question should work if the following command works in a shell:
$ /bin/sh -c './myfile -u username'
To fix your command, you could omit possibly unnecessary shell=True and use check_call():
import subprocess
subprocess.check_call(["./myfile", "-u", "username"])

Try 'bash' instead of './'. If its bash script or else use respective. Its work only if your myfile is shell script.
subprocess.call("bash myfile -u %s" % str(name)), shell=True)