We Keep Coding

Python subprocess.run ignores --exclude clause

I have one issue with subprocess.run.
This command in a Bash shell works without any problem:
tar -C '/home/' --exclude={'/home/user1/.cache','/home/user1/.config'} -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1
But if I execute it through Python:
cmd = "tar -C '/home/' --exclude={'/home/user1/.cache','/home/user1/.config'} -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1"
subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)
The execution works without errors but the --exclude clause is not considered.
Why?

Whether or not curly brace expansion is handled correctly depends on what the standard system shell is. By default, subprocess.run() invokes /bin/sh. On systems like Linux, /bin/sh is bash. On others, such as FreeBSD, it's a different shell that doesn't support brace expansion.
To ensure the subprocess runs with a shell that can handle braces properly, you can tell subprocess.run() what shell to use with the executable argument:
subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, executable='/bin/bash')
As a simple example of this, here's a system where /bin/sh is bash:
>>> subprocess.run("echo foo={a,b}", shell=True)
foo=a foo=b
and one where it's not:
>>> subprocess.run("echo foo={a,b}", shell=True)
foo={a,b}
but specifying another shell works:
>>> subprocess.run("echo foo={a,b}", shell=True, executable='/usr/pkg/bin/bash')
foo=a foo=b

Bash curly expansion doesn't work inside Python and will be sent by subprocess as they are - they will not be expanded, regardless of the arguments you use on run().
Edit: unless of course the argument executable='/bin/bash' as stated on the other answer which seems to work after all
In a bash shell,
--exclude {'/home/user1/.cache','/home/user1/.config'}
becomes:
--exclude=/home/user1/.cache --exclude=/home/user1/.config
So to achieve the same result, in Python it must be expressed like this (one of the possible ways) before sending the command string to subprocess.run:
' '.join(["--exclude=" + path for path in ['/home/user1/.cache','/home/user1/.config']])
cmd = "tar -C '/home/' " + ' '.join(["--exclude=" + path for path in ['/home/user1/.cache','/home/user1/.config']]) + " -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1"
print(cmd) # output: "tar -C '/home/' --exclude=/home/user1/.cache --exclude=/home/user1/.config -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1"
subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)

Running subprocess with spaces in options in python

I tried to search for an answer for a while, but I did not find anything so far for my specific case. I want to run command in python:
ssh -o ConnectTimeout=3 -o ProxyCommand="ssh -q -W %h:%p bastion.host.com" host.com "screen -dmS TEST /bin/bash --login -c 'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log'"
this is my code:
import subprocess
server_command = "screen -dmS TEST /bin/bash --login -c 'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log'"
command = ['ssh', '-o', 'ConnectTimeout=3', 'ProxyCommand="ssh -q -W %h:%p bastion.host.com"', 'host.com', server_command]
p = subprocess.Popen(command, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
stdout, stderr = p.communicate(input=None)
Everything was working (screen was spawned with script running) until I added option with spaces: ProxyCommand="ssh -q -W %h:%p bastion.host.com".
After that I get error:
>>> print(stderr)
b'ssh: Could not resolve hostname ProxyCommand="ssh -q -W %h:%p bastion.host.com": Name or service not known\r\n'
How can I please pass this option to my command?

Your SSH command contains invalid arguments: ProxyCommand is an option, so it needs to be preceded by -o, same as ConnectTimeout (and, as noted by Charles Duffy, the redundant quotes inside that option string need to be removed, since the command is not passed to the shell):
server_command = 'screen -dmS TEST /bin/bash --login -c \'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log\''
command = ['ssh', '-o', 'ConnectTimeout=3', '-o', 'ProxyCommand=ssh -q -W %h:%p bastion.host.com', 'host.com', server_command]
In general when your command line contains spaces and/or quotes and is passed to another command, it may be necessary to shell-quote it. The Python function shlex.quote automates this. In your case it’s not necessary because you (correctly) manually quoted the command you’re passing to screen inside server_command. Alternatively you could have written the following:
script_command = 'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log'
server_command = f'screen -dmS TEST /bin/bash --login -c {shlex.quote(script_command)}'
— Note the absence of manual quotes inside the shell command line. The advantage over manual quoting is that this will also work with nested levels of shell quoting, e.g. when nesting command invocations.

How to pass this tar command as argument to python subprocess

If I am running the command that is specified in the args on the terminal then it goes successfully on terminal but doing the same in python program is not working; I am seeing junk characters in the screen to the size of the input tar file and lot of xterm words too;
I feel the problem is handling the ' ' letters in the args;
import subprocess
try:
args = "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz".split()
subprocess.check_call(args)
except subprocess.CalledProcessError as e:
print e

I am not specialist, but this i found - this commands not working in sh, but working in bash:
$ sh -c "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz"
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz'
$
$ bash -c "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz"
$
Thats a reason why it not work in subprocess directly. This code looks work fine:
import subprocess
command = "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz"
subprocess.Popen(command, shell=True, executable='/bin/bash')

I tried a number of alternatives, and none of them were satisfactory. The best I found was switching over to Popen.
# this should have the a similar signature to check_call
def run_in_shell(*args):
# unfortunately, `args` won't be escaped as it is actually a string argument to bash.
proc = subprocess.Popen(['/bin/bash', '-c', ' '.join(args)])
# This will also work, though I have found users who had problems with it.
# proc = subprocess.Popen(' '.join(args), shell=True, executable='/bin/bash')
stat = proc.wait()
if stat != 0:
subprocess.CalledProcessError(returncode=stat, cmd=command)
return stat
run_in_shell("cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz")
As a note: /bin/sh has problems with the unescaped parentheses. If you don't want to specify '/bin/bash' above, then you will need to escape the paren:
args = 'cat parsing.tgz <\\(echo -n ''| gzip\\)> new-file.tgz'

Automating install of Docker and image pull using python or bash

I am trying to automate the process of installing the docker-engine and then asking the user if he would like to pull rhel/suse/centos images.
Using python was my first idea but I have added bash script to make things easier, python seemed not too friendly to run cli commands.
Now, I am planning to expand the functionality and shell script will not scale.
How do I convert this script to python? Many common command line operations such as "yum install",etc are not easy without using additional python imports.
If you have any easier suggestions, please advice
Thanks!
This is bash script still in the works...
#!/bin/sh
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
fi
echo
echo " - Installing packages"
echo
if [[ -e /usr/bin/yum ]]; then
#Verify packages are up to date
yum update
#Install Docker
yum install docker-engine
else
echo "No yum, lets try apt-get"
sudo apt-get update
#sudo apt-get -y upgrade
#sudo apt-get install linux-image-extra-`uname -r`
#sudo apt-get install docker-engine
fi
if [ $? -eq 0 ]; then
echo success
else
echo failed
exit
fi
#start Docker
echo "Would you like to start Docker and pull Images? Select 1 or 2"
select y1 in "Yes" "No"; do
case $y1 in
Yes ) service docker start; docker pull "rhel:7.2" ;docker pull "mstormo/suse" ;break;;
No ) exit;;
esac
echo " - Complete!"
echo
done

Ok, so I was able to make it work in Python. I will leave the code here, in case anyone needs it.
Cheers!
Rohit
from subprocess import Popen, PIPE
uid = Popen(['id', '-u'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
uid_get = uid.stdout.read(1)
if uid_get != '0':
print "This script needs to be run as root"
print " - Installing packages "
check1 = Popen(['/usr/bin/yum'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
check_yum = check1.stdout.read()
if check_yum == ' ':
print "Yum is not found, trying apt-get"
proc = Popen('apt-get update', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
proc = Popen('apt-get upgrade', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
proc = Popen('apt-get install', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
proc = Popen('apt-get install docker-engine', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
else:
print "Running yum install"
proc = Popen('yum update', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
proc = Popen('yum install docker-engine', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
print "Would you like to start Docker and pull images - RHEL and SUSE? -> y or n ?"
y="yes"
n="no"
choice = raw_input().lower()
if choice in y:
print "Pulling RHEL and SUSE images"
proc = Popen('service docker start; docker pull "rhel:7.2" ;docker pull "mstormo/suse" ; docker run rhel sh -c "cat /etc/*release"; docker run "mstormo/suse" sh -c "cat /etc/*release"', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
elif choice in n:
print "Thank you, exiting...."
else:
print " Invalid selection"
print " - Complete! "

I would suggest trying the sh module:
from sh import docker
for line in docker('build', '-t', some_tag, '.', _iter=True):
sys.stdout.write(str(line))
This is a long running command so it is nice to show its output
line by line. This is what the for loop and _iter does.

Python execute complex shell command

Hi I have to execute a shell command :diff <(ssh -n root#10.22.254.34 cat /vms/cloudburst.qcow2.*) <(ssh -n root#10.22.254.101 cat /vms/cloudburst.qcow2)
I tried
cmd="diff <(ssh -n root#10.22.254.34 cat /vms/cloudburst.qcow2.*) <(ssh -n root#10.22.254.101 cat /vms/cloudburst.qcow2)"
args = shlex.split(cmd)
output,error = subprocess.Popen(args,stdout = subprocess.PIPE, stderr= subprocess.PIPE).communicate()
However I am getting an error diff: extra operand cat
I am pretty new to python. Any help would be appreciated

You are using <(...) (process substitution) syntax, which is interpreted by the shell. Provide shell=True to Popen to get it to use a shell:
cmd = "diff <(ssh -n root#10.22.254.34 cat /vms/cloudburst.qcow2.*) <(ssh -n root#10.22.254.101 cat /vms/cloudburst.qcow2)"
output,error = subprocess.Popen(cmd, shell=True, executable="/bin/bash", stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
Since you don't want the Bourne shell (/bin/sh), use the executable argument to determine the shell to use.

You are using a special syntax called process substitiution in your command line. This is supported by most modern shells (bash, zsh), but not by /bin/sh. Therefore, the method suggested by Ned might not work. (It could, if another shell provides /bin/sh and does not "correctly emulate" sh's behaviour, but it is not guaranteed to).
try this instead:
cmd = "diff <(ssh -n root#10.22.254.34 cat /vms/cloudburst.qcow2.*) <(ssh -n root#10.22.254.101 cat /vms/cloudburst.qcow2)"
output,error = subprocess.Popen(['/bin/bash', '-c', cmd], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
This is basically what the shell=True parameter does, but with /bin/bash instead of /bin/sh (as described in the subprocess docs).