We Keep Coding

Running subprocess with spaces in options in python

I tried to search for an answer for a while, but I did not find anything so far for my specific case. I want to run command in python:
ssh -o ConnectTimeout=3 -o ProxyCommand="ssh -q -W %h:%p bastion.host.com" host.com "screen -dmS TEST /bin/bash --login -c 'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log'"
this is my code:
import subprocess
server_command = "screen -dmS TEST /bin/bash --login -c 'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log'"
command = ['ssh', '-o', 'ConnectTimeout=3', 'ProxyCommand="ssh -q -W %h:%p bastion.host.com"', 'host.com', server_command]
p = subprocess.Popen(command, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
stdout, stderr = p.communicate(input=None)
Everything was working (screen was spawned with script running) until I added option with spaces: ProxyCommand="ssh -q -W %h:%p bastion.host.com".
After that I get error:
>>> print(stderr)
b'ssh: Could not resolve hostname ProxyCommand="ssh -q -W %h:%p bastion.host.com": Name or service not known\r\n'
How can I please pass this option to my command?

Your SSH command contains invalid arguments: ProxyCommand is an option, so it needs to be preceded by -o, same as ConnectTimeout (and, as noted by Charles Duffy, the redundant quotes inside that option string need to be removed, since the command is not passed to the shell):
server_command = 'screen -dmS TEST /bin/bash --login -c \'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log\''
command = ['ssh', '-o', 'ConnectTimeout=3', '-o', 'ProxyCommand=ssh -q -W %h:%p bastion.host.com', 'host.com', server_command]
In general when your command line contains spaces and/or quotes and is passed to another command, it may be necessary to shell-quote it. The Python function shlex.quote automates this. In your case it’s not necessary because you (correctly) manually quoted the command you’re passing to screen inside server_command. Alternatively you could have written the following:
script_command = 'yes | script.sh --option-1 value1 -option2 value2 2>&1 | tee output.log'
server_command = f'screen -dmS TEST /bin/bash --login -c {shlex.quote(script_command)}'
— Note the absence of manual quotes inside the shell command line. The advantage over manual quoting is that this will also work with nested levels of shell quoting, e.g. when nesting command invocations.

Pass python variable into os.system(gnome-terminal) command

How can I pass a python variable into the gnome-terminal command option ? I want to open multiple terminal with a specific command.
Here is the code (an example) of what I want to do :
cmd = "echo OK"
os.system("gnome-terminal -e 'bash -c \"'cmd' ; exec bash\"'")
But it's not working because the shell tries to interpret the command "cmd" (bash : cmd: command not found)
Can you help me please ?
Thank you guys

You could use format to replace your variable into the string :
cmd = "echo OK"
os.system("gnome-terminal -e 'bash -c \"'{}' ; exec bash\"'".format(cmd))

I think you're passing the string "cmd" not the variable cmd = "echo OK". Try out this.
cmd = "echo OK"
os.system("gnome-terminal -e 'bash -c " + cmd + " ; exec bash'")
EDIT>
Maybe the module subprocess can help you. Try this snippet.
import subprocess
cmd_line = "echo Hello!"
p = subprocess.Popen(cmd_line, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out = p.communicate()[0]
print out

How to pass this tar command as argument to python subprocess

If I am running the command that is specified in the args on the terminal then it goes successfully on terminal but doing the same in python program is not working; I am seeing junk characters in the screen to the size of the input tar file and lot of xterm words too;
I feel the problem is handling the ' ' letters in the args;
import subprocess
try:
args = "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz".split()
subprocess.check_call(args)
except subprocess.CalledProcessError as e:
print e

I am not specialist, but this i found - this commands not working in sh, but working in bash:
$ sh -c "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz"
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz'
$
$ bash -c "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz"
$
Thats a reason why it not work in subprocess directly. This code looks work fine:
import subprocess
command = "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz"
subprocess.Popen(command, shell=True, executable='/bin/bash')

I tried a number of alternatives, and none of them were satisfactory. The best I found was switching over to Popen.
# this should have the a similar signature to check_call
def run_in_shell(*args):
# unfortunately, `args` won't be escaped as it is actually a string argument to bash.
proc = subprocess.Popen(['/bin/bash', '-c', ' '.join(args)])
# This will also work, though I have found users who had problems with it.
# proc = subprocess.Popen(' '.join(args), shell=True, executable='/bin/bash')
stat = proc.wait()
if stat != 0:
subprocess.CalledProcessError(returncode=stat, cmd=command)
return stat
run_in_shell("cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz")
As a note: /bin/sh has problems with the unescaped parentheses. If you don't want to specify '/bin/bash' above, then you will need to escape the paren:
args = 'cat parsing.tgz <\\(echo -n ''| gzip\\)> new-file.tgz'

Automating install of Docker and image pull using python or bash

I am trying to automate the process of installing the docker-engine and then asking the user if he would like to pull rhel/suse/centos images.
Using python was my first idea but I have added bash script to make things easier, python seemed not too friendly to run cli commands.
Now, I am planning to expand the functionality and shell script will not scale.
How do I convert this script to python? Many common command line operations such as "yum install",etc are not easy without using additional python imports.
If you have any easier suggestions, please advice
Thanks!
This is bash script still in the works...
#!/bin/sh
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
fi
echo
echo " - Installing packages"
echo
if [[ -e /usr/bin/yum ]]; then
#Verify packages are up to date
yum update
#Install Docker
yum install docker-engine
else
echo "No yum, lets try apt-get"
sudo apt-get update
#sudo apt-get -y upgrade
#sudo apt-get install linux-image-extra-`uname -r`
#sudo apt-get install docker-engine
fi
if [ $? -eq 0 ]; then
echo success
else
echo failed
exit
fi
#start Docker
echo "Would you like to start Docker and pull Images? Select 1 or 2"
select y1 in "Yes" "No"; do
case $y1 in
Yes ) service docker start; docker pull "rhel:7.2" ;docker pull "mstormo/suse" ;break;;
No ) exit;;
esac
echo " - Complete!"
echo
done

Ok, so I was able to make it work in Python. I will leave the code here, in case anyone needs it.
Cheers!
Rohit
from subprocess import Popen, PIPE
uid = Popen(['id', '-u'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
uid_get = uid.stdout.read(1)
if uid_get != '0':
print "This script needs to be run as root"
print " - Installing packages "
check1 = Popen(['/usr/bin/yum'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
check_yum = check1.stdout.read()
if check_yum == ' ':
print "Yum is not found, trying apt-get"
proc = Popen('apt-get update', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
proc = Popen('apt-get upgrade', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
proc = Popen('apt-get install', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
proc = Popen('apt-get install docker-engine', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
else:
print "Running yum install"
proc = Popen('yum update', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
proc = Popen('yum install docker-engine', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
print "Would you like to start Docker and pull images - RHEL and SUSE? -> y or n ?"
y="yes"
n="no"
choice = raw_input().lower()
if choice in y:
print "Pulling RHEL and SUSE images"
proc = Popen('service docker start; docker pull "rhel:7.2" ;docker pull "mstormo/suse" ; docker run rhel sh -c "cat /etc/*release"; docker run "mstormo/suse" sh -c "cat /etc/*release"', shell=True, stdin=None, executable="/bin/bash")
proc.wait()
elif choice in n:
print "Thank you, exiting...."
else:
print " Invalid selection"
print " - Complete! "

I would suggest trying the sh module:
from sh import docker
for line in docker('build', '-t', some_tag, '.', _iter=True):
sys.stdout.write(str(line))
This is a long running command so it is nice to show its output
line by line. This is what the for loop and _iter does.

shell start / stop for python script

I have a simple python script i need to start and stop and i need to use a start.sh and stop.sh script to do it.
I have start.sh:
#!/bin/sh
script='/path/to/my/script.py'
echo 'starting $script with nohup'
nohup /usr/bin/python $script &
and stop.sh
#!/bin/sh
PID=$(ps aux | grep "/path/to/my/script.py" | awk '{print $2}')
echo "killing $PID"
kill -15 $PID
I'm mainly concerned with the stop.sh script. I think that's an appropriate way to find the pid but i wouldn't bet much on it. start.sh successfully starts it. when i run stop.sh, i can no longer find the process by "ps aux | grep 'myscript.py'" but the console outputs:
killing 25052
25058
./stop.sh: 5: kill: No such process
so it seems like it works AND gives an error of sorts with "No such process".
Is this actually an error? Am I approaching this in a sane way? Are there other things I should be paying attention to?
EDIT - I actually ended up with something like this:
start.sh
#!/bin/bash
ENVT=$1
COMPONENTS=$2
TARGETS=("/home/user/project/modules/script1.py" "/home/user/project/modules/script2.py")
for target in "${TARGETS[#]}"
do
PID=$(ps aux | grep -v grep | grep $target | awk '{print $2}')
echo $PID
if [[ -z "$PID" ]]
then
echo "starting $target with nohup for env't: $ENVT"
nohup python $target $ENVT $COMPONENTS &
fi
done
stop.sh
#!/bin/bash
ENVT=$1
TARGETS=("/home/user/project/modules/script1.py" "/home/user/project/modules/script2.py")
for target in "${TARGETS[#]}"
do
pkill -f $target
echo "killing process $target"
done

It is because ps aux |grep SOMETHING also finds the grep SOMETHING process, because SOMETHING matches. After the execution the grep is finished, so it cannot find it.
Add a line: ps aux | grep -v grep | grep YOURSCRIPT
Where -v means exclude. More in man grep.

The "correct" approach would probably be to have your script write its pid to a file in /var/run, and clear it out when you kill the script. If changing the script is not an option, have a look at start-stop-daemon.
If you want to continue with the grep-like approach, have a look at proctools. They're built in on most GNU/Linux machines and readily available on BSD including OS X:
pkill -f /path/to/my/script.py

init-type scripts are useful for this. This is very similar to one I use. You store the pid in a file, and when you want to check if it's running, look into the /proc filesystem.
#!/bin/bash
script_home=/path/to/my
script_name="$script_home/script.py"
pid_file="$script_home/script.pid"
# returns a boolean and optionally the pid
running() {
local status=false
if [[ -f $pid_file ]]; then
# check to see it corresponds to the running script
local pid=$(< "$pid_file")
local cmdline=/proc/$pid/cmdline
# you may need to adjust the regexp in the grep command
if [[ -f $cmdline ]] && grep -q "$script_name" $cmdline; then
status="true $pid"
fi
fi
echo $status
}
start() {
echo "starting $script_name"
nohup "$script_name" &
echo $! > "$pid_file"
}
stop() {
# `kill -0 pid` returns successfully if the pid is running, but does not
# actually kill it.
kill -0 $1 && kill $1
rm "$pid_file"
echo "stopped"
}
read running pid < <(running)
case $1 in
start)
if $running; then
echo "$script_name is already running with PID $pid"
else
start
fi
;;
stop)
stop $pid
;;
restart)
stop $pid
start
;;
status)
if $running; then
echo "$script_name is running with PID $pid"
else
echo "$script_name is not running"
fi
;;
*) echo "usage: $0 <start|stop|restart|status>"
exit
;;
esac

ps aux | grep "/path/to/my/script.py"
will return both the pid for the instance of script.py and also for this instance of grep. That'll probably be why you're getting a no such process: by the time you get around to killing the grep, it's already dead.

I don't have a unix box on at the moment, so i can't test this, but it should be fairly simple to get the idea.
start.sh:
if [ -e ./temp ]
then
pid=`cat temp`
echo "Process already exists; $pid"
else
script='/path/to/my/script.py'
echo 'starting $script with nohup'
nohup /usr/bin/python $script &
echo $! > temp
fi
stop.sh:
if [ -e ./temp ]
then
pid=`cat temp`
echo "killing $pid"
kill -15 $PID
rm temp
else
echo "Process not started"
fi
Try this out.