Print executed command for Python subprocess.Popen - python

I have a script that is automating author re-writes on a number of git repositories.
def filter_history(old, new, name, repoPath):
command = """--env-filter '
an="$GIT_AUTHOR_NAME"
am="$GIT_AUTHOR_EMAIL"
cn="$GIT_COMMITTER_NAME"
cm="$GIT_COMMITTER_EMAIL"
if [[ "$GIT_COMMITTER_NAME" = "|old|" ]]
then
cn="|name|"
cm="|new|"
fi
if [[ "$GIT_AUTHOR_NAME" = "|old|" ]]
then
an="|name|"
am="|new|"
fi
export GIT_AUTHOR_NAME="$an"
export GIT_AUTHOR_EMAIL="$am"
export GIT_COMMITTER_NAME="$cn"
export GIT_COMMITTER_EMAIL="$cm"
'
"""
#DO string replace
command = command.replace("|old|", old)
command = command.replace("|new|", new)
command = command.replace("|name|", name)
print "git filter-branch -f " + command
process = subprocess.Popen(['git filter-branch -f', command],cwd=os.path.dirname(repoPath), shell=True)
process.wait()
The command executes fine, but tells me that nothing changed in the repo history. However, if I take the command that is printed out (which should be what is being executed), drop it in a shell script, and execute it, it changes the history fine. I think that the command is somehow not being executed correctly. Is there any way for be to see exactly what command the subprocess module is executing?

When you use shell = True, subprocess.Popen expects a string as its first argument. It is better not to use shell = True if you can help it, since it can be a security risk (see the Warning.
When you omit shell = True, or use shell = False, subprocess.Popen expects a list of arguments. You can generate that list of arguments from a string using shlex.split:
import shlex
import subprocess
def filter_history(old, new, name, repoPath):
"""Change author info
"""
# http://help.github.com/change-author-info/
# http://stackoverflow.com/a/3880493/190597
command = """git filter-branch -f --env-filter '
an="$GIT_AUTHOR_NAME"
am="$GIT_AUTHOR_EMAIL"
cn="$GIT_COMMITTER_NAME"
cm="$GIT_COMMITTER_EMAIL"
if [[ "$GIT_COMMITTER_NAME" = "{old}" ]]
then
cn="{name}"
cm="{new}"
fi
if [[ "$GIT_AUTHOR_NAME" = "{old}" ]]
then
an="{name}"
am="{new}"
fi
export GIT_AUTHOR_NAME="$an"
export GIT_AUTHOR_EMAIL="$am"
export GIT_COMMITTER_NAME="$cn"
export GIT_COMMITTER_EMAIL="$cm"
'
""".format(old = old, new = new, name = name)
process = subprocess.Popen(
shlex.split(command),
cwd = os.path.dirname(repoPath))
process.communicate()

If your application is running in a Windows environment, as stated in the following answer, subprocess has an undocumented function called subprocess.list2cmdline which you could use. subprocess.list2cmdline translates a sequence of arguments into a command line string, using the same rules as the MS C runtime.
if you are using Python > 3.3 you could also get the args list directly from the subprocess object using .args:
import subprocess
process = subprocess.Popen(...)
subprocess.list2cmdline(process.args)
Since Python 3.8 there is also a possibility to use the shlex.join() function:
Keep in mind though that subprocess does everything via IPC, so the best approach would be to simply examine the args list, as they will be passed to argv in the called program.

Related

Running a command as the first command in subprocess

I have a function that will open a terminal:
def open_next_terminal():
import subprocess
def rand(path="/tmp"):
acc = string.ascii_letters
retval = []
for _ in range(7):
retval.append(random.choice(acc))
return "{}/{}.sh".format(path, ''.join(retval))
file_path = rand()
with open(file_path, "a+") as data:
data.write(
'''
#!/bin/bash
"$#"
exec $SHELL
'''
)
subprocess.call(["sudo", "bash", "{}".format(file_path)])
return file_path
I want to run a command in this newly opened terminal before anything is done in it. For example:
subprocess.call(["sudo", "bash", "{}".format(file_path)]) #<= is called
ls #<= is run
#<= some output of files and folders
root#host:~# #<= the shell is now available
Does subprocess allow a way for me to run a "first command" during the initialization of the shell?
Simply pass the shell=True parameter to subprocess.call, and you can run multiple commands (delimited by semicolons or newlines) as a single string.
subprocess.call('your_first_command.sh; your_real_work.sh', shell=True)

How to source script via python

I can source bash script (without shebang) easy as bash command in terminal but trying to do the same via python command
sourcevars = "cd /etc/openvpn/easy-rsa && . ./vars"
runSourcevars = subprocess.Popen(sourcevars, shell = True)
or
sourcevars = [". /etc/openvpn/easy-rsa/vars"]
runSourcevars = subprocess.Popen(sourcevars, shell = True)
I receive :
Please source the vars script first (i.e. "source ./vars")
Make sure you have edited it to reflect your configuration.
What's the matter, how to do it correctly?I've read some topics here,e.g here but could not solve my problem using given advices. Please explain with examples.
UPDATED:
# os.chdir = ('/etc/openvpn/easy-rsa')
initvars = "cd /etc/openvpn/easy-rsa && . ./vars && ./easy-rsa ..."
# initvars = "cd /etc/openvpn/easy-rsa && . ./vars"
# initvars = [". /etc/openvpn/easy-rsa/vars"]
cleanall = ["/etc/openvpn/easy-rsa/clean-all"]
# buildca = ["printf '\n\n\n\n\n\n\n\n\n' | /etc/openvpn/easy-rsa/build-ca"]
# buildkey = ["printf '\n\n\n\n\n\n\n\n\n\nyes\n ' | /etc/openvpn/easy-rsa/build-key AAAAAA"]
# buildca = "cd /etc/openvpn/easy-rsa && printf '\n\n\n\n\n\n\n\n\n' | ./build-ca"
runInitvars = subprocess.Popen(cmd, shell = True)
# runInitvars = subprocess.Popen(initvars,stdout=subprocess.PIPE, shell = True, executable="/bin/bash")
runCleanall = subprocess.Popen(cleanall , shell=True)
# runBuildca = subprocess.Popen(buildca , shell=True)
# runBuildca.communicate()
# runBuildKey = subprocess.Popen(buildkey, shell=True )
UPDATE 2
buildca = ["printf '\n\n\n\n\n\n\n\n\n' | /etc/openvpn/easy-rsa/build-ca"]
runcommands = subprocess.Popen(initvars+cleanall+buildca, shell = True)
There's absolutely nothing wrong with this in and of itself:
# What you're already doing -- this is actually fine!
sourcevars = "cd /etc/openvpn/easy-rsa && . ./vars"
runSourcevars = subprocess.Popen(sourcevars, shell=True)
# ...*however*, it won't have any effect at all on this:
runOther = subprocess.Popen('./easy-rsa build-key yadda yadda', shell=True)
However, if you subsequently try to run a second subprocess.Popen(..., shell=True) command, you'll see that it doesn't have any of the variables set by sourcing that configuration.
This is entirely normal and expected behavior: The entire point of using source is to modify the state of the active shell; each time you create a new Popen object with shell=True, it's starting a new shell -- their state isn't carried over.
Thus, combine into a single call:
prefix = "cd /etc/openvpn/easy-rsa && . ./vars && "
cmd = "/etc/openvpn/easy-rsa/clean-all"
runCmd = subprocess.Popen(prefix + cmd, shell=True)
...such that you're using the results of sourcing the script in the same shell invocation as that in which you actually source the script.
Alternately (and this is what I'd do), require your Python script to be invoked by a shell which already has the necessary variables in its environment. Thus:
# ask your users to do this
set -a; . ./vars; ./yourPythonScript
...and you can error out if people don't do so very easy:
import os, sys
if not 'EASY_RSA' in os.environ:
print >>sys.stderr, "ERROR: Source vars before running this script"
sys.exit(1)

pass python var to bash

I'm making a script to take pictures and write them to a folder created/named with the "data&time"
I made this part to create the directory and take the pictures
pathtoscript = "/home/pi/python-scripts"
current_time = time.localtime()[0:6]
dirfmt = "%4d-%02d-%02d-%02d-%02d-%02d"
dirpath = os.path.join(pathtoscript , dirfmt)
dirname = dirpath % current_time[0:6] #dirname created with date and time
os.mkdir(dirname) #mkdir
pictureName = dirname + "/image%02d.jpg" #path+name of pictures
camera.capture_sequence([pictureName % i for i in range(9)])
Then I would like to pass the dirname to a bash script (picturesToServer) which uploads the pictures to a server.
How can I do it?
cmd = '/home/pi/python-scripts/picturesToServer >/dev/null 2>&1 &'
call ([cmd], shell=True)
Maybe I could stay in the python script scp the pictures to the server? I have a ssh-agent with the paraphrase set (ssh-add mykey).
Place the variable in the environment (it'll be available as a regular bash variable in the bash script, e.g. as VAR_NAME in the example below) by replacing your call with:
import subprocess
p = subprocess.Popen(cmd, shell=True, env={"VAR_NAME": dirname})
Or pass it as a positional argument (it'll be available in $1 in the script) by replacing your cmd with:
cmd = '/home/pi/python-scripts/picturesToServer >/dev/null 2>&1 "{0}" &'.format(dirname)
As a side note, consider not using shell = True when you call a subprocess. Using shell = True is a bad idea for a lot of reasons that are documented in the Python docs

calling perl script that takes multiple input args in python

Need help with integrating perl script with main python script.
I have a perl script by name: GetHostByVmname.pl
./GetHostByVmname.pl –server 10.0.1.191 –username Administrator –password P#ssword1 –vmname RHTest
I need to call above script from my python main script. Tried below, but doesn’t work:
param = "--server 10.0.1.191 --username Administrator --password P#ssword1 --vmname RHTest"
pipe = subprocess.Popen(["perl", "./GetHostByVmname.pl", param ], stdout=subprocess.PIPE)
You can either provide a shell command
Popen("./GetHostByVmname.pl –server 10.0.1.191 ...", ...)
Or an array where the the first element is the program and the rest are args.
Popen(["./GetHostByVmname.pl", "–server", "10.0.1.191", ... ], ...)
Currently, you are doing the equivalent of the following shell command:
perl ./GetHostByVmname.pl '–server 10.0.1.191 ...'
I think it will be better when you split string
./GetHostByVmname.pl –server 10.0.1.191 –username Administrator –password P#ssword1 –vmname RHTest
to a list, and after call Popen with this list as a first param.
Example:
import shlex, subprocess
args_str = "./GetHostByVmname.pl –server 10.0.1.191 –username Administrator –password P#ssword1 –vmname RHTest"
args = shlex.split(args_str)
p = subprocess.Popen(args, stdout=subprocess.PIPE)

Execute .R script within Python using Rscript.exe shell

I have an .R file saved locally at the following path:
Rfilepath = "C:\\python\\buyback_parse_guide.r"
The command for RScript.exe is:
RScriptCmd = "C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe --vanilla"
I tried running:
subprocess.call([RScriptCmd,Rfilepath],shell=True)
But it returns 1 -- and the .R script did not run successfully. What am I doing wrong? I'm new to Python so this is probably a simple syntax error... I also tried these, but they all return 1:
subprocess.call('"C:\Program Files\R\R-2.15.2\bin\Rscript.exe"',shell=True)
subprocess.call('"C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe"',shell=True)
subprocess.call('C:\Program Files\R\R-2.15.2\bin\Rscript.exe',shell=True)
subprocess.call('C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe',shell=True)
Thanks!
The RScriptCmd needs to be just the executable, no command line arguments. So:
RScriptCmd = "\"C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe\""
Then the Rfilepath can actually be all of the arguments - and renamed:
RArguments = "--vanilla \"C:\\python\\buyback_parse_guide.r\""
It looks like you have a similar problem to mine. I had to reinstall RScript to a path which has no spaces.
See: Running Rscript via Python using os.system() or subprocess()
This is how I worked out the communication between Python and Rscript:
part in Python:
from subprocess import PIPE,Popen,call
p = subprocess.Popen([ path/to/RScript.exe, path/to/Script.R, Arg1], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
out = p.communicate()
outValue = out[0]
outValue contains the output-Value after executing the Script.R
part in the R-Script:
args <- commandArgs(TRUE)
argument1 <- as.character(args[1])
...
write(output, stdout())
output is the variable to send to Python

Categories