I need a python script to call a bash script on windows.
So basically I must make a subprocess call form python, that will call cygwin with the -c option that will call the script I need,
The problem is that this script takes a few arguments and that these arguments are full os spaces and quotes and slashes.
I'm using code like the following
arq_saida_unix = arq_saida.replace("\\","/")
subprocess.call("C:\\cygwin64\\bin\\bash \".\\retirarVirgula.sh\\ \""+arq_saida+"\"")
Or I'm directly escaping, which sometimes takes me to as much as 8 backslashes in a row, for a backslash to get to my script must be escaped i) in bash ii) in cmd.exe iii) in python
all of this is error prone and takes quite some time every time to get it right.
Is there a better way of doing it? Ideally I wouldn't have any escaping backslashes, but anything that avoids the triple-slash double quote above would be nice.
I tried to use re.escape, but could figure out how exactly to use it , except as a replacement to .replace("\","/") and similar.
Don't pass a single string to call; instead, pass a list consisting of the command name and one argument per element. This saves you from needing to protect special characters from shell interpretation.
subprocess.call(["retirarVirgula.sh", arq_saida], executable=r"C:\cygwin64\bin\bash")
Note: I'm assuming arq_saida contains the single argument to pass to the script; if the script takes multiple arguments, then arc_saida should probably be built as a list as well:
arq_saida = ["arg", "arg two", "arg three"]
subprocess.call(["retirarVirgula.sh"] + arq_saida, executable=r"C:\cygwin64\bin\bash")
Related
I am passing as parameter a string; to a Python3 function.
This string is composed by different variables; and in few cases, I need to have the quotes in the string, so I did concatenate various strings to make it work.
This is the string I use in my code, to call an application called conn.app, which use a file called lac.tsd to perform various operations. The whole code logic is not relevant for the purpose of this question; since the root cause of the errors is the fact that path strings and quoted strings behave differently between OS (like Windows and Unix-like systems for example)
execute_string = '-b -m path="/Users/user/lac.tsd" -a app="/build/deploy/conn.app" -o output=/Users/user/out/'
This works without problems on OSX and Linux; but when I run this on Windows machines, I get an error
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '"'
After digging a bit in the differences between OS, it is clear to see that my path for the tsd file is pointing at /Users/.. which is a pure unix construct for the file system; so that has been replaced when I run the code on Windows.
Although, there is still the problem with conn.app, which on windows is conn.exe; the location is the same, so I did fix that too.
Last but not least, 2 issues tied to the OS itself: the / is \ in Windows, so that has to change; and same goes for quotes and double quotes.
Does Python have any construct that help programmers to handle cases like mine, where you build a string of parameters to pass to a function, and have in it single and double quotes, paths and such?
To expand on the original point: how do you handle this between OS? Beside a simple RE to replace a character with another; is there any construct used in Python that allow code to operate independently from the OS used, when dealing with paths, quoted strings and multiple nested quoted strings?
I think the problem may well be that the string is being passed directly to the command interpreter. In UNIX this will work the way you expect because it treats quotes as special characters, but on Windows, depending on the shell, it probably will behave in very different ways. One possible solution is to break the command you want to run into separate argument strings:
execute_args = [
"-b", "-m",
"path=/Users/user/lac.tsd",
"-a",
"app=/build/deploy/conn.app",
"-o",
"output=/Users/user/out/"
]
And then use one of the many functions in subprocess to execute the command directly instead of using the shell (i.e. shell=False, which is the default for most of them).
I have a python script that is calling a bat script called testrunner.bat which in turns executes a TestSuite in SOAPUI. I actually have gotten the external call to work just fine with the following command:
Popen("testrunner.bat -s\"CCT000 - Deploy Software Release\" -R\"TestSuite Report\" -E\"Default environment\" -Ppath.packages.sq=Y:\\NIGHTLY C:\\CI\\HEDeployment\\CI-XXX-DeploySwRelease")
However, I need to be able to have the software "level" to be dynamic and need to pass the variable level into the command in place of "NIGHTLY" so I can specify if it's nightly software, or stable, etc. I have seen that I should break all the arguments up separately, but I am having a hard time.
subprocess.Popen() can take a list of arguments as well as a string. So, this should work for you:
release_type = "NIGHTLY"
Popen(['testrunner.bat',
'-s"CCT000 - Deploy Software Release"',
'-R"TestSuite Report"',
'-E"Default environment"',
'-Ppath.packages.sq=Y:' + release_type,
'C:CIHEDeploymentCI-XXX-DeploySwRelease'])
As mentioned in the docs, shlex.split can be very useful for splitting your original command string into pieces. However, at least in my case, I had to re-add the double quotes.
Also, recall that single-quoted strings can contain double quotes, and vice versa, so you don't need to escape the quotes here.
I'm writing a python plugin for vim and it's looking like the only way to call a specific command is with the vim.command function. However, just substituting values into the function seems like a bad idea. How would I escape values so that I can pass untrusted data as an argument into a vim function? As a simple example, let's say I want to echo out untrusted input (I know I could just use print, but this is just an example). I would do something like:
value = get_data_from_untrusted_source()
vim.command("echo %s" % value)
However, if that untrusted data has a | in it, the command is ended and a new one is executed which is bad. Even if I use quotes, we end up with sql injection like attacks where an attacker can just put an apostrophe in their response to end the string. Then if we double quote, it could be possible to put a backslash somewhere to end the quote. For example if we just double quotes we would go from \' to \'' which escapes the first quote.
Basically what I'm asking is if there's a safe way to call vim functions from a python plugin and would appreciate any help.
I have a bash script that calls a python script. At first I was just returning one variable and that is fine, but now I was told to return two variables and I was wondering if there is a clean and simple way to return more than one variable.
archiveID=$(python glacier_upload.py $archive_file_name $CURRENTVAULT)
Is the call I make from bash
print archive_id['ArchiveId']
archive_id['ArchiveId']
This returns the archive id to the bash script
Normally I know you can use a return statement in python to return multiple variables, but with it just being a script that is the way I found to return a variable. I could make it a function that gets called but even then, how would I receive the multiple variables that I would be passing back?
From your python script, output one variable per line. Then from you bash script, read one variable per line:
Python
print "foo bar"
print 5
Bash
#! /bin/bash
python main.py | while read line ; do
echo $line
done
Final Solution:
Thanks Guillaume! You gave me a great starting point out the soultion. I am just going to post my solution here for others.
#! /bin/bash
array=()
while read line ; do
array+=($line)
done < <(python main.py)
echo ${array[#]}
I found the rest of the solution that I needed here
The safest and cleanest way to parse any input effectively in bash is to map into an array,
mapfile -t -d, <<<"$(python your_script.py)"
Now you just need to make sure you script outputs the data you want to read with the chosen delimiter, "," in my example (-d selects a delimiter, -t trims input like newlines). The quotes are non-optional to ensure the shell doesn't separate things with spaces.
If you have a tuple of things that do not contain commas, this would be enough:
print(str(your_tuple).strip('()'))
Below some easy ways for easy input, before I was more familiar with Bash:
My favorite way is reading straight into a list:
x=($(python3 -c "print('a','b','c')"))
echo ${x[1]}
b
echo ${x[*]}
a b c
for this reason if my_python_function returns a tuple, I would use format to make sure I just get space delimited results:
#Assuming a tuple of length 3 is returned
#Remember to quote in case of a space in a single parameter!
print('"{}" "{}" "{}"'.format(*my_python_function())
If you want this to be generic you would need to construct the format string:
res = my_python_function()
print(("{} "*len(res)).format(*res))
is one way. No need to worry about the extra space, but you could [:-1] on the format string to get rid of it.
Finally, if you are expecting multi-word arguments (i.e. a space in a single argument, you need to add quotes, and a level of indirection (I am assuming you will only be running your own, "safe", scripts):
#myfile.py
res = my_python_function()
print(('"{}" '*len(res)).format(*res))
#my file.bash
eval x=($(python3 myfile.py))
Platform: Windows
Grep: http://gnuwin32.sourceforge.net/packages/grep.htm
Python: 2.7.2
Windows command prompt used to execute the commands.
I am searching for the for the following pattern "2345$" in a file.
Contents of the file are as follows:
abcd 2345
2345
abcd 2345$
grep "2345$" file.txt
grep returns 2 lines (first and second) successfully.
When I try to run the above command through python I don't see any output.
Python code snippet is as follows:
temp = open('file.txt', "r+")
grep_cmd = []
grep_cmd.extend([grep, '"2345$"' ,temp.name])
print grep_cmd
p = subprocess.Popen(grep_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdoutdata = p.communicate()[0]
print stdoutdata
If I have
grep_cmd.extend([grep, '2345$' ,temp.name])
in my python script, I get the correct answer.
The questions is why the grep command with "
grep_cmd.extend([grep, '"2345$"' ,temp.name])
executed from python fails. Isn't python supposed to execute
the command as it is.
Thanks
Gudge.
Do not put double quotes around your pattern. It is only needed on the command line to quote shell metacharacters. When calling a program from python, you do not need this.
You also do not need to open the file yourself - grep will do that:
grep_cmd.extend([grep, '2345$', 'file.txt'])
To understand the reason for the double quotes not being needed and causing your command to fail, you need to understand the purpose of the double quotes and how they are processed.
The shell uses double quotes to prevent special processing of some shell metacharacters. Shell metacharacters are those characters that the shell handles specially and does not pass literally to the programs it executes. The most commonly used shell metacharacter is "space". The shell splits a command on space boundaries to build an argument vector to execute a program with. If you want to include a space in an argument, it must be quoted in some way (single or double quotes, backslash, etc). Another is the dollar sign ($), which is used to signify variable expansion.
When you are executing a program without the shell involved, all these rules about quoting and shell metacharacters are not relevant. In python, you are building the argument vector yourself, so the relevant quoting rules are python quoting rules (e.g. to include a double quote inside a double-quoted string, prefix the double quote with a backslash - the backslash will not be in the final string). The characters in each element of the argument vector when you have completed constructing it are the literal characters that will be passed to the program you are executing.
Grep does not treat double quotes as special characters, so if grep gets double quotes in its search pattern, it will attempt to match double quotes from its input.
My original answer's reference to shell=True was incorrect - first I did not notice that you had originally specified shell=True, and secondly I was coming from the perspective of a Unix/Linux implementation, not Windows.
The python subprocess module page has this to say about shell=True and Windows:
On Windows: the Popen class uses CreateProcess() to execute the child child program, which operates on strings. If args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows.
That linked section on converting an argument sequence to a string on Windows does not make sense to me. First, a string is a sequence, and so is a list, yet the Frequently Used Arguments section says this about arguments:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).
This contradicts the conversion process described in the Python documentation, and given the behaviour you have observed, I'd say the documentation is wrong, and only applied to a argument string, not an argument vector. I cannot verify this myself as I do not have Windows or the source code for Python lying around.
I suspect that if you call subprocess.Popen like:
p = subprocess.Popen(grep + ' "2345$" file.txt', stdout=..., shell_True)
you may find that the double quotes are stripped out as part of the documented argument conversion.
You can use python-textops3 :
from textops import *
print('\n'.join(cat('file.txt') | grep('2345$')))
with python-textops3 you can use unix-like commands with pipes within python
so no need to fork a process which is very heavy