I'm having a problem with subprocess and printing quotes.
My Python script takes user input, mashes it around a bit - and I need it to send it's results to a bash script in this manner.
myscript.sh 'var1 == a var2 == b; othervar == c' /path/to/other/files
Where I'm getting hung up on is the single quotes. Python tries to rip them out.
I used this for my test.
subprocess.Popen([myscript.sh 'var=11; ignore all' /path/to/files], shell=True, executable="/bin/bash")
which returns an invalid syntax pointing at the 2nd single quote. I've also tried the above without the brackets and using single quotes outside and double quotes inside, etc.
Other - would-like.
As I was saying above the 'var == a var == b; othervar == c' is derived from the python script (in string format) - and I'll need to call that in the subprocess like this.
subprocess.Popen([myscript.sh myvariables /path/to/files], shell=True, executable="/bin/bash")
I just have to put the single quotes around the value of myvariables like the first example.
Any pointers as to where I'm going off the correct method?
Thank you.
When shell=True is passed to Popen, you pass whatever you would send on the command line. That means your list should only have one element. So for example:
subprocess.Popen(['myscript.sh "var=11; ignore all" /path/to/files'], shell=True, executable="/bin/bash")
Or if /path/to/files is a variable in your Python environment:
subprocess.Popen(['myscript.sh "var=11; ignore all" %s' % path_to_files], shell=True, executable="/bin/bash")
Having said that I STRONGLY encourage you not to use the shell argument. The reason is fragility. You'll get a much more robust way of doing it like this:
subprocess.Popen(["/bin/bash", "myscript.sh", "var=11; ignore all", path_to_files])
Note that "var=11; ignore all" is passed as one argument to your script. If those are separate arguments, make them separate list elements.
I haven't checked why this works, but it does and without the need for shell=True.
subprocess.Popen(["/bin/bash", "myscript.sh", '""' + string_to_be_quoted + '""', path_to_files])
That's a list, those are strings in it, so they need quotes:
["myscript.sh", "var=11; ignore all", "/path/to/files"]
That should work. If your script really somehow relies on quotes, then try this (I don't know the details of how subprocess works):
["myscript.sh", "'var=11; ignore all'", "/path/to/files"]
Related
i have built a small script that runs a simple shell utility called imapsync, with a bunch of variables taken from a dictionary, the command is as follows:
os.system("imapsync --host1 %s --user1 %s --password1 '%s' --host2 %s --user2 %s --password2 '%s' --ssl1 --no-modulesversion --ssl2" % (fromHost, emails, passwords, toHost, emails, passwords))
the deal is that passwords often contain special characters, example: djDJS*^%%%^&)
this imapsync tool allows such characters if enclosed in single quotes: 'djDJS*^%%%^&)'
what I am trying to achieve is post the single quotes in the command, itself.. I tried "'", backquotes - ``, escaped quotes - \'\', enclosing the command in single quotes, nothing worked thus far
After looking through the documentation of imapsync, I found the recommendation to enclose passwords in double quotes within single quotes to avoid common problems.
Since you already start the string with double quotes, you have to escape the double quotes around your password with a backslash \".
There are also two things you could do to make your code even better.
First, you can use .format syntax for string formatting instead of the old % syntax.
Second replace os.system with subprocess.Popen. This allows you to split your command string into a list of all arguments, which looks more clear.
Your new code would look like
import subprocess
args = [
"imapsync",
"--host1",
fromHost,
"--user1",
emails,
"--password1",
"'\"{}\"'".format(passwords),
"--host2",
toHost,
"--user2",
emails,
"--password2",
"'\"{}\"'".format(passwords),
"--ssl1",
"--no-modulesversion",
"--ssl2"
]
p = subprocess.Popen(args, stdout=subprocess.PIPE)
output = p.communicate()[0]
print(output)
In this example Popen.communicate is used to gather the output of the imapsync command as a string.
The communicate method returns a tuple with the outputs of the subprocess to stdout and stderr streams.
If you also want to read the output to stderr from the subprocess, change the code as following:
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, errors = p.communicate()
print(output)
print(errors)
The best format for passing string parameters in Python is to use the format string method. You could do something like this:
line_command = "imapsync --host1 {fromHost} --user1 {emails} --password1 '\"{passwords}\"' --host2 {toHost} --user2 {emails} --password2 '\"{passwords}\"' --ssl1 --no-modulesversion --ssl2".format(fromHost=fromHost, emails=emails, passwords=passwords, toHost=toHost)
os.system(line_command)
I would like to run ssh with print of python.
The followings are my test code.
import subprocess
# case1:
command_str = "\"print(\'test\')\""
# case 2:
# command_str = "\\\"print(\'test\')\\\""
ssh_command = ['ssh', 'USER_X#localhost', 'python', '-c']
ssh_command.append(command_str)
process = subprocess.run(ssh_command, stdout=subprocess.PIPE)
print(process.stdout)
case 1 and case 2 did not work.
The outputs are followings,
case 1:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c print('test')'
b''
case 2:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c \"print('test')\"'
b''
Please let me know how it works.
It should work with
command_str = "'print(\"test\")'"
or equivalently
command_str = '\'print("test")\''
Explanation
The outermost quotes and the escaping are for the local Python. So in either case, the local Python string will be 'print("test")'.
There is no quoting or escaping required for the local shell, as subcommand.run(...) won't invoke it unless shell=True is passed.
Thus the single quotes within the python string are for the remote shell (presumably bash or other sh-compatible shell). The argument passed to the remote Python is thus print("test"). (And the double quotes in there are to signify the string literal to print to the remote python.)
Can we do without escaping (without \)?
As there are three levels involved (local Python, remote shell, remote Python), I don't think so.
Can we do with a single type of quotes?
Yes, with a bit more escaping. Let's build this from behind (or inside-out).
We want to print
test
This needs to be escaped for the remote Python (to form a string literal instead of an identifier):
"test"
Call this with the print() function:
print("test")
Quite familiar so far.
Now we want to pass this as an argument to python -c on a sh-like shell. To protect the ( and ) to be interpreted by that, we quote the whole thing. For the already present " not to terminate the quotation, we escape them:
"print(\"test\")"
You can try this in a terminal:
$> echo "print(\"test\")"
print("test")
Perfect!
Now we have to represent the whole thing in (the local) Python. We wrap another layer of quotes around it, have to escape the four(!) existing quotation marks as well as the two backslashes:
"\"print(\\\"test\\\")\""
(Done. This can also be used as command_str.)
Can we do with only single quotes (') and escaping?
I don't know, but at least not as easily. Why? Because, other than to Python, double and single quotes aren't interchangeable to sh and bash: Within single quotes, these shells assume a raw string without escaping until the closing ' occurs.
My brain hurts!
If literally, go see a doctor. If figuratively, yeah, mine too. And your code's future readers (including yourself) will probably feel the same, when they try to untangle that quoting-escaping-forest.
But there's a painless alternative in our beloved Python standard library!
import shlex
command_str = shlex.quote('print("test")')
This is much easier to understand. The inner quotes (double quotes here, but doesn't really matter: shlex.quote("print('test')") works just as fine) are for the remote Python. The outer quotes are obviously for the local Python. And all the quoting and escaping beyond that for the remote shell is taken care of by this utility function.
The correct syntax for python 2 and 3 is:
python -c 'print("test")'
I'm trying to send a variable from a python script to a bash script. I'm using popen like as shown below:
subprocess.Popen(["bash", "-c", ". mainGui_functions.sh %d %s" % (commandNum.get(), entryVal)])
However, entryVal can sometimes contain one or more white space characters. In that case I divide the string into multiple arguments ($2,$3..)
How can i get it in one argument?
Simple solution #1: You do it the exact same way you'd do it if you were typing the input on the commandline; put it in quotes:
subprocess.Popen(["bash", "-c", ". mainGui_functions.sh {} '{}'".format(commandNum.get(), entryVal)])
Simple solution #2: If mainGui_functions.sh is already executable, then you can just omit the bash part and pas args to it directly. In this case, subprocess takes care of making sure an entry with whitespace winds up as a single arg for you:
subprocess.Popen(["mainGui_functions.sh", str(commandNum.get()), entryVal])
Put quotes around it in your bash command line – e.g.,
subprocess.Popen(['bash', '-c', '. mainGui_functions.sh %d "%s"' % (commandNum.get(), entryVal)])
I am attempting to create a Python script that in turn runs the shell script "js2coffee" to convert some javascript into coffeescript.
From the command line I can run this, and get coffeescript back again...
echo "var myNumber = 100;" | js2coffee
What I need to do is use this same pattern from Python.
In Python, I've come to something like this:
command = "echo '" + myJavscript + "' | js2coffee"
result = os.popen(command).read()
This works sometimes, but there are issues related to special characters (mostly quotes, I think) not being properly escaped in the myJavascript. There has got to be a standard way of doing this. Any ideas? Thanks!
Use the input stream of a process to feed it the data, that way you can avoid the shell and you don't need to escape your javascript. Additionally, you're not vulnerable to shell injection attacks;
pr = subprocess.Popen(['js2coffee'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
result, stderrdata = pr.communicate('var myNumber = 100;')
subprocess module is the way to go:
http://docs.python.org/library/subprocess.html#frequently-used-arguments
be kindly noted the following:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names)
Platform: Windows
Grep: http://gnuwin32.sourceforge.net/packages/grep.htm
Python: 2.7.2
Windows command prompt used to execute the commands.
I am searching for the for the following pattern "2345$" in a file.
Contents of the file are as follows:
abcd 2345
2345
abcd 2345$
grep "2345$" file.txt
grep returns 2 lines (first and second) successfully.
When I try to run the above command through python I don't see any output.
Python code snippet is as follows:
temp = open('file.txt', "r+")
grep_cmd = []
grep_cmd.extend([grep, '"2345$"' ,temp.name])
print grep_cmd
p = subprocess.Popen(grep_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdoutdata = p.communicate()[0]
print stdoutdata
If I have
grep_cmd.extend([grep, '2345$' ,temp.name])
in my python script, I get the correct answer.
The questions is why the grep command with "
grep_cmd.extend([grep, '"2345$"' ,temp.name])
executed from python fails. Isn't python supposed to execute
the command as it is.
Thanks
Gudge.
Do not put double quotes around your pattern. It is only needed on the command line to quote shell metacharacters. When calling a program from python, you do not need this.
You also do not need to open the file yourself - grep will do that:
grep_cmd.extend([grep, '2345$', 'file.txt'])
To understand the reason for the double quotes not being needed and causing your command to fail, you need to understand the purpose of the double quotes and how they are processed.
The shell uses double quotes to prevent special processing of some shell metacharacters. Shell metacharacters are those characters that the shell handles specially and does not pass literally to the programs it executes. The most commonly used shell metacharacter is "space". The shell splits a command on space boundaries to build an argument vector to execute a program with. If you want to include a space in an argument, it must be quoted in some way (single or double quotes, backslash, etc). Another is the dollar sign ($), which is used to signify variable expansion.
When you are executing a program without the shell involved, all these rules about quoting and shell metacharacters are not relevant. In python, you are building the argument vector yourself, so the relevant quoting rules are python quoting rules (e.g. to include a double quote inside a double-quoted string, prefix the double quote with a backslash - the backslash will not be in the final string). The characters in each element of the argument vector when you have completed constructing it are the literal characters that will be passed to the program you are executing.
Grep does not treat double quotes as special characters, so if grep gets double quotes in its search pattern, it will attempt to match double quotes from its input.
My original answer's reference to shell=True was incorrect - first I did not notice that you had originally specified shell=True, and secondly I was coming from the perspective of a Unix/Linux implementation, not Windows.
The python subprocess module page has this to say about shell=True and Windows:
On Windows: the Popen class uses CreateProcess() to execute the child child program, which operates on strings. If args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows.
That linked section on converting an argument sequence to a string on Windows does not make sense to me. First, a string is a sequence, and so is a list, yet the Frequently Used Arguments section says this about arguments:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).
This contradicts the conversion process described in the Python documentation, and given the behaviour you have observed, I'd say the documentation is wrong, and only applied to a argument string, not an argument vector. I cannot verify this myself as I do not have Windows or the source code for Python lying around.
I suspect that if you call subprocess.Popen like:
p = subprocess.Popen(grep + ' "2345$" file.txt', stdout=..., shell_True)
you may find that the double quotes are stripped out as part of the documented argument conversion.
You can use python-textops3 :
from textops import *
print('\n'.join(cat('file.txt') | grep('2345$')))
with python-textops3 you can use unix-like commands with pipes within python
so no need to fork a process which is very heavy