I know about escaping backslashes in programming. However I'm passing in a command line argument into a Python program to parse. Whitespace is irrelevant, however via the command line, whitespace does matter. So I am trying to concatenate all argv[] arguments except the first element into a string to then parse. However a token of a single '\' is never encountered in the for loop.
The command line argument (notice whitespace around '\'):
((1 * 2) + (3 - (4 \ 5)))
Program:
import sys
string = ""
for token in sys.argv[1:len(sys.argv)]:
print(token)
if "\\" in r"%r" % token:
print("Token with \\ found")
string += token
print(string)
print("done")
This is not a Python problem. The shell will interpret the text you write at the command line and (for typical Unix shells) perform backslash substitution. So by the time Python examines the command line, the backslash will already have been substituted away.
The workaround is to double the backslash or put it in single quotes, if your shell is Bash or some other Bourne-compatible Unix shell. Windows CMD is different and bewildering in its own right; Csh doubly so.
Related
I need to send the following commands to a busybox device via a serial port:
SBC1000 > setenv serverip '192.168.128.100'
SBC1000 > setenv fsfile '1k\root.jffs2-128k'
SBC1000 > saveenv
I can escape the single quotes of the first line without a problem using a backslash:
cmd = 'setenv serverip \'192.168.128.100\''
I've tried various combinations of backslashes for the second line, but couldn't get the 1k\root part to escape properly. I believe it is being interpreted as a return. I tried double and triple escape with no success.
I finally stumbled upon using
cmd = 'setenv fsfile \'1k\\\u0072oot.jffs2-128k\''
to include the \r ( not a return ) for my string.
Is there a more readable way to include this \r ( not a return ) pattern in my string?
The solution was to use double-quotes " " as suggested by John Szakmeister.
I discovered that the command string was being passed to a function inside a private class based on pexpect-serial.
My guess is that my string was being evaluated by pexpect in a greedy way. By using a distinct delimiter, the problem was overcome.
I would like to run ssh with print of python.
The followings are my test code.
import subprocess
# case1:
command_str = "\"print(\'test\')\""
# case 2:
# command_str = "\\\"print(\'test\')\\\""
ssh_command = ['ssh', 'USER_X#localhost', 'python', '-c']
ssh_command.append(command_str)
process = subprocess.run(ssh_command, stdout=subprocess.PIPE)
print(process.stdout)
case 1 and case 2 did not work.
The outputs are followings,
case 1:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c print('test')'
b''
case 2:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c \"print('test')\"'
b''
Please let me know how it works.
It should work with
command_str = "'print(\"test\")'"
or equivalently
command_str = '\'print("test")\''
Explanation
The outermost quotes and the escaping are for the local Python. So in either case, the local Python string will be 'print("test")'.
There is no quoting or escaping required for the local shell, as subcommand.run(...) won't invoke it unless shell=True is passed.
Thus the single quotes within the python string are for the remote shell (presumably bash or other sh-compatible shell). The argument passed to the remote Python is thus print("test"). (And the double quotes in there are to signify the string literal to print to the remote python.)
Can we do without escaping (without \)?
As there are three levels involved (local Python, remote shell, remote Python), I don't think so.
Can we do with a single type of quotes?
Yes, with a bit more escaping. Let's build this from behind (or inside-out).
We want to print
test
This needs to be escaped for the remote Python (to form a string literal instead of an identifier):
"test"
Call this with the print() function:
print("test")
Quite familiar so far.
Now we want to pass this as an argument to python -c on a sh-like shell. To protect the ( and ) to be interpreted by that, we quote the whole thing. For the already present " not to terminate the quotation, we escape them:
"print(\"test\")"
You can try this in a terminal:
$> echo "print(\"test\")"
print("test")
Perfect!
Now we have to represent the whole thing in (the local) Python. We wrap another layer of quotes around it, have to escape the four(!) existing quotation marks as well as the two backslashes:
"\"print(\\\"test\\\")\""
(Done. This can also be used as command_str.)
Can we do with only single quotes (') and escaping?
I don't know, but at least not as easily. Why? Because, other than to Python, double and single quotes aren't interchangeable to sh and bash: Within single quotes, these shells assume a raw string without escaping until the closing ' occurs.
My brain hurts!
If literally, go see a doctor. If figuratively, yeah, mine too. And your code's future readers (including yourself) will probably feel the same, when they try to untangle that quoting-escaping-forest.
But there's a painless alternative in our beloved Python standard library!
import shlex
command_str = shlex.quote('print("test")')
This is much easier to understand. The inner quotes (double quotes here, but doesn't really matter: shlex.quote("print('test')") works just as fine) are for the remote Python. The outer quotes are obviously for the local Python. And all the quoting and escaping beyond that for the remote shell is taken care of by this utility function.
The correct syntax for python 2 and 3 is:
python -c 'print("test")'
I'm trying to run a .exe that requires some parameters by using system().
If there's a space in the .exe's path AND in the path of a file passed in parameters, I get the following error:
The filename, directory name, or volume label syntax is incorrect.
Here is the code that generates that error:
#include <stdlib.h>
#include <conio.h>
int main (){
system("\"C:\\Users\\Adam\\Desktop\\pdftotext\" -layout \"C:\\Users\\Adam\\Desktop\\week 4.pdf\"");
_getch();
}
If the "pdftotext"'s path doesn't use quotation marks (I need them because sometimes the directory will have spaces), everything works fine. Also, if I put what's in "system()" in a string and output it and I copy it in an actual command window, it works.
I thought that maybe I could chain some commands using something like this:
cd C:\Users\Adam\Desktop;
pdftotext -layout "week 4.pdf"
So I would already be in the correct directory, but I don't know how to use multiple commands in the same system() function.
Can anyone tell me why my command doesn't work or if the second way I thought about would work?
Edit: Looks like I needed an extra set of quotation marks because system() passes its arguments to cmd /k, so it needs to be in quotations. I found it here:
C++: How to make a my program open a .exe with optional args
so I'll vote to close as duplicate since the questions are pretty close even though we weren't getting the same error message, thanks!
system() runs command as cmd /C command. And here's citation from cmd doc:
If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:
1. If all of the following conditions are met, then quote characters
on the command line are preserved:
- no /S switch
- exactly two quote characters
- no special characters between the two quote characters,
where special is one of: &<>()#^|
- there are one or more whitespace characters between the
two quote characters
- the string between the two quote characters is the name
of an executable file.
2. Otherwise, old behavior is to see if the first character is
a quote character and if so, strip the leading character and
remove the last quote character on the command line, preserving
any text after the last quote character.
It seems that you are hitting case 2, and cmd thinks that the whole string C:\Users\Adam\Desktop\pdftotext" -layout "C:\Users\Adam\Desktop\week 4.pdf (i.e. without the first and the last quote) is the name of executable.
So the solution would be to wrap the whole command in extra quotes:
//system("\"D:\\test\" nospaces \"text with spaces\"");//gives same error as you're getting
system("\"\"D:\\test\" nospaces \"text with spaces\"\""); //ok, works
And this is very weird. I think it's also a good idea to add /S just to make sure it will always parse the string by the case 2:
system("cmd /S /C \"\"D:\\test\" nospaces \"text with spaces\"\""); //also works
I got here looking for an answer, and this is the code that I came up with (and I was this explicit for the benefit of next person maintaining my code):
std::stringstream ss;
std::string pathOfCommand;
std::string pathOfInputFile;
// some code to set values for paths
ss << "\""; // command opening quote
ss << "\"" << pathOfCommand << "\" "; // Quoted binary (could have spaces)
ss << "\"" << pathOfInputFile << "\""; // Quoted input (could have spaces)
ss << "\""; // command closing quote
system( ss.str().c_str() ); // Execute the command
and it solved all of my problems.
Good learning from here on the internals of System call.Same issue reproducible(of course) with C++ string, TCHARs etc.
One approach that always helped me is SetCurrentDirectory() call. I first set current path and then execute. This has worked for me so far. Any comments welcome.
-Sreejith. D. Menon
I am writing a sublime plugin in Python. But I am still very new to Python language. I am using this line to call meld in Python:
if meld_cmd == None:
meld_cmd = "C:\\Program Files (x86)\\Meld\\meld\\meld"
os.system('"%s" "%s" "%s" ' %(meld_cmd, source_name, dest_name))
It doesn't work well in Windows. The cmd window just flashes for a sec (seem to execute something) then disappears. I print the executed string out in sublime and copy the string into cmd window and execute. It executes well in both admin and non-admin mode. And I tried to add a pause before executing:
os.system('pause && "%s" "%s" "%s" ' %(meld_cmd, source_name, dest_name))
This works well too after clicking enter after the press any key to continue... line.
Not sure how to debug this and what is the reason why it is failing.
cmd.exe /c command has a parsing quirk when the command starts with a quote and has more than two quotes. The exact rules are explained in the online help text available via cmd /?:
If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:
1. If all of the following conditions are met, then quote characters
on the command line are preserved:
- no /S switch
- exactly two quote characters
- no special characters between the two quote characters,
where special is one of: &<>()#^|
- there are one or more whitespace characters between the
two quote characters
- the string between the two quote characters is the name
of an executable file.
2. Otherwise, old behavior is to see if the first character is
a quote character and if so, strip the leading character and
remove the last quote character on the command line, preserving
any text after the last quote character.
To get around this, just wrap the whole command in quotes, e.g. '""%s" "%s" "%s""' % (meld_cmd, source_name, dest_name).
Probably the problem is that you need to execute that script with Admin privileges.
Proceed to question How to run python script with elevated privilege on windows to get more information on that topic.
Platform: Windows
Grep: http://gnuwin32.sourceforge.net/packages/grep.htm
Python: 2.7.2
Windows command prompt used to execute the commands.
I am searching for the for the following pattern "2345$" in a file.
Contents of the file are as follows:
abcd 2345
2345
abcd 2345$
grep "2345$" file.txt
grep returns 2 lines (first and second) successfully.
When I try to run the above command through python I don't see any output.
Python code snippet is as follows:
temp = open('file.txt', "r+")
grep_cmd = []
grep_cmd.extend([grep, '"2345$"' ,temp.name])
print grep_cmd
p = subprocess.Popen(grep_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdoutdata = p.communicate()[0]
print stdoutdata
If I have
grep_cmd.extend([grep, '2345$' ,temp.name])
in my python script, I get the correct answer.
The questions is why the grep command with "
grep_cmd.extend([grep, '"2345$"' ,temp.name])
executed from python fails. Isn't python supposed to execute
the command as it is.
Thanks
Gudge.
Do not put double quotes around your pattern. It is only needed on the command line to quote shell metacharacters. When calling a program from python, you do not need this.
You also do not need to open the file yourself - grep will do that:
grep_cmd.extend([grep, '2345$', 'file.txt'])
To understand the reason for the double quotes not being needed and causing your command to fail, you need to understand the purpose of the double quotes and how they are processed.
The shell uses double quotes to prevent special processing of some shell metacharacters. Shell metacharacters are those characters that the shell handles specially and does not pass literally to the programs it executes. The most commonly used shell metacharacter is "space". The shell splits a command on space boundaries to build an argument vector to execute a program with. If you want to include a space in an argument, it must be quoted in some way (single or double quotes, backslash, etc). Another is the dollar sign ($), which is used to signify variable expansion.
When you are executing a program without the shell involved, all these rules about quoting and shell metacharacters are not relevant. In python, you are building the argument vector yourself, so the relevant quoting rules are python quoting rules (e.g. to include a double quote inside a double-quoted string, prefix the double quote with a backslash - the backslash will not be in the final string). The characters in each element of the argument vector when you have completed constructing it are the literal characters that will be passed to the program you are executing.
Grep does not treat double quotes as special characters, so if grep gets double quotes in its search pattern, it will attempt to match double quotes from its input.
My original answer's reference to shell=True was incorrect - first I did not notice that you had originally specified shell=True, and secondly I was coming from the perspective of a Unix/Linux implementation, not Windows.
The python subprocess module page has this to say about shell=True and Windows:
On Windows: the Popen class uses CreateProcess() to execute the child child program, which operates on strings. If args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows.
That linked section on converting an argument sequence to a string on Windows does not make sense to me. First, a string is a sequence, and so is a list, yet the Frequently Used Arguments section says this about arguments:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).
This contradicts the conversion process described in the Python documentation, and given the behaviour you have observed, I'd say the documentation is wrong, and only applied to a argument string, not an argument vector. I cannot verify this myself as I do not have Windows or the source code for Python lying around.
I suspect that if you call subprocess.Popen like:
p = subprocess.Popen(grep + ' "2345$" file.txt', stdout=..., shell_True)
you may find that the double quotes are stripped out as part of the documented argument conversion.
You can use python-textops3 :
from textops import *
print('\n'.join(cat('file.txt') | grep('2345$')))
with python-textops3 you can use unix-like commands with pipes within python
so no need to fork a process which is very heavy