I'm trying to run a .exe that requires some parameters by using system().
If there's a space in the .exe's path AND in the path of a file passed in parameters, I get the following error:
The filename, directory name, or volume label syntax is incorrect.
Here is the code that generates that error:
#include <stdlib.h>
#include <conio.h>
int main (){
system("\"C:\\Users\\Adam\\Desktop\\pdftotext\" -layout \"C:\\Users\\Adam\\Desktop\\week 4.pdf\"");
_getch();
}
If the "pdftotext"'s path doesn't use quotation marks (I need them because sometimes the directory will have spaces), everything works fine. Also, if I put what's in "system()" in a string and output it and I copy it in an actual command window, it works.
I thought that maybe I could chain some commands using something like this:
cd C:\Users\Adam\Desktop;
pdftotext -layout "week 4.pdf"
So I would already be in the correct directory, but I don't know how to use multiple commands in the same system() function.
Can anyone tell me why my command doesn't work or if the second way I thought about would work?
Edit: Looks like I needed an extra set of quotation marks because system() passes its arguments to cmd /k, so it needs to be in quotations. I found it here:
C++: How to make a my program open a .exe with optional args
so I'll vote to close as duplicate since the questions are pretty close even though we weren't getting the same error message, thanks!
system() runs command as cmd /C command. And here's citation from cmd doc:
If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:
1. If all of the following conditions are met, then quote characters
on the command line are preserved:
- no /S switch
- exactly two quote characters
- no special characters between the two quote characters,
where special is one of: &<>()#^|
- there are one or more whitespace characters between the
two quote characters
- the string between the two quote characters is the name
of an executable file.
2. Otherwise, old behavior is to see if the first character is
a quote character and if so, strip the leading character and
remove the last quote character on the command line, preserving
any text after the last quote character.
It seems that you are hitting case 2, and cmd thinks that the whole string C:\Users\Adam\Desktop\pdftotext" -layout "C:\Users\Adam\Desktop\week 4.pdf (i.e. without the first and the last quote) is the name of executable.
So the solution would be to wrap the whole command in extra quotes:
//system("\"D:\\test\" nospaces \"text with spaces\"");//gives same error as you're getting
system("\"\"D:\\test\" nospaces \"text with spaces\"\""); //ok, works
And this is very weird. I think it's also a good idea to add /S just to make sure it will always parse the string by the case 2:
system("cmd /S /C \"\"D:\\test\" nospaces \"text with spaces\"\""); //also works
I got here looking for an answer, and this is the code that I came up with (and I was this explicit for the benefit of next person maintaining my code):
std::stringstream ss;
std::string pathOfCommand;
std::string pathOfInputFile;
// some code to set values for paths
ss << "\""; // command opening quote
ss << "\"" << pathOfCommand << "\" "; // Quoted binary (could have spaces)
ss << "\"" << pathOfInputFile << "\""; // Quoted input (could have spaces)
ss << "\""; // command closing quote
system( ss.str().c_str() ); // Execute the command
and it solved all of my problems.
Good learning from here on the internals of System call.Same issue reproducible(of course) with C++ string, TCHARs etc.
One approach that always helped me is SetCurrentDirectory() call. I first set current path and then execute. This has worked for me so far. Any comments welcome.
-Sreejith. D. Menon
Related
I'm trying without sucess to pass a Json string to a Python Script using PowerShell Script (.ps1) to automate this task.
spark-submit `
--driver-memory 8g `
--master local[*] `
--conf spark.driver.bindAddress=127.0.0.1 `
--packages mysql:mysql-connector-java:6.0.6,org.elasticsearch:elasticsearch-spark-20_2.11:7.0.0 `
--py-files build/dependencies.zip build/main.py `
$param
When $param='{ \"\"job_start\"\": \"\"jdbc:mysql://127.0.0.1:3307/test\"\"}' works fine, the python receives a valid JSON string and parse correctly.
When I use the character & like $param='{ \"\"job_start\"\": \"\"jdbc:mysql://127.0.0.1:3307/test&serverTimezone=UTC&autoReconnect=true&useSSL=false\"\"}' the string is printed like { "job_start": \jdbc:mysql://127.0.0.1:3307/test? and the rest of the string are reconized as other commands.
'serverTimezone' is not recognized as an internal or external command
'autoReconnect' is not recognized as an internal or external command
'useSSL' is not recognized as an internal or external command
The \"\" is need to maintain the double quots in the Python script, not sure why need two escaped double quotes.
UPDATE:
Now I'm having problems with the ! character, I can't escape this character even with ^ or \.
# Only "" doesn't work
$param='{\"\"job_start\"\": \"\"jdbc:mysql://127.0.0.1:3307/test^&serverTimezone=UTC\"\", \"\"password\"\": \"\"testpassword^!123\"\"}'
spark-submit.cmd `
--driver-memory 8g `
--master local[*] `
--conf spark.driver.bindAddress=127.0.0.1 `
--packages mysql:mysql-connector-java:6.0.6,org.elasticsearch:elasticsearch-spark-20_2.11:7.0.0 `
--py-files build/dependencies.zip build/main.py `
$param
# OUTPUT: misses the ! character
{"job_start": "jdbc:mysql://127.0.0.1:3307/test&serverTimezone=UTC", "password": "testpassword123"}
Thanks you all.
tl;dr
Note: The following does not solve the OP's specific problem (the cause of which is still unknown), but hopefully contains information of general interest.
# Use "" to escape " and - in case of delayed expansion - ^! to escape !
$param = '{ ""job_start"": ""jdbc:mysql://127.0.0.1:3307/test&serverTimezone=UTC&more^!"" }'
There are high-profile utilities (CLIs) such as az (Azure) that are Python-based, but on Windows use an auxiliary batch file as the executable that simply relays arguments to a Python script.
Use Get-Command az, for instance, to discover an executable's full file name; batch files, which are processed by cmd.exe, the legacy command processor, have a filename extension of either .cmd or .bat
To prevent calls to such a batch file from breaking, double quotes embedded in arguments passed from PowerShell must be escaped as ""
Additionally, but only if setlocal enabledelayedexpansion is in effect in a given target batch file or if your computer is configured to use delayed expansion by default, for all batch files:
! characters must be escaped as ^!, which, however, is only effective if cmd.exe considers the ! part of a double-quoted string.
It looks like we have a confluence of two problems:
A PowerShell problem with " chars. embedded in arguments passed to external programs:
In an ideal world, passing JSON text such as '{ "foo": "bar" }' to an external program would work as-is, but due to PowerShell's broken handling of embedded double quotes, that is not enough, and the " chars. must additionally be escaped, for the target program, either as \" (which most programs support), or, in the case of cmd.exe (see below), as "", which Python fortunately recognizes too: '{ ""foo"": ""bar"" }'
Limitations of argument-passing and escaping in cmd.exe batch files:
It sounds like spark-submit is an auxiliary batch file (.cmd or .bat) that passes the arguments through to a Python script.
The problem is that if you use \" for escaping embedded ", cmd.exe doesn't recognize them as escaped, which causes it to consider the & characters unquoted, and they are therefore interpreted as shell metacharacters, i.e. as characters with special syntactic function (command sequencing, in this case).
Additionally, and only if setlocal enabledelayedexpansion is in effect in a given batch file, any literal ! characters in arguments require additional handling:
If cmd.exe thinks the ! is part of an unquoted argument, you cannot escape ! at all.
Inside a quoted argument (which invariably means "..." in cmd.exe), you must escape a literal ! as ^!.
Note that this requirement is the inverse of how all other metacharacters must be escaped (which require ^ when unquoted, but not inside "...").
The unfortunate consequence is that you need to know the implementation details of the target batch file - whether it uses setlocal enabledelayedexpansion or not - in order to formulate your arguments properly.
The same applies if your computer is configured to use delayed expansion by default, for all batch files (and interactively), which is neither common nor advisable. To test if a given computer is configured that way, check the output from the following command for DelayedExpansion : 1: if there's no output at all, delayed expansion is OFF; if there's 1 or 2 outputs, delayed expansion is ON by default if the first or only output reports DelayedExpansion : 1.
Get-ItemProperty -EA Ignore 'registry::HKEY_CURRENT_USER\Software\Microsoft\Command Processor', 'registry::HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor' DelayedExpansion
Workaround:
Since you're technically calling a batch file, use "" to escape literal " chars. inside your single-quoted ('...') PowerShell string.
If you know that the target batch file uses setlocal enabledelayedexpansion or if your computer is configured to use delayed expansion by default, escape ! characters as ^!
Note that this is only effective if cmd.exe considers the ! part of a double-quoted string.
Therefore (note that I've extended the URL to include a token with !, meant to be passed through literally as suffix more!):
$param = '{ ""job_start"": ""jdbc:mysql://127.0.0.1:3307/test&serverTimezone=UTC&more^!"" }'
If you need to escape an existing JSON string programmatically:
# Unescaped JSON string, which in an ideal world you'd be able
# to pass as-is.
$param = '{ "job_start": "jdbc:mysql://127.0.0.1:3307/test&serverTimezone=UTC&more!" }'
# Escape the " chars.
$param = $param -replace '"', '""'
# If needed, also escape the ! chars.
$param = $param -replace '!', '^!'
Ultimately, both problems should be fixed at the source - but that this is highly unlikely, because it would break backward compatibility.
With respect to PowerShell, this GitHub issue contains the backstory, technical details, a robust wrapper function to hide the problems, and discussions about how to fix the problem at least on an opt-in basis.
In this question Which characters need to be escaped when using Bash?
, you will find all the characters that you should escape when passing them as normal characters in the shell, you will also notice that & is one of them.
Now I understand that if you tried to escape it, the JSON parser you are using will probably fail to parse the string. So one quick workaround would be to replace the & by any other special non-escapable symbol like # or %, and do a step in your app where you replace it with & before parsing. Just make sure that the symbol you will use isn't used in your strings, and won't be used at any time.
I am writing a sublime plugin in Python. But I am still very new to Python language. I am using this line to call meld in Python:
if meld_cmd == None:
meld_cmd = "C:\\Program Files (x86)\\Meld\\meld\\meld"
os.system('"%s" "%s" "%s" ' %(meld_cmd, source_name, dest_name))
It doesn't work well in Windows. The cmd window just flashes for a sec (seem to execute something) then disappears. I print the executed string out in sublime and copy the string into cmd window and execute. It executes well in both admin and non-admin mode. And I tried to add a pause before executing:
os.system('pause && "%s" "%s" "%s" ' %(meld_cmd, source_name, dest_name))
This works well too after clicking enter after the press any key to continue... line.
Not sure how to debug this and what is the reason why it is failing.
cmd.exe /c command has a parsing quirk when the command starts with a quote and has more than two quotes. The exact rules are explained in the online help text available via cmd /?:
If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:
1. If all of the following conditions are met, then quote characters
on the command line are preserved:
- no /S switch
- exactly two quote characters
- no special characters between the two quote characters,
where special is one of: &<>()#^|
- there are one or more whitespace characters between the
two quote characters
- the string between the two quote characters is the name
of an executable file.
2. Otherwise, old behavior is to see if the first character is
a quote character and if so, strip the leading character and
remove the last quote character on the command line, preserving
any text after the last quote character.
To get around this, just wrap the whole command in quotes, e.g. '""%s" "%s" "%s""' % (meld_cmd, source_name, dest_name).
Probably the problem is that you need to execute that script with Admin privileges.
Proceed to question How to run python script with elevated privilege on windows to get more information on that topic.
I know about escaping backslashes in programming. However I'm passing in a command line argument into a Python program to parse. Whitespace is irrelevant, however via the command line, whitespace does matter. So I am trying to concatenate all argv[] arguments except the first element into a string to then parse. However a token of a single '\' is never encountered in the for loop.
The command line argument (notice whitespace around '\'):
((1 * 2) + (3 - (4 \ 5)))
Program:
import sys
string = ""
for token in sys.argv[1:len(sys.argv)]:
print(token)
if "\\" in r"%r" % token:
print("Token with \\ found")
string += token
print(string)
print("done")
This is not a Python problem. The shell will interpret the text you write at the command line and (for typical Unix shells) perform backslash substitution. So by the time Python examines the command line, the backslash will already have been substituted away.
The workaround is to double the backslash or put it in single quotes, if your shell is Bash or some other Bourne-compatible Unix shell. Windows CMD is different and bewildering in its own right; Csh doubly so.
)I have confirmed my Linux command works in the terminal, however when I try to call it from python it breaks.
The command is a bit long and has lots of single quotes, so I wrapped it around three double quotes (""") so python can interpret as a raw string (or so I thought). However, when I run it I am getting
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
but I have double and tripple checked my single and double quotes and I have no idea where to go from here.
See the test script below
import os
os.system("""awk -F ' *[[:alnum:]_]*: *' 'BEGIN {h="insert_job;box_name;command;owner;permission;condition;description;std_out_file;std_err_file;alarm_if_fail"; print h; n=split(h,F,/;/)} function pr() {if(F[1] in A) {for(i=1;i<=n;i++)printf "%s%s",A[F[i]],(i<n)?";":RS}} /insert_job/ {pr(); delete A} {for(i in F){if($0~"^"F[i])A[F[i]]=$2}} END {pr()}' ./output/JILS/B7443_dev_jil_20140306104313.csv > /trvapps/autosys/admin/EPS/output/JILS/testout.txt""")
FYI I am using Python 2.4.3, hence why I am using os instead of subprocess.
For your own sanity, try using pipes.quote (or something similar if that doesn't exist in 2.4), ' '.join(words) and '\n'.join(lines) to be able to build up the command rather than using a single complex string if you have to put it in Python. A better solution would be to call a script like #kojiro suggested.
It looks like you are doing some simple CSV munging. How about checking SO for tips on doing that in Python?
In any case, 400+ characters of awk on a single line is enough to make anyone squirm, and doing it in Python, which already has excellent string handling features, is just passing the pain to the next developer. Which will be very angry.
Cramming the awk script into one huge line is awful, and makes it nearly impossible to read and maintain. Don't do that -- if you really must use awk (a dubious claim), write it out on multiple lines, with proper indentation, like you would any other script.
To fix the bug with sh -c interpreting things wrong, use the subprocess module (passing an argument array and not setting shell=True) instead of os.system().
import subprocess
awk_script = r'''
*[[:alnum:]_]*: *
BEGIN {
h="insert_job;box_name;command;owner;permission;condition;description;std_out_file;std_err_file;alarm_if_fail";
print h;
n=split(h,F,/;/)
}
function pr() {
if(F[1] in A) {
for(i=1;i<=n;i++)
printf "%s%s", A[F[i]], (i<n) ? ";" : RS
}
}
/insert_job/ {
pr();
delete A;
}
{
for(i in F) {
if($0~"^"F[i])
A[F[i]]=$2
}
}
END {pr()}
'''
exit_status = subprocess.call(['awk', '-F', awk_script],
stdin=open('./output/JILS/B7443_dev_jil_20140306104313.csv', 'r'),
stdout=open('/trvapps/autosys/admin/EPS/output/JILS/testout.txt', 'w'))
if exit_status != 0:
raise RuntimeException('awk failed')
Platform: Windows
Grep: http://gnuwin32.sourceforge.net/packages/grep.htm
Python: 2.7.2
Windows command prompt used to execute the commands.
I am searching for the for the following pattern "2345$" in a file.
Contents of the file are as follows:
abcd 2345
2345
abcd 2345$
grep "2345$" file.txt
grep returns 2 lines (first and second) successfully.
When I try to run the above command through python I don't see any output.
Python code snippet is as follows:
temp = open('file.txt', "r+")
grep_cmd = []
grep_cmd.extend([grep, '"2345$"' ,temp.name])
print grep_cmd
p = subprocess.Popen(grep_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdoutdata = p.communicate()[0]
print stdoutdata
If I have
grep_cmd.extend([grep, '2345$' ,temp.name])
in my python script, I get the correct answer.
The questions is why the grep command with "
grep_cmd.extend([grep, '"2345$"' ,temp.name])
executed from python fails. Isn't python supposed to execute
the command as it is.
Thanks
Gudge.
Do not put double quotes around your pattern. It is only needed on the command line to quote shell metacharacters. When calling a program from python, you do not need this.
You also do not need to open the file yourself - grep will do that:
grep_cmd.extend([grep, '2345$', 'file.txt'])
To understand the reason for the double quotes not being needed and causing your command to fail, you need to understand the purpose of the double quotes and how they are processed.
The shell uses double quotes to prevent special processing of some shell metacharacters. Shell metacharacters are those characters that the shell handles specially and does not pass literally to the programs it executes. The most commonly used shell metacharacter is "space". The shell splits a command on space boundaries to build an argument vector to execute a program with. If you want to include a space in an argument, it must be quoted in some way (single or double quotes, backslash, etc). Another is the dollar sign ($), which is used to signify variable expansion.
When you are executing a program without the shell involved, all these rules about quoting and shell metacharacters are not relevant. In python, you are building the argument vector yourself, so the relevant quoting rules are python quoting rules (e.g. to include a double quote inside a double-quoted string, prefix the double quote with a backslash - the backslash will not be in the final string). The characters in each element of the argument vector when you have completed constructing it are the literal characters that will be passed to the program you are executing.
Grep does not treat double quotes as special characters, so if grep gets double quotes in its search pattern, it will attempt to match double quotes from its input.
My original answer's reference to shell=True was incorrect - first I did not notice that you had originally specified shell=True, and secondly I was coming from the perspective of a Unix/Linux implementation, not Windows.
The python subprocess module page has this to say about shell=True and Windows:
On Windows: the Popen class uses CreateProcess() to execute the child child program, which operates on strings. If args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows.
That linked section on converting an argument sequence to a string on Windows does not make sense to me. First, a string is a sequence, and so is a list, yet the Frequently Used Arguments section says this about arguments:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).
This contradicts the conversion process described in the Python documentation, and given the behaviour you have observed, I'd say the documentation is wrong, and only applied to a argument string, not an argument vector. I cannot verify this myself as I do not have Windows or the source code for Python lying around.
I suspect that if you call subprocess.Popen like:
p = subprocess.Popen(grep + ' "2345$" file.txt', stdout=..., shell_True)
you may find that the double quotes are stripped out as part of the documented argument conversion.
You can use python-textops3 :
from textops import *
print('\n'.join(cat('file.txt') | grep('2345$')))
with python-textops3 you can use unix-like commands with pipes within python
so no need to fork a process which is very heavy