We Keep Coding

include a string variable that contains spaces into the 'subprocess.run()' [duplicate]

I'm working on a wrapper script that will exercise a vmware executable, allowing for the automation of virtual machine startup/shutdown/register/deregister actions. I'm trying to use subprocess to handle invoking the executable, but the spaces in the executables path and in parameters of the executable are not being handled correctly by subprocess. Below is a code fragment:
vmrun_cmd = r"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"
def vm_start(target_vm):
list_arg = "start"
list_arg2 = "hard"
if vm_list(target_vm):
p = Popen([vmrun_cmd, target_vm, list_arg, list_arg2], stdout=PIPE).communicate()[0]
print p
else:
vm_register(target_vm)
vm_start(target_vm)
def vm_list2(target_vm):
list_arg = "-l"
p = Popen([vmrun_cmd, list_arg], stdout=PIPE).communicate()[0]
for line in p.split('\n'):
print line
If I call the vm_list2 function, I get the following output:
$ ./vmware_control.py --list
C:\Virtual Machines\QAW2K3Server\Windows Server 2003 Standard Edition.vmx
C:\Virtual Machines\ubunturouter\Ubuntu.vmx
C:\Virtual Machines\vacc\vacc.vmx
C:\Virtual Machines\EdgeAS-4.4.x\Other Linux 2.4.x kernel.vmx
C:\Virtual Machines\UbuntuServer1\Ubuntu.vmx
C:\Virtual Machines\Other Linux 2.4.x kernel\Other Linux 2.4.x kernel.vmx
C:\Virtual Machines\QAClient\Windows XP Professional.vmx
If I call the vm_start function, which requires a path-to-vm parameter, I get the following output:
$ ./vmware_control.py --start "C:\Virtual Machines\ubunturouter\Ubuntu.vmx"
'c:\Program' is not recognized as an internal or external command,
operable program or batch file.
Apparently, the presence of a second parameter with embedded spaces is altering the way that subprocess is interpreting the first parameter. Any suggestions on how to resolve this?
python2.5.2/cygwin/winxp

If you have spaces in the path, the easiest way I've found to get them interpreted properly is this.
subprocess.call('""' + path + '""')
I don't know why exactly it needs double double quotes, but that is what works.

I believe that list2cmdline(), which is doing the processing of your list args, splits any string arg on whitespace unless the string contains double quotes. So I would expect
vmrun_cmd = r'"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"'
to be what you want.
You'll also likely want to surround the other arguments (like target_vm) in double quotes on the assumption that they, too, each represent a distinct arg to present to the command line. Something like
r'"%s"' % target_vm
(for example) should suit.
See the list2cmdline documentation

'c:\Program' is not recognized as an internal or external command, operable program or batch file.
To get this message, you are either:
Using shell=True:
vmrun_cmd = r"c:\Program Files\VMware\VMware Server\vmware-cmd.bat"
subprocess.Popen(vmrun_cmd, shell=True)
Changing vmrun_cmd on other part of your code
Getting this error from something inside vmware-cmd.bat
Things to try:
Open a python prompt, run the following command:
subprocess.Popen([r"c:\Program Files\VMware\VMware Server\vmware-cmd.bat"])
If that works, then quoting issues are out of the question. If not, you've isolated the problem.

In Python on MS Windows, the subprocess.Popen class uses the CreateProcess API to started the process. CreateProcess takes a string rather than something like an array of arguments. Python uses subprocess.list2cmdline to convert the list of args to a string for CreateProcess.
If I were you, I'd see what subprocess.list2cmdline(args) returns (where args is the first argument of Popen). It would be interesting to see if it is putting quotes around the first argument.
Of course, this explanation might not apply in a Cygwin environment.
Having said all this, I don't have MS Windows.

One problem is that if the command is surrounded with quotes and doesn't have spaces, that could also confuse the shell.
So I do this:
if ' ' in raw_cmd:
fmt = '"%s"'
else:
fmt = '%s'
cmd = fmt % raw_cmd

That was quite a hard problem for the last three ours....nothing stated so far did work, neither using r"" or Popen with a list and so on. What did work in the end was a combination of format string and r"". So my solution is this:
subprocess.Popen("{0} -f {1}".format(pathToExe, r'"%s"' % pathToVideoFileOrDir))
where both variables pathToExe and pathToVideoFileOrDir have whitespaces in their path. Using \" within the formatted string did not work and resulted in the same error that the first path is not detected any longer correctly.

Possibly stupid suggestion, but perhaps try the following, to remove subprocess + spaces from the equation:
import os
from subprocess Popen, PIPE
os.chdir(
os.path.join("C:", "Program Files", "VMware", "VMware Server")
)
p = Popen(
["vmware-cmd.bat", target_vm, list_arg, list_arg2],
stdout=PIPE
).communicate()[0]
It might also be worth trying..
p = Popen(
[os.path.join("C:", "Program Files", "VMware", "VMware Server", "vmware-cmd.bat"), ...

You probably don't want to use Pipe
If the output of the subprogram is greater than 64KB it is likely your process will crash.
http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/
Subprocess.Popen has a keyword argument shell, making it as if the shell has been parsing your arguments, setting shell=True should do what you want.

Why are you using r""? I believe that if you remove the "r" from the beginning, it will be treated as a standard string which may contain spaces. Python should then properly quote the string when sending it to the shell.

Here's what I don't like
vmrun_cmd = r"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"
You've got spaces in the name of the command itself -- which is baffling your shell. Hence the "'c:\Program' is not recognized as an internal or external command,
operable program or batch file."
Option 1 -- put your .BAT file somewhere else. Indeed, put all your VMWare somewhere else. Here's the rule: Do Not Use "Program Files" Directory For Anything. It's just wrong.
Option 2 -- quote the vmrun_cmd value
vmrun_cmd = r'"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"'

subprocess.call on path with space

the string that contains a file looks like this in the console:
>>> target_file
'src//data//annual_filings//ABB Ltd//ABB_ar_2015.pdf'
I got the target_file from a call to os.walk
The goal is to build a command to run in subprocess.call
Something like:
from subprocess import call
cmd_ = r'qpdf-7.0.0/bin/qpdf --password=%s --decrypt %s %s' %('', target_file, target_file)
call([cmd_])
I tried different variations, setting shell to either True or False.
Replacing the // with /,\ etc.
The issue seems to be with the space in the folder (I can not change the folder name).
The python code needs to run on Windows

you have to define cmd_ as a list of arguments not a list with a sole string in it, or subprocess interprets the string as the command (doesn't even try to split the args):
cmd_ = ['qpdf-7.0.0/bin/qpdf','--password=%s'%'','--decrypt',target_file, target_file]
call(cmd_)
and leave the quoting to subprocess
As a side note, no need to double the slashes. It works, but that's unnecessary.

Passing shell commands with Python os.system() or subprocess.check_call()

I'm trying to call 'sed' from Python and having troubles passing the command line via either subprocess.check_call() or os.system().
I'm on Windows 7, but using the 'sed' from Cygwin (it's in the path).
If I do this from the Cygwin shell, it works fine:
$ sed 's/&nbsp;/\ /g' <"C:foobar" >"C:foobar.temp"
In Python, I've got the full pathname I'm working with in "name". I tried:
command = r"sed 's/&nbsp;/\ /g' " + "<" '\"' + name + '\" >' '\"' + name + '.temp' + '\"'
subprocess.check_call(command, shell=True)
All the concatenation is there to make sure I have double quotes around the input and output filenames (in case there are blank spaces in the Windows file path).
I also tried it replacing the last line with:
os.system(command)
Either way, I get this error:
sed: -e expression #1, char 2: unterminated `s' command
'amp' is not recognized as an internal or external command,
operable program or batch file.
'nbsp' is not recognized as an internal or external command,
operable program or batch file.
Yet, as I said, it works OK from the console. What am I doing wrong?

The shell used by subprocess is probably not the shell you want. You can specify the shell with executable='path/to/executable'. Different shells have different quoting rules.
Even better might be to skip subprocess altogether, and write this as pure Python:
with open("c:foobar") as f_in:
with open("c:foobar.temp", "w") as f_out:
for line in f_in:
f_out.write(line.replace(' ', ' '))

I agree with Ned Batchelder's assessment, but think what you might want to consider using the following code because it likely does what you ultimately want to accomplish which can be done easily with the help of Python's fileinput module:
import fileinput
f = fileinput.input('C:foobar', inplace=1)
for line in f:
line = line.replace(' ', ' ')
print line,
f.close()
print 'done'
This will effectively update the given file in place as use of the keyword suggests. There's also an optional backup= keyword -- not used above -- which will save a copy of the original file if desired.
BTW, a word of caution about using something like C:foobar to specify the file name because on Windows it means a file of that name in whatever the current directory is on drive C:, which might not be what you want.

I think you'll find that, in Windows Python, it's not actually using the CygWin shell to run your command, it's instead using cmd.exe.
And, cmd doesn't play well with single quotes the way bash does.
You only have to do the following to confirm that:
c:\pax> echo hello >hello.txt
c:\pax> type "hello.txt"
hello
c:\pax> type 'hello.txt'
The system cannot find the file specified.
I think the best idea would be to use Python itself to process the file. The Python language is a cross-platform one which is meant to remove all those platform-specific inconsistencies, such as the one you've just found.

Windows error and python

I'm working on a bit of code that is supposed to run an exe file inside a folder on my system and getting an error saying...
WindowsError: [Error 3] The system cannot find the path specified.
Here's a bit of the code:
exepath = os.path.join(EXE file localtion)
exepath = '"' + os.path.normpath(exepath) + '"'
cmd = [exepath, '-el', str(el), '-n', str(z)]
print 'The python program is running this command:'
print cmd
process = Popen(cmd, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
I have imported subprocess and also from subprocess import *
For example, This is how my exe file location looks like in the first line of the code I show:
exepath= os.path.join('/Program Files','next folder','next folder','blah.exe')
Am I missing something?

You need to properly escape the space in the executable path

Besides properly escaping spaces and other characters that could cause problems (such as /), you can also use the 8 character old DOS paths.
For example, Program Files would be:
Progra~1 , making sure to append ~1 for the last two characters.
EDIT: You could add an r to the front of the string, making it a raw literal. Python would read the string character for character. Like this:
r " \Program files"

If I remember correctly, you don't need to quote your executuable file path, like you do in the second line.
EDIT: Well, just grabbed nearby Windows box and tested this. Popen works the same regardless the path is quoted or not. So this is not an issue.

AFAIK, there is no need to surround the path in quotation marks unless cmd.exe is involved in running the program.
In addition, you might want to use the environment variable ProgramFiles to find out the actual location of 'Program Files' because that depends on regional settings and can also be tweaked using TweakUI.

When I write a python script to run Devenv with configure "Debug|Win32" it does nothing

Update: When I use the subprocess.call instead of subprocess.Popen, the problem is solved - does anybody know what's the cause? And there came another problem: I can't seem to find a way to control the output... Is there a way to redirect the output from subprocess.call to a string or something like that? Thanks!
I'm trying to use Devenv to build projects, and it runs just fine when i type it in command prompt like devenv A.sln /build "Debug|Win32" - but when I use a python to run it using Popen(cmd,shell=true) where cmd is the same line as above, it shows nothing. If I remove the |, change it to "Debug" only, it works....
Does anybody know why this happens? I've tried putting a \ before |, but still nothing happened..
This is the code I am using:
from subprocess import Popen, PIPE
cmd = ' "C:\\Program Files\\Microsoft Visual Studio 8\\Common7\\IDE\\devenv" solution.sln /build "Debug|Win32" '
sys.stdout.flush()
p = Popen(cmd,shell=True,stdout=PIPE,stderr=PIPE)
lines = []
for line in p.stdout.readlines():
lines.append(line)
out = string.join(lines)
print out
if out.strip():
print out.strip('\n')
sys.stdout.flush()
...which doesn't work, however, if I swap Debug|Win32 with Debug, it works perfectly..
Thanks for every comment here

There is a difference between devenv.exe and devenv.com, both of which are executable and live in the same directory (sigh). The command lines used in the question and some answers don't say which they want so I'm not sure which will get used.
If you want to call from the command line then you need to ensure you use devenv.com, otherwise you're likely to get a GUI popping up. I think this might be the cause of some (but not all) of the confusion.

See section 17.1.5.1. in the python documentation.
On Windows, Python automatically adds the double quotes around the project configuration argument i.e Debug|win32 is passed as "Debug|win32" to devenv. You DON'T need to add the double quotes and you DON'T need to pass shell=True to Popen.
Use ProcMon to view the argument string passed to devenv.

When shell = False is used, it will treat the string as a single command, so you need to pass the command/arugments as a list.. Something like:
from subprocess import Popen, PIPE
cmd = [
r"C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\devenv", # in raw r"blah" string, you don't need to escape backslashes
"solution.sln",
"/build",
"Debug|Win32"
]
p = Popen(cmd, stdout=PIPE, stderr=PIPE)
out = p.stdout.read() # reads full output into string, including line breaks
print out

try double quoting like: 'devenv A.sln /build "Debug|Win32"'

Looks like Windows' shell is taking that | as a pipe (despite the quotes and escapes). Have you tried shell=False instead?