We Keep Coding

include a string variable that contains spaces into the 'subprocess.run()' [duplicate]

I'm working on a wrapper script that will exercise a vmware executable, allowing for the automation of virtual machine startup/shutdown/register/deregister actions. I'm trying to use subprocess to handle invoking the executable, but the spaces in the executables path and in parameters of the executable are not being handled correctly by subprocess. Below is a code fragment:
vmrun_cmd = r"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"
def vm_start(target_vm):
list_arg = "start"
list_arg2 = "hard"
if vm_list(target_vm):
p = Popen([vmrun_cmd, target_vm, list_arg, list_arg2], stdout=PIPE).communicate()[0]
print p
else:
vm_register(target_vm)
vm_start(target_vm)
def vm_list2(target_vm):
list_arg = "-l"
p = Popen([vmrun_cmd, list_arg], stdout=PIPE).communicate()[0]
for line in p.split('\n'):
print line
If I call the vm_list2 function, I get the following output:
$ ./vmware_control.py --list
C:\Virtual Machines\QAW2K3Server\Windows Server 2003 Standard Edition.vmx
C:\Virtual Machines\ubunturouter\Ubuntu.vmx
C:\Virtual Machines\vacc\vacc.vmx
C:\Virtual Machines\EdgeAS-4.4.x\Other Linux 2.4.x kernel.vmx
C:\Virtual Machines\UbuntuServer1\Ubuntu.vmx
C:\Virtual Machines\Other Linux 2.4.x kernel\Other Linux 2.4.x kernel.vmx
C:\Virtual Machines\QAClient\Windows XP Professional.vmx
If I call the vm_start function, which requires a path-to-vm parameter, I get the following output:
$ ./vmware_control.py --start "C:\Virtual Machines\ubunturouter\Ubuntu.vmx"
'c:\Program' is not recognized as an internal or external command,
operable program or batch file.
Apparently, the presence of a second parameter with embedded spaces is altering the way that subprocess is interpreting the first parameter. Any suggestions on how to resolve this?
python2.5.2/cygwin/winxp

If you have spaces in the path, the easiest way I've found to get them interpreted properly is this.
subprocess.call('""' + path + '""')
I don't know why exactly it needs double double quotes, but that is what works.

I believe that list2cmdline(), which is doing the processing of your list args, splits any string arg on whitespace unless the string contains double quotes. So I would expect
vmrun_cmd = r'"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"'
to be what you want.
You'll also likely want to surround the other arguments (like target_vm) in double quotes on the assumption that they, too, each represent a distinct arg to present to the command line. Something like
r'"%s"' % target_vm
(for example) should suit.
See the list2cmdline documentation

'c:\Program' is not recognized as an internal or external command, operable program or batch file.
To get this message, you are either:
Using shell=True:
vmrun_cmd = r"c:\Program Files\VMware\VMware Server\vmware-cmd.bat"
subprocess.Popen(vmrun_cmd, shell=True)
Changing vmrun_cmd on other part of your code
Getting this error from something inside vmware-cmd.bat
Things to try:
Open a python prompt, run the following command:
subprocess.Popen([r"c:\Program Files\VMware\VMware Server\vmware-cmd.bat"])
If that works, then quoting issues are out of the question. If not, you've isolated the problem.

In Python on MS Windows, the subprocess.Popen class uses the CreateProcess API to started the process. CreateProcess takes a string rather than something like an array of arguments. Python uses subprocess.list2cmdline to convert the list of args to a string for CreateProcess.
If I were you, I'd see what subprocess.list2cmdline(args) returns (where args is the first argument of Popen). It would be interesting to see if it is putting quotes around the first argument.
Of course, this explanation might not apply in a Cygwin environment.
Having said all this, I don't have MS Windows.

One problem is that if the command is surrounded with quotes and doesn't have spaces, that could also confuse the shell.
So I do this:
if ' ' in raw_cmd:
fmt = '"%s"'
else:
fmt = '%s'
cmd = fmt % raw_cmd

That was quite a hard problem for the last three ours....nothing stated so far did work, neither using r"" or Popen with a list and so on. What did work in the end was a combination of format string and r"". So my solution is this:
subprocess.Popen("{0} -f {1}".format(pathToExe, r'"%s"' % pathToVideoFileOrDir))
where both variables pathToExe and pathToVideoFileOrDir have whitespaces in their path. Using \" within the formatted string did not work and resulted in the same error that the first path is not detected any longer correctly.

Possibly stupid suggestion, but perhaps try the following, to remove subprocess + spaces from the equation:
import os
from subprocess Popen, PIPE
os.chdir(
os.path.join("C:", "Program Files", "VMware", "VMware Server")
)
p = Popen(
["vmware-cmd.bat", target_vm, list_arg, list_arg2],
stdout=PIPE
).communicate()[0]
It might also be worth trying..
p = Popen(
[os.path.join("C:", "Program Files", "VMware", "VMware Server", "vmware-cmd.bat"), ...

You probably don't want to use Pipe
If the output of the subprogram is greater than 64KB it is likely your process will crash.
http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/
Subprocess.Popen has a keyword argument shell, making it as if the shell has been parsing your arguments, setting shell=True should do what you want.

Why are you using r""? I believe that if you remove the "r" from the beginning, it will be treated as a standard string which may contain spaces. Python should then properly quote the string when sending it to the shell.

Here's what I don't like
vmrun_cmd = r"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"
You've got spaces in the name of the command itself -- which is baffling your shell. Hence the "'c:\Program' is not recognized as an internal or external command,
operable program or batch file."
Option 1 -- put your .BAT file somewhere else. Indeed, put all your VMWare somewhere else. Here's the rule: Do Not Use "Program Files" Directory For Anything. It's just wrong.
Option 2 -- quote the vmrun_cmd value
vmrun_cmd = r'"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"'

Executing command line from python that contains spaces

I try to automatise an image stitching process from python using the software PTGui.
I can execute the following command which works perfectly in the windows command line :
C:\Users\mw4168> "C:\Program Files\PTGui\ptgui.exe" -batch "C:\Users\mw4168\OneDrive\Desktop\PTGui Tests\3 rows\Panorama.pts"
command screenshot here
However, when I try to execute this command using os.system or subprocess.run in Python:
import os
os.system("C:\Program Files\PTGui\ptgui.exe" + "-batch" +"C:\Users\mw4168\OneDrive\Desktop\PTGui Tests\3 rows\panorama.pts")
I get this error :
'C:\Program' is not recognized as an internal or external command, operable program or batch file.
error screenshot here
It seems that there is an issue with the spaces within the string... Any idea on how to fix this?
Thanks a lot in advance,
Paul

Like the os.system documentation already tells you, a better solution altogether is to use subprocess instead.
subprocess.run([
r"C:\Program Files\PTGui\ptgui.exe",
"-batch",
r"C:\Users\mw4168\OneDrive\Desktop\PTGui Tests\3 rows\panorama.pts"])
The string you created also lacked spaces between the indvidual arguments; but letting Python pass the arguments to the OS instead also gives you more control over quoting etc.

The issue is that you're probably (as you have posted no code) not passing the C:\Program Files\PTGui\ptgui.exe as a string to the terminal but just a plain command
Use
import os
os.system('\"C:\Program Files\PTGui\ptgui.exe\" -batch')

Testing 7-Zip archives from a python script

So I've got a python script that, at it's core, makes .7z archives of selected directories for the purpose of backing up data. For simplicty sake I've simply invoked 7-zip through the windows command line, like so:
def runcompressor(target, contents):
print("Compressing {}...".format(contents))
archive = currentmodule
archive += "{}\\{}.7z".format(target, target)
os.system('7z u "{}" "{}" -mx=9 -mmt=on -ssw -up1q0r2x2y2z1w2'.format(archive, contents))
print("Done!")
Which creates a new archive if one doesn't exist and updates the old one if it does, but if something goes wrong the archive will be corrupted, and if this command hits an existing, corrupted archive, it just gives up. Now 7zip has a command for testing the integrity of an archive, but the documentation says nothing about giving an output, and then comes the trouble of capturing that output in python.
Is there a way I can test the archives first, to determine if they've been corrupted?

The 7z executable returns a value of two or greater if it encounters a problem. In a batch script, you would generally use errorlevel to detect this. Unfortunately, os.system() under Windows gives the return value of the command interpreter used to run your program, not the exit value of your program itself.
If you want the latter, you'll probably going to have to get your hands a little dirtier with the subprocess module, rather than using the os.system() call.
If you have version 3.5 (or better), this is as simple as:
import subprocess as sp
x = sp.run(['7z', 'a', 'junk.7z', 'junk.txt'], stdout=sp.PIPE, stderr=sp.STDOUT)
print(x.returncode)
That junk.txt in my case is a real file but junk.7z is just a copy of one of my text files, hence an invalid archive. The output from the program is 2 so it's easily detectable if something went wrong.
If you print out x rather than just x.returncode, you'll see something like (reformatted and with \r\n sequences removed for readability):
CompletedProcess(
args=['7z', 'a', 'junk.7z', 'junk.txt'],
returncode=2,
stdout=b'
7-Zip [64] 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
Error: junk.7z is not supported archive
System error:
Incorrect function.
'
)

Applying a perl script to every file in a directory and obtain output using Python

I am trying to make a python script that will open a directory, apply a perl script to every file in that directory and get its out put in either multiple text files or just one.
I currently have:
import shlex, subprocess
arg_str = "perl tilt.pl *.pdb > final.txt"
arg = shlex.split(arg_str)
import os
framespdb = os.listdir("prac_frames")
for frames in framespdb:
subprocess.Popen(arg, stdout=True)
I keep getting *.pdb not found. I am very new to all of this so any help trying to complete this script would help.

*.pdb not found means exactly that - there won't be a *.pdb in whatever directory you're running the script... and as I read the code - I don't see anything to imply it's within 'frames' when it runs the perl script.
you probably need os.chdir(path) before the Popen.
How do I "cd" in Python?
...using a python script to run somewhat dubious syscalls to perl may offend some people but everyone's done it.. aside from that I'd point out:
always specify full paths (this becomes a problem if you will later say, want to run your job automatically from cron or an environment that doesn't have your PATH).
i.e. 'which perl' - put that full path in.
./.pdb would be better but not as good as the fullpath/.pdb (which you could use instead of the os.chdir option).

subprocess.Popen(arg, stdout=True)
does not expand filename wildcards. To handle your *.pdb, use shell=True.

How to use subprocess when multiple arguments contain spaces?

I'm working on a wrapper script that will exercise a vmware executable, allowing for the automation of virtual machine startup/shutdown/register/deregister actions. I'm trying to use subprocess to handle invoking the executable, but the spaces in the executables path and in parameters of the executable are not being handled correctly by subprocess. Below is a code fragment:
vmrun_cmd = r"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"
def vm_start(target_vm):
list_arg = "start"
list_arg2 = "hard"
if vm_list(target_vm):
p = Popen([vmrun_cmd, target_vm, list_arg, list_arg2], stdout=PIPE).communicate()[0]
print p
else:
vm_register(target_vm)
vm_start(target_vm)
def vm_list2(target_vm):
list_arg = "-l"
p = Popen([vmrun_cmd, list_arg], stdout=PIPE).communicate()[0]
for line in p.split('\n'):
print line
If I call the vm_list2 function, I get the following output:
$ ./vmware_control.py --list
C:\Virtual Machines\QAW2K3Server\Windows Server 2003 Standard Edition.vmx
C:\Virtual Machines\ubunturouter\Ubuntu.vmx
C:\Virtual Machines\vacc\vacc.vmx
C:\Virtual Machines\EdgeAS-4.4.x\Other Linux 2.4.x kernel.vmx
C:\Virtual Machines\UbuntuServer1\Ubuntu.vmx
C:\Virtual Machines\Other Linux 2.4.x kernel\Other Linux 2.4.x kernel.vmx
C:\Virtual Machines\QAClient\Windows XP Professional.vmx
If I call the vm_start function, which requires a path-to-vm parameter, I get the following output:
$ ./vmware_control.py --start "C:\Virtual Machines\ubunturouter\Ubuntu.vmx"
'c:\Program' is not recognized as an internal or external command,
operable program or batch file.
Apparently, the presence of a second parameter with embedded spaces is altering the way that subprocess is interpreting the first parameter. Any suggestions on how to resolve this?
python2.5.2/cygwin/winxp

If you have spaces in the path, the easiest way I've found to get them interpreted properly is this.
subprocess.call('""' + path + '""')
I don't know why exactly it needs double double quotes, but that is what works.

I believe that list2cmdline(), which is doing the processing of your list args, splits any string arg on whitespace unless the string contains double quotes. So I would expect
vmrun_cmd = r'"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"'
to be what you want.
You'll also likely want to surround the other arguments (like target_vm) in double quotes on the assumption that they, too, each represent a distinct arg to present to the command line. Something like
r'"%s"' % target_vm
(for example) should suit.
See the list2cmdline documentation

'c:\Program' is not recognized as an internal or external command, operable program or batch file.
To get this message, you are either:
Using shell=True:
vmrun_cmd = r"c:\Program Files\VMware\VMware Server\vmware-cmd.bat"
subprocess.Popen(vmrun_cmd, shell=True)
Changing vmrun_cmd on other part of your code
Getting this error from something inside vmware-cmd.bat
Things to try:
Open a python prompt, run the following command:
subprocess.Popen([r"c:\Program Files\VMware\VMware Server\vmware-cmd.bat"])
If that works, then quoting issues are out of the question. If not, you've isolated the problem.

In Python on MS Windows, the subprocess.Popen class uses the CreateProcess API to started the process. CreateProcess takes a string rather than something like an array of arguments. Python uses subprocess.list2cmdline to convert the list of args to a string for CreateProcess.
If I were you, I'd see what subprocess.list2cmdline(args) returns (where args is the first argument of Popen). It would be interesting to see if it is putting quotes around the first argument.
Of course, this explanation might not apply in a Cygwin environment.
Having said all this, I don't have MS Windows.

One problem is that if the command is surrounded with quotes and doesn't have spaces, that could also confuse the shell.
So I do this:
if ' ' in raw_cmd:
fmt = '"%s"'
else:
fmt = '%s'
cmd = fmt % raw_cmd

That was quite a hard problem for the last three ours....nothing stated so far did work, neither using r"" or Popen with a list and so on. What did work in the end was a combination of format string and r"". So my solution is this:
subprocess.Popen("{0} -f {1}".format(pathToExe, r'"%s"' % pathToVideoFileOrDir))
where both variables pathToExe and pathToVideoFileOrDir have whitespaces in their path. Using \" within the formatted string did not work and resulted in the same error that the first path is not detected any longer correctly.

Possibly stupid suggestion, but perhaps try the following, to remove subprocess + spaces from the equation:
import os
from subprocess Popen, PIPE
os.chdir(
os.path.join("C:", "Program Files", "VMware", "VMware Server")
)
p = Popen(
["vmware-cmd.bat", target_vm, list_arg, list_arg2],
stdout=PIPE
).communicate()[0]
It might also be worth trying..
p = Popen(
[os.path.join("C:", "Program Files", "VMware", "VMware Server", "vmware-cmd.bat"), ...

You probably don't want to use Pipe
If the output of the subprogram is greater than 64KB it is likely your process will crash.
http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/
Subprocess.Popen has a keyword argument shell, making it as if the shell has been parsing your arguments, setting shell=True should do what you want.

Why are you using r""? I believe that if you remove the "r" from the beginning, it will be treated as a standard string which may contain spaces. Python should then properly quote the string when sending it to the shell.

Here's what I don't like
vmrun_cmd = r"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"
You've got spaces in the name of the command itself -- which is baffling your shell. Hence the "'c:\Program' is not recognized as an internal or external command,
operable program or batch file."
Option 1 -- put your .BAT file somewhere else. Indeed, put all your VMWare somewhere else. Here's the rule: Do Not Use "Program Files" Directory For Anything. It's just wrong.
Option 2 -- quote the vmrun_cmd value
vmrun_cmd = r'"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"'