I have a file name that I want to pass to a program or a bash script. For example if it's my car's picture.jpg, I have to change it to my\ car\'s picture.jpg to pass it to os.system like show my\ car\'s picture.jpg. Is there a function to do it the backslashes automatically?
You should use the subprocess module to call shell scripts from Python. Then you don't have to worry about escaping things yourself.
import subprocess
subprocess.call(['script_name', "my car's picture.jpg"])
subprocess.call() will escape everything correctly for you. If you need to read the output of the shell script, use subprocess.check_output() instead.
You can simply pass as is and use subprocess, os.system is depreciated.
c = check_output(["file","/home/padraic/Pictures/my cars' picture.png"])
print(c)
b"/home/padraic/Pictures/my cars' picture.png: PNG image data, 1366 x 768, 8-bit/color RGB, non-interlaced\n"
To call a script use check_call, if you want to pipe you can use Popen, there are lots of example in the docs linked above including replacing-os-system.
I can offer several incomplete suggestions that could be helpful to you.
Use "my car's picture.jpg" -- double quotes escape single ones
Using spaces in a UNIX file system generates only headaches. You could pass the filename inside of double-quotes.
os.system('cp "my car\'s picture" myCarPicture.jpg')
If you are using filenames with backslashes in a Windows system, use raw string
r"C:\Foo\bah\baz.jpg"
Related
I am trying to read from a file which has contents like this:
#\5\5\5
...
#\5\5\10
This file content is then fed into subprocess module of python like this:
for lines in file.readlines():
print(lines)
cmd = ls
p = subprocess.run([cmd, lines])
The output turns into something like this:
CompletedProcess(args=['ls', "'#5\\5\\5'\n"], returncode=1)
I don't understand why the contents of the file is appended with a double quote and another backward slash is getting appended.
The real problem here isn't Python or the subprocess module. The problem the use of subprocess to invoke shell commands, and then trying to parse the results. In this case, it looks like the command is ls, and the plan appears to be to read some filesystem paths from a text file (each path on a separate line), and list the files at that location on the filesystem.
Using subprocess to invoke ls is really, really, really not the way to accomplish that in Python. This is basically an attempt to use Python like a shell script (this use of ls would still be problematic, but that's a different discussion).
If a shell script is the right tool for the job, then write a shell script. If you want to use Python, then use one of the API's that it provides for interacting with the OS and the filesystem. There is no need to bring in external programs to achieve this.
import os
with open("list_of_paths.txt", "r") as fin:
for line in fin.readlines():
w = os.listdir(line.strip())
print(w)
Note the use of .strip(), this is a string method that will remove invisible characters like spaces and newlines from the ends of the input.
The listdir method provided by the os module will return a list of the files in a directory. Other options are os.scandir, os.walk, and the pathlib module.
But please do not use subprocess. 95% of the time, when someone thinks "should I use Python's subprocess module for this?" the ansewr is "NO".
It is because \ with a relevant character or digit becomes something else other than the string. For example, \n is not just \ and n but it means next line. If you really want a \n, then you would add another backslash to it (\\n). Likewise \5 means something else. here is what I found when i ran \5:
and hence the \\ being added, if I am not wrong
I want to use metaflac (https://linux.die.net/man/1/metaflac) command from within a python script.
from subprocess import run
flac_files = "/home/fricadelle/Artist - Album (2008)/*.flac"
run(['metaflac', '--add-replay-gain', flac_files])
I get
The FLAC file could not be opened. Most likely the file does not exist
or is not readable.
if I add shell = True to the run function I'd get:
ERROR: you must specify at least one FLAC file;
metaflac cannot be used as a pipe
So what do I do wrong? Thanks!
PS: of course the command works fine in a shell:
metaflac --add-replay-gain /home/fricadelle/Artist\ -\ Album \(2008\)/*.flac
Unless you specify shell=True (and as a first approximation, you should never specify shell=True), the arguments you provide are passed as is, with no shell expansions, word-splitting or dequoting. So the filename you pass as an argument is precisely /home/fricadelle/Artist - Album (2008)/*.flac, which is not the name of any file. (That's why you don't need to add backslashes before the spaces and parentheses. If you specified shell=True -- and I repeat, you really should avoid that -- then you would need to include backslashes so that the shell doesn't split the name into several different words.)
When you type
flac_files = "/home/fricadelle/Artist - Album (2008)/*.flac unquoted in a shell, the shell will try to expand that to a list of all the files whose names match then pattern, and will then pass that list as separate arguments. Since subprocess.run doesn't do this, you will have to do it yourself, which you would normally do with glob.glob. For example,
from subprocess import run
from glob import glob
flac_files = "/home/fricadelle/Artist - Album (2008)/*.flac"
run(['metaflac', '--add-replay-gain'] + glob(flac_files))
Note: unlike the shell, glob.glob will return an empty list if the pattern matches no files. You really should check for this error rather than invoke metaflac with no filename options.
See the answer here for a better explanation.
Globbing doesn't work the way you're expecting it to here, you need to specify shell=True, but then you'll need to drop the list.
run('metaflac --add-replay-gain ' + flac_files, shell=True)
Should do the trick.
I am trying to execute a system executable on UNIX with python. I have used op.system() to do this, but really need to use subprocess.call() instead. My op.System call is below:
os.system('gmsh default.msh_timestep%06d* animation_options.geo' %(timestep));
and works fine. It calls the program gmsh and gmsh reads a series of files specified in default.msh_timestep%06d*. I then try to do the equivalent thing with subprocess, but I get errors saying that the files are not there. Below is the subprocesses call:
call(["gmsh", "default.msh_timestep%06d*" %(timestep), "animation_options.geo"],shell=True);
Does anyone know what could be going on here? I'm admittedly a Python noob, so this might be a silly question.
Globbing is done by the shell for you. In Python, you need to do it yourself. You can use glob.glob to get file list that match the pattern:
import glob
call(["gmsh"] + glob.glob("default.msh_timestep%06d*" % (timestep,)) +
["animation_options.geo"])
If you want to use shell=True, pass a string isntead of a list of strings:
call("gmsh default.msh_timestep%06d* animation_options.geo" % (timestep,), shell=True)
I'm working on a wrapper script that will exercise a vmware executable, allowing for the automation of virtual machine startup/shutdown/register/deregister actions. I'm trying to use subprocess to handle invoking the executable, but the spaces in the executables path and in parameters of the executable are not being handled correctly by subprocess. Below is a code fragment:
vmrun_cmd = r"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"
def vm_start(target_vm):
list_arg = "start"
list_arg2 = "hard"
if vm_list(target_vm):
p = Popen([vmrun_cmd, target_vm, list_arg, list_arg2], stdout=PIPE).communicate()[0]
print p
else:
vm_register(target_vm)
vm_start(target_vm)
def vm_list2(target_vm):
list_arg = "-l"
p = Popen([vmrun_cmd, list_arg], stdout=PIPE).communicate()[0]
for line in p.split('\n'):
print line
If I call the vm_list2 function, I get the following output:
$ ./vmware_control.py --list
C:\Virtual Machines\QAW2K3Server\Windows Server 2003 Standard Edition.vmx
C:\Virtual Machines\ubunturouter\Ubuntu.vmx
C:\Virtual Machines\vacc\vacc.vmx
C:\Virtual Machines\EdgeAS-4.4.x\Other Linux 2.4.x kernel.vmx
C:\Virtual Machines\UbuntuServer1\Ubuntu.vmx
C:\Virtual Machines\Other Linux 2.4.x kernel\Other Linux 2.4.x kernel.vmx
C:\Virtual Machines\QAClient\Windows XP Professional.vmx
If I call the vm_start function, which requires a path-to-vm parameter, I get the following output:
$ ./vmware_control.py --start "C:\Virtual Machines\ubunturouter\Ubuntu.vmx"
'c:\Program' is not recognized as an internal or external command,
operable program or batch file.
Apparently, the presence of a second parameter with embedded spaces is altering the way that subprocess is interpreting the first parameter. Any suggestions on how to resolve this?
python2.5.2/cygwin/winxp
If you have spaces in the path, the easiest way I've found to get them interpreted properly is this.
subprocess.call('""' + path + '""')
I don't know why exactly it needs double double quotes, but that is what works.
I believe that list2cmdline(), which is doing the processing of your list args, splits any string arg on whitespace unless the string contains double quotes. So I would expect
vmrun_cmd = r'"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"'
to be what you want.
You'll also likely want to surround the other arguments (like target_vm) in double quotes on the assumption that they, too, each represent a distinct arg to present to the command line. Something like
r'"%s"' % target_vm
(for example) should suit.
See the list2cmdline documentation
'c:\Program' is not recognized as an internal or external command, operable program or batch file.
To get this message, you are either:
Using shell=True:
vmrun_cmd = r"c:\Program Files\VMware\VMware Server\vmware-cmd.bat"
subprocess.Popen(vmrun_cmd, shell=True)
Changing vmrun_cmd on other part of your code
Getting this error from something inside vmware-cmd.bat
Things to try:
Open a python prompt, run the following command:
subprocess.Popen([r"c:\Program Files\VMware\VMware Server\vmware-cmd.bat"])
If that works, then quoting issues are out of the question. If not, you've isolated the problem.
In Python on MS Windows, the subprocess.Popen class uses the CreateProcess API to started the process. CreateProcess takes a string rather than something like an array of arguments. Python uses subprocess.list2cmdline to convert the list of args to a string for CreateProcess.
If I were you, I'd see what subprocess.list2cmdline(args) returns (where args is the first argument of Popen). It would be interesting to see if it is putting quotes around the first argument.
Of course, this explanation might not apply in a Cygwin environment.
Having said all this, I don't have MS Windows.
One problem is that if the command is surrounded with quotes and doesn't have spaces, that could also confuse the shell.
So I do this:
if ' ' in raw_cmd:
fmt = '"%s"'
else:
fmt = '%s'
cmd = fmt % raw_cmd
That was quite a hard problem for the last three ours....nothing stated so far did work, neither using r"" or Popen with a list and so on. What did work in the end was a combination of format string and r"". So my solution is this:
subprocess.Popen("{0} -f {1}".format(pathToExe, r'"%s"' % pathToVideoFileOrDir))
where both variables pathToExe and pathToVideoFileOrDir have whitespaces in their path. Using \" within the formatted string did not work and resulted in the same error that the first path is not detected any longer correctly.
Possibly stupid suggestion, but perhaps try the following, to remove subprocess + spaces from the equation:
import os
from subprocess Popen, PIPE
os.chdir(
os.path.join("C:", "Program Files", "VMware", "VMware Server")
)
p = Popen(
["vmware-cmd.bat", target_vm, list_arg, list_arg2],
stdout=PIPE
).communicate()[0]
It might also be worth trying..
p = Popen(
[os.path.join("C:", "Program Files", "VMware", "VMware Server", "vmware-cmd.bat"), ...
You probably don't want to use Pipe
If the output of the subprogram is greater than 64KB it is likely your process will crash.
http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/
Subprocess.Popen has a keyword argument shell, making it as if the shell has been parsing your arguments, setting shell=True should do what you want.
Why are you using r""? I believe that if you remove the "r" from the beginning, it will be treated as a standard string which may contain spaces. Python should then properly quote the string when sending it to the shell.
Here's what I don't like
vmrun_cmd = r"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"
You've got spaces in the name of the command itself -- which is baffling your shell. Hence the "'c:\Program' is not recognized as an internal or external command,
operable program or batch file."
Option 1 -- put your .BAT file somewhere else. Indeed, put all your VMWare somewhere else. Here's the rule: Do Not Use "Program Files" Directory For Anything. It's just wrong.
Option 2 -- quote the vmrun_cmd value
vmrun_cmd = r'"c:/Program Files/VMware/VMware Server/vmware-cmd.bat"'
I am trying to use os.system() to call another program that takes an input and an output file. The command I use is ~250 characters due to the long folder names.
When I try to call the command, I'm getting an error: The input line is too long.
I'm guessing there's a 255 character limit (its built using a C system call, but I couldn't find the limitations on that either).
I tried changing the directory with os.chdir() to reduce the folder trail lengths, but when I try using os.system() with "..\folder\filename" it apparently can't handle relative path names. Is there any way to get around this limit or get it to recognize relative paths?
Even it's a good idea to use subprocess.Popen(), this does not solve the issue.
Your problem is not the 255 characters limit, this was true on DOS times, later increased to 2048 for Windows NT/2000, and increased again to 8192 for Windows XP+.
The real solution is to workaround a very old bug in Windows APIs: _popen() and _wpopen().
If you ever use quotes during the command line you have to add the entire command in quoates or you will get the The input line is too long error message.
All Microsoft operating systems starting with Windows XP had a 8192 characters limit which is now enough for any decent command line usage but they forgot to solve this bug.
To overcome their bug just include your entire command in double quotes, and if you want to know more real the MSDN comment on _popen().
Be careful because these works:
prog
"prog"
""prog" param"
""prog" "param""
But these will not work:
""prog param""
If you need a function that does add the quotes when they are needed you can take the one from http://github.com/ssbarnea/tendo/blob/master/tendo/tee.py
You should use the subprocess module instead. See this little doc for how to rewrite os.system calls to use subprocess.
You should use subprocess instead of os.system.
subprocess has the advantage of being able to change the directory for you:
import subprocess
my_cwd = r"..\folder\"
my_process = subprocess.Popen(["command name", "option 1", "option 2"], cwd=my_cwd)
my_process.wait() # wait for process to end
if my_process.returncode != 0:
print "Something went wrong!"
The subprocess module contains some helper functions as well if the above looks a bit verbose.
Assuming you're using windows, from the backslashes, you could write a .bat file from python and then os.system() on that. It's a hack.
Make sure when you're using '\' in your strings that they're being properly escaped.
Python uses the '\' as the escape character, so the string "..\folder\filename" evaluates to "..folderfilename" since an escaped f is still an f.
You probably want to use
r"..\folder\filename"
or
"..\\folder\\filename"
I got the same message but it was strange because the command was not that long (130 characters) and it used to work, it just stopped working one day.
I just closed the command window and opened a new one and it worked.
I have had the command window opened for a long time (maybe months, it's a remote virtual machine).
I guess is some windows bug with a buffer or something.