We Keep Coding

How do I run this cmd command in python [duplicate]

My code is simply as follows:
file = 'C:\\Exe\\First Version\\filename.exe'
os.system(file)
When I run this program, a Windows error is raised: can't find the file specified.
I found out the problem has to with the whitespace in the middle of "First Version". How could I find a way to circumvent the problem?
P.S.: what if the variable 'file' is being passed as an argument into another function?

Putting quotes around the path will work:
file = 'C:\\Exe\\First Version\\filename.exe'
os.system('"' + file + '"')
but a better solution is to use the subprocess module instead:
import subprocess
file = 'C:\\Exe\\First Version\\filename.exe'
subprocess.call([file])

I used this:
import subprocess, shlex
mycmd='"C:\\Program Files\\7-Zip\\7z" x "D:\\my archive.7z" -o"D:\\extract folder" -aou'
subprocess.run(shlex.split(mycmd))

Try enclosing it with double quotes.
file = '"C:\\Exe\\First Version\\filename.exe"'
os.system(file)

It is true that os.system will launch a binary which has spaces in the path by wrapping that path in quotes. (That should be a pretty obvious solution if you are accustom to using a terminal.) By itself, however, that doesn't resolve the more painful problem with this function... Once you do that, you can then run into troubles adding arguments to your command! (Ahh!)
All current recommendations are to use the subprocess module now instead of this old, frowned upon function. One can also use shlx to convert flat strings into lists for those subprocess functions. I've encountered issues or pains with those methods too, which I won't ramble on about... Also, it's sometimes just easier use os.system when all you want is a thin wrapper over the shell, which implicitly displays the output streams on the console, works synchronously, etc. I sure wish there was a built-in function to execute a command on the shell like this, with absolutely 0 parsing, wrapping, abstracting...
Since there is no built-in without "filters", here's my solution patch for os.system. This is lifted from an open source library of mine. This has been tested on Windows, Mac, and Ubuntu Linux. I'm aware it's not 100% foolproof, and it's more involved than one would hop for, but it's not too bad.
When you call this _system() wrapper (passing a string to execute), just surround your long path in quotes and include any arguments that needs with or without quotes as well (i.e. exactly as you would enter the command in a terminal!). On the first "token" in the command, this will remove the quotes and escape spaces in the path on Mac or Linux. On Windows, it uses the "short name" by actually resolving what that is on the given environment. That part of the code is a bit tricky. Basically it uses a batch mechanism for the name resolution, and it sends the results back over stderr for the purpose of parsing what you'd get otherwise for the Popen() results on stdout.
I think I included all the imports and defines that you'll need. If I missed any (copying and pasting spinets of source), let me know.
from os import system, getcwd, chdir
from subprocess import Popen, PIPE
import platform
__plat = platform.system()
IS_WINDOWS = __plat == "Windows"
IS_LINUX = __plat == "Linux"
IS_MACOS = __plat == "Darwin"
__SCRUB_CMD_TMPL = "{0}{1}"
__DBL_QUOTE = '"'
__SPACE = ' '
__ESC_SPACE = '\\ '
if IS_WINDOWS :
__BATCH_RUN_AND_RETURN_CMD = ["cmd","/K"] # simply assuming cmd is on the system path...
__BATCH_ONE_LINER_TMPLT = "{0} 1>&2\n" # the newline triggers execution when piped in via stdin
__BATCH_ESCAPE_PATH_TMPLT = 'for %A in ("{0}") do #echo %~sA'
from subprocess import STARTUPINFO, STARTF_USESHOWWINDOW
__BATCH_ONE_LINER_STARTUPINFO = STARTUPINFO()
__BATCH_ONE_LINER_STARTUPINFO.dwFlags |= STARTF_USESHOWWINDOW
def _system( cmd, wrkDir=None ):
if wrkDir is not None:
initWrkDir = getcwd()
print( 'cd "%s"' % (wrkDir,) )
chdir( wrkDir )
cmd = __scrubSystemCmd( cmd )
print( cmd )
system( cmd )
print('')
if wrkDir is not None: chdir( initWrkDir )
def __scrubSystemCmd( cmd ):
"""
os.system is more convenient than the newer subprocess functions
when the intention is to act as very thin wrapper over the shell.
There is just one MAJOR problem with it:
If the first character in the command is a quote (to escape a long path
to the binary you are executing), then the limited (undesirable) parsing
built into the function can all fall apart. So, this scrub function
solves that...
"""
if not cmd.startswith( __DBL_QUOTE ): return cmd
cmdParts = cmd[1:].split( __DBL_QUOTE )
safeBinPath = _escapePath( cmdParts[0] )
args = __DBL_QUOTE.join( cmdParts[1:] ) # (the leading space will remain)
return __SCRUB_CMD_TMPL.format( safeBinPath, args )
def _escapePath( path ):
if not IS_WINDOWS: return path.replace(__SPACE, __ESC_SPACE)
return( path if __SPACE not in path else
__batchOneLinerOutput( __BATCH_ESCAPE_PATH_TMPLT.format(path) ) )
def __batchOneLinerOutput( batch ):
cmd = __BATCH_ONE_LINER_TMPLT.format( batch )
p = Popen( __BATCH_RUN_AND_RETURN_CMD, shell=False,
startupinfo=__BATCH_ONE_LINER_STARTUPINFO,
stdin=PIPE, stdout=PIPE, stderr=PIPE )
# pipe cmd to stdin, return stderr, minus a trailing newline
return p.communicate( cmd )[1].rstrip()
UPDATE
A better trick for the Windows context dawned on me recently. You don't need any of the conversion to short file name, or an escape sequence for a space. All you need to do is thwart the Python source which throws a monkey wrench into this by checking if the first character in the command is a double quote. Well, on Windows cmd / Batch, you can prefix any command with an # to indicate not to "echo" that line. So, if you simply slap that on the front of your command, there will no longer be a leading quote! You probably don't want the command echoed either, so it's an improvement in its self anyway.
Essentially, replace __scrubSystemCmd above with the following. If you want, you could drop all that code which gets the short file name (which is like half the code I orginally posted!)...
if IS_WINDOWS:
...
__NO_ECHO_PREFIX = "#"
def __scrubSystemCmd( cmd ):
if IS_WINDOWS:
if not cmd.startswith( __NO_ECHO_PREFIX ):
return __NO_ECHO_PREFIX + cmd
elif not cmd.startswith( __DBL_QUOTE ): return cmd
cmdParts = cmd[1:].split( __DBL_QUOTE )
safeBinPath = _escapePath( cmdParts[0] )
args = __DBL_QUOTE.join( cmdParts[1:] ) # (the leading space will remain)
return __SCRUB_CMD_TMPL.format( safeBinPath, args )

You can use the short name of the file which has spaces in its name.
file = 'C:\\Exe\\FirstV~1\\filename.exe'
os.system(file)

How to pass variable of Python to bash code within same Python Program?

I have written a python code to check if file exists in Hadoop file system or not. Python function receives location passed from another function and bash code within checks if location exists.
def check_file_exists_in_hadoop(loc):
yourdir = "/somedirectory/inhadoop/"+loc
cmd = '''
hadoop fs -test -d ${yourdir};
if [ $? -eq 0 ]
then
echo "Directory exists!"
else
echo "Directory does not exists!"
fi
'''
res = subprocess.check_output(cmd, shell=True)
output = (str(res, "utf-8").strip())
print(output)
if output == "Directory exists!":
print("Yay!!!!")
else:
print("Oh no!!!!")
How to pass 'yourdir' variable inside bash portion of code.

All that playing around in shells looks awkward, why not just do:
def check_file_exists_in_hadoop(loc):
path = "/somedirectory/inhadoop/" + loc
res = subprocess.run(["hadoop", "fs", "-test", "-d", path])
return res.returncode == 0
You can execute as:
if check_file_exists_in_hadoop('foo.txt'):
print("Yay!!!!")
else:
print("Oh noes!!!!")
When you execute/run a process/program in a Unix-like system, it receives an array of arguments (exposed as e.g., sys.argv in Python). you can construct these in various ways but passing them to run gives you the most direct control. You can of course use a shell to do this, but starting up a shell just to do this seems unnecessary. Given that this argument list is just a list of strings in Python you can use normal list/string manipulations to construct whatever you need.
Using a shell can be useful, but as Gilles says you need to be careful to sanitise/escape your input — not everybody loves little bobby tables!

Pass the string as an argument to the shell. Instead of using shell=True, which runs ['sh', '-c', cmd] under the hood, invoke a shell explicitly. After the shell code, the first argument is the shell or script name (which is unused here), then the next argument is available as "$1" in the shell snippet, the next argument as "$2", etc.
cmd = '''
hadoop fs -test -d "$1";
…
'''
res = subprocess.check_output(['sh', '-c', cmd, 'sh', yourdir])
Alternatively, pass the string as an environment variable.
cmd = '''
hadoop fs -test -d "$yourdir";
…
'''
env = os.environ.copy()
env['yourdir'] = yourdir
res = subprocess.check_output(cmd, shell=True, env=env)
In the shell snippet, note the double quotes around $1 or $yourdir.
Do not interpolate the string into the shell command directly, i.e. don't use things like 'test -d {}'.format(yourdir). That doesn't work if the string contains shell special characters: it's a gaping security hole. For example if yourdir is a; rm -rf ~ then you've just kissed your data goodbye.

Python script : Running a script with multiple arguments using subprocess

My question is related to this earlier question - Python subprocess usage
I am trying to run this command using python
nccopy -k 4 "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]" foo.nc
When I run the above command I should be able to see a file called foo.nc on my disk or a network error stating unable to access that URL or remote URL not found.
Currently the ESRL NOAA server is down - so when I run the above command I get
syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: ^
NetCDF: Access failure
Location: file nccopy.c; line 1348
I should get the same error when I run the python script
This is the code I have and I am unable to figure out exactly how to proceed further -
I tried splitting up "-k 4" into two arguments and removing the quotes and I still get this error nccopy : invalid format : 4
Results of print(sys.argv) data.py
['data.py', '-k', '4', 'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][20:34][26:40]', 'foo.nc']
import numpy as np
import subprocess
import sys
url = '"http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]"'
outputFile = 'foo.nc'
arg1 = "-k 4"
arg3 = url
arg4 = outputFile
print (input)
subprocess.check_call(["nccopy",arg1,arg3,arg4])

There's two dilemmas here.
One being that subprocess processes your arguments and tries to use 4 as a separate argument.
The other being that system calls still goes under normal shell rules, meaning that parameters and commands will be parsed for metacharacters aka special characters. In this case you're wrapping [ and ].
There for you need to separate each parameters and it's value into separate objects in the parameter-list, for instance -k 4 should be ['-k', '4'] and you need to wrap parameters/values in '...' instead of "...".
Try this, shlex.split() does the grunt work for you, and i swapped the encapsulation characters around the URL:
import numpy as np
import subprocess
import sys
import shlex
url = "'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]'"
outputFile = 'foo.nc'
command_list = shlex.split('nccopy -k 4 ' + url + ' ' + outpufFile)
print(command_list)
subprocess.check_call(command_list)

Instead of arg1 = "-k 4", use two arguments instead.
import subprocess
url = 'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]'
outputFile = 'foo.nc'
arg1 = "-k"
arg2 = "4"
arg3 = url
arg4 = outputFile
subprocess.check_call(["nccopy", arg1, arg2, arg3, arg4])
See also here Python subprocess arguments

If you have a working shell command that runs a single program with multiple arguments and you want to parameterized it e.g., to use a variable filename instead of the hardcoded value then you could use shlex.split() to create a list of command-line arguments that you could pass to subprocess module and replace the desired argument with a variable e.g.:
>>> shell_command = "python -c 'import sys; print(sys.argv)' 1 't w o'"
>>> import shlex
>>> shlex.split(shell_command)
['python', '-c', 'import sys; print(sys.argv)', '1', 't w o']
To run the command using the same Python interpreter as the parent script, sys.executable could be used and we can pass a variable instead of '1':
#!/usr/bin/env python
import random
import sys
import subprocess
variable = random.choice('ab')
subprocess.check_call([sys.executable, '-c', 'import sys; print(sys.argv)',
variable, 't w o'])
Note:
one command-line argument per list item
no shlex.split() in the final code
there are no quotes inside 't w o' i.e., 't w o' is used instead of '"t w o"' or "'t w o'"
subprocess module does not run the shell by default and therefore you don't need to escape shell meta-characters such as a space inside the command-line arguments. And in reverse, if your command uses some shell functionality (e.g., file patterns) then either reimplement the corresponding features in Python (e.g., using glob module) or use shell=True and pass the command as a string as is. You might need pipes.quote(), to escape variable arguments in this case. Wildcard not working in subprocess call using shlex

Failed to run shell commands with subprocess [duplicate]

This question already has answers here:
Why subprocess.Popen doesn't work when args is sequence?
(3 answers)
Closed 6 years ago.
In my terminal if I run: echo $(pwd), I got /home/abr/workspace, but when I tried to run this script in python like this:
>>> import subprocess
>>> cmd = ['echo', '$(pwd)']
>>> subprocess.check_output(cmd, shell=True)
I get '\n'. How to fix this?

Use os package:
import os
print os.environ.get('PWD', '')

From the documentation on the subprocess module:
If args is a sequence, the first item specifies the command string,
and any additional items will be treated as additional arguments to
the shell itself.
You want:
subprocess.check_output("echo $(pwd)", shell=True)

Try this:
cmd = 'echo $(pwd)'
subprocess.check_output(cmd, shell=True)
In subprocess doc it specified that cmd should be a string when shell=True.
From the documentation:
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence.
A better way to achieve this is probably to use the os module from the python standard library, like this:
import os
print os.getcwd()
>> "/home/abr/workspace"
The getcwd() function returns a string representing the current working directory.

The command subpreocess.check_output will return the output of the command you are calling:
Example:
#echo 2
2
from python
>>>subprocess.check_output(['echo', '2'], shell=True)
>>>'2\n'
the '\n' is included because that is what the command does it prints the output sting and then puts the current on a new line.
now back to your problem; assuming you want the output of 'PWD', first of all you have to get rid of the shell. If you provide the shell argument, the command will be run in a shell environment and you won't see the returned string.
subprocess.check_output(['pwd'])
Will return the current directory + '\n'
On a personal note, I have a hard time understanding what you are trying to do, but I hope this helps solve it.

Python subprocess.Popen fails on shell command as argument

Struggling with subprocess.Popen() - why is First and Third working as expected, while the second does not find any of multiple files or directories? Error message is:
>ls: Zugriff auf * nicht möglich: Datei oder Verzeichnis nicht gefunden
English translation:
File not found or directory: access to * not possible
Here is the code.
#!/usr/bin/python
# -*- coding: UTF-8 -*-
import subprocess
args = []
args.append ('ls')
args.append ('-al')
# First does work
cmd1 = subprocess.Popen(args)
cmd1.wait()
# Second does NOT work
args.append ('*')
cmd2 = subprocess.Popen(args)
cmd2.wait()
# Third does work
shellcmd = "ls -al *"
cmd3 = subprocess.Popen(shellcmd, shell=True )
cmd3.wait()

This is because by default subprocess.Popen() doesn't have the shell interpret the commands, so the "*" isn't being expanded into the required list of files. Try adding shell=True as a final argument to the call.
Also note the warning in the documentation about not trusting user input to be processed in this way.

This is happening because of shell globbing.
Basically, the * in ls -al * is expanded by your shell, to match all available files.
When you run the subprocess without the shell=True flag, python is not able to parse the * on its own, and hence, the error message ls: cannot access *: No such file or directory is displayed.
When you run the command with shell=True, python actually passes the control to the shell, and hence the correct output is displayed.
As an aside, executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution, so it should be used with caution (see warning here).
EDIT 1
Both shell globbing and the way Popen consumes args is causing the issue here
From subprocess module,
class subprocess.Popen
args should be a sequence of program arguments or else a single string.
If shell is True, it is recommended to pass argsas astring` rather than as a sequence.
To understand that shell globbing and the manner in which Popen consumes args is the issue here, compare the output of the following. Note that in 2 cases, when shell=True, only ls is executed since the input passed is a list and not a string, against the recommendation
subprocess.Popen(['ls']) #works
subprocess.Popen('ls') #works
subprocess.Popen(['ls', '-al']) #works
subprocess.Popen(['ls -al']) #doesn't work raises OSError since not a single command
subprocess.Popen('ls -al') #doesn't work raises OSError since not a single command
subprocess.Popen(['ls -al'], shell=True) #works since in shell mode
subprocess.Popen('ls -al', shell=True) #works since in shell mode & string is single command
subprocess.Popen(['ls', '-al'], shell=True) #output corresponds to ls only, list passed instead of string, against recommendation
subprocess.Popen(['ls', '-al', '*']) #doesn't work because of shell globbing for *
subprocess.Popen(['ls -al *']) #doesn't work raises OSError since not a single commandfor *
subprocess.Popen('ls -al *') #doesn't work raises OSError since not a single commandvalid arg
subprocess.Popen(['ls', '-al', '*'], shell=True) #output corresponds to ls only, list passed instead of string, against recommendation
subprocess.Popen(['ls -al *'], shell=True) #works
subprocess.Popen('ls -al *', shell=True) #works

Not a direct answer to your question, but you can also try using the python library sh
example:
from sh import ls
print ls("-al")
link to more examples