Python script : Running a script with multiple arguments using subprocess - python

My question is related to this earlier question - Python subprocess usage
I am trying to run this command using python
nccopy -k 4 "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]" foo.nc
When I run the above command I should be able to see a file called foo.nc on my disk or a network error stating unable to access that URL or remote URL not found.
Currently the ESRL NOAA server is down - so when I run the above command I get
syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: ^
NetCDF: Access failure
Location: file nccopy.c; line 1348
I should get the same error when I run the python script
This is the code I have and I am unable to figure out exactly how to proceed further -
I tried splitting up "-k 4" into two arguments and removing the quotes and I still get this error nccopy : invalid format : 4
Results of print(sys.argv) data.py
['data.py', '-k', '4', 'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][20:34][26:40]', 'foo.nc']
import numpy as np
import subprocess
import sys
url = '"http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]"'
outputFile = 'foo.nc'
arg1 = "-k 4"
arg3 = url
arg4 = outputFile
print (input)
subprocess.check_call(["nccopy",arg1,arg3,arg4])

There's two dilemmas here.
One being that subprocess processes your arguments and tries to use 4 as a separate argument.
The other being that system calls still goes under normal shell rules, meaning that parameters and commands will be parsed for metacharacters aka special characters. In this case you're wrapping [ and ].
There for you need to separate each parameters and it's value into separate objects in the parameter-list, for instance -k 4 should be ['-k', '4'] and you need to wrap parameters/values in '...' instead of "...".
Try this, shlex.split() does the grunt work for you, and i swapped the encapsulation characters around the URL:
import numpy as np
import subprocess
import sys
import shlex
url = "'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]'"
outputFile = 'foo.nc'
command_list = shlex.split('nccopy -k 4 ' + url + ' ' + outpufFile)
print(command_list)
subprocess.check_call(command_list)

Instead of arg1 = "-k 4", use two arguments instead.
import subprocess
url = 'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[408:603][2][20:34][26:40]'
outputFile = 'foo.nc'
arg1 = "-k"
arg2 = "4"
arg3 = url
arg4 = outputFile
subprocess.check_call(["nccopy", arg1, arg2, arg3, arg4])
See also here Python subprocess arguments

If you have a working shell command that runs a single program with multiple arguments and you want to parameterized it e.g., to use a variable filename instead of the hardcoded value then you could use shlex.split() to create a list of command-line arguments that you could pass to subprocess module and replace the desired argument with a variable e.g.:
>>> shell_command = "python -c 'import sys; print(sys.argv)' 1 't w o'"
>>> import shlex
>>> shlex.split(shell_command)
['python', '-c', 'import sys; print(sys.argv)', '1', 't w o']
To run the command using the same Python interpreter as the parent script, sys.executable could be used and we can pass a variable instead of '1':
#!/usr/bin/env python
import random
import sys
import subprocess
variable = random.choice('ab')
subprocess.check_call([sys.executable, '-c', 'import sys; print(sys.argv)',
variable, 't w o'])
Note:
one command-line argument per list item
no shlex.split() in the final code
there are no quotes inside 't w o' i.e., 't w o' is used instead of '"t w o"' or "'t w o'"
subprocess module does not run the shell by default and therefore you don't need to escape shell meta-characters such as a space inside the command-line arguments. And in reverse, if your command uses some shell functionality (e.g., file patterns) then either reimplement the corresponding features in Python (e.g., using glob module) or use shell=True and pass the command as a string as is. You might need pipes.quote(), to escape variable arguments in this case. Wildcard not working in subprocess call using shlex

Related

Python: get last command line in linux terminal

I would like to write a python script which access the last command executed in terminal, i.e the command that launched the program.
For example, I want the terminal to output 'python myfile.py' if i typed python myfile.py
First I tried:
import os
os.system("touch command.txt")
os.system("history > command.txt")
with open("command.txt", "r") as f:
command = f.read()[-1]
print(command)
but this is not working since history is a bash built-in function.
Then I tried :
import os, subprocess
command = subprocess.check_output(["tail","-n","1",os.path.expanduser("~/.bash_history")]).decode("utf-8").rstrip()
print(command)
but this does not meet my expectations, because bash history is only updated when the terminal is closed.
To improve this behavior I tried os.putenv("PROMPT_COMMAND", "history-a"), but it didn't help neither, because bash history update is still one step behind, as my variable command would now contain the command line just before python myfile.py
Now I'm stuck and I need your help pls
You can't get the original shell command line in a reliable way without participation of the shell itself, but you can generate an equivalent command line using sys.argv. (It won't include things like redirections, but if you're just re-executing from inside the existing copy of the program, all those executions will have been already performed before you're started anyhow, so when you re-exec yourself the new copy will inherit their effect).
So:
#!/usr/bin/env python
import os.path, sys
try:
from shlex import quote # Python 3
except ImportError:
from pipes import quote # Python 2
sys_argv_str = ' '.join(quote(x) for x in sys.argv)
print("We can be restarted by calling the argv: %r" % (sys.argv,))
print("As a shell-syntax string, that would be: %s" % (sys_argv_str,))
print("...or, if your shell is bash, you can specify the interpreter directly:")
print(' ' + ' '.join(quote(x) for x in (['exec', '-a', sys.argv[0], os.path.abspath(sys.executable), os.path.abspath(__file__)] + sys.argv[1:])))
If someone calls ./yourprogram "first argument" "second argument", that output might look like:
We can be restarted by calling the argv: ['./yourprogram', 'first argument', 'second argument']
As a shell-syntax string, that would be: ./yourprogram 'first argument' 'second argument'
...or, if your shell is bash, you can specify the interpreter directly:
exec -a ./yourprogram /usr/bin/python /home/charles/tmp/yourprogram 'first argument' 'second argument'
Note that argv[0] is not guaranteed to be identical to __file__! When a program is starting another program it can pass any string it likes in the argv[0] slot; it's merely convention, not a firm guarantee, that that will contain the name that was used to start the software at hand.

how to pass string path in subprocess command line?

So i have a code where i use the subprocess.popen method, i want to pass a string path on settings but i dont know how to do it.
there is my code :
screenCommand = rep_reference+"\osgedit\osgviewer.exe "
iterationsPath = str(path + "\sreplace.osg")
command = "{0} --screen 0 --window 0 0 1920 1080 path+\sreplace.osg {1} " .format(screenCommand, listFix[e])
process = subprocess.Popen(command)
time.sleep(3.0)
process.kill()
how to path the iterationsPath variable on my command settings ?
You pass the various arguments and options in a list along with the command.
For example, to open VSCode in the current directory, in the shell you would do:
$ code .
Using subprocess.Popen() (or subprocess.call()):
>>> import subprocess
>>> subprocess.Popen(['code', '.'])
>>>
Essentially you split the shell command by spaces.
From the docs
On POSIX, if args is a string, the string is interpreted as the name
or path of the program to execute. However, this can only be done if
not passing arguments to the program.
https://docs.python.org/3/library/subprocess.html#subprocess.Popen

python argparse how to get entire command as string

I have a script named patchWidth.py and it parses command line arguments with argparse:
# read command line arguments -- the code is able to process multiple files
parser = argparse.ArgumentParser(description='angle simulation trajectories')
parser.add_argument('filenames', metavar='filename', type=str, nargs='+')
parser.add_argument('-vec', metavar='v', type=float, nargs=3)
Suppose this script is run with the following:
>>> python patchWidth.py file.dat -vec 0. 0. 1.
Is there a way to get this entire thing as a string in python? I would like to be able to print to the output file what command was run with what arguments.
Yes, you can use the sys module:
import sys
str(sys.argv) # arguments as string
Note that argv[0] is the script name. For more information, take a look at the sys module documentation.
I do not know if it would be the best option, but...
import sys
" ".join(sys.argv)
Will return a string like /the/path/of/file/my_file.py arg1 arg2 arg3
This will work with commands that have space-separated strings in them.
import sys
" ".join("\""+arg+"\"" if " " in arg else arg for arg in sys.argv)
Sample output:
$ python3 /tmp/derp.py "my arg" 1 2 3
python3 /tmp/derp.py "my arg" 1 2 3
This won't work if there's a string argument with a quotation mark in it, to get around that you'd have to delimit the quotes like: arg.replace("\"", "\\\""). I left it out for brevity.

Forward a shell command using python

So to be more precise, what I am trying to do is :
read a full shell command as argument of my python script like : python myPythonScript.py ls -Fl
Call that command within my python script when I'd like to (Make some loops on some folders and apply the command etc ...)
I tried this :
import subprocess
from optparse import OptionParser
from subprocess import call
def execCommand(cmd):
call(cmd)
if __name__ == '__main__':
parser = OptionParser()
(options,args) = parser.parse_args()
print args
execCommand(args)
The result is that now I can do python myPythonScript.py ls , but I don't know how to add options. I know I can use parser.add_option , but don't know how to make it work for all options as I don't want to make only specific options available, but all possible options depending on the command I am running.
Can I use something like parser.add_option('-*') ? How can I parse the options then and call the command with its options ?
EDIT
I need my program to parse all type of commands passed as argument : python myScript.py ls -Fl , python myScript.py git pull, python myScript rm -rf * etc ...
OptionParser is useful when your own program wants to process the arguments: it helps you turn string arguments into booleans or integers or list items or whatever. In your case, you just want to pass the arguments on to the program you're invoking, so don't bother with OptionParser. Just pass the arguments as given in sys.argv.
subprocess.call(sys.argv[1:])
Depending on how much your program depends on command line arguments, you can go with simple route.
Simple way of reading command line arguments
Use sys to obtain all the arguments to python command line.
import sys
print sys.argv[1:]
Then you can use subprocess to execute it.
from subprocess import call
# e.g. call(["ls", "-l"])
call(sys.argv[1:])
This sample below works fine for me.
import sys
from subprocess import call
print(sys.argv[1:])
call(sys.argv[1:])

Failed to run shell commands with subprocess [duplicate]

This question already has answers here:
Why subprocess.Popen doesn't work when args is sequence?
(3 answers)
Closed 6 years ago.
In my terminal if I run: echo $(pwd), I got /home/abr/workspace, but when I tried to run this script in python like this:
>>> import subprocess
>>> cmd = ['echo', '$(pwd)']
>>> subprocess.check_output(cmd, shell=True)
I get '\n'. How to fix this?
Use os package:
import os
print os.environ.get('PWD', '')
From the documentation on the subprocess module:
If args is a sequence, the first item specifies the command string,
and any additional items will be treated as additional arguments to
the shell itself.
You want:
subprocess.check_output("echo $(pwd)", shell=True)
Try this:
cmd = 'echo $(pwd)'
subprocess.check_output(cmd, shell=True)
In subprocess doc it specified that cmd should be a string when shell=True.
From the documentation:
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence.
A better way to achieve this is probably to use the os module from the python standard library, like this:
import os
print os.getcwd()
>> "/home/abr/workspace"
The getcwd() function returns a string representing the current working directory.
The command subpreocess.check_output will return the output of the command you are calling:
Example:
#echo 2
2
from python
>>>subprocess.check_output(['echo', '2'], shell=True)
>>>'2\n'
the '\n' is included because that is what the command does it prints the output sting and then puts the current on a new line.
now back to your problem; assuming you want the output of 'PWD', first of all you have to get rid of the shell. If you provide the shell argument, the command will be run in a shell environment and you won't see the returned string.
subprocess.check_output(['pwd'])
Will return the current directory + '\n'
On a personal note, I have a hard time understanding what you are trying to do, but I hope this helps solve it.

Categories