have a small issue with a program I am trying to launch from a Python script via Popen() (I understand Popen() may not be ideal, but I am working with somewhat of a template used in other instances, and want to follow convention).
I am a bit confused, as I can't seem to get the following to run:
root = os.getcwd()
bin = 'my_executable.exe'
bin_fullpath = os.path.join(root,bin)
params = 'Option C -f Module -y -q'
p = subprocess.Popen([bin_fullpath,params])
out = p.communicate()
The program launches, but exits with error code 1 (I checked with check_call).
However, when I forgo the above method, and simply provide the entire string I need to run, as follows:
subprocess.Popen(r'C:\Users\me\Desktop\path\to\tool\my_executable.exe Option C -f Module -y -q')
The program executes as expected. Obviously I have something wrong with the sytntax, but I can't figure out what . . .
Any insight would be greatly appreciated!
When you use the "list of arguments" format, each one has to be its own string, as a separate member of the list, like this:
params = ['Option', 'C', '-f', 'Module', '-y', '-q']
p = subprocess.Popen([bin_fullpath, *params])
When you put them all in a single string, you're telling subprocess they're all one big argument.1 So, rather than being the rough equivalent of this command line:
C:\Users\me\Desktop\path\to\tool\my_executable.exe Option C -f Module -y -q
… it's the rough equivalent of this one:
C:\Users\me\Desktop\path\to\tool\my_executable.exe "Option C -f Module -y -q"
If it's not clear why those are different, consider these examples instead:
fix.exe "My Pictures\picture1.jpg"
fix.exe My Pictures\picture1.jpg
The first one is fixing one picture, My Pictures\picture1.jpg. The second is fixing two pictures, My, and Pictures\picture1.jpg.
For more details on the args argument, see Frequently Used Arguments. Notice the "one big string" version is actually not valid without shell=True—even though it happens to usually work on Windows.
1. Things are a little more complicated than this on Windows, because subprocess actually has to take all of the args and work out how to put them together in a string so that they can be parsed back into the actual separate values the way you asked for them. But never mind that.
Related
I am trying to use Python to run an executable (Windows 7) with parameters. I have been able to make the program run, but the amount of parameters I can use that will prove the Python script worked with parameters is limited. The best one is formatted like so:
-debugoutput debug.txt
I have tested this using a windows shortcut with an edited target and it works, it creates a debug output in the program directory.
Here is the code I am using:
import subprocess
args = [r"C:\Users\MyName\LevelEditor\LevelEditor.exe", "-debugoutput debug.txt"]
subprocess.call(args)
This does run the program, but the debug output is not created. I have tried putting an "r" in front of the parameter but this made no difference. I assume it is a simple formatting error but I can't find any examples to learn from that are doing the same thing.
UPDATE:
Thanks for the answers everyone, all the same, simple formatting error indeed.
In-code definition results in invocation of shell command line:
C:\Users\MyName\LevelEditor\LevelEditor.exe "-debugoutput debug.txt"
As you can see, by merging -debugoutput debug.txt to single list element, you explicitly stated that space between them shouldn't be parsed as command line argument separator.
To achieve expected behavior put file name string as separate element to argument list.
[r"C:\Users\MyName\LevelEditor\LevelEditor.exe", "-debugoutput", "debug.txt"]
As far as I know you need to split the arguments by the space, so your args would look like:
args = [r"C:\Users\MyName\LevelEditor\LevelEditor.exe", "-debugoutput", "debug.txt"]
Does that work?
I do not know if it works, but
import subprocess
args = [r"C:\Users\MyName\LevelEditor\LevelEditor.exe", "-debugoutput", "debug.txt"]
subprocess.run(args)
Following the docs
I need to execute a command line in the bakground in python 2.7. I need to fire and forget.
Here is the command:
cmd = "/usr/local/bin/fab -H %s aws_bootstrap initial_chef_run:%s,%s,%s -w" % (...)
How do I use the subproccess module?
e.g. is it
subprocess.call([cmd])
or
subprocess.call(["/usr/local/bin/fab", "-H %s aws_bootstrap initial_chef_run:%s,%s,%s -w"])
I dont get how to use the list. Or is every element of the list what would be a white space.
Thanks
each thing that would be seperated by whitespace is a seperate entity of the list
subprocess.call is blocking however
subprocess.popen is non-blocking
cmd = ["/usr/local/bin/fab", "-H",var1,"aws_bootstrap initial_chef_run:%s,%s,%s"%(var2,var3,var4), "-w"]
subprocess.popen(cmd) # dopnt wait just keep going
#or
subprocess.call(cmd) # wait until the command returns
you may however alternatively pass the command as one big string
cmd = "/usr/local/bin/fab -H %s aws_bootstrap initial_chef_run:%s,%s,%s -w" % (...)
subprocess.call(cmd)
in general this method(passing a single string) is frowned upon for some reason that has never been explained sufficiently to me
I used this recently to fire a perl script, like so:
var = "C:\Users\user\Desktop"
retcode = subprocess.call(["perl", '.\hgncDL.pl',var])
Working code
Define hParam and runParams in following code and you're good to go:
hParam = 'hParam'
runParams = (a,b,c)
args = ('/usr/local/bin/fab', '-H', hParam, 'aws_bootstrap', 'initial_chef_run:%s,%s,%s' % runParams, '-w')
subprocess.Popen(args)
Details
How do I use <any python module> module?
https://docs.python.org is a good starting point.
In particular, docs for subprocess module available here.
I can't provide direct links for each case later in this answer due to restriction imposed by low reputation. Each time I will be referring to 'docs', look for a section in docs on the module.
I need to execute a command line in the background in python 2.7. I need to fire and forget
Consider subprocess.Popen(args). Note capital 'P'.
See docs for more details.
subprocess.call(args) works in similar way, but it would block until the command completes. As stated in docs:
Run the command described by args. Wait for command to complete, then return the returncode attribute.
How to use the sequence form of args parameter?
This is covered in "Frequently used arguments" section of docs:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).
Also, passing an args in a string form has its limitation:
If passing a single string, either shell must be True or else the string must simply name the program to be executed without specifying any arguments.
Despite mentioned limitation, subprocess.Popen('cmd.exe /?') works for me. Win7, Python 2.7.8 64bit.
HTH, cheers.
I did hours on research regarding the following question but I wasn't able to find an answer at all. Though there seem to be many fellows having problems with that. I hope I will recieve some help from the community. ;)
I have a Cshell script where I need to call a Python3 script from. Also I am passing a variable.
.csh
#!/bin/csh -f
set variable = value
/../geos.py $variable
So far so fine. In my Python3 script I take this variable, do some calculations and now want to pass back the 'new_variable' to the VERY SAME C shell script in order to proceed my set of data.
.py
import os
...
new_variable = 'foobar'
os.environ['new_variable'] = new_variable
return new_variable
My actual goal is that my C Shell script:
#!/bin/csh -f
set variable = value
/../geos.py $variable
echo $new_variable
doesn't return 'Undefined variable'. So obviously my code doesn't work. Sure, I might be able to temporarily save the python calculations into a file but this seems quite unconvincingly. Also, I understand that it is just not possible to manipulate an environmental variable of the shell through a child process, but still I only want to pass a normal variable. There should be one way, no?
If it is possible, I wasn't able to figure out any solution using subprocess.check_call. What am I missing?
E D I T:
Merci beaucoup.
I knew that there must have been an easy solution. Thanks a lot!
For CSHELL the following code worked:
set new_variable=`../geos.py $variable`
echo $new_variable
For BASH the following code worked:
new_variable=`../geos.py $variable`
echo $new_variable
In the python script itself you don't need to do anything but putting your desired variable into standard output, e.g. print(you_even_can_name_them_as_you_want). No os.environ oo whatever necessary. Made my day. SOLVED
in bash I'd use:
new_variable=$(../geos.py $variable)
Have the python script produce the new value as standard out (i.e. print(new_variable) )
In csh I don't know, maybe you would have to use backquotes instead of $() ?
Question can be related to Use python subprocess module like a command line simulator
I have written some infrastructure code called my_shell to which you can pass shell commands of my application that looks like this
class ApplicationTestShell(object):
def __init__(self):
'''
Constructor
'''
self.play_ground_dir = "/var/tmp/MyAppDir"
ensure_dir_exists_and_empty(self.play_ground_dir)
def execute_command(self, command, on_success = None, on_failure = None):
p = create_shell_process(self, self.play_ground_dir)
sout, serr = p.communicate(input = command)
if p.returncode == 0:
on_success(sout)
else:
on_failure(serr)
def create_shell_process(self, cwd):
return Popen("/bin/bash", env= {WHAT DO I DO HERE?},cwd = test_dir, stdout=PIPE, stderr=PIPE, stdin=PIPE)
The interesting bit to me here is the env parameter. Python expects like a 'map' datastructure of all environment variable. My application requires several variables exported and set. The script for setting and exporting is generated by running say '/bin/appload myapp' (Assume appload is always available on the path). What I do currently
is when I call p.communicate I do the following
p.communicate(input = "eval `/bin/appload myapp`;" + command)
So basically before running the command I call the infrastructure setup.
Is there any way to do this in a better fashion in Python. I somehow want to push the eval /bin/appload part to the env parameter on the Popen class OR as part of the shell creation process.
What are the problems with my current implementation? (I feel it is hacky but I may be wrong)
It depends on how /bin/appload myapp works. If it only guarantees that it will output bash syntax, then parsing that output in Python in order to construct the environment object there is almost certainly more trouble than it's worth (you might need to support parameter and variable expansion, subshells, process substitution, etc, etc). On the other hand, if you are sure that /bin/appload myapp will only ever output lines of the form "VARIABLENAME=someword", then that's pretty trivial to parse in Python and you could move it into your Python code if you like.
There are an awful lot of different directions you could go with these requirements; you could capture the output of appload myapp into a tempfile and set the subprocess's $BASH_ENV to that filename; that would cause the shell to source your environment setup before running your command in a way that some might consider cleaner. You could give your command (with the eval-ing prefix) as the first argument to Popen and pass shell=True, and let Popen do the bash invocation on its own (setting $SHELL explicitly to bash if necessary). You could use bash's -c option to specify the code to run on the command line rather than via stdin. You could have a multi-tiered approach by invoking a shell from Python which eval's the appload myapp environment and then exec's another shell underneath it, so that the first doesn't show up in ps listings and the command given to create_shell_process has the shell all to itself (although that shouldn't really matter). You could do a lot of things, depending on what your concerns are with respect to how the shell is invoked, how it looks in ps listings, whether you want your command to still be run if the appload myapp output produces an error when eval'd, etc. But for a general solution, I think what you have is perfectly fine.
I don't see any real problems with the implementation, besides cosmetic things or minor things that probably only came from copying and pasting the code: create_shell_process doesn't use its cwd parameter, and the on_success and on_failure parameters look like they're optional but the defaults will break things (you can't call None).
friends,
I have a simple script
import subprocess
subprocess.call(["./run_xf"])
old=open('./inv.mt0','r')
lines=old.readlines()
lines=lines[3:]
new=open('./inv.mt1','w')
new.writelines(lines)
old.close()
new.close()
subprocess.call(["rm", "inv.mt0"], shell=True)
All the codes work except the last one.
run_xf runs hspice and generate inv.mt0. then i copy part of inv.mt0 to inv.mt1. then i want to delete inv.mt0. But this doesn't work.
For this specific example, it complains rm can't find operand. But if i write them together, it doesn't delete the file as well.
thanks
xf
If you are using shell=True you must pass a string to subprocess.call, not a list. See http://docs.python.org/library/subprocess.html#subprocess.call for more details.
However, invoking subprocess with shell=True is not recommended due to security implications. You should remove shell=True and leave the list-style args.