I'm having trouble using mutilple subprocess calls back to back.
These 2 work fine:
subprocess.call(["gmake", "boot-tilera"], cwd="/home/ecorbett/trn_fp")
p = subprocess.Popen(["gmake", "run-tilera"], stdout=subprocess.PIPE, cwd="/home/ecorbett/trn_fp")
However, I get an error when I try to run this call directly after:
time.sleep(10)
subprocess.call(["./go2.sh"], cwd="/home/ecorbett/trn_fp/kem_ut")
I added sleep in there because I need a few seconds before I run the "./go2.sh" program. Not sure if that is the issue.
Any advice?
A possible reason why your shell script is working on the command-line is that the shebang line was not written correctly (or not written at all). See an example in which the script would work from a command line but not as a Python subprocess: Is this the right way to run a shell script inside Python?
If your shell script did not have a shebang line specified, it would work from command line because $SHELL is set in your environment and the script is taking that as a default. When running from a python subprocess, python does not know what it is and fails with OSError: [Errno 8] Exec format error. The subprocess.call() to gmake worked because it is a binary program and not a shell script. Using the argument shell=True gave an instruction to interpret the argument exactly as it would in a shell.
However, be careful about using shell=True in subprocess.call() as it may be insecure in some cases: subprocess Python docs.
Related
I am trying to write a python script that will execute a bash script I have on my Windows machine. Up until now I have been using the Cygwin terminal so executing the bash script RunModels.scr has been as easy as ./RunModels.scr. Now I want to be able to utilize subprocess of Python, but because Windows doesn't have the built in functionality to handle bash I'm not sure what to do.
I am trying to emulate ./RunModels.scr < validationInput > validationOutput
I originally wrote this:
os.chdir(atm)
vin = open("validationInput", 'r')
vout = open("validationOutput", 'w')
subprocess.call(['./RunModels.scr'], stdin=vin, stdout=vout, shell=True)
vin.close()
vout.close()
os.chdir(home)
But after spending a while trying to figure out why my access was denied, I realized my issue wasn't the file permissions but the fact that I was trying to execute a bash file on Windows in general. Can someone please explain how to execute a bash script with directed input/output on windows using a python script?
Edit (Follow up Question):
Thanks for the responses, I needed the full path to my bash.exe as the first param. Now however, command line calls from within RunModels.scr come back in the python output as command not found. For example, ls, cp, make. Any suggestions for this?
Follow up #2:
I updated my call to this:
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
The error I now get is /usr/bin/bash: RunModels.scr: No such file or directory.
Using cwd does not seem to have any effect on this error, either way the subprocess is looking in /usr/bin/bash for RunModels.scr.
SELF-ANSWERED
I needed to specify the path to RunModels.scr in the call as well as using cwd.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'C:\\path\\dir_where_RunModels\\RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
But another problem...
Regardless of specifying cwd, the commands executed by RunModels.scr are throwing errors as if RunModels.scr is in the wrong directory. The script executes, but cp and cd throw the error no such file or directory. If I navigate to where RunModels.scr is through the command line and execute it the old fashioned way I don't get these errors.
Python 3.4 and below
Just put bash.exe in first place in your list of subprocess.call arguments. You can remove shell=True, that's not necessary in this case.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Depending on how bash is installed (is it in the PATH or not), you might have to use the full path to the bash executable.
Python 3.5 and above
subprocess.call() has been effectively replaced by subprocess.run().
subprocess.run(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Edit:
With regard to the second question, you might need to add the -l option to the shell invocation to make sure it reads all the restart command files like /etc/profile. I presume these files contain settings for the $PATH in bash.
Edit 2:
Add something like pwd to the beginning of RunModels.scr so you can verify that you are really in the right directory. Check that there is no cd command in the rc-files!
Edit 3:
The error /usr/bin/bash: RunModels.scr: No such file or directory can also be generated if bash cannot find one of the commands that are called in the script. Try adding the -v option to bash to see if that gives more info.
A better solution than the accepted answer is to use the executable keyword argument to specify the path to your shell. Behind the curtain, Python does something like
exec([executable, '-c`, subprocess_arg_string])
So, concretely, in this case,
subprocess.call(
'./RunModels.scr',
stdin=vin, stdout=vout,
shell=True,
executable="C:/cygwin64/bin/bash.exe")
(Windows thankfully lets you use forward slashes instead of backslashes, so you can avoid the requirement to double the backslashes or use a raw string.)
I'm trying to capture the output of a bash line I'm executing in my python script using the following code:
call_bash = subprocess.Popen(['/space/jazz/1/users/gwarner/test.sh'],stdout=subprocess.PIPE)
output = call_bash.communicate()[0]
This works fine but I'd like to change it so that that bash code is written directly into this python script (eliminating the need for a separate .sh file). I know that I can run bash code from python with subprocess.call() but I can't figure out how to capture the output (what normally gets printed to the terminal incase I'm using the wrong terminology). Can anyone help?
I'm running python 2.7 on CentOS
The first argument to subprocess.Popen is a list of arguments (unless you specify shell=True).
So you could do your example like
call_bash = subprocess.Popen(['ssh', 'me#cli.globusonline.org', 'scp me#comp:/~/blah.text ep#comp:/register/'], stdout=subprocess.PIPE)
output = call_bash.communicate()[0]
This will invoke the ssh command with 2 arguments, me#cli.globusonline.org and scp me#comp:/~/blah.text ep#comp:/register/.
Yesterday, I wrote and ran a python script which executes a shell using subprocess.Popen(command.split()) where command is string which constitutes .sh script and its argument. This script was working fine until yesterday. Today, I ran the same script and now I am continuously hitting this error.
p=subprocess.Popen(shell_command.split())
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
I know there are similar questions that have been asked before related to this question, but in my case I tried everything which doesn't solve my purpose. Using shell=True does not work because my shell script calls an another shell script before which some environment has to be set in order to run that script. I am badly stuck in this. I just restart my system once. I am using ubuntu 12.04
EDIT:
import subprocess
import os
import sys
arg1=sys.argv[1]
arg2=sys.argve[2]
shell_command = 'my_path/my_shell.sh ' + arg1 + ' '+ arg2
P = subprocess.Popen(shell_command.split())
P.wait()
my_shell.sh:
arg1=$1
arg2=$2
cd $TOP
setup the environment and run shell script
build the kernel ...
execute shell command .....
I solved this by putting this line at the top of the called shell script:
#!/bin/sh
That will guarantee that the system always uses the correct interpreter when running your script.
Following statement worked for me
subprocess.Popen(['sh','my_script.sh'])
The error message suggests that the external program is not a valid executable.
As #tripleee said, there is an issue executing your script. Make sure:
Change the shell command to "./my_path/my_script.sh" or "/bin/bash my_path/my_script.sh". Account for environment variables, if necessary.
Both scripts have execute bit set (chmod +x)
The files exist at the location you think they do. (Use abspath or verify environment)
The files have contents
Try removing and re-typing the first line. I recommend killing the whole line, and hitting backspace several times in case there's a non-printable character before the #!
See also: Why is '#!/usr/bin/env python' supposedly more correct than just '#!/usr/bin/python'?
This can also happen if the binary is not meant to run on your system.
I'm on OSX, but the binary I was running is not meant for OSX, as I saw from using file path/to/binary:
webui/bin/wkhtmltopdf: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=b6566a9e44c43a0eebf18d8c1dc6cb616801a77e, stripped
The error is because the executables are not given in the prescribed format for subprocess to execute it.
example:
shell_command1 = r"/path/to/executable/shell/file"
shell_command2 = r"./path/to/executable/shell/file"
subprocess.call(shell_command1) or subprocess.Popen(shell_command1) will not be able to run shell_command1 because subprocess needs an executable command like (mailx, ls, ping, etc.) or executable script like shell_command2 or you need to specify command like this
subprocess.call(["sh", shell_command1])
subprocess.Popen(["sh", shell_command1])
but however, you can use os.system() or os.Popen() as well
I am currently facing the same issue. I noticed that using shell=True argument, like subprocess.Popen(shell_command.split(), shell=True) works as inteded.
It is recommended to install the package binfmt-support to help the system better recognize the scipts. It helps regardless of whether they have a shebang line.
I need to execute the simple command below in windows 7 command prompt using Python26.
cd C:\Python26\main project files\Process
C:\Aster\runtime\waster Analysis.comm
It runs a FEM simulation and I tried it manually and it worked well. Now, I want to automate the write procedure using Python26.
I studied the other questions and found that the os.system works but it didn't. Also I saw subprocess module but it didn't work.
The current directory is a process property: Every single process has its own current directory. A line like
os.system("cd xyz")
starts a command interpreter (cmd.exe on Windows 7) and execute the cd command in this subprocess, not affecting the calling process in any way. To change the directory of the calling process, you can use os.chdir() or the cwd keyword parameter to subprocess.Popen().
Example code:
p = subproces.Popen(["C:/Aster/runtime/waster", "Analysis.comm"],
cwd="C:/Python26/main project files/Process")
p.wait()
(Side notes: Use forward slashes in path names in Python files. You should avoid os.system() and passing shell=True to the function in the subprocess module unless really necessary.)
I have written a nice python script that parses XML and adds some sophisticated logic to then interface with an external command via subprocess module.
Most of the subprocess.Popen calls do exactly what they're supposed to, but the last one simply refuses to execute. No error message , it just doesn't do what it's supposed to. I even put the actual CMD into a shell script surrounded by debugging statements, and the shell script gets executed, but not the actual CMD.
More infuriatingly, the very same line of code in a separate .py file executes just fine.
I have no idea why or how this could be?
The python code is generating a file and tries to invoke the external command with options
p = subprocess.Popen([CMD,'object','new_host','--file','/tmp/add.1234'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
r = p.communicate()
print r
this logic works in the standalone file, but not in the larger python script (which has other working Popen calls in it).
Does anybody have an idea why this could be?
PS: I can not update python to a more recent version