I'm trying to call the SLURM squeue from a python script. The command,
/usr/bin/squeue --Format=username,jobid,name,timeleft
Works fine from the command line, but fails from subprocess.Popen with:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/n/home00/DilithiumMatrix/.conda/envs/py35/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/n/home00/DilithiumMatrix/.conda/envs/py35/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/squeue --Format=username,jobid,name,timeleft'
MWE:
import subprocess
command = "/usr/bin/squeue --Format=username,jobid,name,timeleft"
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
text = p.stdout.read()
print(text)
/usr/bin/squeue works fine from both the command line or Popen.
Could it be failing because it requires some information about the user/group that's executing the squeue command and that is (somehow) lost when running via python? What else could be causing this?
The first argument to subprocess.Popen is either a String, or a list of Strings. If it is a single String, it will be interpreted as a filename. This is the reason for the error you get.
To pass a list of Strings, it should match how a shell would pass your arguments to the process. A standard shell will split your command line by whitespace, so instead of this:
command = "/usr/bin/squeue --Format=username,jobid,name,timeleft"
You need this:
command = ["/usr/bin/squeue", "--Format=username,jobid,name,timeleft"]
Splitting the second argument at the "=" as you mentioned in your comment just confuses squeue, which would then see two arguments.
Related
I made an attempt at parsing running programs in my computer (Debian OS) with the Subprocess python module. Here is my code:
import subprocess
cmd = "ps -A" # Unix command to get running processes
runningprox = subprocess.check_output(cmd) #returns output as byte string
rpstring = runningprox.decode("utf-8")
#converts byte string to string and puts it in a variable
print(rpstring)
However, when I run the code, I get the following error message:
Traceback (most recent call last): File "ratalert.py", line 6, in
runningprox = subprocess.check_output(cmd) #returns output as byte string File "/usr/local/lib/python3.6/subprocess.py", line 336, in
check_output
**kwargs).stdout File "/usr/local/lib/python3.6/subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process: File "/usr/local/lib/python3.6/subprocess.py", line 707, in init
restore_signals, start_new_session) File "/usr/local/lib/python3.6/subprocess.py", line 1333, in _execute_child
raise child_exception_type(errno_num, err_msg) FileNotFoundError: [Errno 2] No such file or directory: 'ps -A'
I don't understand why I get this error message. Considering 'ps -A' is neither a file nor a directory, but just the Unix command I put in a variable as a string.
How can I fix this? Thank you.
The subprocess.check_output function expects a list with the command and its arguments, like so:
runningprox = subprocess.check_output(['ps', '-A'])
Otherwise, it will treat the string you passed as a single command, and will look up for an executable file with name ps -A, space included and all.
You can use shlex to do the splitting as the shell would do:
import shlex, subprocess
cmd = 'ps -A'
runningprox = subprocess.check_output(shlex.split(cmd))
I am trying to list the contents of an s3 bucket using Python 2.7.13. This does not work:
>>> args
['aws', 's3', 'ls', 's3://mybucket']
>>> p = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Error is:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\subprocess.py", line 711, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 959, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Why is this so ?
But this works:
>>> p = subprocess.Popen(args, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Why is shell=False failing but shell=True working ?
You are totally right to try to use subprocess with shell=False. It is the best way to ensure portability, and is probably faster to launch.
In your case, your arguments look okay (no redirection, no pipe, multiple command):
['aws', 's3', 'ls', 's3://mybucket']
so the only thing that prevents it to work with shell=False is the fact that aws is not really an executable, but rather a file whose extension (not shown here) is contained in PATHEXT
On my machine PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY
Means that any aws.js, aws.bat, ... file can be executed. But you need the shell for that.
To locate your program, type where aws in a command prompt, you'll get the full path & extension of the command.
If you don't want to use shell=True, there's an alternative which amounts to the same thing
args = ['cmd','/c','aws', 's3', 'ls', 's3://mybucket']
Since you're already running a cmd you don't need shell=True
I am running into following error when trying to run a command using Popen,what is wrong here?
cmd = "export COMMANDER_SERVER=commander.company.com"
Pipe = Popen(cmd, stdout=PIPE, stderr=PIPE)
(output, error) = Pipe.communicate()
Error:-
Traceback (most recent call last):
File "test_ectool.py", line 26, in <module>
main()
File "test_ectool.py", line 13, in main
Pipe = Popen(cmd, stdout=PIPE, stderr=PIPE)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
You need to separate the arguments from the command and give a list to Popen.
As Kenster's comment said, even if your command worked, you would only succeed in modifying an environmental variable inside a sub-shell not the main shell.
You will not be able run run export this way, because it is not a program. It is a bash built-in.
Here is an example of a command that does work, with the correct semantics.
from subprocess import Popen,PIPE
cmd = "echo COMMANDER_SERVER=commander.company.com"
Pipe = Popen(cmd.split(), stdout=PIPE, stderr=PIPE)
(output, error) = Pipe.communicate()
print output
merlin2011's answer is incorrect regarding the command string for Popen (point #1). From the Python docs:
args should be a sequence of program arguments or else a single string.
As other people have stated, the environment variable will not be saved. For that, you need to use os.environ['VARNAME'] = value. If you want to use other bash builtins, then you must pass shell=True as an argument to Popen.
I have written a script that checks if an SVN Repo is up and running, the result is based on the return value.
import subprocess
url = " validurl"
def check_svn_status():
subprocess.call(['svn info'+url],shell=True)
def get_status():
subprocess.call('echo $?',shell=True)
def main():
check_svn_status()
get_status()
if __name__ == '__main__':
main()
The problem I'm facing is that if I change the url to something that does't exist I still get the return value as 0, but if I were to run this outside the script, i.e go to the terminal type svn info wrong url and then do a echo $? I get a return value of 1. But I can't re-create this in the python. Any guidelines ?
TraceBack after updating
Traceback (most recent call last):
File "svn_status.py", line 21, in <module>
main()
File "svn_status.py", line 15, in main
check_svn_status()
File "svn_status.py", line 8, in check_svn_status
p = sp.Popen(['svn info'], stdout=sp.PIPE, stderr=sp.PIPE)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1202, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or director
y
Why your approach does not work:
You invoke two independent subshells. The second shell does not know of the first shell and therefore does not have any information about the returncode of the process that was executed in the first shell.
Solution:
Use the subprocess module, spawn your subprocess directly (not through a subshell) and retrieve the returncode. Help yourself by reading the documentation of the module: http://docs.python.org/library/subprocess.html
There are several ways to achieve your goal. One simple way could be:
import subprocess as sp
p = sp.Popen(['command', 'arg1', 'arg2'], stdout=sp.PIPE, stderr=sp.PIPE)
stdout, stderr = p.communicate()
returncode = p.returncode
This way, you don't go through a subshell (shell=False by default), which is the recommended approach for various reasons. You directly catch the returncode of the spawned subprocess and you have full access to the subprocess' standard output and standard error.
subprocess.call returns the retcode, just store the result of your subprocess.call(['svn info'+url],shell=True)
http://docs.python.org/library/subprocess.html
I have already read the previous questions posted on the same argument but I really haven't figured it out yet.
I am trying to run a command that works without issues from the command line :
xyz#klm:~/python-remoteWorkspace/PyLogParser/src:18:43>ush -o PPP -p PRD -n log 'pwd'
6:43PM PPP:prd:lgsprdppp:/ama/log/PRD/ppp
but when I do the same in python I always get errors :
stringa = Popen(["ush -o PPP -p PRD -n log 'pwd'"], stdout=PIPE, stdin=PIPE).communicate()[0]
Here the error.
Traceback (most recent call last): File "getStatData.py", line 134, in <module>
retrieveListOfFiles(infoToRetList) File "getStatData.py", line 120, in retrieveListOfFiles
stringa = Popen(["ush -o PPP -p PRD -n log 'pwd'"], stdout=PIPE, stdin=PIPE).communicate()[0] File "/opt/python-2.6-64/lib/python2.6/subprocess.py", line 595, in __init__
errread, errwrite) File "/opt/python-2.6-64/lib/python2.6/subprocess.py", line 1092, in _execute_child
raise child_exception OSError: [Errno 2] No such file or directory
I've tried also different solutions like
stringa = Popen(["ush", "-o", "PPP", "-p" "PRD", "-n", "log", '"pwd"'], stdout=PIPE, stdin=PIPE).communicate()[0]
but nothing seems to work. I have also tried to put the absolute path to ush but nothing...
Can somebody please explain me what am I doing wrong ?
Thanks in advance, AM.
EDIT :
I have a strange thing happening, when I do
which ush
I get
ush: aliased to nocorrect /projects/aaaaaaa/local/ush/latest/ush.py
But why is it working then ???
!!! Thank you all for the answers !!!
Popen(["ush", "-o", "PPP", "-p", "PRD", "-n", "log", "pwd"])
should be right. The extra quoting around 'pwd' in the shell command makes it a single argument, but the quotes aren't actually passed along. Since you're already splitting the arguments, leave the extra quotes out.
Apparently (in an update from OP) ush is a shell alias. Thus, it only expands in the shell; anywhere else, it won't work. Expand it yourself:
Popen(["nocorrect", "/projects/aaaaaaa/local/ush/latest/ush.py",
"-o", "PPP", "-p", "PRD", "-n", "log", "pwd"])
If ush on your system is an alias, popen won't work. popen requires an executable file as the first parameter: either an absolute path or the name of something that is in your PATH.