We Keep Coding

python subprocess check_output

I am try to execute below command using python subprocess but it is failing.
Please help
import subprocess
cmd = "bash /opt/health_check -t 2>/dev/null"
retcode = subprocess.call([cmd])
print retcode
I am getting below output:
Traceback (most recent call last):
File "./script.py", line 65, in <module>
retcode = subprocess.call([cmd])
File "/usr/lib64/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

If you call check_output with a list, you would need to tokenize the command yourself, as in:
import subprocess
cmd = ["bash", "/opt/health_check", "-t"]
retcode = subprocess.call([cmd])
print retcode
This doesn't use a shell so you can't use input redirection. If you really want to execute your command with a shell, then pass a string and set shell=True:
import subprocess
cmd = "bash /opt/health_check -t 2>/dev/null"
retcode = subprocess.call(cmd, shell=True)
print retcode

This is an incorrect way to call subprocess. A quick and dirty way to do it is to change it like:
subprocess.call(cmd, shell=True)
This will invoke the system's shell to execute the command, and you'll have access to all the goodies it provides. It's not something to use ligthly, however, so check security considerations from the docs before doing so.
Otherwise, you can simply provide the command as a list, like so:
subprocess.call(["bash", "/opt/health_check"])

subprocess error on windows while listing an s3 bucket

I am trying to list the contents of an s3 bucket using Python 2.7.13. This does not work:
>>> args
['aws', 's3', 'ls', 's3://mybucket']
>>> p = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Error is:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\subprocess.py", line 711, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 959, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Why is this so ?
But this works:
>>> p = subprocess.Popen(args, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Why is shell=False failing but shell=True working ?

You are totally right to try to use subprocess with shell=False. It is the best way to ensure portability, and is probably faster to launch.
In your case, your arguments look okay (no redirection, no pipe, multiple command):
['aws', 's3', 'ls', 's3://mybucket']
so the only thing that prevents it to work with shell=False is the fact that aws is not really an executable, but rather a file whose extension (not shown here) is contained in PATHEXT
On my machine PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY
Means that any aws.js, aws.bat, ... file can be executed. But you need the shell for that.
To locate your program, type where aws in a command prompt, you'll get the full path & extension of the command.
If you don't want to use shell=True, there's an alternative which amounts to the same thing
args = ['cmd','/c','aws', 's3', 'ls', 's3://mybucket']
Since you're already running a cmd you don't need shell=True

SLURM squeue format argument fails from subprocess.Popen

I'm trying to call the SLURM squeue from a python script. The command,
/usr/bin/squeue --Format=username,jobid,name,timeleft
Works fine from the command line, but fails from subprocess.Popen with:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/n/home00/DilithiumMatrix/.conda/envs/py35/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/n/home00/DilithiumMatrix/.conda/envs/py35/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/squeue --Format=username,jobid,name,timeleft'
MWE:
import subprocess
command = "/usr/bin/squeue --Format=username,jobid,name,timeleft"
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
text = p.stdout.read()
print(text)
/usr/bin/squeue works fine from both the command line or Popen.
Could it be failing because it requires some information about the user/group that's executing the squeue command and that is (somehow) lost when running via python? What else could be causing this?

The first argument to subprocess.Popen is either a String, or a list of Strings. If it is a single String, it will be interpreted as a filename. This is the reason for the error you get.
To pass a list of Strings, it should match how a shell would pass your arguments to the process. A standard shell will split your command line by whitespace, so instead of this:
command = "/usr/bin/squeue --Format=username,jobid,name,timeleft"
You need this:
command = ["/usr/bin/squeue", "--Format=username,jobid,name,timeleft"]
Splitting the second argument at the "=" as you mentioned in your comment just confuses squeue, which would then see two arguments.

How to use Subprocess in Windows

I am trying to save the result or function runcmd in the variable Result.
Here is what I have tried:
import subprocess
def runcmd(cmd):
x = subprocess.Popen(cmd, stdout=subprocess.PIPE)
Result = x.communicate(stdout)
return Result
runcmd("dir")
When I run ths code, I get this result:
Traceback (most recent call last):
File "C:\Python27\MyPython\MyCode.py", line 7, in <module>
runcmd("dir")
File "C:\Python27\MyPython\MyCode.py", line 4, in runcmd
x = subprocess.Popen(cmd, stdout=subprocess.PIPE)
File "C:\Python27\lib\subprocess.py", line 679, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 893, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
What could I do to fix this?

I think what you are looking for is os.listdir()
check out the os module for more info
an example:
>>> import os
>>> l = os.listdir()
>>> print (l)
['DLLs', 'Doc', 'google-python-exercises', 'include', 'Lib', 'libs', 'LICENSE.txt', 'NEWS.txt', 'python.exe', 'pythonw.e
xe', 'README.txt', 'tcl', 'Tools', 'VS2010Cmd.lnk']
>>>
You could also read the output into a list:
result = []
process = subprocess.Popen('dir',
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE )
for line in process.stdout:
result.append(line)
errcode = process.returncode
for line in result:
print(line)

As far as I know, dir is a built in command of the shell in Windows and thus not a file available for execution as a program. Which is probably why subprocess.Popen cannot find it. But you can try adding shell=True to the Popen() construtor call like this:
def runcmd(cmd):
x = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
return x.communicate(stdout)
runcmd("dir")
If shell=True doesn't help, you're out of luck executing dir directly. But then you can make a .bat file and put a call to dir there instead, and then invoke that .bat file from Python instead.
btw also check out the PEP8!
P.S As Mark Ransom pointed out in a comment, you could just use ['cmd', '/c', 'dir'] as the value of cmd instead of the .bat hack if shell=True fails to fix the issue.

checking the return code using python (MAC)

I have written a script that checks if an SVN Repo is up and running, the result is based on the return value.
import subprocess
url = " validurl"
def check_svn_status():
subprocess.call(['svn info'+url],shell=True)
def get_status():
subprocess.call('echo $?',shell=True)
def main():
check_svn_status()
get_status()
if __name__ == '__main__':
main()
The problem I'm facing is that if I change the url to something that does't exist I still get the return value as 0, but if I were to run this outside the script, i.e go to the terminal type svn info wrong url and then do a echo $? I get a return value of 1. But I can't re-create this in the python. Any guidelines ?
TraceBack after updating
Traceback (most recent call last):
File "svn_status.py", line 21, in <module>
main()
File "svn_status.py", line 15, in main
check_svn_status()
File "svn_status.py", line 8, in check_svn_status
p = sp.Popen(['svn info'], stdout=sp.PIPE, stderr=sp.PIPE)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1202, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or director
y

Why your approach does not work:
You invoke two independent subshells. The second shell does not know of the first shell and therefore does not have any information about the returncode of the process that was executed in the first shell.
Solution:
Use the subprocess module, spawn your subprocess directly (not through a subshell) and retrieve the returncode. Help yourself by reading the documentation of the module: http://docs.python.org/library/subprocess.html
There are several ways to achieve your goal. One simple way could be:
import subprocess as sp
p = sp.Popen(['command', 'arg1', 'arg2'], stdout=sp.PIPE, stderr=sp.PIPE)
stdout, stderr = p.communicate()
returncode = p.returncode
This way, you don't go through a subshell (shell=False by default), which is the recommended approach for various reasons. You directly catch the returncode of the spawned subprocess and you have full access to the subprocess' standard output and standard error.

subprocess.call returns the retcode, just store the result of your subprocess.call(['svn info'+url],shell=True)
http://docs.python.org/library/subprocess.html