We Keep Coding

subprocess error on windows while listing an s3 bucket

I am trying to list the contents of an s3 bucket using Python 2.7.13. This does not work:
>>> args
['aws', 's3', 'ls', 's3://mybucket']
>>> p = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Error is:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\subprocess.py", line 711, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 959, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Why is this so ?
But this works:
>>> p = subprocess.Popen(args, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Why is shell=False failing but shell=True working ?

You are totally right to try to use subprocess with shell=False. It is the best way to ensure portability, and is probably faster to launch.
In your case, your arguments look okay (no redirection, no pipe, multiple command):
['aws', 's3', 'ls', 's3://mybucket']
so the only thing that prevents it to work with shell=False is the fact that aws is not really an executable, but rather a file whose extension (not shown here) is contained in PATHEXT
On my machine PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY
Means that any aws.js, aws.bat, ... file can be executed. But you need the shell for that.
To locate your program, type where aws in a command prompt, you'll get the full path & extension of the command.
If you don't want to use shell=True, there's an alternative which amounts to the same thing
args = ['cmd','/c','aws', 's3', 'ls', 's3://mybucket']
Since you're already running a cmd you don't need shell=True

SLURM squeue format argument fails from subprocess.Popen

I'm trying to call the SLURM squeue from a python script. The command,
/usr/bin/squeue --Format=username,jobid,name,timeleft
Works fine from the command line, but fails from subprocess.Popen with:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/n/home00/DilithiumMatrix/.conda/envs/py35/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/n/home00/DilithiumMatrix/.conda/envs/py35/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/squeue --Format=username,jobid,name,timeleft'
MWE:
import subprocess
command = "/usr/bin/squeue --Format=username,jobid,name,timeleft"
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
text = p.stdout.read()
print(text)
/usr/bin/squeue works fine from both the command line or Popen.
Could it be failing because it requires some information about the user/group that's executing the squeue command and that is (somehow) lost when running via python? What else could be causing this?

The first argument to subprocess.Popen is either a String, or a list of Strings. If it is a single String, it will be interpreted as a filename. This is the reason for the error you get.
To pass a list of Strings, it should match how a shell would pass your arguments to the process. A standard shell will split your command line by whitespace, so instead of this:
command = "/usr/bin/squeue --Format=username,jobid,name,timeleft"
You need this:
command = ["/usr/bin/squeue", "--Format=username,jobid,name,timeleft"]
Splitting the second argument at the "=" as you mentioned in your comment just confuses squeue, which would then see two arguments.

Why subprocess.call () is showing error in linux?

Let us consider Linux platform where I need to execute a program called smart.exe which uses input.dat file. Both the files are placed in the same directory with each file having the same file permission 777.
Now if I run the following command in the terminal window smart.exe is fully executed without any error.
$./smart.exe input.dat
On the other hand, if I use the following python script called my_script.py placed in the same directory, then I get an error.
my_script.py has the following code:
#!/usr/bin/python
import os, subprocess
exit_code = subprocess.call("./smart.exe input.dat", shell = False)
The error is as follows:
File "my_script.py", line 4, in <module>
exit_code = subprocess.call("./smart.exe input.dat", shell = False)
File "/usr/lib64/python2.6/subprocess.py", line 478, in call
p = Popen(*popenargs, **kwargs)
File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Can someone please tell me why this is happening. Please note that the smart.exe should take around 10 sec to fully complete. This may be a clue for the problem.
Please also advise if there is any other way to run smart.exe from my_script.py. Your solution is much appreciated!

You should decide if you want shell support or not.
If you want the shell to be used (which is not necessary here), you should use exit_code = subprocess.call("./smart.exe input.dat", shell=True). Then the shell interprets your command line.
If you don't want it (as you don't need it and want to avoid unnecessary complexity), you should do exit_code = subprocess.call(["./smart.exe", "input.dat"], shell=False).
(And there is no point naming your binarys .exe under Linux.)

error while running Popen

I am running into following error when trying to run a command using Popen,what is wrong here?
cmd = "export COMMANDER_SERVER=commander.company.com"
Pipe = Popen(cmd, stdout=PIPE, stderr=PIPE)
(output, error) = Pipe.communicate()
Error:-
Traceback (most recent call last):
File "test_ectool.py", line 26, in <module>
main()
File "test_ectool.py", line 13, in main
Pipe = Popen(cmd, stdout=PIPE, stderr=PIPE)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

You need to separate the arguments from the command and give a list to Popen.
As Kenster's comment said, even if your command worked, you would only succeed in modifying an environmental variable inside a sub-shell not the main shell.
You will not be able run run export this way, because it is not a program. It is a bash built-in.
Here is an example of a command that does work, with the correct semantics.
from subprocess import Popen,PIPE
cmd = "echo COMMANDER_SERVER=commander.company.com"
Pipe = Popen(cmd.split(), stdout=PIPE, stderr=PIPE)
(output, error) = Pipe.communicate()
print output

merlin2011's answer is incorrect regarding the command string for Popen (point #1). From the Python docs:
args should be a sequence of program arguments or else a single string.
As other people have stated, the environment variable will not be saved. For that, you need to use os.environ['VARNAME'] = value. If you want to use other bash builtins, then you must pass shell=True as an argument to Popen.

Python OSError: [Errno 2]

I have the following code that is attempting to start each of the "commands" below in Linux. The module attempts to keep each of the 2 commands running if either should crash for whatever reason.
#!/usr/bin/env python
import subprocess
commands = [ ["screen -dmS RealmD top"], ["screen -DmS RealmD top -d 5"] ]
programs = [ subprocess.Popen(c) for c in commands ]
while True:
for i in range(len(programs)):
if programs[i].returncode is None:
continue # still running
else:
# restart this one
programs[i]= subprocess.Popen(commands[i])
time.sleep(1.0)
Upon executing the code the following exception is thrown:
Traceback (most recent call last):
File "./marp.py", line 82, in <module>
programs = [ subprocess.Popen(c) for c in commands ]
File "/usr/lib/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1092, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I think I'm missing something obvious, can anyone see what's wrong with the code above?

Use ["screen", "-dmS", "RealmD", "top"] instead of ["screen -dmS RealmD top"].
Maybe also use the complete path to screen.

Only guess is that it can't find screen. Try /usr/bin/screen or whatever which screen gives you.

The problem is that your command should be split. subprocces requires that the cmd is a list, not a string.
It shouldn't be:
subprocess.call('''awk 'BEGIN {FS="\t";OFS="\n"} {a[$1]=a [$1] OFS $2 FS $3 FS $4} END
{for (i in a) {print i a[i]}}' 2_lcsorted.txt > 2_locus_2.txt''')
That won't work. If you feed subprocess a string, it assumes that that is the path to the command you want to execute. The command needs to be a list. Check out http://www.gossamer-threads.com/lists/python/python/724330. Also, because you're using file redirection, you should use subprocess.call(cmd, shell=True). You can also use shlex.

I got same error when i wrote like this :-
subprocess.Popen("ls" ,shell = False , stdout = subprocess.PIPE ,stderr = subprocess.PIPE)
And problem is solved when i made shell=True .It will work
subprocess.Popen("ls" ,shell = False , stdout = subprocess.PIPE ,stderr = subprocess.PIPE, shell=True)

commands = [ "screen -dmS RealmD top", "screen -DmS RealmD top -d 5" ]
programs = [ subprocess.Popen(c.split()) for c in commands ]

Just in case.. I also got stuck with this error and the issue was that my files were in DOS instead of UNIX so at :
return subprocess.call(lst_exp)
where lst_exp is a list of args, one of them was "not found" because it was in DOS instead of UNIX but error thrown was the same :
File "/var/www/run_verifier.py", line 59, in main
return subprocess.call(lst_exp)
File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory