We Keep Coding

Python subprocess running from mingw fails to find binaries

I have a set of python modules that I want to work both on linux and on MinGW (Windows).
While subprocess works fine with on Linux, on windows subprocess.call/Popen keep throwing errors saying the required binary can not be found.
For example, I use Xilinx's Vivado a lot (EDA tool). While just typing "vivado" in mingw prompt works fine and vivado is launched, if I try to call vivado through subprocess, I get an error saying the file can not be found.
My mingw environment path variable contains: /c/Xilinx/Vivado/2015.2/bin which is where the vivado binary is found. If I launch python and do this,
print os.environ['PATH']
I see it contains: C:\\Xilinx\\Vivado\\2015.2\\bin;
So I guess python performed the appropriate translations upon launch.
Also, if instead of vivado I try to launch vivado.bat (file contained in the same directory as the binary) from python's subprocess, everything works and the binary is launched.
The vivado binary has execution permissions...
So my question is. Is this problem something known? what is the cleanest way to solve it?
Thanks!

Not sure if this answers the OP exactly, but I think it is highly related: consider this example, where the executable path is written Windows-style, with backslashes:
#!/usr/bin/env python3
import sys, os
import subprocess
NOTEPADPATH = r"C:\WINDOWS\system32\notepad.exe"
print("NOTEPADPATH is {}".format(NOTEPADPATH))
proc_notepad = subprocess.Popen([NOTEPADPATH], shell=False)
If I run this in MSYS2 shell, it fails:
$ python3 test.py
NOTEPADPATH is C:\WINDOWS\system32\notepad.exe
Traceback (most recent call last):
File "test4.py", line 11, in <module>
proc_notepad = subprocess.Popen([NOTEPADPATH], shell=False)
File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\WINDOWS\\system32\\notepad.exe': 'C:\\WINDOWS\\system32\\notepad.exe'
If I run this in Anaconda 3 shell Python3 (which is apparently compiled for Windows natively), it succeeds (Notepad is started):
(base) C:\msys64\tmp>python test.py
NOTEPADPATH is C:\WINDOWS\system32\notepad.exe
If we now change the executable path, so it uses forward slash as path separator, instead of backslash:
#!/usr/bin/env python3
import sys, os
import subprocess
NOTEPADPATH = r"C:/WINDOWS/system32/notepad.exe"
print("NOTEPADPATH is {}".format(NOTEPADPATH))
proc_notepad = subprocess.Popen([NOTEPADPATH], shell=False)
... then it succeeds both in MSYS2, where Notepad is started:
$ python3 test.py
NOTEPADPATH is C:/WINDOWS/system32/notepad.exe
... and in Anaconda 3 shell Python3:
(base) C:\msys64\tmp>python test.py
NOTEPADPATH is C:/WINDOWS/system32/notepad.exe

Python: Trouble with cross platform running of subprocesses

I have the following helper method which executes commands perfectly on OSX and only with some commands on Windows:
def exec_cmd(cmd):
"""Run a command and return the status, standard output and error."""
proc = Popen(shlex.split(cmd), stdout=PIPE, stderr=PIPE)
stdout, stderr = proc.communicate()
# I like to get True or False rather than 0 (True) or 1 (False)
# which is just backwards as usually 0 is False and 1 is True
status = not bool(proc.returncode)
return (status, stdout, stderr)
For example, the following sample commands all work perfectly on Mac using my exec_cmd helper:
osascript -e 'tell application "Microsoft PowerPoint" to activate
osascript -e 'tell application "Microsoft PowerPoint" to quit
For example, the following sample commands all work perfectly on Windows using my exec_cmd helper:
"C:\Program Files\Microsoft Office\Office15\Powerpnt.exe" /S "C:\Users\MyUser\example.pptx"
Taskkill /IM POWERPNT.EXE /F
However, the following does not work on Windows:
START "" "C:\Program Files\Microsoft Office\Office15\Powerpnt.exe"
It errors out with:
WindowsError: [Error 2] The system cannot find the file specified
Even this doesn't work:
p = Popen(["START", "", "C:\Program Files\Microsoft Office\Office15\Powerpnt.exe"], stdout=PIPE, stderr=PIPE)
However running that same command on the command line works fine, or even stranger just doing this works:
os.system('START "" "C:\Program Files\Microsoft Office\Office15\Powerpnt.exe"')
Why does os.system work, but not the Popen version? These are just simple open and close app examples, but I'd like to do more as I need to get the stdout output for some commands I plan on running.
Any help on sorting this out is appreciated. I can't seem to understand the underlying mechanic of os.system vs. subprocess.Popen.

You are seeing this issue because START isn't a program, its a shell command. According to the documentation, os.system() "Executes the command (a string) in a subshell", where as popen doesn't. os.system() effectively spawns a new cmd.exe instance, and passes the command to that, where as popen just spawns a new process.
You are getting the The system cannot find the file specified error because there isn't a program called START

"No such file or directory" error when calling fc-list in python

I am attempting to scrape a terminal window of the list of fonts installed on the curent hosting server. I have written the following code:
import subprocess
cmd = 'fc-list'
output = subprocess.Popen(cmd, stdout=subprocess.PIPE ).communicate()[0]
but when i call this code, an exception is raised:
[Errno 2] No such file or directory
I can open a terminal window, and this works fine. What am i doing wrong?

You need to provide the absolute path to the executable. When you open a terminal window you then have a shell running which will search in $PATH to find the program. When you run the program directly, via subprocess, you do not have a shell to search $PATH. (note: it is possible to tell subprocess that you do want a shell, but usually this leads to security vulnerabilities)
Here is what you would want to use:
import subprocess
cmd = '/usr/local/bin/fc-list'
output = subprocess.Popen(cmd, stdout=subprocess.PIPE ).communicate()[0]

Using ruby gem/command from python

I've installed the Ruby gem 'haml' on my mac, which I can use to compile haml files into html files using the following command at the terminal:
haml 'path/to/haml/file.haml' 'desired/html/path/file.html'
This command simply creates an html file at the second path, and gives no output in the terminal. So for example, this command:
haml "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.haml" "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.html"
Creates a sugar.html file at the given path. Now I'm trying to use this functionality from a python script. When I type this into IDLE's interactive python shell:
>>>import subprocess
>>>subprocess.Popen('haml "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.haml" "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.html"', shell=True, executable='/bin/bash')
<subprocess.Popen object at 0x159d6f0>
I get output suggesting that the process has been run, however, there is no file outputted. Why is this happening? I even put in the Shell argument, but no interactive shell shows up. Also, I read somewhere that the default shell used is not bash, which is what the Mac terminal uses, so I put that in too for good measure.
Following icktoofay's advice, I ran check_call. Here is the traceback I received:
Traceback (most recent call last):
File
"/Users/neil/Desktop/subprocesstest.py",
line 7, in
p = subprocess.check_call(x, shell=True) File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 504, in check_call
raise CalledProcessError(retcode, cmd) CalledProcessError: Command 'haml
"/Volumes/Macintosh
HD/Users/neil/Sites/ICSP/sugar.haml"
"/Volumes/Macintosh
HD/Users/neil/Sites/ICSP/sugar.html"'
returned non-zero exit status 127
According to the bash reference manual, while searching for a command to be executed,
If the name is neither a shell
function nor a builtin, and contains
no slashes, Bash searches each element
of $PATH for a directory containing an
executable file by that name. ... If
that function is not defined, the
shell prints an error message and
returns an exit status of 127.
However, I thought it was indeed finding the haml command after adding the shell and executable arguments, because before that, it was giving a 'file or directory not found error', which indicates that the function is not executable directly but rather in a shell only.
Now how do I make python find this haml command? Or would I have to use some ugly workaround like an applescript which then invokes the haml command.

I see that you are using, shell=True, so I would have expected things to just work. Checked it locally here with Python 2.7.1 and haml 3.1.1 and I had no problems executing it. There are also some python implementations you might be interested in, PyHAML, HamlPy, djaml or django-haml.
import subprocess
subprocess.Popen(['haml', 'hello.haml', 'hello.html'], shell=True)
% cat hello.html
<strong class='code' id='message'>Hello, World!</strong>

shlex.split() is your friend, when you want to build args list suitable for Popen and its ilk.
>>> import subprocess
>>> import shlex
>>> p = subprocess.Popen(shlex.split('haml "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.haml" "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.html"'))
>>> p.wait()
0

Using subprocess to run Python script on Windows

Is there a simple way to run a Python script on Windows/Linux/OS X?
On the latter two, subprocess.Popen("/the/script.py") works, but on Windows I get the following error:
Traceback (most recent call last):
File "test_functional.py", line 91, in test_functional
log = tvnamerifiy(tmp)
File "test_functional.py", line 49, in tvnamerifiy
stdout = PIPE
File "C:\Python26\lib\subprocess.py", line 595, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 804, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application
monkut's comment: The use case isn't clear. Why use subprocess to run a python script? Is there something preventing you from importing the script and calling the necessary function?
I was writing a quick script to test the overall functionality of a Python-command-line tool (to test it on various platforms). Basically it had to create a bunch of files in a temp folder, run the script on this and check the files were renamed correctly.
I could have imported the script and called the function, but since it relies on sys.argv and uses sys.exit(), I would have needed to do something like..
import sys
import tvnamer
sys.argv.append("-b", "/the/folder")
try:
tvnamer.main()
except BaseException, errormsg:
print type(errormsg)
Also, I wanted to capture the stdout and stderr for debugging incase something went wrong.
Of course a better way would be to write the script in more unit-testable way, but the script is basically "done" and I'm doing a final batch of testing before doing a "1.0" release (after which I'm going to do a rewrite/restructure, which will be far tidier and more testable)
Basically, it was much easier to simply run the script as a process, after finding the sys.executable variable. I would have written it as a shell-script, but that wouldn't have been cross-platform. The final script can be found here

Just found sys.executable - the full path to the current Python executable, which can be used to run the script (instead of relying on the shbang, which obviously doesn't work on Windows)
import sys
import subprocess
theproc = subprocess.Popen([sys.executable, "myscript.py"])
theproc.communicate()

How about this:
import sys
import subprocess
theproc = subprocess.Popen("myscript.py", shell = True)
theproc.communicate() # ^^^^^^^^^^^^
This tells subprocess to use the OS shell to open your script, and works on anything that you can just run in cmd.exe.
Additionally, this will search the PATH for "myscript.py" - which could be desirable.

Yes subprocess.Popen(cmd, ..., shell=True) works like a charm. On Windows the .py file extension is recognized, so Python is invoked to process it (on *NIX just the usual shebang). The path environment controls whether things are seen. So the first arg to Popen is just the name of the script.
subprocess.Popen(['myscript.py', 'arg1', ...], ..., shell=True)

It looks like windows tries to run the script using its own EXE framework rather than call it like
python /the/script.py
Try,
subprocess.Popen(["python", "/the/script.py"])
Edit: "python" would need to be on your path.

For example, to execute following with command prompt or BATCH file we can use this:
C:\Python27\python.exe "C:\Program files(x86)\dev_appserver.py" --host 0.0.0.0 --post 8080 "C:\blabla\"
Same thing to do with Python, we can do this:
subprocess.Popen(['C:/Python27/python.exe', 'C:\\Program files(x86)\\dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:\\blabla'], shell=True)
or
subprocess.Popen(['C:/Python27/python.exe', 'C:/Program files(x86)/dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:/blabla'], shell=True)

You are using a pathname separator which is platform dependent. Windows uses "\" and Unix uses "/".

When you are running a python script on windows in subprocess you should use python in front of the script name. Try:
process = subprocess.Popen("python /the/script.py")

Supplemental info: It is worth noting that the documentation states that you need to use shell=True if you are using a dos shell command like dir
without it you get something like this.
>>> import subprocess
>>> subprocess.run(['dir'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 489, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in __ini
t__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _exe
cute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
>>> subprocess.run(['dir'], shell=True)
Volume in drive J is garbage
Volume Serial Number is 5EE7-B084
Also you can use path like objects for the args which is recent addition.
from pathlib import Path
subprocess.run(Path('c:/proj/myfile.bat'))
Also worth noting there is a whole set of windows specific controls that allow you to control how a process is spawned which concurrent operations can use.
So controlling subprocesses on windows is not as simple as posix style.