I have a set of python modules that I want to work both on linux and on MinGW (Windows).
While subprocess works fine with on Linux, on windows subprocess.call/Popen keep throwing errors saying the required binary can not be found.
For example, I use Xilinx's Vivado a lot (EDA tool). While just typing "vivado" in mingw prompt works fine and vivado is launched, if I try to call vivado through subprocess, I get an error saying the file can not be found.
My mingw environment path variable contains: /c/Xilinx/Vivado/2015.2/bin which is where the vivado binary is found. If I launch python and do this,
print os.environ['PATH']
I see it contains: C:\\Xilinx\\Vivado\\2015.2\\bin;
So I guess python performed the appropriate translations upon launch.
Also, if instead of vivado I try to launch vivado.bat (file contained in the same directory as the binary) from python's subprocess, everything works and the binary is launched.
The vivado binary has execution permissions...
So my question is. Is this problem something known? what is the cleanest way to solve it?
Thanks!
Not sure if this answers the OP exactly, but I think it is highly related: consider this example, where the executable path is written Windows-style, with backslashes:
#!/usr/bin/env python3
import sys, os
import subprocess
NOTEPADPATH = r"C:\WINDOWS\system32\notepad.exe"
print("NOTEPADPATH is {}".format(NOTEPADPATH))
proc_notepad = subprocess.Popen([NOTEPADPATH], shell=False)
If I run this in MSYS2 shell, it fails:
$ python3 test.py
NOTEPADPATH is C:\WINDOWS\system32\notepad.exe
Traceback (most recent call last):
File "test4.py", line 11, in <module>
proc_notepad = subprocess.Popen([NOTEPADPATH], shell=False)
File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\WINDOWS\\system32\\notepad.exe': 'C:\\WINDOWS\\system32\\notepad.exe'
If I run this in Anaconda 3 shell Python3 (which is apparently compiled for Windows natively), it succeeds (Notepad is started):
(base) C:\msys64\tmp>python test.py
NOTEPADPATH is C:\WINDOWS\system32\notepad.exe
If we now change the executable path, so it uses forward slash as path separator, instead of backslash:
#!/usr/bin/env python3
import sys, os
import subprocess
NOTEPADPATH = r"C:/WINDOWS/system32/notepad.exe"
print("NOTEPADPATH is {}".format(NOTEPADPATH))
proc_notepad = subprocess.Popen([NOTEPADPATH], shell=False)
... then it succeeds both in MSYS2, where Notepad is started:
$ python3 test.py
NOTEPADPATH is C:/WINDOWS/system32/notepad.exe
... and in Anaconda 3 shell Python3:
(base) C:\msys64\tmp>python test.py
NOTEPADPATH is C:/WINDOWS/system32/notepad.exe
Related
I have a python file test.py that must be executed in the python2.7 interpreter, while some other code in main.py requires the python3.6 interpreter.
I had to call the python3.6 interpreter through test.py to execute the code in main.py because main.py relies on some specific versions of libraries that are not available in python2.7. Also, since I'm writing an app plugin, the plugin's interpreter is only version 2.7. So I have to make cross-version calls. And there are parameters to be passed between them.
I tried opening a subprocess through the subprocess module in test.py. But the following error was encountered.
Codes in test.py:
import subprocess
epath = r'E:\Anaconda3\envs\***\python.exe'
fpath = r'D:\Codes\_Projects\***\main.py'
p = subprocess.call(
[epath, fpath],
stdin=None,
stdout=None,
stderr=None,
creationflags=subprocess.CREATE_NEW_CONSOLE,
shell=False
)
for line in iter(p.stdout.readline, b''):
print line
p.stdout.close()
p.wait()
And I got an error (in the cmd window where the subprocess executes):
Python error: Py_Initialize: unable to load the file system codec
File "E:/Slicer_4.10.2/bin/../lib/Python/Lib\encodings\__init__.py", line 123
raise CodecRegistryError,\
^
SyntaxError: invalid syntax
Current thread 0x00006a00 (most recent call first):
The path E:/Slicer_4.10.2/bin/ is the directory of the python2.7 interpreter. Slicer is the software I'm developing the plugin for. It comes with a python2.7 interpreter. My test.py must be executed with this interpreter.
I googled this error and someone say that this is due to a mismatch of environmental variables. So I printed out the environmental variables in test.py and here are some of them:
'PYTHONPATH': 'E:/Slicer_4.10.2/bin/../lib/Slicer-4.10;E:/Slicer_4.10.2/bin/../lib/Slicer-4.10/qt-scripted-modules;E:/Slicer_4.10.2/bin/../lib/Slicer-4.10/qt-loadable-modules;E:/Slicer_4.10.2/bin/../lib/vtkTeem;E:/Slicer_4.10.2/bin/../bin/Python;E:/Slicer_4.10.2/bin/../lib/Slicer-4.10/qt-loadable-modules/Python;E:/Slicer_4.10.2/bin/../lib/Python/Lib;E:/Slicer_4.10.2/bin/../lib/Python/Lib/lib-dynload;E:/Slicer_4.10.2/bin/../lib/Python/Lib/site-packages;E:/Slicer_4.10.2/bin/.././bin/Lib/site-packages;E:/Slicer_4.10.2/bin/.././lib/Slicer-4.10/python2.7/site-packages;E:/Slicer_4.10.2/bin/../lib/Python/lib/Python/Lib;E:/Slicer_4.10.2/bin/../lib/Python/lib/Python/Lib/lib-dynload;E:/Slicer_4.10.2/bin/../lib/Python/lib/Python/Lib/site-packages;E:/Slicer_4.10.2/bin/../bin/Python',
'PYTHONHOME': 'E:/Slicer_4.10.2/bin/../lib/Python',
'PYTHONNOUSERSITE': '1',
It can be seen that all environment variables related to python are under the Slicer path, which means that these are environment variables added by the python2.7 interpreter, during runtime.I'm sure for that because I didn't add them manually to the system's environment variables. Obviously they are not the environment that python3.6 intepreter needs.
So, is there any way to reset environment variables, or run subprocesses in a cmd/powershell that has no environment variables like PYTHONPATH set?
My OS is windows 10.0.
I have a self-installed python in my user directory in a corporate UNIX SUSE computer (no sudo privilege):
which python
<user>/bin/python/Python-3.6.1/python
I have an executable (chmod 777) sample.py file with this line at the top of the file:
#!<user>/bin/python/Python-3.6.1/python
I can execute the file like this:
python sample.py
But when I run it by itself I get an error:
/full/path/sample.py
/full/path/sample.py: Command not found
I have no idea why it's not working. I'm discombobulated as what might be going wrong since the file is executable, the python path is correct, and the file executes if I put a python command in the front. What am I missing?
EDIT:
I tried putting this on top of the file:
#!/usr/bin/env python
Now, I get this error:
: No such file or directory
I tried this to make sure my env is correct
which env
/usr/bin/env
EDIT2:
Yes, I can run the script fine using the shebang command like this:
<user>/bin/python/Python-3.6.1/python /full/path/sample.py
Your file has DOS line endings (CR+LF). It works if you run python sample.py but doesn't work if you run ./sample.py. Recode the file so it has Unix line endings (pure LF at the end of every line).
Try using #!/usr/bin/env python as described in this post. Let the OS do the work.
I want get into virtual environment in python files.But it raise no such files.
import subprocess
subprocess.Popen(['source', '/Users/XX/Desktop/mio/worker/venv/bin/activate'])
Traceback (most recent call last):
File "/Users/Ru/Desktop/mio/worker/run.py", line 3, in
subprocess.Popen(['source', '/Users/Ru/Desktop/mio/worker/venv/bin/activate'])
File"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in init
errread, errwrite)
File"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I think your code doesn't work because you are separating the 'source' command from the virtualenv path argument, from the documentation:
"Note in particular that options (such as -input) and arguments (such
as eggs.txt) that are separated by whitespace in the shell go in
separate list elements, while arguments that need quoting or
backslash escaping when used in the shell (such as filenames
containing spaces or the echo command shown above) are single list
elements."
You should try one of two things:
First, write the source and the virtualenv file path as a single string argument:
import subprocess
subprocess.Popen(['source '/Users/XX/Desktop/mio/worker/venv/bin/activate'])
I'm working on OSX and that doesn't seem to work, but it might be due to the shell you are using. To ensure this will work, you can use the shell=True flag:
import subprocess
subprocess.Popen(['source '/Users/XX/Desktop/mio/worker/venv/bin/activate'],shell=True)
This will use /bin/sh shell by default. Again, you can read more in the documentation.
Tom.
There is another simpler way to do what you want.
If you want a python script to use a virtualenv you can always use the python interpreter from the virualenv itself.
/Users/Ru/Desktop/mio/worker/venv/bin/python my_python_file.py
This will run the my_python_file.py with the properties/libraries of the virtualenv.
If you want to run that file inside a subprocess you can do something similar to the method I described above:
import subprocess
subprocess.Popen(['/Users/Ru/Desktop/mio/worker/venv/bin/python my_python_file.py])
and have my_python_file.py import pika and do other actions you wish to do.
Tom.
I am trying to write Python scripts to do some Maven commands automatically.
The following Python code does not work:
import subprocess
args = ['mvn', '-version']
process = subprocess.Popen(args, stdout=subprocess.PIPE)
resulting in the following error:
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
process = subprocess.Popen(args, stdout=subprocess.PIPE)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
As a next step, I supply subprocess the full path to the mvn binary:
import subprocess
args = ['/usr/local/Cellar/maven/3.2.3/libexec/bin/mvn', '-version']
process = subprocess.Popen(args, stderr=subprocess.PIPE)
out, err = process.communicate()
This command gets a bit further, but printing err reveals the following complaint:
Error: JAVA_HOME is not defined correctly.
We cannot execute /usr/libexec/java_home/bin/java
As a sanity check, I verify on my own terminal that JAVA_HOME is correctly set:
$ mvn -version
Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T16:58:10-04:00)
Maven home: /usr/local/Cellar/maven/3.2.3/libexec
Java version: 1.8.0_25, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"
I am also able to run mvn without hassle on the terminal.
My question is: why is subprocess not able to run mvn, and how can I get it to do so?
Use
process = subprocess.Popen(args,shell=True)
Python inherits the environment from your terminal, and a subprocess spawned from Python should inherit the environment from the parent process. So I am not sure where the error regarding your JAVA_HOME env variable is coming from. Could it be that you do not spawn Python from the same terminal as you did the mvn -version test with?
Note that you can explicitly modify the environment for the child, via the env parameter to Popen().
I faced the similar issue & spent almost an hour however it was easy to fix. Most of the people make mistake to use mvn rather mvn.cmd We must use mvn.cmd as mvn is window command application....
import subprocess
proc = subprocess.Popen(['mvn.cmd' , '-version'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
stdout, stderr = proc.communicate()
print(stdout, stderr)
Is there a simple way to run a Python script on Windows/Linux/OS X?
On the latter two, subprocess.Popen("/the/script.py") works, but on Windows I get the following error:
Traceback (most recent call last):
File "test_functional.py", line 91, in test_functional
log = tvnamerifiy(tmp)
File "test_functional.py", line 49, in tvnamerifiy
stdout = PIPE
File "C:\Python26\lib\subprocess.py", line 595, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 804, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application
monkut's comment: The use case isn't clear. Why use subprocess to run a python script? Is there something preventing you from importing the script and calling the necessary function?
I was writing a quick script to test the overall functionality of a Python-command-line tool (to test it on various platforms). Basically it had to create a bunch of files in a temp folder, run the script on this and check the files were renamed correctly.
I could have imported the script and called the function, but since it relies on sys.argv and uses sys.exit(), I would have needed to do something like..
import sys
import tvnamer
sys.argv.append("-b", "/the/folder")
try:
tvnamer.main()
except BaseException, errormsg:
print type(errormsg)
Also, I wanted to capture the stdout and stderr for debugging incase something went wrong.
Of course a better way would be to write the script in more unit-testable way, but the script is basically "done" and I'm doing a final batch of testing before doing a "1.0" release (after which I'm going to do a rewrite/restructure, which will be far tidier and more testable)
Basically, it was much easier to simply run the script as a process, after finding the sys.executable variable. I would have written it as a shell-script, but that wouldn't have been cross-platform. The final script can be found here
Just found sys.executable - the full path to the current Python executable, which can be used to run the script (instead of relying on the shbang, which obviously doesn't work on Windows)
import sys
import subprocess
theproc = subprocess.Popen([sys.executable, "myscript.py"])
theproc.communicate()
How about this:
import sys
import subprocess
theproc = subprocess.Popen("myscript.py", shell = True)
theproc.communicate() # ^^^^^^^^^^^^
This tells subprocess to use the OS shell to open your script, and works on anything that you can just run in cmd.exe.
Additionally, this will search the PATH for "myscript.py" - which could be desirable.
Yes subprocess.Popen(cmd, ..., shell=True) works like a charm. On Windows the .py file extension is recognized, so Python is invoked to process it (on *NIX just the usual shebang). The path environment controls whether things are seen. So the first arg to Popen is just the name of the script.
subprocess.Popen(['myscript.py', 'arg1', ...], ..., shell=True)
It looks like windows tries to run the script using its own EXE framework rather than call it like
python /the/script.py
Try,
subprocess.Popen(["python", "/the/script.py"])
Edit: "python" would need to be on your path.
For example, to execute following with command prompt or BATCH file we can use this:
C:\Python27\python.exe "C:\Program files(x86)\dev_appserver.py" --host 0.0.0.0 --post 8080 "C:\blabla\"
Same thing to do with Python, we can do this:
subprocess.Popen(['C:/Python27/python.exe', 'C:\\Program files(x86)\\dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:\\blabla'], shell=True)
or
subprocess.Popen(['C:/Python27/python.exe', 'C:/Program files(x86)/dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:/blabla'], shell=True)
You are using a pathname separator which is platform dependent. Windows uses "\" and Unix uses "/".
When you are running a python script on windows in subprocess you should use python in front of the script name. Try:
process = subprocess.Popen("python /the/script.py")
Supplemental info: It is worth noting that the documentation states that you need to use shell=True if you are using a dos shell command like dir
without it you get something like this.
>>> import subprocess
>>> subprocess.run(['dir'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 489, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in __ini
t__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _exe
cute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
>>> subprocess.run(['dir'], shell=True)
Volume in drive J is garbage
Volume Serial Number is 5EE7-B084
Also you can use path like objects for the args which is recent addition.
from pathlib import Path
subprocess.run(Path('c:/proj/myfile.bat'))
Also worth noting there is a whole set of windows specific controls that allow you to control how a process is spawned which concurrent operations can use.
So controlling subprocesses on windows is not as simple as posix style.