Is there a simple way to run a Python script on Windows/Linux/OS X?
On the latter two, subprocess.Popen("/the/script.py") works, but on Windows I get the following error:
Traceback (most recent call last):
File "test_functional.py", line 91, in test_functional
log = tvnamerifiy(tmp)
File "test_functional.py", line 49, in tvnamerifiy
stdout = PIPE
File "C:\Python26\lib\subprocess.py", line 595, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 804, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application
monkut's comment: The use case isn't clear. Why use subprocess to run a python script? Is there something preventing you from importing the script and calling the necessary function?
I was writing a quick script to test the overall functionality of a Python-command-line tool (to test it on various platforms). Basically it had to create a bunch of files in a temp folder, run the script on this and check the files were renamed correctly.
I could have imported the script and called the function, but since it relies on sys.argv and uses sys.exit(), I would have needed to do something like..
import sys
import tvnamer
sys.argv.append("-b", "/the/folder")
try:
tvnamer.main()
except BaseException, errormsg:
print type(errormsg)
Also, I wanted to capture the stdout and stderr for debugging incase something went wrong.
Of course a better way would be to write the script in more unit-testable way, but the script is basically "done" and I'm doing a final batch of testing before doing a "1.0" release (after which I'm going to do a rewrite/restructure, which will be far tidier and more testable)
Basically, it was much easier to simply run the script as a process, after finding the sys.executable variable. I would have written it as a shell-script, but that wouldn't have been cross-platform. The final script can be found here
Just found sys.executable - the full path to the current Python executable, which can be used to run the script (instead of relying on the shbang, which obviously doesn't work on Windows)
import sys
import subprocess
theproc = subprocess.Popen([sys.executable, "myscript.py"])
theproc.communicate()
How about this:
import sys
import subprocess
theproc = subprocess.Popen("myscript.py", shell = True)
theproc.communicate() # ^^^^^^^^^^^^
This tells subprocess to use the OS shell to open your script, and works on anything that you can just run in cmd.exe.
Additionally, this will search the PATH for "myscript.py" - which could be desirable.
Yes subprocess.Popen(cmd, ..., shell=True) works like a charm. On Windows the .py file extension is recognized, so Python is invoked to process it (on *NIX just the usual shebang). The path environment controls whether things are seen. So the first arg to Popen is just the name of the script.
subprocess.Popen(['myscript.py', 'arg1', ...], ..., shell=True)
It looks like windows tries to run the script using its own EXE framework rather than call it like
python /the/script.py
Try,
subprocess.Popen(["python", "/the/script.py"])
Edit: "python" would need to be on your path.
For example, to execute following with command prompt or BATCH file we can use this:
C:\Python27\python.exe "C:\Program files(x86)\dev_appserver.py" --host 0.0.0.0 --post 8080 "C:\blabla\"
Same thing to do with Python, we can do this:
subprocess.Popen(['C:/Python27/python.exe', 'C:\\Program files(x86)\\dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:\\blabla'], shell=True)
or
subprocess.Popen(['C:/Python27/python.exe', 'C:/Program files(x86)/dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:/blabla'], shell=True)
You are using a pathname separator which is platform dependent. Windows uses "\" and Unix uses "/".
When you are running a python script on windows in subprocess you should use python in front of the script name. Try:
process = subprocess.Popen("python /the/script.py")
Supplemental info: It is worth noting that the documentation states that you need to use shell=True if you are using a dos shell command like dir
without it you get something like this.
>>> import subprocess
>>> subprocess.run(['dir'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 489, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in __ini
t__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _exe
cute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
>>> subprocess.run(['dir'], shell=True)
Volume in drive J is garbage
Volume Serial Number is 5EE7-B084
Also you can use path like objects for the args which is recent addition.
from pathlib import Path
subprocess.run(Path('c:/proj/myfile.bat'))
Also worth noting there is a whole set of windows specific controls that allow you to control how a process is spawned which concurrent operations can use.
So controlling subprocesses on windows is not as simple as posix style.
Related
I have a set of python modules that I want to work both on linux and on MinGW (Windows).
While subprocess works fine with on Linux, on windows subprocess.call/Popen keep throwing errors saying the required binary can not be found.
For example, I use Xilinx's Vivado a lot (EDA tool). While just typing "vivado" in mingw prompt works fine and vivado is launched, if I try to call vivado through subprocess, I get an error saying the file can not be found.
My mingw environment path variable contains: /c/Xilinx/Vivado/2015.2/bin which is where the vivado binary is found. If I launch python and do this,
print os.environ['PATH']
I see it contains: C:\\Xilinx\\Vivado\\2015.2\\bin;
So I guess python performed the appropriate translations upon launch.
Also, if instead of vivado I try to launch vivado.bat (file contained in the same directory as the binary) from python's subprocess, everything works and the binary is launched.
The vivado binary has execution permissions...
So my question is. Is this problem something known? what is the cleanest way to solve it?
Thanks!
Not sure if this answers the OP exactly, but I think it is highly related: consider this example, where the executable path is written Windows-style, with backslashes:
#!/usr/bin/env python3
import sys, os
import subprocess
NOTEPADPATH = r"C:\WINDOWS\system32\notepad.exe"
print("NOTEPADPATH is {}".format(NOTEPADPATH))
proc_notepad = subprocess.Popen([NOTEPADPATH], shell=False)
If I run this in MSYS2 shell, it fails:
$ python3 test.py
NOTEPADPATH is C:\WINDOWS\system32\notepad.exe
Traceback (most recent call last):
File "test4.py", line 11, in <module>
proc_notepad = subprocess.Popen([NOTEPADPATH], shell=False)
File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\WINDOWS\\system32\\notepad.exe': 'C:\\WINDOWS\\system32\\notepad.exe'
If I run this in Anaconda 3 shell Python3 (which is apparently compiled for Windows natively), it succeeds (Notepad is started):
(base) C:\msys64\tmp>python test.py
NOTEPADPATH is C:\WINDOWS\system32\notepad.exe
If we now change the executable path, so it uses forward slash as path separator, instead of backslash:
#!/usr/bin/env python3
import sys, os
import subprocess
NOTEPADPATH = r"C:/WINDOWS/system32/notepad.exe"
print("NOTEPADPATH is {}".format(NOTEPADPATH))
proc_notepad = subprocess.Popen([NOTEPADPATH], shell=False)
... then it succeeds both in MSYS2, where Notepad is started:
$ python3 test.py
NOTEPADPATH is C:/WINDOWS/system32/notepad.exe
... and in Anaconda 3 shell Python3:
(base) C:\msys64\tmp>python test.py
NOTEPADPATH is C:/WINDOWS/system32/notepad.exe
I want get into virtual environment in python files.But it raise no such files.
import subprocess
subprocess.Popen(['source', '/Users/XX/Desktop/mio/worker/venv/bin/activate'])
Traceback (most recent call last):
File "/Users/Ru/Desktop/mio/worker/run.py", line 3, in
subprocess.Popen(['source', '/Users/Ru/Desktop/mio/worker/venv/bin/activate'])
File"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in init
errread, errwrite)
File"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I think your code doesn't work because you are separating the 'source' command from the virtualenv path argument, from the documentation:
"Note in particular that options (such as -input) and arguments (such
as eggs.txt) that are separated by whitespace in the shell go in
separate list elements, while arguments that need quoting or
backslash escaping when used in the shell (such as filenames
containing spaces or the echo command shown above) are single list
elements."
You should try one of two things:
First, write the source and the virtualenv file path as a single string argument:
import subprocess
subprocess.Popen(['source '/Users/XX/Desktop/mio/worker/venv/bin/activate'])
I'm working on OSX and that doesn't seem to work, but it might be due to the shell you are using. To ensure this will work, you can use the shell=True flag:
import subprocess
subprocess.Popen(['source '/Users/XX/Desktop/mio/worker/venv/bin/activate'],shell=True)
This will use /bin/sh shell by default. Again, you can read more in the documentation.
Tom.
There is another simpler way to do what you want.
If you want a python script to use a virtualenv you can always use the python interpreter from the virualenv itself.
/Users/Ru/Desktop/mio/worker/venv/bin/python my_python_file.py
This will run the my_python_file.py with the properties/libraries of the virtualenv.
If you want to run that file inside a subprocess you can do something similar to the method I described above:
import subprocess
subprocess.Popen(['/Users/Ru/Desktop/mio/worker/venv/bin/python my_python_file.py])
and have my_python_file.py import pika and do other actions you wish to do.
Tom.
I am trying to write Python scripts to do some Maven commands automatically.
The following Python code does not work:
import subprocess
args = ['mvn', '-version']
process = subprocess.Popen(args, stdout=subprocess.PIPE)
resulting in the following error:
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
process = subprocess.Popen(args, stdout=subprocess.PIPE)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
As a next step, I supply subprocess the full path to the mvn binary:
import subprocess
args = ['/usr/local/Cellar/maven/3.2.3/libexec/bin/mvn', '-version']
process = subprocess.Popen(args, stderr=subprocess.PIPE)
out, err = process.communicate()
This command gets a bit further, but printing err reveals the following complaint:
Error: JAVA_HOME is not defined correctly.
We cannot execute /usr/libexec/java_home/bin/java
As a sanity check, I verify on my own terminal that JAVA_HOME is correctly set:
$ mvn -version
Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T16:58:10-04:00)
Maven home: /usr/local/Cellar/maven/3.2.3/libexec
Java version: 1.8.0_25, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"
I am also able to run mvn without hassle on the terminal.
My question is: why is subprocess not able to run mvn, and how can I get it to do so?
Use
process = subprocess.Popen(args,shell=True)
Python inherits the environment from your terminal, and a subprocess spawned from Python should inherit the environment from the parent process. So I am not sure where the error regarding your JAVA_HOME env variable is coming from. Could it be that you do not spawn Python from the same terminal as you did the mvn -version test with?
Note that you can explicitly modify the environment for the child, via the env parameter to Popen().
I faced the similar issue & spent almost an hour however it was easy to fix. Most of the people make mistake to use mvn rather mvn.cmd We must use mvn.cmd as mvn is window command application....
import subprocess
proc = subprocess.Popen(['mvn.cmd' , '-version'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
stdout, stderr = proc.communicate()
print(stdout, stderr)
I would like to run an executable that performs some processing on a dataset located on a remote filer. As part of the design, I'd like the location of the filer to be flexible and something that's passed to my python program at runtime.
I've put together the following bit of code that illustrates my problem, but using the python command, so anyone can run this:
#!/usr/bin/env python
import os
import subprocess
def runMySubProcess(cmdstr, iwd):
p = subprocess.Popen(cmdstr,
shell=True,
cwd=iwd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
(stdout, stderr) = p.communicate()
if stderr:
raise IOError, stderr
return stdout
if __name__ == '__main__':
print runMySubProcess('python -h', 'C:\\')
print runMySubProcess('python -h', '\\\\htpc\\nas')
This works great as long as iwd is on a share that's be mapped to a drive letter on the machine. But if iwd is a UNC path the subprocess.Popen() call ends up with stderr output, which in turn throws the IOError exception:
Traceback (most recent call last):
File "test.py", line 19, in <module>
print runMySubProcess('dir', '\\\\htpc\\nas')
File "test.py", line 14, in runMySubProcess
raise IOError, stderr
IOError: '\\htpc\nas'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported. Defaulting to Windows directory.
Is there a way to make this subprocess call work without resorting to parsing iwd and making a temporary drive mount on the machine that exists while the subprocess command executes? I want to avoid having to manage the creation and cleanup of drive mounts. And of course, I'd rather not have to deal with (albeit unlikely) case where all drive letters are currently in use on the machine.
The problem is not with Popen,, but with cmd.exe, which does not allow the working directory to be a UNC path. It just does not; try it. You may have better luck specifying shell=False on your Popen() call, assuming that whatever executable you're running can handle a UNC path, but of course if what you're trying to run is a command that's built in to cmd.exe you don't have a choice.
I have Python 3.1, Subversion 1.6.12, and PySVN installed on Windows XP.
If I open a Python terminal and do
import subprocess
print subprocess.check_output(['svnlook','youngest','D:/svn-repos/myrepo'])
I get the expected revision number.
However, if I add this to Subversion's post-commit.bat, it fails with the error "The handle is invalid":
File "C:\Program Files\Python31\lib\subprocess.py", line 472, in check_output
process = Popen(*popenargs, stdout=PIPE, **kwargs)
File "C:\Program Files\Python31\lib\subprocess.py", line 651, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\Program Files\Python31\lib\subprocess.py", line 750, in _get_handles
p2cread = GetStdHandle(STD_INPUT_HANDLE)
WindowsError: [Error 6] The handle is invalid
What could be causing this, and how would I fix it? I tried changing the Subversion service to run as my user, thinking it was some sort of permissions issue with the default systems account, but that had no effect.
Assuming there's no direct fix for this, how would I work around this? I need some way to retrieve the youngest revision number from a SVN repository without have a local working copy. I've dug through PySVN's Programmer's Reference, but I can't find the equivalent call to "svnlook youngest".
Edit: I'm calling the script from the post-commit.bat like:
#ECHO OFF
"C:\Program Files\Python31\python.exe" "D:\svn-repos\myrepo\hooks\myscript.py"
I ended up using a different SVN binding, svn-python, and that worked. I can only guess there was some mismatch between the Windows binaries for the version of subversion and PySVN.
i think you don't need to use the subprocess (just for this) , you see you can just use :
import os
stdout = os.popen('svnlook youngest D:/svn-repos/myrepo')
print stdout.read()
Occam's razor :)
Because as i see it from here .bat file are old stuff, and subprocess that deal with a lot of redirection , processing i don't think this will work , but maybe i'm mistaken , maybe i just want to found you an excuse, but well ...
By the way, in the python script you do a print and you have #ECHO OFF in your .bat so i don't think it will work maybe you can wrap your command with something like this:
#ECHO ON
"C:\Program Files\Python31\python.exe" "D:\svn-repos\myrepo\hooks\myscript.py"
#ECHO OFF
well good luck :)