I am trying to call mvn commands from Python, for this subprocess module has to be used.
The problem is, this has been working for a long time and all of a sudden does not work anymore, because the executed Maven commands complain about JAVA_HOME not being set, even though it is when i manually type echo $JAVA_HOME into the shell.
I have no idea why it stopped working all of a sudden.
What i would expect
command= "echo $JAVA_HOME"
proc = subprocess.Popen(['bash', '-c', command],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
output, err = proc.communicate()
print(str(output))
prints the path to my Java JDK.
$ echo $JAVA_HOME
prints the path to my Java JDK.
What happens instead
command= "echo $JAVA_HOME"
proc = subprocess.Popen(['bash', '-c', command],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
output, err = proc.communicate()
print(str(output))
prints b'\n'
$ echo $JAVA_HOME
prints epath/to/my/java/jdk
What i already tried
Using shell=True in Popen: Works, but is discouraged due to security risks and it seems to use /bin/sh when executed on our Jenkins, which makes the script crash because some commands are only executable when using bash. It worked without it too, so there must be a way to get along without it.
adding env=os.environ.copy() as argument to Popen: No effect..Even when specifying the JAVA_HOME explicitly using env
Moving the JDK to a path with no weird spaces or anything like that: No effect...
Checking the output of os.environ['JAVA_HOME']: Prints the path to my Java JDK
Information
I am still using the same python version. I did not update anything that could have caused this weird behavior all of a sudden, at least i wouldnt know what it is.
I am using Windows 10 Enterprise, x64 based
I am using GIT Bash
I am using Python 3.8.5
Update 1:
After reading something about problems of shared environment variables between WSL and Windows, i discovered that i can specify shared variables by setting a environment variable 'WSLENV'. I added JAVA_HOME/p to it and now Python subprocess no longer prints b'\n', but b'/tmp/docker-desktop-root/mnt/host/c/Users/user/Desktop/jdk11\n'. So the problem seems to be WSL (?).
Unfortunately, Maven still says JAVA_HOME should point to a JDK not a JRE, so this path seems not to work.
Update 2:
By changing the WSLENV variable's content from JAVA_HOME/p to JAVA_HOME/u, the subprocess now prints the correct path to the JDK. Still, Maven fails with the same error message..
Update 3:
For making it work with WSL enabled, check out my answer below
I found a way to make it work with WSL enabled, it is kinda ugly but it seems to work.
command = "mvn --version"
proc = subprocess.Popen(['wsl', 'bash.exe', '-c', command],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
output, err = proc.communicate()
print(str(output))
print(str(err))
By appending wsl and bash.exe i managed to make it work, the output is the basic output from mvn --version just like expected. Notice the .exe which seems to tell WSL to use the same bash executable like in normal usage without subprocess.
Without the .exe, WSL seems to use a different bash executable, where JAVA_HOME is not defined or at least maven complains about it with the error message that i already mentioned above.
Notice that this code probably won't work when WSL is not enabled, so you would need to programmatically test if WSL is enabled and then modify the command accordingly.
Im still searching for any solution where i dont need to modify the process args, i am gonna update if i will find one.
Related
From python, I need to run a python file inside of git bash, while running in Windows.
That is, I have a configuration script written in python that calls other python scripts. Unfortunately, some of them use Unix commands, so they must be run using git bash in Windows.
Currently I'm using this:
cmd = f'{sys.executable} mydependency.py'
pipe = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
# waiting for pipe is handled later...
However, this doesn't work, giving me a cannot execute binary file message. How can I get it to run?
PS: For slightly more context, mydependency.py is actually the amalgamate.py script from the simdjson (https://github.com/simdjson/simdjson) project.
EDIT:
I have also attempted the following:
Switch to run or call instead of subprocess.Popen
Use f'{git_bash_path} {sys.executable} mydependency.py'
Change the shell and executable parameters of Popen,run and call
I found a solution:
cmd = git_bash_path # Found with glob.
pipe = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
pipe.communicate(input=f'{sys.executable} mydependency.py'.encode())
I'm not entirely sure why this works, if anyone has an explanation I'd be glad to hear it.
I am trying to run a shell script using through Python using subprocess.Popen().
The shell script just has the following lines:
#!/bin/sh
echo Hello World
Following is the Python code:
print("RUNNNING SHELL SCRIPT NOW")
shellscript = subprocess.Popen(['km/example/example1/source/test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
shellscript.wait()
for line in shellscript.stdout.readlines():
print(line)
print("SHELL SCRIPT RUN ENDED")
However, on running this, I am only getting the following output:
RUNNNING SHELL SCRIPT NOW
SHELL SCRIPT RUN ENDED
i.e. I am not getting the shell script output in between these 2 lines.
Moreover, when I remove the stderr=subprocess.PIPE part from the subprocess, I get the following output:
RUNNNING SHELL SCRIPT NOW
'km' is not defined as an internal or external command.
SHELL SCRIPT RUN ENDED
I am not able to understand how to resolve this, and run the shell script properly. Kindly guide. Thanks.
UPDATE:
I also tried the following change:
print("RUNNNING SHELL SCRIPT NOW")
shellscript = subprocess.Popen(['km/example/example1/source/test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
out, err = shellscript.communicate()
print(out)
print("SHELL SCRIPT RUN ENDED")
I get the following output:
RUNNNING SHELL SCRIPT NOW
b''
SHELL SCRIPT RUN ENDED
The simple and straightforward fix is to not use bare Popen for this.
You also don't need a shell to run a subprocess; if the subprocess is a shell script, that subprocess itself will be a shell, but you don't need the help of the shell to run that script.
proc = subprocess.run(
['km/example/example1/source/test.sh'],
check=True, capture_output=True, text=True)
out = proc.stdout
If you really need to use Popen, you need to understand its processing model. But if you are just trying to get the job done, the simple answer is don't use Popen.
The error message actually looks like you are on Windows, and it tries to run km via cmd which thinks the slashes are option separators, not directory separators. Removing the shell=True avoids this complication, and just starts a process with the requested name. (This of course still requires that the file exists in the relative file name you are specifying. Perhaps see also What exactly is current working directory? and also perhaps switch to native Windows backslashes, with an r'...' string to prevent Python from trying to interpret the backslashes.)
Im using Ubuntu 20 with zsh. When I using subprocess.call, it always using bash to exec command but not zsh. How should I do to fix this?
No, it uses sh regardless of which shell is your login shell.
There is a keyword argument to select a different shell, but you should generally run as little code as possible in a subshell; mixing nontrivial shell script with Python means the maintainer has to understand both languages.
whatever = subprocess.run(
'echo $SHELL',
shell=True, executable='/usr/bin/zsh',
check=True)
(This will echo your login shell, so the output would be /usr/bin/zsh even if you ran this without executable, or with Bash instead.)
In many situations, you should avoid shell=True entirely if you can.
I'm trying to automate driver installation using a python script, previously i was using a batch file for the same task, since i'm now building a GUI using python i'd like to incorporate everything into python.
I was using pnputil.exe to install driver: 'pnputil -i -a path_to_inf'
but for some reason i can't make it work in python, i've tried subprocess.call, os.system, but nothing works, i always get some kind of error, using os.system i can run registry commands to read/write/add/remove keys, but with pnputil it just gives me errors.
os.system error = 'pnputil' is not recognized as an internal or external command,
operable program or batch file.
subprocess.call error = subprocess.Popen(['pnputil -i -a path_to_inf'], shell=True) = The filename, directory name or volume label syntax is incorrect.
You have to use the whole address of pnputil.exe to execute in python..
Try this
subprocess.Popen(['C:\\Windows\\System32\\PNPUTIL.exe -i -a path_to_inf'], shell=True)
Or
subprocess.Popen(['C:\\Windows\\SYSNATIVE\\PNPUTIL.exe -i -a path_to_inf'], shell=True)
Either should work, because it is based on 32-bit and 64-bit version
I ran across this issue myself and banged my head against it for quite a while. Putting this up here for anyone else that hits the issue as I wasn't able to find a good explanation of why this problem was occurring.
This problem happens because a 32-bit Python application is attempting to access a 64-bit Windows resource and is being automatically redirected to the incorrect path. More info on that relationship here:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa384187%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396
So, if your Python installation is 64-bit, you're already in the right context, and just a call to pnputil should work. If you're on 32-bit Python, you must redirect to sysnative. The following is the code I'm currently using to grab a list of driver store drivers:
import platform
import subprocess
if '32bit' in platform.architecture():
process = '%systemroot%\Sysnative\pnputil.exe -e'
else:
process = 'pnputil.exe -e'
p = subprocess.Popen(process, shell=True,
stdout=subprocess.PIPE, universal_newlines=True)
I seem to have discovered a problem with subprocess.Popen() on a Windows XP x64 machine. If Popen() is called with Shell=True, instead of a subprocess being created, the error
The syntax of the command is incorrect
is printed to the command prompt.
The problem was discovered when trying to use Mercurial on the command line. 'hg resolve' would crash with the above error. I've narrowed it down to a problem with Popen(), and have made a <10 line script that reproduces the issue. The problem also exists in tortoisehg in a variety of places, also due to the fact it makes use of Popen() with Shell=True.
Interestingly, if I run my test-code with Shell=False, it works fine (expected behaviour occurs). I've tested the code on a Windows x86 build, and with both Shell=True & False it works as expected. This specific machine has some customizations to the PATH, however nothing serious.
Does anybody know what could be different about this x64 machine to cause this error. The machine was already installed/setup before I came into the job, so I am unsure what has been changed. The only idea I've had so far is that there's some error occurring due to incorrect escaping of parameters (or something related), however I'm unsure where this could occur.
Just for informational purposes, I've posted one of Mercurial's calls to Popen, in case anybody spots something which might be the issue
p = subprocess.Popen(cmd, shell=True, bufsize=-1,
close_fds=closefds,
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
universal_newlines=newlines,
env=env)
Unfortunately I can't post the test-code right now (it's at work), however I will post an update the moment I can get my hands on it.
Update, the code for the test is below...
import os
import sys
import subprocess
if 'THG_HGTK_SPAWN' in os.environ:
print 'spawn successful'
sys.exit(0)
os.environ['THG_HGTK_SPAWN'] = '1'
subprocess.Popen(args={'python', r'test.py'}, shell=True)
print 'spawning...'
sys.exit(0)