We Keep Coding

Python Subprocess.Popen does not inherit JAVA_HOME

I am trying to call mvn commands from Python, for this subprocess module has to be used.
The problem is, this has been working for a long time and all of a sudden does not work anymore, because the executed Maven commands complain about JAVA_HOME not being set, even though it is when i manually type echo $JAVA_HOME into the shell.
I have no idea why it stopped working all of a sudden.
What i would expect
command= "echo $JAVA_HOME"
proc = subprocess.Popen(['bash', '-c', command],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
output, err = proc.communicate()
print(str(output))
prints the path to my Java JDK.
$ echo $JAVA_HOME
prints the path to my Java JDK.
What happens instead
command= "echo $JAVA_HOME"
proc = subprocess.Popen(['bash', '-c', command],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
output, err = proc.communicate()
print(str(output))
prints b'\n'
$ echo $JAVA_HOME
prints epath/to/my/java/jdk
What i already tried
Using shell=True in Popen: Works, but is discouraged due to security risks and it seems to use /bin/sh when executed on our Jenkins, which makes the script crash because some commands are only executable when using bash. It worked without it too, so there must be a way to get along without it.
adding env=os.environ.copy() as argument to Popen: No effect..Even when specifying the JAVA_HOME explicitly using env
Moving the JDK to a path with no weird spaces or anything like that: No effect...
Checking the output of os.environ['JAVA_HOME']: Prints the path to my Java JDK
Information
I am still using the same python version. I did not update anything that could have caused this weird behavior all of a sudden, at least i wouldnt know what it is.
I am using Windows 10 Enterprise, x64 based
I am using GIT Bash
I am using Python 3.8.5
Update 1:
After reading something about problems of shared environment variables between WSL and Windows, i discovered that i can specify shared variables by setting a environment variable 'WSLENV'. I added JAVA_HOME/p to it and now Python subprocess no longer prints b'\n', but b'/tmp/docker-desktop-root/mnt/host/c/Users/user/Desktop/jdk11\n'. So the problem seems to be WSL (?).
Unfortunately, Maven still says JAVA_HOME should point to a JDK not a JRE, so this path seems not to work.
Update 2:
By changing the WSLENV variable's content from JAVA_HOME/p to JAVA_HOME/u, the subprocess now prints the correct path to the JDK. Still, Maven fails with the same error message..
Update 3:
For making it work with WSL enabled, check out my answer below

I found a way to make it work with WSL enabled, it is kinda ugly but it seems to work.
command = "mvn --version"
proc = subprocess.Popen(['wsl', 'bash.exe', '-c', command],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
output, err = proc.communicate()
print(str(output))
print(str(err))
By appending wsl and bash.exe i managed to make it work, the output is the basic output from mvn --version just like expected. Notice the .exe which seems to tell WSL to use the same bash executable like in normal usage without subprocess.
Without the .exe, WSL seems to use a different bash executable, where JAVA_HOME is not defined or at least maven complains about it with the error message that i already mentioned above.
Notice that this code probably won't work when WSL is not enabled, so you would need to programmatically test if WSL is enabled and then modify the command accordingly.
Im still searching for any solution where i dont need to modify the process args, i am gonna update if i will find one.

Using pyinstaller to convert Python3 and Python2 to .exe

I am currently writing a script, on Python 3.
I need to call an already existing Python 2 script, from within my Python 3 script.
In the Python3 script I am doing this using something like:
import subprocess
my_command = 'python script_in_py2.py arg1 arg2'
process = subprocess.Popen(my_command.split(), stdout = subprocess.PIPE)
output, error = process.communicate()
Thus, variable output will get the output that would be displayed on screen if I had just written my_command in terminal.
Although I am currently developing in Ubuntu, eventually I will migrate it to Windows where I will need to convert it to an .exe. For this I have previously used pyinstaller.
I know that running the resulting .exe (of a purely Python3 script) on a Windows machine without Python installed at all works.
However I need to know before going on, whether it will also work like this. My intuition is no (in my knowledge creating a process like that is similar to running the command directly in Terminal/Command Prompt), however I would like to know if anybody knows of a way/workaround. (Installing Python2 on the machine that would run the script is not an option).

A possible solution could be like this:
Convert your python2 script i.e. 'script_in_py2.py' into an executable i.e. 'script_in_py2.exe' using pyinstaller.
pyinstaller script_in_py2.py
In your python3 script, call this python 2 executable instead of calling python 2 script.
import subprocess
my_command = 'script_in_py2.exe arg1 arg2'
process = subprocess.Popen(my_command.split(), stdout = subprocess.PIPE)
output, error = process.communicate()
Now convert your python3 script into an executable using pyinstaller.
pyinstaller script_in_py3.py
Paste the executable 'script_in_py2.exe' and 'python27.dll' in dist folder of 'script_in_py3' (i.e. parallel to location of executable 'script_in_py3.exe')
Run "script_in_py3.exe" and you will get desired output.
I have tried it on my system and its working smoothly.

Formatting Windows Command Line Call For Executing R Script from Python

This is similar to a number of questions, but could still use some clarity. My example is still unable to run, so I may be missing something.
I'm trying to run a full R script from Python. I don't need to see it, or get output, but need it to run and preferably know when it's completed.
I am using Python 3.7.0 (Run from Spyder), R 3.5.2, and Windows 10.
I can run this from Windows Command Line with:
Rscript test_script.R
When I'm in the location with the script, or if I'm not in that location:
Rscript C:/Path/to/test_script.R
From Python I tried:
import subprocess
command = 'Rscript'
script = 'C:/Path/to/test_script.R'
subprocess.Popen([command,script], shell=True)
This does not give an error, but I can see from the directories where the Rscript would create files, that nothing has run (And it would take awhile to run). In the Python console, the 'output' displays '1'
I have also tried:
import subprocess
command = 'C:/Path/to/Rscript.exe --vanilla -q -f C:/Path/to/test_script.R'
subprocess.Popen(command, shell=True)
I'm not sure what the --Vannilla -q -f means, but I saw it used often in other questions.
Any ideas on where I am going wrong or how to get this to execute?

Why does this Python subprocess command only work when shell=True on Windows?

I have a python script that involves multiple subprocess.call commands. I wrote the script on a Mac, and it runs perfectly. I just tried running it on Windows and I am baffled by an error.
The following command to call ImageMagick returns "exit status 4":
file1 = "D:/Temp/OCR_test/sample/images/crops/time_0011.png"
subprocess.call(['convert', file1, '-resize', '200%', file1])
Changing the command to the following works:
subprocess.call(['convert', file1, '-resize', '200%', file1], shell=True)
I'm a little wary of using shell=True because of the warnings in the documentation.
I also need the command to work on both Mac and Windows, and I am confused as to why it wouldn't work on Windows (I checked and the command does work using Windows CMD).
Interestingly, the following line worked earlier in the script (where file, lat_crop1, and croplat are defined variables):
subprocess.call(['ffmpeg', '-loglevel', 'panic', '-i', file, '-vf', lat_crop1, '-n', croplat])
I read this SO question and tried all the suggestions (shlex, variations of my command, etc...), but I still get the same result.
Anyone have any idea how I could modify that line so it can work without shell=True?
Also, what does "exit status 4" mean? I Googled and read so much documentation but found nothing about it.
EDIT: Based on the information provided in the answer, I changed to the command that was not working to subprocess.call(['mogrify', file1, '-resize', '200%', file1]) and that runs successfully in Python on Windows. Luckily ImageMagick provides mogrify as an alternative to convert.

I suspect you're calling C:\Windows\System32\convert.exe (NTFS/FAT partition converter) instead of imagemagick.
When shell=True, the path finds the convert.bat or convert.cmd script from imagemagick, but without it, the path can only find the .exe file, which is a completely different program, and you get error 4: invalid parameter.
In that particular case, it doesn't work even with an executable, because the "wrong" convert is located in a system path. shell=False only searches in system paths (python subprocess Popen environment PATH?). So that's bad luck that a program named convert is located in the system path.
Try to explicitly add .bat extension like this:
subprocess.call(['convert.bat', file1, '-resize', '200%', file1])
To know which executables are likely to be run you can type:
where convert
in a command prompt.
In your case (an executable), that could be workarounded by passing the absolute path of the executable you want to run.
Another way would be to copy/rename ImageMagick convert to imconvert. Which program calls itself convert and doesn't expect conflicts anyway ?
Or in that case, it's legitimate to leave shell=True, with a nice comment explaining that Microsoft left a confusing (and seldom used convert program in a system path for us to trip into)
Solutions are not pretty, at least there are some.

Execute bash script from Python on Windows

I am trying to write a python script that will execute a bash script I have on my Windows machine. Up until now I have been using the Cygwin terminal so executing the bash script RunModels.scr has been as easy as ./RunModels.scr. Now I want to be able to utilize subprocess of Python, but because Windows doesn't have the built in functionality to handle bash I'm not sure what to do.
I am trying to emulate ./RunModels.scr < validationInput > validationOutput
I originally wrote this:
os.chdir(atm)
vin = open("validationInput", 'r')
vout = open("validationOutput", 'w')
subprocess.call(['./RunModels.scr'], stdin=vin, stdout=vout, shell=True)
vin.close()
vout.close()
os.chdir(home)
But after spending a while trying to figure out why my access was denied, I realized my issue wasn't the file permissions but the fact that I was trying to execute a bash file on Windows in general. Can someone please explain how to execute a bash script with directed input/output on windows using a python script?
Edit (Follow up Question):
Thanks for the responses, I needed the full path to my bash.exe as the first param. Now however, command line calls from within RunModels.scr come back in the python output as command not found. For example, ls, cp, make. Any suggestions for this?
Follow up #2:
I updated my call to this:
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
The error I now get is /usr/bin/bash: RunModels.scr: No such file or directory.
Using cwd does not seem to have any effect on this error, either way the subprocess is looking in /usr/bin/bash for RunModels.scr.
SELF-ANSWERED
I needed to specify the path to RunModels.scr in the call as well as using cwd.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'C:\\path\\dir_where_RunModels\\RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
But another problem...
Regardless of specifying cwd, the commands executed by RunModels.scr are throwing errors as if RunModels.scr is in the wrong directory. The script executes, but cp and cd throw the error no such file or directory. If I navigate to where RunModels.scr is through the command line and execute it the old fashioned way I don't get these errors.

Python 3.4 and below
Just put bash.exe in first place in your list of subprocess.call arguments. You can remove shell=True, that's not necessary in this case.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Depending on how bash is installed (is it in the PATH or not), you might have to use the full path to the bash executable.
Python 3.5 and above
subprocess.call() has been effectively replaced by subprocess.run().
subprocess.run(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Edit:
With regard to the second question, you might need to add the -l option to the shell invocation to make sure it reads all the restart command files like /etc/profile. I presume these files contain settings for the $PATH in bash.
Edit 2:
Add something like pwd to the beginning of RunModels.scr so you can verify that you are really in the right directory. Check that there is no cd command in the rc-files!
Edit 3:
The error /usr/bin/bash: RunModels.scr: No such file or directory can also be generated if bash cannot find one of the commands that are called in the script. Try adding the -v option to bash to see if that gives more info.

A better solution than the accepted answer is to use the executable keyword argument to specify the path to your shell. Behind the curtain, Python does something like
exec([executable, '-c`, subprocess_arg_string])
So, concretely, in this case,
subprocess.call(
'./RunModels.scr',
stdin=vin, stdout=vout,
shell=True,
executable="C:/cygwin64/bin/bash.exe")
(Windows thankfully lets you use forward slashes instead of backslashes, so you can avoid the requirement to double the backslashes or use a raw string.)