I have a python file test.py that must be executed in the python2.7 interpreter, while some other code in main.py requires the python3.6 interpreter.
I had to call the python3.6 interpreter through test.py to execute the code in main.py because main.py relies on some specific versions of libraries that are not available in python2.7. Also, since I'm writing an app plugin, the plugin's interpreter is only version 2.7. So I have to make cross-version calls. And there are parameters to be passed between them.
I tried opening a subprocess through the subprocess module in test.py. But the following error was encountered.
Codes in test.py:
import subprocess
epath = r'E:\Anaconda3\envs\***\python.exe'
fpath = r'D:\Codes\_Projects\***\main.py'
p = subprocess.call(
[epath, fpath],
stdin=None,
stdout=None,
stderr=None,
creationflags=subprocess.CREATE_NEW_CONSOLE,
shell=False
)
for line in iter(p.stdout.readline, b''):
print line
p.stdout.close()
p.wait()
And I got an error (in the cmd window where the subprocess executes):
Python error: Py_Initialize: unable to load the file system codec
File "E:/Slicer_4.10.2/bin/../lib/Python/Lib\encodings\__init__.py", line 123
raise CodecRegistryError,\
^
SyntaxError: invalid syntax
Current thread 0x00006a00 (most recent call first):
The path E:/Slicer_4.10.2/bin/ is the directory of the python2.7 interpreter. Slicer is the software I'm developing the plugin for. It comes with a python2.7 interpreter. My test.py must be executed with this interpreter.
I googled this error and someone say that this is due to a mismatch of environmental variables. So I printed out the environmental variables in test.py and here are some of them:
'PYTHONPATH': 'E:/Slicer_4.10.2/bin/../lib/Slicer-4.10;E:/Slicer_4.10.2/bin/../lib/Slicer-4.10/qt-scripted-modules;E:/Slicer_4.10.2/bin/../lib/Slicer-4.10/qt-loadable-modules;E:/Slicer_4.10.2/bin/../lib/vtkTeem;E:/Slicer_4.10.2/bin/../bin/Python;E:/Slicer_4.10.2/bin/../lib/Slicer-4.10/qt-loadable-modules/Python;E:/Slicer_4.10.2/bin/../lib/Python/Lib;E:/Slicer_4.10.2/bin/../lib/Python/Lib/lib-dynload;E:/Slicer_4.10.2/bin/../lib/Python/Lib/site-packages;E:/Slicer_4.10.2/bin/.././bin/Lib/site-packages;E:/Slicer_4.10.2/bin/.././lib/Slicer-4.10/python2.7/site-packages;E:/Slicer_4.10.2/bin/../lib/Python/lib/Python/Lib;E:/Slicer_4.10.2/bin/../lib/Python/lib/Python/Lib/lib-dynload;E:/Slicer_4.10.2/bin/../lib/Python/lib/Python/Lib/site-packages;E:/Slicer_4.10.2/bin/../bin/Python',
'PYTHONHOME': 'E:/Slicer_4.10.2/bin/../lib/Python',
'PYTHONNOUSERSITE': '1',
It can be seen that all environment variables related to python are under the Slicer path, which means that these are environment variables added by the python2.7 interpreter, during runtime.I'm sure for that because I didn't add them manually to the system's environment variables. Obviously they are not the environment that python3.6 intepreter needs.
So, is there any way to reset environment variables, or run subprocesses in a cmd/powershell that has no environment variables like PYTHONPATH set?
My OS is windows 10.0.
Related
I got a Node.js CLI program called meyda installed (Mac OS 10.14) using:
sudo npm install --global meyda
From the Terminal I can call the program and it works as expected; like:
meyda --bs=256 --o=apagodis2.csv DczN6842.wav rms
Now, I want to call it from inside a python script (using Spyder) at the same location and tried this – but getting error:
import os
os.system ('/usr/local/bin/meyda --bs=256 --o=apagodis4.csv samples_training/DczN6842.wav rms')
>>> env: node: No such file or directory
I can issue more "traditional" shell commands like this from the same Python script and it works:
os.system ('cp samples_training/DczN6842.wav copy.wav')
Also tried subprocess call with same result. I confirmed the executable is at /usr/local/bin/
To make sure I also removed all file arguments calling the program using only the help flag but same, error.
os.system ('/usr/local/bin/meyda -h')
>>> env: node: No such file or directory
Why is the command not found from inside Python but sucessfully in the macOS Terminal?
I have a source file having environment variables (few environmental variables are conditional based as well).
source file is like (filename: global_setup)
if( -f /tools/tool/bin) then
setenv SHELL /bin/csh
endif
setenv LICENCE_FILE 2457#abc
setenv ...........
I want to source this file using python script. Code is like this:
import subprocess
import os
if os.path.isfile('global_setup'):
subprocess.call(['/bin/csh', '-c', 'source ~/global_setup'], shell=True)
else:
print("file not found");
But this is not working. Showing error as:
/home/global_setup: line 53: syntax error: unexpected end of file
Also, I tried to remove this EOF error by changing file format to Unix.
Even it works, i doubt setup will be done in some other terminal as i am using subprocess.call() in python.
Sourcing a file with subprocess.call() won't have an effect on the environment variables in Python, since subprocess runs, well, a subprocess.
You will need to parse the file yourself if you wish to affect os.environ within your Python interpreter.
Since your file is csh-style, the otherwise very nice envparse module won't do here.
Something you could do, though, is invoke
env = subprocess.check_output(['/bin/csh', '-c', 'source ~/global_setup; printenv'], shell=True)
and parse the resulting env variable line by line and add the contents into os.environ. This way the conditionals within global_setup would also work as expected.
Instead of doing it this way, you should be doing one of two things:
Source your files in the linux shell before invoking your script.
Convert your file into a dotenv file that Python can read using the Python-dotenv module.
Sourcing it before is easier IMO, instead of sourcing it during runtime using subprocess.
I am using Pycharm for Python development. Code is running in Pycharm venv from where I want to call a bash script which activates and deactivates conda environments (this one: https://github.com/lykaust15/DeepSimulator).
So I need to deactivate the Pycharm venv when calling the script via subprocess.call(), as far as I understand. Otherwise I get errors. How do I do this?
My call is:
result = subprocess.call([deppsim_path, "-i", fasta, "-c", "4",
"-C", "1", "-n", "100",
"-H", os.path.dirname(deppsim_path),
"-o", tmp_path], shell=False)
EDIT:
Some of the errors:
/home/user/path/bin/venv/bin/python /home/user/path/bin/sim_seq.py
Pre-process input genome...
/home/user/path/bin/DeepSimulator/deep_simulator.sh: Zeile 207: deactivate: Datei oder Verzeichnis nicht gefunden
Pre-process input genome done!
Executing the preprocessing step...
Traceback (most recent call last):
File "/home/user/path/bin/DeepSimulator/util/genome_sampling.py", line 5, in <module>
import scipy.stats as st
ImportError: No module named scipy.stats
You can always call a particular install of python, including virtual environments, using the full path to the python executable. So that's your answer. Use a full path to a python binary that is in the actual Python installation you want to use to run the subprocesses, and you won't involve any virtual environments in running the subprocesses.
As an example, here's what two Python binary paths look like on my system:
> which python
/usr/local/bin/python
> venv development
> which python
/Users/stevenjohnson/envs/development/bin/python
So /usr/local/bin/python is my actual Python 2.7 installation, and /Users/stevenjohnson/envs/development/bin/python is my "development" virtual environment. So if I run the main program in my virtual environment, but if I call /usr/local/bin/python in my subprocess call, The subprocess will run using the base Python 2.7 installation.
I have 2 python interpreters installed on a windows 10 computer. Python 3.6 from ArcGIS pro 2.2 and Python 2.7 from ArcMap 10.6, using the cmd on Windows:
"C:/Program Files/ArcGIS/Pro/bin/Python/Scripts/propy" -c "import arcpy;print(arcpy.__path__)"
The output is:
'C:\\Program Files\\ArcGIS\\Pro\\Resources\\ArcPy\\arcpy'
This is the intended result, but I need to do the same using subprocess.Popen from Python2.7 and also need to use shell=True
cmd = """ "C:/Program Files/ArcGIS/Pro/bin/Python/Scripts/propy" -c "import arcpy;print(arcpy.__path__)" """
subprocess.Popen(cmd, shell=True,stdout = subprocess.PIPE)
process.stdout.read()
This will raise the error:
Fatal Python error: Py_Initialize: unable to load the file system codec
File "C:\Python27\ArcGIS10.5\lib\encodings\__init__.py", line 123
raise CodecRegistryError,\
Basically the interpreter being used is Python2.7, I have tried to set %PYYHONPATH% to Python3.6 but still the same problem
Just a note, propy is a .bat that activates Python's virtualenv and therefore should be setting everything properly
How could I have the proper output using subprocess.Popen ?
We had a similar problem when trying to run python scripts within ArcGIS Workflow Manager within ArcGIS Pro. The problem is ArcGIS Workflow Manager loads up Python 2.X but we needed Python 3.X
To resolve the problem we added the following in our __init__.py
import sys
if sys.version_info[0] >= 3:
# When run through Workflow Manager 10.5, the PYTHONPATH environment
# variable is being set to the Python 2 environment, and that needs to be removed
# when running in the Python 3 environment.
sys.path = list(filter(lambda p: 'arcgis\\desktop' not in p, sys.path))
I'm currently working on Pycharm with remote python Interpreter(miniconda3/bin/python).
So when I type echo $PATH in remote server, it prints
/home/woosung/bin:/home/woosung/.local/bin:/home/woosung/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
I created project in Pycharm and set remote python Interpreter as miniconda3 python, it works well when I just run some *.py files.
But when I typed some os.system() lines, weird things happened.
For instance, in test.py from Pycharm project
import os
os.system('echo $PATH')
os.system('python --version')
Output is
ssh://woosung#xxx.xxx.xxx.xxx:xx/home/woosung/miniconda3/bin/python -u /tmp/pycharm_project_203/test.py
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
Python 2.7.12
Process finished with exit code 0
I tried same command in remote server,
woosung#test-pc:~$ echo $PATH
/home/woosung/bin:/home/woosung/.local/bin:/home/woosung/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
woosung#test-pc:~$ python --version
Python 3.6.6 :: Anaconda, Inc.
PATH and the version of python are totally different! How can I fix this?
I've already tried add os.system('export PATH="$PATH:$HOME/miniconda3/bin"') to test.py. But it still gives same $PATH.(/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games)
EDIT
Thanks to the comment of #Dietrich Epp, I successfully add interpreter path to the shell $PATH.
(os.environ["PATH"] += ":/home/woosung/miniconda3/bin")
But I stuck the more basic problem. When I add the path and execute command the some *.py file including import library which is only in miniconda3, the shell gives ImportError.
For instance, in test.py
import matplotlib
os.environ["PATH"] += ":/home/woosung/miniconda3/bin"
os.system("python import_test.py")
and import_test.py
import matplotlib
And when I run test.py,
Traceback (most recent call last):
File "import_test.py", line 1, in <module>
import matplotlib
ImportError: No module named matplotlib
Looks like the shell doesn't understand how to utilize modified $PATH.
I find the solution.
It is not direct but quite simple.
I changed os.system("python import_test.py") to os.system(sys.executable + ' import_test.py').
This makes the shell uses the Pycharm remote interpreter(miniconda3), not original.