Resolving errors in python environment on Windows - python

I just started having some problems with python, and would like to fix this the proper way, as a last resort, reinstalling if I have to.
Before Python 3, I had Python 2.7 installed. I then installed Python 3.7 along side it. I don't remember, if the path environment was automatically set, or if i manually set it.
My system variables path contains :
C:\Python27\;C:\Python27\Scripts;
So I believe this was set automatically.
My user variables path however contains :
C:\Python37\;C:\Python37\Scripts;
Would the python installer use the user environment? I'm not sure, so I don't know if I set that myself.
However, the user variables path also contains :
%PYTHON_DIR%\Python37\Scripts\;%PYTHON_DIR%\Python37\;%PYTHON_DIR%\Python36\Scripts\;%PYTHON_DIR%\Python36\
Again, I don't know if I set this myself, and I have no recollection of setting this in my user variables :
PYTHON_DIR = C:\Users\pcuser\AppData\Local\Programs\Python
...but it's there, and I have two folders in that path.
I also have this in my user variables path :
PYTHON = os.path.expanduser(os.getenv('PYTHON', 'C:\\Python37\\python.exe'))
I know I created a python environment. I just have to look for it, as I don't remember where to find it right now.
Here is the problem I am having.
I also am using Python in MSYS, and get this error from Python in C:\msys64\usr\bin :
Could not find platform independent libraries <prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000000800018040 (most recent call first):
0 [main] python 1489 cygwin_exception::open_stackdumpfile: Dumping stack t
race to python.exe.stackdump
I set PYTHONHOME to C:\msys64\usr\bin. That doesn't solve the problem.
I also removed PYTHON_DIR to see if there was a conflict there, but that didn't solve the problem.
Rather, I got another error :
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000000800018040 (most recent call first):
0 [main] python 143 cygwin_exception::open_stackdumpfile: Dumping stack tr
ace to python.exe.stackdump
So it seems to me my Python environment is messed up, and I want to learn how to set it up correctly, hopefully without having to start over.
How can I get rid of these errors, and get a proper python environment?

Personally, I use miniconda.
Miniconda is the small version of anaconda which is an excellent environment manager. Miniconda does not include a gui and I feel it is nicer to work with.
To create an environment with
conda create -n my-test-env python=3.9
You can activate it using
conda activate my-test-env
You can install other packages using
conda install numpy.
However, not all python packages are available in conda. You can any pip packages by installing pip
conda install pip
pip install numpy

Setting PYTHONHOME to a value like C:\msys64\usr\bin is definitely wrong; it should be something like C:\Python37\Lib if that's where Python installed its libraries. But try simply unsetting it.

After a while of troubleshooting, I was able to discover where the problem was.
Some files in C:\msys64\usr\lib\python3.8 somehow got deleted, or python 3.8 was installed, but not properly. So I did a reinstall, and python stopped complaining about missing encodings and modules.
Running a test on all python installations
MINGW64 ~
# python
Python 3.8.2 (default, Feb 27 2020, 05:27:33) [GCC 9.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
# python setup_build.py install
running install
running build
running build_ext
skipping 'geotools/geotools.c' Cython extension (up-to-date)
running install_lib
copying build/lib.mingw-3.8/geotools-cpython-38.dll -> C:/msys64/mingw64/lib/python3.8/site-packages
running install_egg_info
Writing C:/msys64/mingw64/lib/python3.8/site-packages/geotools-0.1.0-py3.8.egg-info
MINGW32 ~
# python
Python 3.8.2 (default, Feb 27 2020, 06:39:26) [GCC 9.2.0 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
# python setup_build.py install
running install
running build
running build_ext
dllwrap: WARNING: dllwrap is deprecated, use gcc -shared or ld -shared instead
skipping 'geotools/geotools.c' Cython extension (up-to-date)
running install_lib
running install_egg_info
Removing C:/msys64/mingw32/lib/python3.8/site-packages/geotools-0.1.0-py3.8.egg-info
Writing C:/msys64/mingw32/lib/python3.8/site-packages/geotools-0.1.0-py3.8.egg-info
C:\WINDOWS\system32>py -3.7 -m pip list
Package Version
----------------------------- ---------
...
C:\WINDOWS\system32>py -2.7 -m pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please
upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop s
upport for Python 2.7 in January 2021. More details about Python 2 support in pi
p can be found at https://pip.pypa.io/en/latest/development/release-process/#pyt
hon-2-support pip 21.0 will remove support for this functionality.
Package Version
----------------------------- ----------
...
C:\WINDOWS\system32>python
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AM
D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', 'C:\\WINDOWS\\SYSTEM32\\python27.zip', 'C:\\Python27\\DLLs', 'C:\\Python27\
\lib', 'C:\\Python27\\lib\\plat-win', 'C:\\Python27\\lib\\lib-tk', 'C:\\Python27
', 'C:\\Users\\pcUser\\AppData\\Roaming\\Python\\Python27\\site-packages', 'C:\\Py
thon27\\lib\\site-packages', 'C:\\Python27\\lib\\site-packages\\pybind11-2.6.2-p
y2.7.egg']
>>>
C:\WINDOWS\system32>py -3
Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSC v.1900 64 bit
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', 'C:\\Users\\pcUser\\AppData\\Local\\Programs\\Python\\Python37\\python37.zip'
, 'C:\\Users\\pcUser\\AppData\\Local\\Programs\\Python\\Python37\\DLLs', 'C:\\User
s\\pcUser\\AppData\\Local\\Programs\\Python\\Python37\\lib', 'C:\\Users\\pcUser\\App
Data\\Local\\Programs\\Python\\Python37', 'C:\\Users\\pcUser\\AppData\\Roaming\\Py
thon\\Python37\\site-packages', 'C:\\Users\\pcUser\\AppData\\Local\\Programs\\Pyth
on\\Python37\\lib\\site-packages']
>>>
C:\WINDOWS\system32>py -3.6 -m pip list
Package Version
--------------- --------
...
My python environment is good. All working nicely. :)
What I learned...
Using PYTHONHOME and PYTHONPATH is not neccesary, as was suggested by many... although equally suggessted by many.
From my experience - speaking fron a novice perspective - I would not recommend using them.
Maybe they work for others, but I got errors regardless of how I set it up - whether
C:\Python37 or
C:\Python37\lib or
C:\Users\pcUser\AppData\Local\Programs\Python\Python37 or
C:\msys64\usr\lib\python3.8;C:\msys64\usr\lib\python3.8\site-packages;C:\msys64\usr\lib\python3.8\lib-dynload;C:\msys64\usr\lib\python3.8\distutils
They all return a heap of errors on one installment or other.
So it does not work for me. However, probably works for others.
I am happy though that after probably more than 60 hours, my python environment is back in working order. :)

Related

ModuleNotFoundError on virtual environment

After installing miniconda, my python modules stopped working, throwing ModuleNotFoundError. From what I can tell, miniconda changed my default environment settings. I checked both .bash_profile and .bashrc and updated the files to give conda the lowest priority. This fixed my default python version but didn't fix any of the broken modules.
Next I checked my PYTHONPATH with python3 -c "import sys;print(sys.path)". I discovered that the PYTHONPATH consisted entirely of conda python paths instead of the python version I had called. For reference, my default python version should be 3.8 (now set in .bashrc), and the conda version is 3.9.
['', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload']
I then manually changed my PYTHONPATH in the .bashrc file to include the appropriate library paths. After reloading .bashrc:
['', '/Users/Ghoti/venv/3.8/lib/python3.8/site-packages', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload']
My modules now work! However, I haven't been able to figure out how to stop the conda 3.9 libraries from being appended to my PYTHONPATH. In addition, my printed python version is wrong.
Ghoti$ python --version
Python 3.9.6
Ghoti$ which python
/Users/Ghoti/venv/3.8/bin/python
I was able to "fix" my ModuleNotFoundError problem. However, the solution is only temporary. If I ever need to switch python version/environment, I'll have to go through the process again. I'd like to figure out what is overriding my PYTHONPATH, causing it to call conda 3.9 libraries, and fix the python version irregularity. I've considered that there might be a script/process running in the background, but I haven't found any related to conda/miniconda. I've also been looking for a python setting/config file. No luck. Any suggestions on where I should look?
Edit - Did some more digging. It looks like my version 3.8 python executable was entirely overwritten, and the only existing python installation that is version 3.9.6 is in my "/usr/bin". The two conda environments have versions "3.9.12" and "3.8.13". I feel more confident the issue isn't due to conda, but unsure what could have caused the problem.
Final Edit
I don't think the problem was miniconda. I did start having problems within a few days of using miniconda and I original assumed that it just took me a while to notice the issues. However, I now think that my virtual environment was created using a shared python. Problems were noticed on the same day that I connected to network. The shared python version changed, and that broke my environment. I don't have a solution to salvage the broken environment, but rebuilding it from scratch shouldn't take too long.
Sounds like you only want to use conda when you explicitly need it, in other words, the default Python is the system Python.
If that's the case, you should disable the auto-activation of the base environment:
conda config --set auto_activate_base false
<restart shell>
Now you'll need to explicitly activate the conda environment before you can use the conda Python:
$ python
Python 3.10.6 (main, Aug 11 2022, 13:49:25) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ conda activate base
(base) $ python
Python 3.9.12 (main, Jun 1 2022, 06:36:29)
[Clang 12.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Python 3.6.x + Windows: Interpreter just shuts down for any external library

Due to external dependencies I need to run python 3.6.x (I know it is EOL) on Windows.
However, any external library causes the interpreter to just silently quit.
As an example I have installed numpy in the following way
py -m pip install numpy
And then running Python through PowerShell:
PS C:\Users\USER> py
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
PS C:\Users\USER>
As soon as I run import numpy the python interpreter simply silently returns.
I have tried to install every python 3.6.x available from Pythons official website, and the issue persists.
I fully removed old versions of python between each install.
I cannot find any error logs or anything.
What can be the issue here and how can I debug further?
Any internal library (that I have tried) works
I have tried with the following libraries:
Numpy
Tensorflow
Protobuf
matplotlib
scipy
pandas
nose
sympy

Anaconda installed but No module named 'conda' after installing flake8

While trying to install atom-lint package in Atom editor I somehow corrupted my conda installation.
I did the following things that might have caused the issue:
Installed a python dependency Flake8 using conda install.
Messed around with Atom Init Script (I can provide more info if needed)
After I did these things I encountered the following problem:
If I run conda I get
$ conda
Traceback (most recent call last):
File "/Users/me/miniconda3/bin/conda", line 12, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Strangely enough if I run python in my command line anaconda still seems to be installed an working.
$ python
Python 3.7.1 (default, Oct 23 2018, 14:07:42)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
I have also noticed that the Python3 bin has disappeared from the conda environment folder ~/miniconda3/envs/my_env/bin/
This is very strange and I thought might be related.
Any help would be much appreciated.
I ended up making a backup copy of the miniconda3/envs folder, reinstalling miniconda and copying the environment back in. It works now, not sure what caused the issue.

How to use Python Dbus bindings in Anaconda

I am trying to install dbus on Anaconda python environment and I am struggling.
Here is the error message I am getting:
e#gateway:~$ python
Python 3.5.4 |Anaconda custom (64-bit)| (default, Oct 13 2017, 11:22:58)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbus
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/e/anaconda3/lib/python3.5/site-packages/dbus/__init__.py", line 77, in <module>
import dbus.types as types
File "/home/e/anaconda3/lib/python3.5/site-packages/dbus/types.py", line 6, in <module>
from _dbus_bindings import (
ImportError: /home/e/anaconda3/lib/python3.5/site-packages/_dbus_bindings.so: undefined symbol: _Py_ZeroStruct
>>>
Here are some of the outputs I think may be asked:
e#gateway:~$ conda install dbus
Fetching package metadata ...........
Solving package specifications: .
# All requested packages already installed.
# packages in environment at /home/e/anaconda3:
#
dbus 1.10.22 h3b5a359_0
e#gateway:~$ sudo apt-get install libdbus-glib-1-dev libdbus-1-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libdbus-glib-1-dev is already the newest version (0.106-1).
libdbus-1-dev is already the newest version (1.10.6-1ubuntu3.3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
e#gateway:~$ sudo apt-get install dbus
Reading package lists... Done
Building dependency tree
Reading state information... Done
dbus is already the newest version (1.10.6-1ubuntu3.3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
e#gateway:~$ which python
/home/e/anaconda3/bin/python
e#gateway:~$ conda --version
conda 4.3.31
e#gateway:~$ sudo /home/e/anaconda3/bin/python -m pip install dbus-python
The directory '/home/e/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/e/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: dbus-python in ./anaconda3/lib/python3.5/site-packages
DBus is working fine on the system python, however not working on Anaconda Python.
Python 2.7:
e#gateway:~$ which python
/usr/bin/python
e#gateway:~$ python
Python 2.7.12 (default, Nov 20 2017, 18:23:56)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbus
>>>
Python 3.5:
e#gateway:~$ which python3
/usr/bin/python3
e#gateway:~$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbus
>>>
Can anyone help me? Am I missing something blatantly obvious here?
Thanks in advance.
I had similar issues, there are few cases where dbus and python don't work well out-of-the-box. The consensus appears to be that you need a system-level install (i.e. apt-get) to get dbus to work. I believe the /home/e/anaconda3/lib/python3.5/site-packages/_dbus_bindings.so: undefined symbol: _Py_ZeroStruct error you're seeing is directly related to that.
conda install dbus does not add anything to ~/anaconda3/lib/python3.6/site-packages, but instead appears to install some executables in ~/anaconda3/bin/ like dbus-run-session, dbus-daemon, etc. This makes some sense when you analyze the contents of the dbus tarball https://anaconda.org/conda-forge/dbus, as it's all C files and executables. I'm not sure it's supposed to be the dbus python module, but I could be wrong.
EDIT:
I searched the conda repositories and found a few individuals that uploaded a version of dbus-python, presumably that they compiled and installed. I tried this one out in a py3.6 conda environment via:
conda install -c scottwales dbus-python
I was then able to import dbus. This is a hacky approach and should not be used in production, I'd recommend listening to
Carlos Cordoba's post below. But if you need a solution now, search through some user conda packages or try to compile the library yourself.
Can anyone help me? Am I missing something blatantly obvious here?
Yes, you are. There's one thing people still don't understand about conda: conda is not a pip replacement. It is a general package manager, in the same vein as apt-get, yum, brew, emerge, etc, but cross-platform and based on Python.
In this case, that means that conda install dbus does not install the Python Dbus bindings, as you would expect with pip . It installs the Dbus C package itself, which is needed by Qt 5 (again, the C++ library, not the Python bindings to it).
Unfortunately, there are no Conda packages for dbus-python. To make matters worse, it seems there's no easy way to create packages for it, as pointed out here.
Finally, you said
Here is the error message I am getting
The (most probable) cause of that error is because you added your system Python dist-packages path to the PYTHONPATH of Anaconda or because you blindly copied the dbus module from system Python to Anaconda. Please don't do that ever again. System Python and Anaconda packages are compiled with different compilers and under different conditions. So mixing them is the cause of incomprehensible errors, just like the one you reported.

default python won't switch to homebew installation

I know this question has been asked - but none of the solutions there seem to be working. I recently found out that on Mac OS, the native installation of python isn't very good. So, I installed homebrew, and installed Python 2 through there. My path is as follows:
~ echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
However, I still get the following:
~ which python
/usr/bin/python
~ which python2
/usr/local/bin/python2
It's also probably important to note that my ~/.bash_profile, ~/.profile and ~/.zshrc files are empty.
Somehow, I think I got pip to work with the python2 (homebrew) installation. Basically, I did pip install pandas and I get the following outputs for the two python interpreters:
~ python
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named pandas
That is, it doesn't appear with the default python installation. However, I also have the following:
~ python2
Python 2.7.14 (default, Sep 25 2017, 09:53:22)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>>
So, pandas is imported in the python2 (homebrew) python, but not the standard one. How can I get it so that the default python is my homebrew python? I also read somewhere that brew link python would work, but when I do that I get
~ brew link python
Warning: Already linked: /usr/local/Cellar/python/2.7.14
To relink: brew unlink python && brew link python
I want to be able to just use python my_file.py and have it use the homebrew installation. (Not sure if it's related, but hopefully this would also allow Sublime to use the homebrew install when I hit command + B).
Sorry for the long post, new computer and I'm trying to get this all correct before I do any big projects and find out halfway through that things aren't working the way I need them to.
Thanks!
The macOS system Python is installed at /usr/bin/python. This is the only one called python on your path.
Homebrew doesn't create a link for python in /usr/local/bin. If you want one you can create it yourself:
ln -s /usr/local/bin/python2 /usr/local/bin/python
Alternatively, just update your shebang lines to specify python2 or python3, which is probably better anyway (explicit is better than implicit).

Categories