We Keep Coding

Issue with Spyder Interpreting Mingw64 GCC compiled version of Python - DLL load failure

Motivation: The package of interest only works with GCC compiled Python, and I would like to use an IDE (Spyder) to interpret the GCC compiled Python which is not the default MSC version of Python included with Anaconda 3 on windows.
Issue: DLL load failed while importing [package name]: The specified procedure could not be found.
System information:
Windows 10 Home Build 19043.1526 with WSL Enabled.
Bash: MSYS2 MinGW x64
Spyder 5.1.5
gcc version 11.2.0 targeting x86_64-w64-ming32
Python 3.9.10 (main, Jan 25 2022, 17:58:23) [GCC 11.2.0 64 bit (AMD64)]
Reproducible Example:
Download Anaconda 3 which includes Spyder and default MSC compiled
Python
Use Msys2 64 bit with Mingw64 to obtain GCC compiled python. To do
so, you would need to use pacman to obtain mingw-w64-x86_64-toolchain
and python for mingw64.
Install common python packages such as numpy, scipy, and matplotlib
using pacman. You will also need spyder dependencies such as
spyder-kernels and IPython. If the dependencies are not satisfied, Spyder will fail to start the kernel.
At this stage, running python in mingw64 (by typing "python" in
bash) should have no problems. Importing packages should also have
no problems. e.g, import matplotlib.pyplot
Try using Anaconda 3 Spyder to interpret the GCC python by going to:
Tool > Preferences > Python interpreter > Select Use the following
python interpreter > Set the location to
"path to
msys2"\mingw64\bin\python.exe
Depending on the matplotlib graphical settings, you may or may not
already receive an error in the console regarding a DLL load
failure. If not, then importing matplotlib.pyplot or importing
scipy.optimize, should give you the error. However, starting GCC
python in mingw64 will have no problem importing these packages.
Errors:
Python 3.9.10 (main, Jan 25 2022, 17:58:23) [GCC 11.2.0 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.
IPython -- An enhanced Interactive Python.
Output from spyder call 'show_mpl_backend_errors':
=========================================================================
NOTE: The following error appeared when setting your Matplotlib backend!!
=========================================================================
Traceback (most recent call last):
File "C:\msys64\mingw64\lib\python3.9\site-packages\spyder_kernels\console\kernel.py", line 739, in _set_mpl_backend
get_ipython().run_line_magic(magic, backend)
File "C:\msys64\mingw64\lib\python3.9\site-packages\IPython\core\interactiveshell.py", line 2364, in run_line_magic
result = fn(*args, **kwargs)
File "C:\msys64\mingw64\lib\python3.9\site-packages\decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "C:\msys64\mingw64\lib\python3.9\site-packages\IPython\core\magic.py", line 187, in <lambda>
call = lambda f, *a, **k: f(*a, **k)
File "C:\msys64\mingw64\lib\python3.9\site-packages\IPython\core\magics\pylab.py", line 99, in matplotlib
gui, backend = self.shell.enable_matplotlib(args.gui.lower() if isinstance(args.gui, str) else args.gui)
File "C:\msys64\mingw64\lib\python3.9\site-packages\ipykernel\zmqshell.py", line 601, in enable_matplotlib
gui, backend = super(ZMQInteractiveShell, self).enable_matplotlib(gui)
File "C:\msys64\mingw64\lib\python3.9\site-packages\IPython\core\interactiveshell.py", line 3533, in enable_matplotlib
from matplotlib_inline.backend_inline import configure_inline_support
File "C:\msys64\mingw64\lib\python3.9\site-packages\matplotlib_inline\backend_inline.py", line 7, in <module>
from matplotlib.backends.backend_agg import ( # noqa
File "C:\msys64\mingw64\lib\python3.9\site-packages\matplotlib\backends\backend_agg.py", line 35, in <module>
from PIL import Image
File "C:\msys64\mingw64\lib\python3.9\site-packages\PIL\Image.py", line 89, in <module>
from . import _imaging as core
ImportError: DLL load failed while importing _imaging: The specified procedure could not be found.
import matplotlib.pyplot
Traceback (most recent call last):
File "<ipython-input-1-864e826dab68>", line 1, in <module>
import matplotlib.pyplot
File "C:\msys64\mingw64\lib\python3.9\site-packages\matplotlib\pyplot.py", line 36, in <module>
import matplotlib.colorbar
File "C:\msys64\mingw64\lib\python3.9\site-packages\matplotlib\colorbar.py", line 44, in <module>
import matplotlib.contour as contour
File "C:\msys64\mingw64\lib\python3.9\site-packages\matplotlib\contour.py", line 17, in <module>
import matplotlib.text as text
File "C:\msys64\mingw64\lib\python3.9\site-packages\matplotlib\text.py", line 16, in <module>
from .textpath import TextPath # Unused, but imported by others.
File "C:\msys64\mingw64\lib\python3.9\site-packages\matplotlib\textpath.py", line 11, in <module>
from matplotlib.mathtext import MathTextParser
File "C:\msys64\mingw64\lib\python3.9\site-packages\matplotlib\mathtext.py", line 27, in <module>
from PIL import Image
File "C:\msys64\mingw64\lib\python3.9\site-packages\PIL\Image.py", line 89, in <module>
from . import _imaging as core
ImportError: DLL load failed while importing _imaging: The specified procedure could not be found.
import scipy.optimize
Traceback (most recent call last):
File "<ipython-input-2-77fb0d19797b>", line 1, in <module>
import scipy.optimize
File "C:\msys64\mingw64\lib\python3.9\site-packages\scipy\optimize\__init__.py", line 413, in <module>
from ._linprog import linprog, linprog_verbose_callback
File "C:\msys64\mingw64\lib\python3.9\site-packages\scipy\optimize\_linprog.py", line 22, in <module>
from ._linprog_highs import _linprog_highs
File "C:\msys64\mingw64\lib\python3.9\site-packages\scipy\optimize\_linprog_highs.py", line 20, in <module>
from ._highs._highs_wrapper import _highs_wrapper
ImportError: DLL load failed while importing _highs_wrapper: The specified procedure could not be found.

ImportError: cannot import name '_remove_dead_weakref'

Yesterday updated my ubuntu 17.04 to ubuntu 17.10. any comments? Appear when I try to run server in pycharm. #django project.
bash -cl "/home/encuentrum/venv-encuentrum3/bin/python /usr/share/pycharm/helpers/pycharm/django_manage.py check /home/encuentrum/GitLab/encuentrum3/ENCUENTRUM/packers_"
Traceback (most recent call last):
File "/usr/share/pycharm/helpers/pycharm/django_manage.py", line 5, in <module>
from pycharm_run_utils import adjust_django_sys_path
File "/usr/share/pycharm/helpers/pycharm/pycharm_run_utils.py", line 4, in <module>
import imp
File "/home/encuentrum/venv-encuentrum3/lib/python3.6/imp.py", line 19, in <module>
from importlib._bootstrap import _ERR_MSG, _exec, _load, _builtin_from_name
File "/home/encuentrum/venv-encuentrum3/lib/python3.6/importlib/__init__.py", line 57, in <module>
import types
File "/home/encuentrum/venv-encuentrum3/lib/python3.6/types.py", line 171, in <module>
import functools as _functools
File "/home/encuentrum/venv-encuentrum3/lib/python3.6/functools.py", line 23, in <module>
from weakref import WeakKeyDictionary
File "/home/encuentrum/venv-encuentrum3/lib/python3.6/weakref.py", line 12, in <module>
from _weakref import (
ImportError: cannot import name '_remove_dead_weakref'

Maybe you have mixed your multiple Python installations, the newer version of weakref are not compatible with older version python binary, try to remove any one (the older one is recommended) of Python installation.
Analysis
For my case, I have installed older version Python （3.5.1) before, and upgrade my Debian installation. The newer Debian upgrade it's Python3.5 to 3.5.3 which have _remove_dead_weakref in _weakref in its Python binary
When I type $ where python3.5, I get
/usr/local/bin/python3.5
/usr/local/bin/python3.5
/usr/bin/python3.5
The /usr/local/bin/python3.5 is my own older installation, and /usr/bin/python3.5 is Debian offical Python3.5
When I update my Python3.5 installation by apt-get, apt-get execute python3.5 -E -S /usr/lib/python3.5/py_compile.py $files (post-install script) in the deb package.`, it triggers the weakref issue, here is my log
Setting up python3.5-minimal (3.5.3-1+deb9u1) ...
Traceback (most recent call last):
File "/usr/lib/python3.5/py_compile.py", line 6, in <module>
import importlib._bootstrap_external
File "/usr/lib/python3.5/importlib/__init__.py", line 57, in <module>
import types
File "/usr/lib/python3.5/types.py", line 166, in <module>
import functools as _functools
File "/usr/lib/python3.5/functools.py", line 23, in <module>
from weakref import WeakKeyDictionary
File "/usr/lib/python3.5/weakref.py", line 12, in <module>
from _weakref import (
ImportError: cannot import name '_remove_dead_weakref'
I tested Python 3.5.1 and Python 3.5.3 with same import action, here are the compares
Official Python 3.5.3 from apt-get
Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516] on linux
>>> from _weakref import _remove_dead_weakref
>>>
My own Python 3.5.1 installation
Python 3.5.1 (default, Apr 23 2016, 16:40:21)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> from _weakref import _remove_dead_weakref
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name '_remove_dead_weakref'
>>>
So, I confirm that python3.5 in /usr/local/bin/ cannot use _remove_dead_weakref.
But which python did apt-get use in post-installation script? Try it.
$ which python3.5
/usr/local/bin/python3.5
So, here is why. The post-installation script use my custom installation of python, along with newer python library (/usr/lib/python3.5/weakref.py)
Fix it!
As I said, disable older version of python
sudo mv /usr/local/bin/python3.5 /usr/local/bin/python3.5.bak
Test
$ which python3.5
/usr/bin/python3.5

Adding to #Comzyh answer, this indeed is due to mixed python versions when an upgrade happen due to any reason. A quick fix is to remove your venv python binary i.e. rm <path-to-your-env>/bin/python and copying your system python binary to your venv like cp /usr/bin/python <path-to-your-env>/bin/python. This will fix the weak ref error

AttributeError: module 'functools' has no attribute 'wraps'

I am trying to test 3rd party code with Anaconda 4.2 / Python 3.5 When I execute tests I get following exception:
Traceback (most recent call last):
File "pyspark/sql/tests.py", line 25, in <module>
import subprocess
File "/home/user/anaconda3/lib/python3.5/subprocess.py", line 364, in <module>
import signal
File "/home/user/anaconda3/lib/python3.5/signal.py", line 3, in <module>
from functools import wraps as _wraps
File "/home/user/anaconda3/lib/python3.5/functools.py", line 22, in <module>
from types import MappingProxyType
File "/home/user/Spark/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/types.py", line 22, in <module>
import calendar
File "/home/user/anaconda3/lib/python3.5/calendar.py", line 10, in <module>
import locale as _locale
File "/home/user/anaconda3/lib/python3.5/locale.py", line 108, in <module>
#functools.wraps(_localeconv)
AttributeError: module 'functools' has no attribute 'wraps'
Normally I would assume some module is shadowing built-in modules but as far as I can tell this is not the issue:
I logged module path (functools.__file__) from the tests and it yields expected path. Also there is nothing strange in the path I get in the exception.
To exclude possible module corruption I tested completely new Anaconda installation.
When I execute tests, with the same configuration and path, from IPython shell (%run pyspark/sql/tests.py) problem disappears.
functools.wraps can be imported in the shell started in the same directory and with the same configuration.
When I replace Python 3 environment with Python 2 environment problem disappears.
Problem cannot be reproduced with environment create using virtualenv.
With different version of the same project I get:
Traceback (most recent call last):
File "pyspark/sql/tests.py", line 25, in <module>
import pydoc
File "/home/user/anaconda3/lib/python3.5/pydoc.py", line 55, in <module>
import importlib._bootstrap
File "/home/user/anaconda3/lib/python3.5/importlib/__init__.py", line 57, in <module>
import types
File "/home/user/Spark/spark-1.6.3-bin-hadoop2.6/python/pyspark/sql/types.py", line 22, in <module>
import calendar
File "/home/user/anaconda3/lib/python3.5/calendar.py", line 10, in <module>
import locale as _locale
File "/home/user/anaconda3/lib/python3.5/locale.py", line 19, in <module>
import functools
File "/home/user/anaconda3/lib/python3.5/functools.py", line 22, in <module>
from types import MappingProxyType
ImportError: cannot import name 'MappingProxyType'
Is there something obvious I missed here?
Edit:
Dockerfile which can be used to reproduce the problem:
FROM debian:latest
RUN apt-get update
RUN apt-get install -y wget bzip2
RUN wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
RUN bash Anaconda3-4.2.0-Linux-x86_64.sh -b -p /anaconda3
RUN wget ftp://ftp.piotrkosoft.net/pub/mirrors/ftp.apache.org/spark/spark-2.1.0/spark-2.1.0-bin-hadoop2.7.tgz
RUN tar xf spark-2.1.0-bin-hadoop2.7.tgz
ENV PATH /anaconda3/bin:$PATH
ENV SPARK_HOME /spark-2.1.0-bin-hadoop2.7
ENV PYTHONPATH $PYTHONPATH:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$SPARK_HOME/python
WORKDIR /spark-2.1.0-bin-hadoop2.7
RUN python python/pyspark/sql/tests.py

I suspect this is happening because, the functools module of python3 has the following import: from types import MappingProxyType and, instead of picking up this module from ${CONDA_PREFIX}/lib/python3.5/types.py, it tries to import the module from inside the sql directory: ${SPARK_HOME}/python/pyspark/sql/types.py . The functools module of python2 does not have this import and hence does not throw the error.
A workaround to this, is to somehow import the required types module first and then invoke the script. As a proof of concept:
(root) ~/condaexpts$ PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$SPARK_HOME/python python
Python 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import types
>>> import os
>>> sqltests=os.environ['SPARK_HOME'] + '/python/pyspark/sql/tests.py'
>>> exec(open(sqltests).read())
.....Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/01/30 05:59:43 WARN SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0
17/01/30 05:59:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
...
----------------------------------------------------------------------
Ran 128 tests in 372.565s
Also note that there is nothing special about conda. One can see the same thing in a normal virtualenv (with python3):
~/condaexpts$ virtualenv -p python3 venv
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in venv/bin/python3
Also creating executable in venv/bin/python
Installing setuptools, pip...done.
~/condaexpts$ source venv/bin/activate
(venv)~/condaexpts$ python --version
Python 3.4.3
(venv)~/condaexpts$ python $WORKDIR/python/pyspark/sql/tests.py
Traceback (most recent call last):
File "/home/ubuntu/condaexpts/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/tests.py", line 26, in <module>
import pydoc
File "/usr/lib/python3.4/pydoc.py", line 59, in <module>
import importlib._bootstrap
File "/home/ubuntu/condaexpts/venv/lib/python3.4/importlib/__init__.py", line 40, in <module>
import types
File "/home/ubuntu/condaexpts/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/types.py", line 22, in <module>
import calendar
File "/usr/lib/python3.4/calendar.py", line 10, in <module>
import locale as _locale
File "/home/ubuntu/condaexpts/venv/lib/python3.4/locale.py", line 20, in <module>
import functools
File "/home/ubuntu/condaexpts/venv/lib/python3.4/functools.py", line 22, in <module>
from types import MappingProxyType
ImportError: cannot import name 'MappingProxyType'

Anaconda Installation error on MAc OS

I am tyying to install anaconda on my mac but at the last step of installtion it gives me following error:
installing: conda-env-2.4.5-py27_0 ...
Python 2.7.11 :: Continuum Analytics, Inc.
creating default environment...
Traceback (most recent call last):
File "/Users/sdb99/anaconda2/pkgs/conda-4.0.5-py27_0/lib/python2.7/site-packages/conda/install.py", line 37, in <module>
import subprocess
File "/Users/sdb99/anaconda2/pkgs/python-2.7.11-0/lib/python2.7/subprocess.py", line 430, in <module>
import pickle
File "/Users/sdb99/anaconda2/pkgs/python-2.7.11-0/lib/python2.7/pickle.py", line 1272, in <module>
import binascii as _binascii
ImportError: dlopen(/Users/sdb99/anaconda2/pkgs/python-2.7.11-0/lib/python2.7/lib-dynload/binascii.so, 2): Library not loaded: #loader_path/../../libz.1.dylib
Referenced from: /Users/sdb99/anaconda2/pkgs/python-2.7.11-0/lib/python2.7/lib-dynload/binascii.so
Reason: image not found
Any idea how i can get ahead from here?
Thanks!

Just update python to the next version and try installing Anaconda again.
I had python 2.7 and then I updated it to python 3 and I was able to install Anaconda graphical installer successfully.

Image not found error when importing QtGui from PyQt4

I have installed ActivePython 2.7 on Mac OS X 10.5. The main reason for this is to use pypm, the package manager. However, after installing ActivePython and setting up my .bash_profile file to use that version of Python, I encounter a lot of problems with the pypm-installed libraries. To install all of these modules, I just type
pypm install <package-name>
and this is how I got matplotlib, PyQt4, PIL, etc. The problem is that all of these appear to install with no problems, they place the correct .pth and .so files into the correct site-packages folder, and basic imports work fine. But I get the following errors.
First, here is what I see when checking for Python on my machine:
new-host:site-packages ely$ which python
/usr/local/bin/python
new-host:site-packages ely$ ls -l /usr/local/bin/python
lrwxr-xr-x 1 root wheel 60 May 25 22:15 /usr/local/bin/python -> /Library/Frameworks/Python.framework/Versions/2.7/bin/python
Next, I invoke Python and attempt to import things.
new-host:site-packages ely$ python
ActivePython 2.7.1.4 (ActiveState Software Inc.) based on
Python 2.7.1 (r271:86832, Feb 7 2011, 11:33:10)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import PyQt4
>>> from PyQt4 import QtGui
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: dlopen(PyQt4/QtGui.so, 2): Library not loaded: QtGui.framework/Versions/4/QtGui
Referenced from: /Users/ely/Library/Python/2.7/lib/python/site-packages/PyQt4/QtGui.so
Reason: image not found
So the above shows that PyQt4 imports with no complaints, but then I get the 'image not found' error when I try to use any of the sub-modules. For another example, here is the standard matplotlib.
>>> import matplotlib
>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "matplotlib/pyplot.py", line 23, in <module>
from matplotlib.figure import Figure, figaspect
File "matplotlib/figure.py", line 16, in <module>
import artist
File "matplotlib/artist.py", line 6, in <module>
from transforms import Bbox, IdentityTransform, TransformedBbox, TransformedPath
File "matplotlib/transforms.py", line 34, in <module>
from matplotlib._path import affine_transform
ImportError: dlopen(matplotlib/_path.so, 2): Symbol not found: __ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_l
Referenced from: /Users/ely/Library/Python/2.7/lib/python/site-packages/matplotlib/_path.so
Expected in: dynamic lookup
>>> from matplotlib import pyplot
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "matplotlib/pyplot.py", line 23, in <module>
from matplotlib.figure import Figure, figaspect
File "matplotlib/figure.py", line 16, in <module>
import artist
File "matplotlib/artist.py", line 6, in <module>
from transforms import Bbox, IdentityTransform, TransformedBbox, TransformedPath
File "matplotlib/transforms.py", line 34, in <module>
from matplotlib._path import affine_transform
ImportError: dlopen(matplotlib/_path.so, 2): Symbol not found: __ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_l
Referenced from: /Users/ely/Library/Python/2.7/lib/python/site-packages/matplotlib/_path.so
Expected in: dynamic lookup
Any thoughts on why the pypm-installed versions are so problematic? I've used ActivePython and pypm on my Ubuntu work computer for a long time and never had any of these issues. Everything I've ever installed with pypm has worked beautifully. Why is it so different on Mac?

Make sure python and qt have same architecture (32bit vs 64bit)
for mathplot looks like a different g++ used to compile python and mathplot
have you tried with stock osx python?