conda cannot find installed module - python

I've been trying to install "mkldnn" with conda, but came into this old issue of "conda installed but python cannot find installed module". The weird thing is after I added the path to python import, it still cannot find it.
Do you happen to have an idea why? Here are the methods I've tried and the results.
Step 1: I have tried building a new conda environment with Python3.6 and re-install mkldnn with conda. Still cannot find.
[yl5090#log-0 ~]$ module purge
[yl5090#log-0 ~]$ module load anaconda3/5.3.0
[yl5090#log-0 ~]$ conda create --name pytorch-mpi python=3.6
Solving environment: done
## Package Plan ##
environment location: /home/yl5090/.conda/envs/pytorch-mpi
added / updated specs:
- python=3.6
The following NEW packages will be INSTALLED:
ca-certificates: 2018.03.07-0
certifi: 2018.11.29-py36_0
libedit: 3.1.20170329-h6b74fdf_2
libffi: 3.2.1-hd88cf55_4
libgcc-ng: 8.2.0-hdf63c60_1
libstdcxx-ng: 8.2.0-hdf63c60_1
ncurses: 6.1-he6710b0_1
openssl: 1.1.1a-h7b6447c_0
pip: 18.1-py36_0
python: 3.6.7-h0371630_0
readline: 7.0-h7b6447c_5
setuptools: 40.6.2-py36_0
sqlite: 3.25.3-h7b6447c_0
tk: 8.6.8-hbc83047_0
wheel: 0.32.3-py36_0
xz: 5.2.4-h14c3975_4
zlib: 1.2.11-h7b6447c_3
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate pytorch-mpi
#
# To deactivate an active environment, use:
# > source deactivate
#
[yl5090#log-0 ~]$ source activate pytorch-mpi
(pytorch-mpi) [yl5090#log-0 ~]$ conda install -c mingfeima mkldnn
Solving environment: done
## Package Plan ##
environment location: /home/yl5090/.conda/envs/pytorch-mpi
added / updated specs:
- mkldnn
The following NEW packages will be INSTALLED:
intel-openmp: 2019.1-144
mkl: 2019.1-144
mkldnn: 0.16.1-0 mingfeima
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(pytorch-mpi) [yl5090#log-0 ~]$ python
Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mkldnn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mkldnn'
>>>
Step 2: I have checked there is no mkldnn package in the sys.path of this Python.
>>> import sys
>>> sys.path
['', '/home/yl5090/.conda/envs/pytorch-mpi/lib/python36.zip', '/home/yl5090/.conda/envs/pytorch-mpi/lib/python3.6', '/home/yl5090/.conda/envs/pytorch-mpi/lib/python3.6/lib-dynload', '/home/yl5090/.conda/envs/pytorch-mpi/lib/python3.6/site-packages']
[yl5090#log-1 lab4]$ cd /home/yl5090/.conda/envs/pytorch-mpi/lib
[yl5090#log-1 lib]$ pwd
/home/yl5090/.conda/envs/pytorch-mpi/lib
[yl5090#log-1 lib]$ find . -type d -name "*mkldnn*" -print
[yl5090#log-1 lib]$
Step 3: I checked the import path of conda installed numpy and cannot find mkldnn in that path.
>>> import numpy
>>> import inspect
>>> inspect.getfile(numpy)
'/home/yl5090/.conda/envs/pytorch-mpi/lib/python3.6/site-packages/numpy/__init__.py'
[yl5090#log-1 lib]$ cd /home/yl5090/.conda/envs/pytorch-mpi/lib/python3.6/site-packages/
[yl5090#log-1 site-packages]$ ls
certifi pip
certifi-2018.11.29-py3.6.egg-info pip-18.1-py3.6.egg-info
easy_install.py pkg_resources
mkl_fft __pycache__
mkl_fft-1.0.6-py3.6.egg-info README.txt
mkl_random setuptools
mkl_random-1.0.2-py3.6.egg-info setuptools-40.6.2-py3.6.egg-info
numpy wheel
numpy-1.15.4-py3.6.egg-info wheel-0.32.3-py3.6.egg-info
Step 4: I did find an mkldnn installation in the /.conda/pkgs directory and added it to the python import path. But still cannot import it from python.
[yl5090#log-1 pkgs]$ ls /home/yl5090/.conda/pkgs | grep mkldnn
mkldnn-0.16.1-0
[yl5090#log-1 pkgs]$ cd mkldnn*
[yl5090#log-1 mkldnn-0.16.1-0]$ ls
include info lib share
(load in python...)
>>> path = "/home/yl5090/.conda/pkgs"
>>> import sys
>>> sys.path.insert(0, path)
>>> sys.path
['/home/yl5090/.conda/pkgs', '', '/home/yl5090/.conda/envs/pytorch-mpi/lib/python36.zip', '/home/yl5090/.conda/envs/pytorch-mpi/lib/python3.6', '/home/yl5090/.conda/envs/pytorch-mpi/lib/python3.6/lib-dynload', '/home/yl5090/.conda/envs/pytorch-mpi/lib/python3.6/site-packages']
>>> import mkldnn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mkldnn'
The remaining reason I can think of now is Python cannot import .so files without an __init__.py file, but I'm not sure if that's the correct direction to go for this issue.
Finally, here are what's inside the mkldnn directory that I found, not sure if this is what conda has installed.
[yl5090#log-1 mkldnn-0.16.1-0]$ ls
include info lib share
Thanks!!!

I figured out. This is not a python package : https://github.com/intel/mkl-dnn It's a C/C++ module... Thanks!

Related

Unable to install tkinter with pyenv Pythons on MacOS

Versions of Python installed via pyenv fail to import tkinter:
※ python
Python 3.8.1 (default, Feb 29 2020, 11:45:59)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tkinter
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/factor/.pyenv/versions/3.8.1/lib/python3.8/tkinter/__init__.py", line 36, in <module>
import _tkinter # If this fails your Python may not be configured for Tk
ModuleNotFoundError: No module named '_tkinter'
>>>
or you might get a message about the header version doesn't match the binary:
DEPRECATION WARNING: The system version of Tk is deprecated and may be removed in a future release. Please don't rely on it. Set TK_SILENCE_DEPRECATION=1 to suppress this warning.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/factor/.pyenv/versions/3.8.1/lib/python3.8/tkinter/__init__.py", line 4552, in _test
root = Tk()
File "/Users/factor/.pyenv/versions/3.8.1/lib/python3.8/tkinter/__init__.py", line 2263, in __init__
self._loadtk()
File "/Users/factor/.pyenv/versions/3.8.1/lib/python3.8/tkinter/__init__.py", line 2279, in _loadtk
raise RuntimeError("tk.h version (%s) doesn't match libtk.a version (%s)"
RuntimeError: tk.h version (8.6) doesn't match libtk.a version (8.5)
TL;DR set the env. vars. mentioned in tcl-tk's caveats and this GitHub comment when installing new Pythons via pyenv to get tkinter.
First, ensure you have the latest tcl-tk via homebrew and then pay attention to its caveats:
※ brew install tcl-tk
※ brew info tcl-tk
tcl-tk: stable 8.6.10 (bottled) [keg-only]
...
==> Caveats
tcl-tk is keg-only, which means it was not symlinked into /usr/local,
because tk installs some X11 headers and macOS provides an (older) Tcl/Tk.
If you need to have tcl-tk first in your PATH run:
echo 'export PATH="/usr/local/opt/tcl-tk/bin:$PATH"' >> ~/.zshrc
For compilers to find tcl-tk you may need to set:
export LDFLAGS="-L/usr/local/opt/tcl-tk/lib"
export CPPFLAGS="-I/usr/local/opt/tcl-tk/include"
For pkg-config to find tcl-tk you may need to set:
export PKG_CONFIG_PATH="/usr/local/opt/tcl-tk/lib/pkgconfig"
...
You'll also need to know about pyenv's PYTHON_CONFIGURE_OPTS, --with-tcltk-includes, and --with-tcltk-libs, e.g. from this comment.
Next, reinstall Python with the environment variables active:
※ pyenv uninstall 3.8.1
※ env \
PATH="$(brew --prefix tcl-tk)/bin:$PATH" \
LDFLAGS="-L$(brew --prefix tcl-tk)/lib" \
CPPFLAGS="-I$(brew --prefix tcl-tk)/include" \
PKG_CONFIG_PATH="$(brew --prefix tcl-tk)/lib/pkgconfig" \
CFLAGS="-I$(brew --prefix tcl-tk)/include" \
PYTHON_CONFIGURE_OPTS="--with-tcltk-includes='-I$(brew --prefix tcl-tk)/include' --with-tcltk-libs='-L$(brew --prefix tcl-tk)/lib -ltcl8.6 -ltk8.6'" \
pyenv install 3.8.1
It should work now:
※ pyenv global 3.8.1
※ python
Python 3.8.1 (default, Feb 29 2020, 11:56:10)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tkinter
>>> tkinter.TclVersion, tkinter.TkVersion
(8.6, 8.6)
>>> tkinter._test()
# You should get a GUI
If you get the following error, you might be missing the PYTHON_CONFIGURE_OPTS env. var. above.
DEPRECATION WARNING: The system version of Tk is deprecated and may be removed in a future release. Please don't rely on it. Set TK_SILENCE_DEPRECATION=1 to suppress this warning.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/factor/.pyenv/versions/3.8.1/lib/python3.8/tkinter/__init__.py", line 4552, in _test
root = Tk()
File "/Users/factor/.pyenv/versions/3.8.1/lib/python3.8/tkinter/__init__.py", line 2263, in __init__
self._loadtk()
File "/Users/factor/.pyenv/versions/3.8.1/lib/python3.8/tkinter/__init__.py", line 2279, in _loadtk
raise RuntimeError("tk.h version (%s) doesn't match libtk.a version (%s)"
RuntimeError: tk.h version (8.6) doesn't match libtk.a version (8.5)
Here is step by step guide to make tkinter (and IDLE) work if you use pyenv for Python environments management on macOS:
install tcl-tk with Homebrew. In shell run brew install tcl-tk
in shell run echo 'export PATH="/usr/local/opt/tcl-tk/bin:$PATH"' >> ~/.zshrc
reload shell by quitting Terminal app or run source ~/.zshrc
after reloaded check that tcl-tk is in $PATH. Run echo $PATH | grep --color=auto tcl-tk. As the result you should see your $PATH contents with tcl-tk highlighted
now we run three commands from Homebrew's output from step #1
in shell run export LDFLAGS="-L/usr/local/opt/tcl-tk/lib"
in shell run export CPPFLAGS="-I/usr/local/opt/tcl-tk/include"
in shell run export PKG_CONFIG_PATH="/usr/local/opt/tcl-tk/lib/pkgconfig"
if you have your Python version already installed with pyenv then uninstall it with pyenv uninstall <your python version>. E.g. pyenv uninstall 3.8.2
set environment variable that will be used by python-build. In shell run PYTHON_CONFIGURE_OPTS="--with-tcltk-includes='-I/usr/local/opt/tcl-tk/include' --with-tcltk-libs='-L/usr/local/opt/tcl-tk/lib -ltcl8.6 -ltk8.6'"
Note: in future use tck-tk version that actually installed with Homebrew. At the moment of posting 8.6 was the actual
finally install Python with pyenv with pyenv install <version>. E.g. pyenv install 3.8.2
Test
in shell run pyenv global <verion that you've just installed>
now check IDLE. In shell run idle. You should see IDLE window without any warnings and "text printed in red".
now check tkinter. In shell run python -m tkinter -c "tkinter._test()". You should see test window like on the image:
That's it!
My environment:
check this is something went wrong executing steps above:
macOS Catalina
zsh (included in macOS Catalina) = "shell" above
Homebrew (installed with instructions from Homebrew official website)
pyenv (installed with Homebrew and PATH updated according to pyenv official readme from GitHub)
Python 3.8.x - 3.9.x (installed with pyenv install <version> command)
For MacOS Big Sur (11.2.3), Carl G's answer didn't work for me because I got a zlib error. Building off of this answer and this blog post, I found success with
brew install bzip2
export LDFLAGS="-L $(xcrun --show-sdk-path)/usr/lib -L brew --prefix bzip2/lib"
export CFLAGS="-L $(xcrun --show-sdk-path)/usr/include -L brew --prefix bzip2/include"
export PYTHON_CONFIGURE_OPTS="--with-tcltk-includes='-I/usr/local/opt/tcl-tk/include' --with-tcltk-libs='-L/usr/local/opt/tcl-tk/lib -ltcl8.6 -ltk8.6' --enable-framework"
pyenv install 3.8.6
I had the same issue when I tried to install tkinter through pyenv. I was able to fix it using the following in case someone has the same problem and still want to stick with pyenv.
The Fix
I followed #nickolay instructions to install tkinter and set the path the proper way.
Then, I installed anaconda3-2020.07 using the pyenv install anaconda3-2020.07 command.
Because I am using pyenv python 3.8.6 globally: I navigated to the folder I want to use tkinter and used the anaconda3-2020.07 locally by utilizing the command pyenv local anaconda3-2020.07to use this version of pyenv in that specific folder. It ran without errors!
Note: I am using the following script in the .bash_profile to trigger the virtualenv automatically when cd the desired directory
# manage python version using pyenv
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi
# add pyenv virtualenv
eval "$(pyenv virtualenv-init -)"
For Python 3.9 and above version, you can use the following formula
brew install python-tk#3.9
It will instal python#3.9 and tcl-tk and bind them for you.

Spyder: importerror: cannot import name constants

Ubuntu 16.04
Python 2.7
Anaconda 2
I have Anaconda2 installed. This includes the spyder IDE, according to google.
$ spyder
Gives:
Traceback (most recent call last):
File "/home/username/anaconda2/bin/spyder", line 7, in <module>
from spyder.app.start import main
File "/home/username/anaconda2/lib/python2.7/site-packages/spyder/app/start.py", line 13, in <module>
import zmq
File "/usr/lib/python2.7/dist-packages/zmq/__init__.py", line 66, in <module>
from zmq import backend
File "/usr/lib/python2.7/dist-packages/zmq/backend/__init__.py", line 40, in <module>
reraise(*exc_info)
File "/usr/lib/python2.7/dist-packages/zmq/backend/__init__.py", line 27, in <module>
_ns = select_backend(first)
File "/usr/lib/python2.7/dist-packages/zmq/backend/select.py", line 27, in select_backend
mod = __import__(name, fromlist=public_api)
File "/usr/lib/python2.7/dist-packages/zmq/backend/cython/__init__.py", line 6, in <module>
from . import (constants, error, message, context,
ImportError: cannot import name constants
Doesn't work:
$ sudo apt-get purge spyder
$ sudo apt-get install spyder
Doesn't work:
$ conda uninstall spyder
$ conda install spyder
Googling the error doesn't help me along. I'm not a programmer. Python libraries and Anaconda directories and IDE repositories, are all Greek to me. Might be useful information for those who actually know how things work: importing scipy.io also gives me importerrors.
Where is the fault and how do I resolve it?
EDITS
$ conda update chardet
gives:
/home/pieter/anaconda2/lib/python2.7/site-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.22) or chardet (2.3.0) doesn't match a supported version!
RequestsDependencyWarning)
Solving environment: done
# All requested packages already installed.
and
$ conda update urllib3
gives:
/home/pieter/anaconda2/lib/python2.7/site-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.22) or chardet (2.3.0) doesn't match a supported version!
RequestsDependencyWarning)
Solving environment: done
# All requested packages already installed.
$ conda info spyder
gives:
absurdly long, long, long list of replies
of which the last is:
spyder 3.2.4 py27h04a3490_0
---------------------------
file name : spyder-3.2.4-py27h04a3490_0.tar.bz2
name : spyder
version : 3.2.4
build string: py27h04a3490_0
build number: 0
channel : https://repo.continuum.io/pkgs/main/linux-64
size : 2.5 MB
arch : None
constrains : ()
license : MIT
md5 : ed9729850ec1ab3c83d87b422e687bc5
platform : None
subdir : linux-64
timestamp : 1508356637551
url : https://repo.continuum.io/pkgs/main/linux-64/spyder-3.2.4-py27h04a3490_0.tar.bz2
dependencies:
chardet >=2.0
jedi >=0.9
nbconvert
numpydoc
pickleshare
psutil
pycodestyle
pyflakes
pygments >=2.0
pylint
pyqt
python >=2.7,<2.8.0a0
pyzmq
qtawesome >=0.4.1
qtconsole >=4.2
qtpy >=1.2.0
rope >=0.9.4
sphinx
myEnv:
(myEnv) pieter#t:~/bagfile_analysis$ spyder
Traceback (most recent call last):
File "/home/pieter/anaconda2/envs/myEnv/bin/spyder", line 4, in <module>
import re
File "/home/pieter/anaconda2/envs/myEnv/lib/python3.6/re.py", line 142, in <module>
class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'
Hmmm... have you tried THIS already?
Also, how exactly did you install spyder? Maybe you didn't install it in the right environemnt(in your case probably the default root/base environment)
I'd recommend installing it via conda in the terminal! Also make sure you have installed it in the right environment:
get a list of all your environments:
$ conda env list
install it in a specific environment:
$ conda install --name "your env" spyder
and of course make sure you got the latest anaconda version:
$ conda update -n base conda
EDIT
ok now we know you only have your base environment.
since chardet is a dependency of spyder let's update those two packages first to be sure...
$ conda update chardet
$ conda update urllib3
Let's go on and check your spyder.
Does it work now?
if not try this:
$ conda info spyder
what do you get?
you should get a very long list, but only the last entry is of importance, something like that:
spyder 3.2.5 py27_0
-------------------
file name : spyder-3.2.5-py27_0.tar.bz2
name : spyder
version : 3.2.5
build string: py27_0
build number: 0
channel : https://repo.continuum.io/pkgs/main/osx-64
size : 2.5 MB
arch : None
constrains : ()
license : MIT
md5 : 5ec8ffcda103c23c81005a5aed0c91e7
platform : None
subdir : osx-64
timestamp : 1514270782022
url : https://repo.continuum.io/pkgs/main/osx-64/spyder-3.2.5-py27_0.tar.bz2
dependencies:
chardet >=2.0
cloudpickle
jedi >=0.9
nbconvert
numpydoc
pickleshare
psutil
pycodestyle
pyflakes
pygments >=2.0
pylint
pyqt
python >=2.7,<2.8.0a0
python.app
pyzmq
qtawesome >=0.4.1
qtconsole >=4.2
qtpy >=1.2.0
rope >=0.9.4
sphinx
if you don't get anything back you have installed spyder in the wrong environment and you can skip step 1:
STEP 1:
uninstall spyder again.
$ conda uninstall spyder
STEP 2:
reinstall it
$ conda install spyder
EDIT II
Have you had a look at the link I posted?
the only other thing that comes is in my mind is trying a different environment (for example called "myEnv").
$ conda create --name myEnv spyder
then activate the environment and try to launch spyder. conda tells you how to activate the environment, but i guess it's the same as for mac os:
$ source activate myEnv
$ spyder
$ source deactivate
if that doesn't work you can remove the environment again.
$ conda remove --name myEnv *
In any case the dependency error is weird...

Can not install VTK with Anaconda

I can not install VTK by anaconda in Mac OS X. I use
conda install -c anaconda vtk=6.3.0
Python 2.7 but get the following error:
UnsatisfiableError: The following specifications were found to be in conflict: - python 3.6* - vtk 6.3.0* -> python 2.7* -> openssl 1.0.1* Use "conda info " to see the dependencies for each package.
But I've updated Python to 3.6, and the same error.
Conda Search
$ conda search vtk
Fetching package metadata ...........
vtk 5.10.1 py27_0 defaults
5.10.1 py27_1 defaults
6.3.0 py27_0 defaults
6.3.0 py27_1 defaults
Create the choicest env
$ conda create -n vtk python=2.7 vtk=6.3.0
Fetching package metadata ...........
Solving package specifications: .
Package plan for installation in environment /home/sidon/opt/anaconda3/envs/vtk:
The following NEW packages will be INSTALLED:
openssl: 1.0.2k-1
pip: 9.0.1-py27_1
python: 2.7.13-0
readline: 6.2-2
setuptools: 27.2.0-py27_0
sqlite: 3.13.0-0
tk: 8.5.18-0
vtk: 6.3.0-py27_1
wheel: 0.29.0-py27_0
zlib: 1.2.8-3
Proceed ([y]/n)? y
vtk-6.3.0-py27 100% |#############################################################################################| Time: 0:00:15 1.97 MB/s
#
# To activate this environment, use:
# > source activate vtk
#
# To deactivate this environment, use:
# > source deactivate vtk
#
Active the new env
$ source activate vtk
Test
$ python -c "import vtk; print vtk.vtkSphereSource()"
vtkSphereSource (0xcfdcc0)
Debug: Off
Modified Time: 39
Reference Count: 2
Registered Events: (none)
Executive: 0xae5e00
ErrorCode: Success
Information: 0xf6e210
AbortExecute: Off
Progress: 0
Progress Text: (None)
Theta Resolution: 8
Phi Resolution: 8
Theta Start: 0
Phi Start: 0
Theta End: 360
Phi End: 180
Radius: 0.5
Center: (0, 0, 0)
LatLong Tessellation: 0
Output Points Precision: 0
I would suggest installing directly from the wheel file. You can get the VTK wheel file from here (wheel file directory site). This site has pretty much every wheel file so it is your friend! just search for it and you will see a bunch of .whl VTK files.
just download the wheel file that matches your anaconda python root version (either 3.5, 3.6, 2.7 etc.) and either 32 or 64 bit (depending on your python version).
To install from a wheel file, open a terminal in the directory the wheel is in and type
pip install "your_wheel_file_name"
If you want to install it for a conda environment, just activate the environment first (activate "env_name") and then install from the wheel file.
Try this out for me:
conda create --name vtk_test python=2.7 vtk
This will create a new environment with VTK, Python and their dependencies inside.

pip not installing numba/llvmlite properly within conda environment

I create a new conda environment
user#machine:~/project$ conda create -n test-env -c numba python=3.5.2 llvmdev=3.8
Fetching package metadata ...........
Solving package specifications: .
Package plan for installation in environment /home/user/anaconda2/envs/test-env:
The following NEW packages will be INSTALLED:
llvmdev: 3.8.1-7 numba
openssl: 1.0.2k-0
pip: 9.0.1-py35_1
python: 3.5.2-0
readline: 6.2-2
setuptools: 27.2.0-py35_0
sqlite: 3.13.0-0
system: 5.8-2 numba
tk: 8.5.18-0
wheel: 0.29.0-py35_0
xz: 5.2.2-1
zlib: 1.2.8-3
Proceed ([y]/n)? y
#
# To activate this environment, use:
# > source activate test-env
#
# To deactivate this environment, use:
# > source deactivate test-env
#
and then activate it and attempt to use specifically pip (not conda) to install llvmlite and numba, which appears to succeed. (Note: I've tried also --no-cache-dir and it doesn't change anything.)
user#machine:~/project$ source activate test-env
(test-env) user#machine:~/project$ pip install llvmlite==0.15 numba==0.30.1
Collecting llvmlite==0.15
Collecting numba==0.30.1
Collecting numpy (from numba==0.30.1)
Using cached numpy-1.12.0-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: llvmlite, numpy, numba
Successfully installed llvmlite-0.15.0 numba-0.30.1 numpy-1.12.0
But the library is not installed properly,
(test-env) user#machine:~/project$ python
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
Traceback (most recent call last):
File "/home/user/anaconda2/envs/test-env/lib/python3.5/site-packages/llvmlite/binding/ffi.py", line 42, in <module>
lib = ctypes.CDLL(os.path.join(_lib_dir, _lib_name))
File "/home/user/anaconda2/envs/test-env/lib/python3.5/ctypes/__init__.py", line 347, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /home/user/anaconda2/envs/test-env/lib/python3.5/site-packages/llvmlite/binding/libllvmlite.so: undefined symbol: _ZNKSt14error_category23default_error_conditionEi
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/user/anaconda2/envs/test-env/lib/python3.5/site-packages/numba/__init__.py", line 9, in <module>
from . import config, errors, runtests, types
File "/home/user/anaconda2/envs/test-env/lib/python3.5/site-packages/numba/config.py", line 11, in <module>
import llvmlite.binding as ll
File "/home/user/anaconda2/envs/test-env/lib/python3.5/site-packages/llvmlite/binding/__init__.py", line 6, in <module>
from .dylib import *
File "/home/user/anaconda2/envs/test-env/lib/python3.5/site-packages/llvmlite/binding/dylib.py", line 4, in <module>
from . import ffi
File "/home/user/anaconda2/envs/test-env/lib/python3.5/site-packages/llvmlite/binding/ffi.py", line 47, in <module>
lib = ctypes.CDLL(_lib_name)
File "/home/user/anaconda2/envs/test-env/lib/python3.5/ctypes/__init__.py", line 347, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libllvmlite.so: cannot open shared object file: No such file or directory
Why does the conda installation of llvmdev from the numba channel fail to "just work".
In my use case, I'm coming to a project that has a pip-styled requirements.txt file, and I need to create conda environments from that file. Some project developers will use venv+pip, some will use conda, and some packages it contains are not found in any anaconda channels, so pip installation is mandatory. We don't want to maintain a separate envrionment.yaml in addition to the requirements.txt, so installing from requirements.txt inside of a conda envrionment is part of my constraints.
Everything seems OK, except for the pip installation of numba/llvmlite, which expects a system dependency of llvm 3.8+. I want to satisfy that as part of the conda environment though.
How can I ensure from a conda environment only that the right llvmdev exists for installing numba and llvmlite?
Installing llvmdev:
Installing llvmdev from the conda-forge channel can be achieved by adding conda-forge to your channels with:
conda config --add channels conda-forge
Once the conda-forge channel has been enabled, llvmdev can be installed with:
conda install llvmdev
It is possible to list all of the versions of llvmdev available on your platform with:
conda search llvmdev --channel conda-forge
HELP:I cloned the relevant files from their GitHub sources and did
python setup.py install
for more without conda use pip
sudo pip install -U llvmlite
sudo pip install -U numba
This is what worked for me using the Anaconda environment:
pip uninstall llvmlite
pip install -U --ignore-installed numba
In an ideal scenario, the required shared library should be present after the installation of llvmlite.
(test-env) ~/condaexpts$ conda create -n test-env -c numba python=3.5.2 llvmdev=3.8
(test-env) ~/condaexpts$ source activate test-env
(test-env) ~/condaexpts$ pip install numpy==1.12.0 llvmlite==0.15 numba==0.30.1
(test-env) ~/condaexpts$ find $CONDA_PREFIX | grep libllvmlite
/home/ubuntu/condaexpts/m3/envs/test-env/lib/python3.5/site-packages/llvmlite/binding/libllvmlite.so
If this file is not present in the test-env, then something went wrong while installing llvmlite.
Also, think about this approach the other way round. You don't need to have a requirements file separate from conda environment file. You can have both dependencies in the conda environment file itself:
(test-env) ~/condaexpts$ conda env export
name: test-env
channels:
- !!python/unicode
'numba'
- !!python/unicode
'defaults'
dependencies:
- !!python/unicode
'openssl=1.0.2k=0'
- !!python/unicode
'pip=9.0.1=py35_1'
- !!python/unicode
'python=3.5.2=0'
- !!python/unicode
'readline=6.2=2'
- !!python/unicode
'setuptools=27.2.0=py35_0'
- !!python/unicode
'sqlite=3.13.0=0'
- !!python/unicode
'tk=8.5.18=0'
- !!python/unicode
'wheel=0.29.0=py35_0'
- !!python/unicode
'xz=5.2.2=1'
- !!python/unicode
'zlib=1.2.8=3'
- !!python/unicode
'llvmdev=3.8.1=7'
- !!python/unicode
'system=5.8=2'
- pip:
- llvmlite==0.15.0
- numba==0.30.1
- numpy==1.12.0
prefix: !!python/unicode '/home/ubuntu/condaexpts/m3/envs/test-env'
On second note, you can install the pre-built binaries required from the 'numba' conda channel itself. With:
(root) ~/condaexpts$ ./Miniconda3-latest-Linux-x86_64.sh -b -p m3
(root) ~/condaexpts$ source ./m3/bin/activate
(root) ~/condaexpts$ conda create -n test-env -c numba python=3.5.2 llvmdev=3.8 numba=0.30.1 llvmlite=0.15.0
(root) ~/condaexpts$ source activate test-env
(test-env) ~/condaexpts$ python
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
>>> numba.__version__
'0.30.1'

ImportError when using Python Anaconda package grib_api

I am trying to use the ECMWF GRIB_API to access GRIB files from Python. I get this error
Traceback (most recent call last):
File "/home/martin/markj/JustImportGRIB.py", line 1, in <module>
from gribapi import *
ImportError: No module named gribapi
One line of code :)
from gribapi import *
I am using Anaconda Python 2.7.12, Linux Centos 64-bit, installed as user and not added Anaconda to the path if that's relevant - the system Python (2.6) is on the path. I installed GRIB_API from the Pingu Carsti channel which was the top Google hit. The conda install did not report any errors.
[martin#bonnie ~]$ anaconda2/bin/conda install -c pingucarsti grib_api=1.10.4
Fetching package metadata .........
Solving package specifications: ..........
Package plan for installation in environment /home/martin/anaconda2:
The following packages will be downloaded:
package | build
---------------------------|-----------------
grib_api-1.10.4 | 3 1.7 MB pingucarsti
The following NEW packages will be INSTALLED:
grib_api: 1.10.4-3 pingucarsti
Proceed ([y]/n)? y
Fetching packages ...
grib_api-1.10. 100% |################################| Time: 0:00:01 1.24 MB/s
Extracting packages ...
[ COMPLETE ]|###################################################| 100%
Linking packages ...
[ COMPLETE ]|###################################################| 100%
I found some bug reports about this error being caused by a bug in the ECMWF GRIB_API, but I don't know enough about Anaconda (or perhaps Python) to figure out how to apply the workarounds.
I read GRIB files in Python with pygrib. I have successfully installed it inside a conda virtual environment (Miniconda2 on Ubuntu 16.04). Here is what has worked for me:
Install system dependencies for pygrib:
sudo apt-get install libjpeg9
sudo apt-get install libgrib-api-dev
Install pygrib and ecmwf_grib from conda-forge channel.
conda install -c conda-forge pygrib ecmwf_grib
Try conda-forge's build at python-eccodes (note: that's not eccodes which is the C library).
How it is built: https://github.com/conda-forge/python-eccodes-feedstock/blob/master/recipe/meta.yaml
The ECMWF's ecCodes library is the next version of their GRIB API, and is extremely similar to the old GRIB API.
pelson> conda create -n gribby -c conda-forge python-eccodes
Fetching package metadata .............
Solving package specifications: .
Package plan for installation in environment /Users/pelson/miniconda/envs/gribby:
The following NEW packages will be INSTALLED:
ca-certificates: 2017.7.27.1-0 conda-forge
curl: 7.54.1-0 conda-forge
eccodes: 2.4.0-0 conda-forge
hdf4: 4.2.12-0 conda-forge
hdf5: 1.8.18-1 conda-forge
intel-openmp: 2018.0.0-h68bdfb3_7 defaults
jasper: 1.900.1-4 conda-forge
jpeg: 9b-1 conda-forge
krb5: 1.14.2-0 conda-forge
libgfortran: 3.0.1-h93005f0_2 defaults
libnetcdf: 4.4.1.1-8 conda-forge
libpng: 1.6.28-1 conda-forge
libssh2: 1.8.0-1 conda-forge
mkl: 2018.0.0-h5ef208c_6 defaults
ncurses: 5.9-10 conda-forge
numpy: 1.13.3-py27h62f9060_0 defaults
openssl: 1.0.2l-0 conda-forge
python: 2.7.14-0 conda-forge
python-eccodes: 2.4.0-py27_1 conda-forge
readline: 6.2-0 conda-forge
sqlite: 3.13.0-1 conda-forge
tk: 8.5.19-2 conda-forge
zlib: 1.2.8-3 conda-forge
#
# To activate this environment, use:
# > source activate gribby
#
# To deactivate an active environment, use:
# > source deactivate
#
pelson> source activate gribby
python(gribby) pelson> python
Python 2.7.14 | packaged by conda-forge | (default, Oct 5 2017, 23:08:53)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import gribapi
>>> gribapi.__version__
'2.4.0'

Categories