I'm working on applying a clustering algorithm (sklearn.AgglomerativeClustering) on a dataset. I've tried running this same block of code in Spyder IDE and VScode, each time the cell runs for about 35-45s then returns a message that the kernel crashed unexpectedly and and new kernel is created. Im using Python 3.10 with Anaconda package manager.
Spyder had no information about the kernel crash but in VScode I was pointed to this github post Kernel-crashes. I figured somehow my numpy installation was affecting kernel performance or execution. I created a new virtual env using conda; re-installed numpy, pandas, sci-kit learn, etc. Same kernel crash occurring with the same message.
I am using a new Macbook with the m1 chip. Unsure if that's has an influence or is a hint at how to solve this issue. Did see the sci-kit learn installation docs had separate section sci-kit learn install docs about installing with m1 chip but to be honest wasn't exactly sure how to interpret and use that info.
this is the code Im trying to run that causes the kernel to crash. the pca_features is the numpy array return after running .fit_transform method on my raw data. PCA executed fine for what it's worth and showed no issues. Tried using subset of rows, first 5000, to see if that helped cell run at all but still no luck.
Error Message:
The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click here for more info. View Jupyter log for further details.
Canceled future for execute_request message before replies were done
Im open to other clustering algorithms but fact that I can't get this to execute makes me less confident any others (e.g. KMeans) would successfully run. But I could be wrong.
My goal is get a unsupervised clustering algorithm fitted to the data so I can get labels for each observation and compare it against primary components from PCA.
```
aggclus = AgglomerativeClustering(n_clusters = 4, affinity="euclidean",linkage="ward")
subset_pcafeatures = pca_features[:5000,5]
cluster_labels = aggclus.fit_predict(pca_features)
```
Any tips/advice/help/assistance would be much appreciated. Thank you
Updated 29-01-23
output from $ conda list
# packages in environment at ../anaconda3/envs/work-env:
#
# Name Version Build Channel
appnope 0.1.3 pyhd8ed1ab_0 conda-forge
asttokens 2.2.1 pyhd8ed1ab_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
blas 2.116 openblas conda-forge
blas-devel 3.9.0 16_osxarm64_openblas conda-forge
bottleneck 1.3.5 py310h96f19d2_0
brotli 1.0.9 h1a28f6b_7
brotli-bin 1.0.9 h1a28f6b_7
bzip2 1.0.8 h620ffc9_4
ca-certificates 2023.01.10 hca03da5_0
certifi 2022.12.7 py310hca03da5_0
comm 0.1.2 pyhd8ed1ab_0 conda-forge
contourpy 1.0.5 py310h525c30c_0
cycler 0.11.0 pyhd3eb1b0_0
debugpy 1.5.1 py310hc377ac9_0
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pyhd8ed1ab_0 conda-forge
fftw 3.3.9 h1a28f6b_1
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.12.1 h1192e45_0
giflib 5.2.1 h80987f9_1
ipykernel 6.20.2 pyh736e0ef_0 conda-forge
ipython 8.8.0 pyhd1c38e8_0 conda-forge
jedi 0.18.2 pyhd8ed1ab_0 conda-forge
joblib 1.1.1 py310hca03da5_0
jpeg 9e h1a28f6b_0
jupyter_client 7.4.9 pyhd8ed1ab_0 conda-forge
jupyter_core 5.1.1 py310hca03da5_0
kiwisolver 1.4.4 py310h313beb8_0
lcms2 2.12 hba8e193_0
lerc 3.0 hc377ac9_0
libblas 3.9.0 16_osxarm64_openblas conda-forge
libbrotlicommon 1.0.9 h1a28f6b_7
libbrotlidec 1.0.9 h1a28f6b_7
libbrotlienc 1.0.9 h1a28f6b_7
libcblas 3.9.0 16_osxarm64_openblas conda-forge
libcxx 14.0.6 h848a8c0_0
libdeflate 1.8 h1a28f6b_5
libffi 3.4.2 hca03da5_6
libgfortran 5.0.0 11_3_0_hca03da5_28
libgfortran5 11.3.0 h009349e_28
liblapack 3.9.0 16_osxarm64_openblas conda-forge
liblapacke 3.9.0 16_osxarm64_openblas conda-forge
libopenblas 0.3.21 openmp_hc731615_3 conda-forge
libpng 1.6.37 hb8d0fd4_0
libsodium 1.0.18 h27ca646_1 conda-forge
libtiff 4.5.0 h2fd578a_0
libwebp 1.2.4 h68602c7_0
libwebp-base 1.2.4 h1a28f6b_0
llvm-openmp 14.0.6 hc6e5704_0
lz4-c 1.9.4 h313beb8_0
matplotlib 3.6.2 py310hca03da5_0
matplotlib-base 3.6.2 py310h8bbb115_0
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
missingno 0.4.2 pyhd3eb1b0_1
munkres 1.1.4 py_0
ncurses 6.4 h313beb8_0
nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge
numexpr 2.8.4 py310hecc3335_0
numpy 1.23.5 py310hb93e574_0
numpy-base 1.23.5 py310haf87e8b_0
openblas 0.3.21 openmp_hf78f355_3 conda-forge
openssl 1.1.1s h1a28f6b_0
packaging 23.0 pyhd8ed1ab_0 conda-forge
pandas 1.5.2 py310h46d7db6_0
parso 0.8.3 pyhd8ed1ab_0 conda-forge
pexpect 4.8.0 pyh1a96a4e_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.3.0 py310hf4a492f_1
pip 22.3.1 py310hca03da5_0
platformdirs 2.6.2 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.36 pyha770c72_0 conda-forge
psutil 5.9.0 py310h1a28f6b_0
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
pygments 2.14.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 py310hca03da5_0
python 3.10.9 hc0d8a6c_0
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
pytz 2022.7 py310hca03da5_0
pyzmq 23.2.0 py310hc377ac9_0
readline 8.2 h1a28f6b_0
scikit-learn 1.2.0 py310h313beb8_0
scipy 1.9.3 py310h20cbe94_0
seaborn 0.12.2 py310hca03da5_0
setuptools 65.6.3 py310hca03da5_0
six 1.16.0 pyh6c4a22f_0 conda-forge
sqlite 3.40.1 h7a7dc30_0
stack_data 0.6.2 pyhd8ed1ab_0 conda-forge
threadpoolctl 2.2.0 pyh0d69192_0
tk 8.6.12 hb8d0fd4_0
tornado 6.2 py310h1a28f6b_0
traitlets 5.8.1 pyhd8ed1ab_0 conda-forge
typing-extensions 4.4.0 hd8ed1ab_0 conda-forge
typing_extensions 4.4.0 pyha770c72_0 conda-forge
tzdata 2022g h04d1e81_0
wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge
wheel 0.37.1 pyhd3eb1b0_0
xz 5.2.10 h80987f9_1
zeromq 4.3.4 hbdafb3b_1 conda-forge
zlib 1.2.13 h5a0b063_0
zstd 1.5.2 h8574219_0
Related
When I run the pip install in the new conda env:
(base) brando9~ $ pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.9.1+cu111 (from versions: 1.11.0, 1.11.0+cpu, 1.11.0+cu102, 1.11.0+cu113, 1.11.0+cu115, 1.11.0+rocm4.3.1, 1.11.0+rocm4.5.2, 1.12.0, 1.12.0+cpu, 1.12.0+cu102, 1.12.0+cu113, 1.12.0+cu116, 1.12.0+rocm5.0, 1.12.0+rocm5.1.1, 1.12.1, 1.12.1+cpu, 1.12.1+cu102, 1.12.1+cu113, 1.12.1+cu116, 1.12.1+rocm5.0, 1.12.1+rocm5.1.1, 1.13.0, 1.13.0+cpu, 1.13.0+cu116, 1.13.0+cu117, 1.13.0+cu117.with.pypi.cudnn, 1.13.0+rocm5.1.1, 1.13.0+rocm5.2, 1.13.1, 1.13.1+cpu, 1.13.1+cu116, 1.13.1+cu117, 1.13.1+cu117.with.pypi.cudnn, 1.13.1+rocm5.1.1, 1.13.1+rocm5.2)
ERROR: No matching distribution found for torch==1.9.1+cu111
the other env with that pytorch version:
(metalearning3.9) [pzy2#vision-submit ~]$ pip list
Package Version Location
---------------------------------- -------------------- ------------------
absl-py 1.0.0
aiohttp 3.8.3
aiosignal 1.3.1
alabaster 0.7.12
anaconda-client 1.9.0
anaconda-project 0.10.1
antlr4-python3-runtime 4.8
anyio 2.2.0
appdirs 1.4.4
argcomplete 2.0.0
argh 0.26.2
argon2-cffi 20.1.0
arrow 0.13.1
asn1crypto 1.4.0
astroid 2.6.6
astropy 4.3.1
asttokens 2.0.7
astunparse 1.6.3
async-generator 1.10
async-timeout 4.0.2
atomicwrites 1.4.0
attrs 21.2.0
autopep8 1.5.7
Babel 2.9.1
backcall 0.2.0
backports.shutil-get-terminal-size 1.0.0
beautifulsoup4 4.10.0
binaryornot 0.4.4
bitarray 2.3.0
bkcharts 0.2
black 19.10b0
bleach 4.0.0
bokeh 2.4.1
boto 2.49.0
Bottleneck 1.3.2
brotlipy 0.7.0
cached-property 1.5.2
cachetools 5.0.0
certifi 2021.10.8
cffi 1.14.6
chardet 4.0.0
charset-normalizer 2.0.4
cherry-rl 0.1.4
click 8.0.3
cloudpickle 2.0.0
clyent 1.2.2
colorama 0.4.4
conda 4.12.0
conda-content-trust 0+unknown
conda-pack 0.6.0
conda-package-handling 1.8.0
conda-token 0.3.0
configparser 5.3.0
contextlib2 0.6.0.post1
cookiecutter 1.7.2
crc32c 2.3
crcmod 1.7
cryptography 3.4.8
cycler 0.10.0
Cython 0.29.24
cytoolz 0.11.0
daal4py 2021.3.0
dask 2021.10.0
debugpy 1.4.1
decorator 5.1.0
defusedxml 0.7.1
diff-match-patch 20200713
dill 0.3.4
distributed 2021.10.0
docker-pycreds 0.4.0
docutils 0.17.1
entrypoints 0.3
et-xmlfile 1.1.0
executing 0.9.1
fairseq 0.12.2 /home/pzy2/fairseq
fastcache 1.1.0
fastcluster 1.2.6
fasteners 0.17.3
filelock 3.3.1
flake8 3.9.2
Flask 1.1.2
flatbuffers 2.0.7
fonttools 4.25.0
frozenlist 1.3.0
fsspec 2021.8.1
gast 0.4.0
gcs-oauth2-boto-plugin 3.0
gevent 21.8.0
gitdb 4.0.9
GitPython 3.1.27
glob2 0.7
gmpy2 2.0.8
google-apitools 0.5.32
google-auth 2.6.3
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
google-reauth 0.1.1
gql 0.2.0
graphql-core 1.1
greenlet 1.1.1
grpcio 1.44.0
gsutil 5.9
gym 0.22.0
gym-notices 0.0.6
h5py 3.3.0
HeapDict 1.0.1
higher 0.2.1
html5lib 1.1
httplib2 0.20.4
huggingface-hub 0.5.1
hydra-core 1.0.7
idna 3.2
imagecodecs 2021.8.26
imageio 2.9.0
imagesize 1.2.0
importlib-metadata 4.12.0
inflection 0.5.1
iniconfig 1.1.1
intervaltree 3.1.0
ipykernel 6.4.1
ipython 7.29.0
ipython-genutils 0.2.0
ipywidgets 7.6.5
isort 5.9.3
itsdangerous 2.0.1
jdcal 1.4.1
jedi 0.18.0
jeepney 0.7.1
Jinja2 2.11.3
jinja2-time 0.2.0
joblib 1.1.0
json5 0.9.6
jsonschema 3.2.0
jupyter 1.0.0
jupyter-client 6.1.12
jupyter-console 6.4.0
jupyter-core 4.8.1
jupyter-server 1.4.1
jupyterlab 3.2.1
jupyterlab-pygments 0.1.2
jupyterlab-server 2.8.2
jupyterlab-widgets 1.0.0
keras 2.10.0
Keras-Preprocessing 1.1.2
keyring 23.1.0
kiwisolver 1.3.1
lark-parser 0.12.0
lazy-object-proxy 1.6.0
learn2learn 0.1.7
libarchive-c 2.9
libclang 14.0.6
littleutils 0.2.2
llvmlite 0.37.0
locket 0.2.1
loguru 0.6.0
lxml 4.6.3
Markdown 3.3.6
MarkupSafe 1.1.1
matplotlib 3.4.3
matplotlib-inline 0.1.2
mccabe 0.6.1
mistune 0.8.4
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
mock 4.0.3
monotonic 1.6
more-itertools 8.10.0
mpmath 1.2.1
msgpack 1.0.2
multidict 6.0.2
multipledispatch 0.6.0
munkres 1.1.4
mypy-extensions 0.4.3
nbclassic 0.2.6
nbclient 0.5.3
nbconvert 6.1.0
nbformat 5.1.3
nest-asyncio 1.5.1
networkx 2.6.3
nltk 3.6.5
nose 1.3.7
notebook 6.4.5
numba 0.54.1
numexpr 2.7.3
numpy 1.20.3
numpydoc 1.1.0
nvidia-ml-py3 7.352.0
nvidia-smi 0.1.3
oauth2client 4.1.3
oauthlib 3.2.0
olefile 0.46
omegaconf 2.0.6
opencv-python 4.6.0.66
openpyxl 3.0.9
opt-einsum 3.3.0
ordered-set 4.1.0
packaging 21.0
pandas 1.3.4
pandocfilters 1.4.3
parso 0.8.2
partd 1.2.0
path 16.0.0
pathlib2 2.3.6
pathspec 0.7.0
pathtools 0.1.2
patsy 0.5.2
pep8 1.7.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.4.0
pip 22.2.2
pkginfo 1.7.1
plotly 5.7.0
pluggy 0.13.1
ply 3.11
portalocker 2.5.1
poyo 0.5.0
progressbar2 4.0.0
prometheus-client 0.11.0
promise 2.3
prompt-toolkit 3.0.20
protobuf 3.19.6
psutil 5.8.0
ptyprocess 0.7.0
py 1.10.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycodestyle 2.7.0
pycosat 0.6.3
pycparser 2.20
pycurl 7.44.1
pydocstyle 6.1.1
pyerfa 2.0.0
pyflakes 2.3.1
Pygments 2.10.0
pylint 2.9.6
pyls-spyder 0.4.0
pyodbc 4.0.0-unsupported
pyOpenSSL 21.0.0
pyparsing 3.0.4
pyrsistent 0.18.0
PySocks 1.7.1
pytest 6.2.4
python-dateutil 2.8.2
python-lsp-black 1.0.0
python-lsp-jsonrpc 1.0.0
python-lsp-server 1.2.4
python-slugify 5.0.2
python-utils 3.1.0
pytz 2021.3
pyu2f 0.1.5
PyWavelets 1.1.1
pyxdg 0.27
PyYAML 6.0
pyzmq 22.2.1
QDarkStyle 3.0.2
qpth 0.0.15
qstylizer 0.1.10
QtAwesome 1.0.2
qtconsole 5.1.1
QtPy 1.10.0
regex 2021.8.3
requests 2.26.0
requests-oauthlib 1.3.1
retry-decorator 1.1.1
rope 0.19.0
rsa 4.7.2
Rtree 0.9.7
ruamel-yaml-conda 0.15.100
sacrebleu 2.2.0
sacremoses 0.0.49
scikit-image 0.18.3
scikit-learn 0.24.2
scikit-learn-intelex 2021.20210714.170444
scipy 1.7.1
seaborn 0.11.2
SecretStorage 3.3.1
Send2Trash 1.8.0
sentry-sdk 1.5.9
setproctitle 1.2.2
setuptools 58.0.4
shortuuid 1.0.8
simplegeneric 0.8.1
singledispatch 3.7.0
sip 4.19.13
six 1.16.0
sklearn 0.0
smmap 5.0.0
sniffio 1.2.0
snowballstemmer 2.1.0
sorcery 0.2.2
sortedcollections 2.1.0
sortedcontainers 2.4.0
soupsieve 2.2.1
Sphinx 4.2.0
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
sphinxcontrib-websupport 1.2.4
spyder 5.1.5
spyder-kernels 2.1.3
SQLAlchemy 1.4.22
statsmodels 0.12.2
subprocess32 3.5.4
sympy 1.9
tables 3.6.1
TBB 0.2
tblib 1.7.0
tensorboard 2.10.1
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorflow-estimator 2.10.0
tensorflow-gpu 2.10.1
tensorflow-io-gcs-filesystem 0.27.0
termcolor 2.0.1
terminado 0.9.4
testpath 0.5.0
text-unidecode 1.3
textdistance 4.2.1
tfrecord 1.14.1
threadpoolctl 2.2.0
three-merge 0.1.1
tifffile 2021.7.2
timm 0.6.11
tinycss 0.4
tokenizers 0.11.6
toml 0.10.2
toolz 0.11.1
torch 1.9.1+cu111
torchaudio 0.9.1
torchmeta 1.8.0
torchtext 0.10.1
torchvision 0.10.1+cu111
tornado 6.1
tqdm 4.62.3
traitlets 5.1.0
transformers 4.18.0
typed-ast 1.4.3
typing-extensions 3.10.0.2
ujson 4.0.2
ultimate-anatome 0.1.1
ultimate-aws-cv-task2vec 0.0.1
unicodecsv 0.14.1
Unidecode 1.2.0
urllib3 1.26.7
wandb 0.13.5
watchdog 2.1.3
wcwidth 0.2.5
webencodings 0.5.1
Werkzeug 2.0.2
wheel 0.37.0
whichcraft 0.6.1
widgetsnbextension 3.5.1
wrapt 1.12.1
wurlitzer 2.1.1
xlrd 2.0.1
XlsxWriter 3.0.1
xlwt 1.3.0
yapf 0.31.0
yarl 1.7.2
zict 2.0.0
zipp 3.6.0
zope.event 4.5.0
zope.interface 5.4.0
WARNING: You are using pip version 22.2.2; however, version 22.3.1 is available.
You should consider upgrading via the '/home/pzy2/miniconda3/envs/metalearning3.9/bin/python -m pip install --upgrade pip' command.
(metalearning3.9) [pzy2#vision-submit ~]$
I asked a related question because I can't install pytorch with cuda with conda, see details here: why does conda install the pytorch CPU version despite me putting explicitly to download the cuda toolkit version?
I think this works:
# -- Install PyTorch sometimes requires more careful versioning due to cuda, ref: official install instruction https://pytorch.org/get-started/previous-versions/
# you need python 3.9 for torch version 1.9.1 to work, due to torchmeta==1.8.0 requirement
if ! python -V 2>&1 | grep -q 'Python 3\.9'; then
echo "Error: Python 3.9 is required!"
exit 1
fi
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
From looking at the link you've provided, I can see
cu111/torch-1.9.1%2Bcu111-cp36-cp36m-linux_x86_64.whl
cu111/torch-1.9.1%2Bcu111-cp36-cp36m-win_amd64.whl
cu111/torch-1.9.1%2Bcu111-cp37-cp37m-linux_x86_64.whl
cu111/torch-1.9.1%2Bcu111-cp37-cp37m-win_amd64.whl
cu111/torch-1.9.1%2Bcu111-cp38-cp38-linux_x86_64.whl
cu111/torch-1.9.1%2Bcu111-cp38-cp38-win_amd64.whl
cu111/torch-1.9.1%2Bcu111-cp39-cp39-linux_x86_64.whl
cu111/torch-1.9.1%2Bcu111-cp39-cp39-win_amd64.whl
cp39 means Python version 3.9. Notice there is no support for 3.10 or 3.11. In your new environment, you probably are running a newer version of Python, whereas in the other environment you have Python 3.6, 3.7, 3.8, or 3.9
To install pytorch 1.9.1cu11 you need python 3.9 to be avaiable. Added that to my bash install.sh
# - create conda env
conda create -n metalearning_gpu python=3.9
conda activate metalearning_gpu
## conda remove --name metalearning_gpu --all
# - make sure pip is up to date
which python
pip install --upgrade pip
pip3 install --upgrade pip
which pip
which pip3
# -- Install PyTorch sometimes requires more careful versioning due to cuda, ref: official install instruction https://pytorch.org/get-started/previous-versions/
# you need python 3.9 for torch version 1.9.1 to work, due to torchmeta==1.8.0 requirement
if ! python -V 2>&1 | grep -q 'Python 3\.9'; then
echo "Error: Python 3.9 is required!"
exit 1
fi
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
I am trying to install pytorch-geometric for a deep-learning project. Torch-sparse is throwing segmentation faults when I attempt to import it (see below). Initially I tried different versions of each required library, as I thought it might be a GPU issue, but I've since tried to simplify by installing cpu-only versions.
Python 3.9.12 (main, Apr 5 2022, 06:56:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import torch_scatter
>>> import torch_cluster
>>> import torch_sparse
Segmentation fault (core dumped)
And the same issue, presumably due to torch_sparse, when importing pytorch_geometric:
Python 3.9.12 (main, Apr 5 2022, 06:56:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch_geometric
Segmentation fault (core dumped)
I'm on an Ubuntu distribution:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy
Here's my (lightweight for DL) conda installs:
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
blas 1.0 mkl
brotlipy 0.7.0 py310h7f8727e_1002
bzip2 1.0.8 h7b6447c_0
ca-certificates 2022.07.19 h06a4308_0
certifi 2022.9.24 py310h06a4308_0
cffi 1.15.1 py310h74dc2b5_0
charset-normalizer 2.0.4 pyhd3eb1b0_0
cpuonly 2.0 0 pytorch
cryptography 37.0.1 py310h9ce1e76_0
fftw 3.3.9 h27cfd23_1
idna 3.4 py310h06a4308_0
intel-openmp 2021.4.0 h06a4308_3561
jinja2 3.0.3 pyhd3eb1b0_0
joblib 1.1.1 py310h06a4308_0
ld_impl_linux-64 2.38 h1181459_1
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgfortran-ng 11.2.0 h00389a5_1
libgfortran5 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.0.3 h7f8727e_2
markupsafe 2.1.1 py310h7f8727e_0
mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py310h7f8727e_0
mkl_fft 1.3.1 py310hd6ae3a3_0
mkl_random 1.2.2 py310h00e6091_0
ncurses 6.3 h5eee18b_3
numpy 1.23.3 py310hd5efca6_0
numpy-base 1.23.3 py310h8e6c178_0
openssl 1.1.1q h7f8727e_0
pip 22.2.2 py310h06a4308_0
pycparser 2.21 pyhd3eb1b0_0
pyg 2.1.0 py310_torch_1.12.0_cpu pyg
pyopenssl 22.0.0 pyhd3eb1b0_0
pyparsing 3.0.9 py310h06a4308_0
pysocks 1.7.1 py310h06a4308_0
python 3.10.6 haa1d7c7_0
pytorch 1.12.1 py3.10_cpu_0 pytorch
pytorch-cluster 1.6.0 py310_torch_1.12.0_cpu pyg
pytorch-mutex 1.0 cpu pytorch
pytorch-scatter 2.0.9 py310_torch_1.12.0_cpu pyg
pytorch-sparse 0.6.15 py310_torch_1.12.0_cpu pyg
readline 8.1.2 h7f8727e_1
requests 2.28.1 py310h06a4308_0
scikit-learn 1.1.2 py310h6a678d5_0
scipy 1.9.1 py310hd5efca6_0
setuptools 63.4.1 py310h06a4308_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.39.3 h5082296_0
threadpoolctl 2.2.0 pyh0d69192_0
tk 8.6.12 h1ccaba5_0
tqdm 4.64.1 py310h06a4308_0
typing_extensions 4.3.0 py310h06a4308_0
tzdata 2022e h04d1e81_0
urllib3 1.26.12 py310h06a4308_0
wheel 0.37.1 pyhd3eb1b0_0
xz 5.2.6 h5eee18b_0
zlib 1.2.13 h5eee18b_0
Any help would be greatly appreciated!
I've found a combination of packages that works for me - hopefully someone else will have this issue at some point and be able to reproduce the steps from me talking to myself here. The full process for getting stuff working was:
Fresh conda environment with forced Python=3.9 (conda create -n ENVNAME python=3.9)
Activate that environment
Install basic python packages (conda install numpy pandas matplotlib scikit-learn)
Check CUDA version if working with a GPU (nvidia-smi in terminal prints these details for NVIDIA cards)
Install Pytorch using their suggested conda command (conda install pytorch torchvision torchaudio cudatoolkit=CUDA_VERSION -c pytorch -c conda-forge). This had to go through the env solving process on my machine.
Install pytorch geometric (or just torch sparse if that's all you need) with conda install pyg -c pyg. Again this had a solving process.
Check that torch_sparse imports without fault
Here's the conda list for this working combination of packages:
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
blas 1.0 mkl
bottleneck 1.3.5 py39h7deecbd_0
brotli 1.0.9 h5eee18b_7
brotli-bin 1.0.9 h5eee18b_7
brotlipy 0.7.0 py39hb9d737c_1004 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2022.9.24 ha878542_0 conda-forge
certifi 2022.9.24 py39h06a4308_0
cffi 1.14.6 py39he32792d_0 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
cryptography 37.0.2 py39hd97740a_0 conda-forge
cudatoolkit 11.6.0 hecad31d_10 conda-forge
cycler 0.11.0 pyhd3eb1b0_0
dbus 1.13.18 hb2f20db_0
expat 2.4.9 h6a678d5_0
ffmpeg 4.3 hf484d3e_0 pytorch
fftw 3.3.9 h27cfd23_1
fontconfig 2.13.1 h6c09931_0
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.11.0 h70c0345_0
giflib 5.2.1 h7b6447c_0
glib 2.69.1 h4ff587b_1
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
gst-plugins-base 1.14.0 h8213a91_2
gstreamer 1.14.0 h28cd5cc_2
icu 58.2 he6710b0_3
idna 3.4 pyhd8ed1ab_0 conda-forge
intel-openmp 2021.4.0 h06a4308_3561
jinja2 3.0.3 pyhd3eb1b0_0
joblib 1.1.1 py39h06a4308_0
jpeg 9e h7f8727e_0
kiwisolver 1.4.2 py39h295c915_0
krb5 1.19.2 hac12032_0
lame 3.100 h7f98852_1001 conda-forge
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libbrotlicommon 1.0.9 h5eee18b_7
libbrotlidec 1.0.9 h5eee18b_7
libbrotlienc 1.0.9 h5eee18b_7
libclang 10.0.1 default_hb85057a_2
libdeflate 1.8 h7f8727e_5
libedit 3.1.20210910 h7f8727e_0
libevent 2.1.12 h8f2d780_0
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgfortran-ng 11.2.0 h00389a5_1
libgfortran5 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libiconv 1.17 h166bdaf_0 conda-forge
libllvm10 10.0.1 hbcb73fb_5
libpng 1.6.37 hbc83047_0
libpq 12.9 h16c4e8d_3
libstdcxx-ng 11.2.0 h1234567_1
libtiff 4.4.0 hecacb30_0
libuuid 1.0.3 h7f8727e_2
libwebp 1.2.4 h11a3e52_0
libwebp-base 1.2.4 h5eee18b_0
libxcb 1.15 h7f8727e_0
libxkbcommon 1.0.1 hfa300c1_0
libxml2 2.9.14 h74e7548_0
libxslt 1.1.35 h4e12654_0
lz4-c 1.9.3 h295c915_1
markupsafe 2.1.1 py39h7f8727e_0
matplotlib 3.5.2 py39h06a4308_0
matplotlib-base 3.5.2 py39hf590b9c_0
mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py39h7f8727e_0
mkl_fft 1.3.1 py39hd3c417c_0
mkl_random 1.2.2 py39h51133e4_0
munkres 1.1.4 py_0
ncurses 6.3 h5eee18b_3
nettle 3.6 he412f7d_0 conda-forge
nspr 4.33 h295c915_0
nss 3.74 h0370c37_0
numexpr 2.8.3 py39h807cd23_0
numpy 1.23.3 py39h14f4228_0
numpy-base 1.23.3 py39h31eccc5_0
openh264 2.1.1 h780b84a_0 conda-forge
openssl 1.1.1q h7f8727e_0
packaging 21.3 pyhd3eb1b0_0
pandas 1.4.4 py39h6a678d5_0
pcre 8.45 h295c915_0
pillow 9.2.0 py39hace64e9_1
pip 22.2.2 py39h06a4308_0
ply 3.11 py39h06a4308_0
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyg 2.1.0 py39_torch_1.12.0_cu116 pyg
pyopenssl 22.0.0 pyhd8ed1ab_1 conda-forge
pyparsing 3.0.9 py39h06a4308_0
pyqt 5.15.7 py39h6a678d5_1
pyqt5-sip 12.11.0 py39h6a678d5_1
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.9.13 haa1d7c7_2
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.9 2_cp39 conda-forge
pytorch 1.12.1 py3.9_cuda11.6_cudnn8.3.2_0 pytorch
pytorch-cluster 1.6.0 py39_torch_1.12.0_cu116 pyg
pytorch-mutex 1.0 cuda pytorch
pytorch-scatter 2.0.9 py39_torch_1.12.0_cu116 pyg
pytorch-sparse 0.6.15 py39_torch_1.12.0_cu116 pyg
pytz 2022.1 py39h06a4308_0
qt-main 5.15.2 h327a75a_7
qt-webengine 5.15.9 hd2b0992_4
qtwebkit 5.212 h4eab89a_4
readline 8.2 h5eee18b_0
requests 2.28.1 pyhd8ed1ab_1 conda-forge
scikit-learn 1.1.2 py39h6a678d5_0
scipy 1.9.1 py39h14f4228_0
setuptools 63.4.1 py39h06a4308_0
sip 6.6.2 py39h6a678d5_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.39.3 h5082296_0
threadpoolctl 2.2.0 pyh0d69192_0
tk 8.6.12 h1ccaba5_0
toml 0.10.2 pyhd3eb1b0_0
torchaudio 0.12.1 py39_cu116 pytorch
torchvision 0.13.1 py39_cu116 pytorch
tornado 6.2 py39h5eee18b_0
tqdm 4.64.1 py39h06a4308_0
typing_extensions 4.4.0 pyha770c72_0 conda-forge
tzdata 2022e h04d1e81_0
urllib3 1.26.11 pyhd8ed1ab_0 conda-forge
wheel 0.37.1 pyhd3eb1b0_0
xz 5.2.6 h5eee18b_0
zlib 1.2.13 h5eee18b_0
zstd 1.5.2 ha4553b6_0
I tried to install gdal package on my conda environment. I activated gcpy environment and installed the gdal package using conda install -c conda-forge gdal. The package installs successfully. But, when I tried to import the package, I get error:
In [1]: import gdal
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-ee38efafc30b> in <module>
----> 1 import gdal
ModuleNotFoundError: No module named 'gdal'
I can see the package in the conda list.
gdal 3.2.1 py39h409cc32_1 conda-forge
geos 3.8.1 he1b5a44_0 conda-forge
geoschem-gcpy 1.0.0 py39hf3d152e_0 conda-forge
geotiff 1.6.0 h5d11630_3 conda-forge
gettext 0.19.8.1 h0b5b191_1005 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
glib 2.66.4 hc4f0c31_2 conda-forge
glib-tools 2.66.4 hc4f0c31_2 conda-forge
gst-plugins-base 1.14.5 h0935bb2_2 conda-forge
gstreamer 1.18.3 h3560a44_0 conda-forge
h5netcdf 0.8.1 py_0 conda-forge
h5py 3.1.0 nompi_py39h25020de_100 conda-forge
hdf4 4.2.13 h10796ff_1004 conda-forge
hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge
heapdict 1.0.1 py_0 conda-forge
helpdev 0.7.1 pyhd8ed1ab_0 conda-forge
icu 68.1 h58526e2_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
imagesize 1.2.0 py_0 conda-forge
importlib-metadata 3.4.0 py39hf3d152e_0 conda-forge
importlib_metadata 3.4.0 hd8ed1ab_0 conda-forge
intervaltree 3.0.2 py_0 conda-forge
ipykernel 5.4.2 py39hef51801_0 conda-forge
ipython 7.19.0 py39hef51801_2 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
isort 5.7.0 pyhd8ed1ab_0 conda-forge
jedi 0.17.2 py39hf3d152e_1 conda-forge
jeepney 0.6.0 pyhd8ed1ab_0 conda-forge
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
joblib 1.0.0 pyhd8ed1ab_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.13.1 hbfbb72e_1002 conda-forge
jsonschema 3.2.0 py_2 conda-forge
jupyter_client 6.1.11 pyhd8ed1ab_1 conda-forge
jupyter_core 4.7.0 py39hf3d152e_1 conda-forge
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
kealib 1.4.14 h0042707_0 conda-forge
keyring 22.0.1 py39hf3d152e_0 conda-forge
kiwisolver 1.3.1 py39h1a9c180_1 conda-forge
krb5 1.17.2 h926e7f8_0 conda-forge
lazy-object-proxy 1.4.3 py39h07f9747_2 conda-forge
lcms2 2.11 hcbb858e_1 conda-forge
ld_impl_linux-64 2.35.1 hea4e1c9_1 conda-forge
libblas 3.9.0 7_openblas conda-forge
libcblas 3.9.0 7_openblas conda-forge
libclang 11.0.1 default_ha53f305_1 conda-forge
libcurl 7.71.1 hcdd3856_8 conda-forge
libdap4 3.20.6 h1d1bd15_1 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 hcdb4288_3 conda-forge
libffi 3.3 h58526e2_2 conda-forge
libgcc-ng 9.3.0 h2828fa1_18 conda-forge
libgdal 3.2.1 h52563cd_1 conda-forge
libgfortran-ng 9.3.0 hff62375_18 conda-forge
libgfortran5 9.3.0 hff62375_18 conda-forge
libglib 2.66.4 h748fe8e_2 conda-forge
libgomp 9.3.0 h2828fa1_18 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 h74f7ee3_1012 conda-forge
liblapack 3.9.0 7_openblas conda-forge
libllvm11 11.0.1 hf817b99_0 conda-forge
libnetcdf 4.7.4 nompi_h56d31a8_107 conda-forge
libnghttp2 1.41.0 h8cfc5f6_2 conda-forge
libopenblas 0.3.12 pthreads_h4812303_1 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 12.3 h255efa7_3 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libspatialindex 1.9.3 he1b5a44_3 conda-forge
libspatialite 5.0.0 heaf302f_0 conda-forge
libssh2 1.9.0 hab1572f_5 conda-forge
libstdcxx-ng 9.3.0 h6de172a_18 conda-forge
libtiff 4.2.0 hdc55705_0 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp-base 1.1.0 h36c2ea0_3 conda-forge
libxcb 1.13 h7f98852_1003 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.10 h72842e0_3 conda-forge
I tried conda update --all as said Here . I also tried all similar question. But, still no solution.
Reference 1
Reference 2
There are five major modules that are included with the GDAL Python bindings.:
from osgeo import gdal
from osgeo import ogr
from osgeo import osr
from osgeo import gdal_array
from osgeo import gdalconst
Additionally, there are five compatibility modules that are included but provide notices to state that they are deprecated and will be going away. If you are using GDAL 1.7 bindings, you should update your imports to utilize the usage above, but the following will work until at least GDAL 2.1.
import gdal
import ogr
import osr
import gdalnumeric
import gdalconst
If you have previous code that imported the global module and still need to support the old import, a simple try…except import can silence the deprecation warning and keep things named essentially the same as before:
try:
from osgeo import gdal
except ImportError:
import gdal
I did conda list but I can't see what I am looking for. Are conda packages installed with conda developed . not shown?
# packages in environment at /Users/pinocchio/anaconda3/envs/myenv:
#
# Name Version Build Channel
absl-py 0.9.0 py37_0
appnope 0.1.0 py37hc8dfbb8_1001 conda-forge
asn1crypto 1.3.0 py37_0
astroid 2.3.3 py37_0
attrs 19.3.0 py_0 conda-forge
backcall 0.1.0 py_0 conda-forge
beautifulsoup4 4.8.2 py37_0
blas 1.0 mkl
bleach 3.1.4 pyh9f0ad1d_0 conda-forge
bzip2 1.0.8 h1de35cc_0
c-ares 1.15.0 h1de35cc_1001
ca-certificates 2020.1.1 0
cairo 1.14.12 hc4e6be7_4
certifi 2020.4.5.1 py37_0
cffi 1.14.0 py37hb5b8e2f_0
chardet 3.0.4 py37_1003
conda 4.8.3 py37_0
conda-build 3.18.11 py37_0
conda-package-handling 1.6.0 py37h1de35cc_0
cryptography 2.8 py37ha12b0ac_0
cycler 0.10.0 py37_0
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
entrypoints 0.3 py37hc8dfbb8_1001 conda-forge
expat 2.2.6 h0a44026_0
filelock 3.0.12 py_0
fontconfig 2.13.0 h5d5b041_1
freetype 2.9.1 hb4e5f40_0
fribidi 1.0.5 h1de35cc_0
gettext 0.19.8.1 h15daf44_3
glib 2.63.1 hd977a24_0
glob2 0.7 py_0
graphite2 1.3.13 h2098e52_0
graphviz 2.40.1 hefbbd9a_2
grpcio 1.16.1 py37h044775b_1
harfbuzz 1.8.8 hb8d4a28_0
icu 58.2 h4b95b61_1
idna 2.9 py_1
importlib-metadata 1.6.0 py37hc8dfbb8_0 conda-forge
importlib_metadata 1.6.0 0 conda-forge
intel-openmp 2019.4 233
ipykernel 5.2.0 py37h43977f1_1 conda-forge
ipython 7.13.0 py37hc8dfbb8_2 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
isort 4.3.21 py37_0
jedi 0.16.0 py37hc8dfbb8_1 conda-forge
jinja2 2.11.1 py_0
jpeg 9b he5867d9_2
json5 0.9.0 py_0 conda-forge
jsonschema 3.2.0 py37hc8dfbb8_1 conda-forge
jupyter_client 6.1.2 py_0 conda-forge
jupyter_core 4.6.3 py37hc8dfbb8_1 conda-forge
jupyterlab 2.0.1 py_0 conda-forge
jupyterlab_server 1.0.7 py_0 conda-forge
kiwisolver 1.1.0 py37h0a44026_0
lazy-object-proxy 1.4.3 py37h1de35cc_0
libarchive 3.3.3 h786848e_5
libcxx 4.0.1 hcfea43d_1
libcxxabi 4.0.1 hcfea43d_1
libedit 3.1.20181209 hb402a30_0
libffi 3.2.1 h475c297_4
libgfortran 3.0.1 h93005f0_2
libiconv 1.15 hdd342a3_7
liblief 0.9.0 h2a1bed3_2
libpng 1.6.37 ha441bb4_0
libprotobuf 3.11.4 hd9629dc_0
libsodium 1.0.17 h01d97ff_0 conda-forge
libtiff 4.1.0 hcb84e12_0
libxml2 2.9.9 hf6e021a_1
lz4-c 1.8.1.2 h1de35cc_0
lzo 2.10 h362108e_2
markdown 3.1.1 py37_0
markupsafe 1.1.1 py37h1de35cc_0
matplotlib 3.1.3 py37_0
matplotlib-base 3.1.3 py37h9aa3819_0
mccabe 0.6.1 py37_1
mistune 0.8.4 py37h0b31af3_1000 conda-forge
mkl 2019.4 233
mkl-service 2.3.0 py37hfbe908c_0
mkl_fft 1.0.15 py37h5e564d8_0
mkl_random 1.1.0 py37ha771720_0
nbconvert 5.6.1 py37_0 conda-forge
nbformat 5.0.4 py_0 conda-forge
ncurses 6.2 h0a44026_0
ninja 1.9.0 py37h04f5b5a_0
notebook 6.0.3 py37_0 conda-forge
numpy 1.18.1 py37h7241aed_0
numpy-base 1.18.1 py37h6575580_1
olefile 0.46 py37_0
openssl 1.1.1g h1de35cc_0
pandoc 2.9.2 0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
pango 1.42.4 h060686c_0
parso 0.6.2 py_0 conda-forge
pcre 8.43 h0a44026_0
pexpect 4.8.0 py37hc8dfbb8_1 conda-forge
pickleshare 0.7.5 py37hc8dfbb8_1001 conda-forge
pillow 6.2.1 py37hb68e598_0
pip 20.0.2 py37_1
pixman 0.38.0 h1de35cc_0
pkginfo 1.5.0.1 py37_0
prometheus_client 0.7.1 py_0 conda-forge
prompt-toolkit 3.0.5 py_0 conda-forge
protobuf 3.11.4 py37h0a44026_0
psutil 5.7.0 py37h1de35cc_0
ptyprocess 0.6.0 py_1001 conda-forge
py-lief 0.9.0 py37h1413db1_2
pycosat 0.6.3 py37h1de35cc_0
pycparser 2.20 py_0
pygments 2.6.1 py_0 conda-forge
pylint 2.4.4 py37_0
pyopenssl 19.1.0 py37_0
pyparsing 2.4.6 py_0
pyrsistent 0.16.0 py37h9bfed18_0 conda-forge
pysocks 1.7.1 py37_0
python 3.7.7 hc70fcce_0_cpython
python-dateutil 2.8.1 py_0 conda-forge
python-graphviz 0.13.2 pypi_0 pypi
python-libarchive-c 2.8 py37_13
python_abi 3.7 1_cp37m conda-forge
pytorch 1.4.0 py3.7_0 pytorch
pytz 2019.3 py_0
pyyaml 5.3.1 py37h1de35cc_0
pyzmq 18.1.1 py37h0a44026_0
readline 8.0 h1de35cc_0
requests 2.23.0 py37_0
ripgrep 11.0.2 he32d670_0
ruamel_yaml 0.15.87 py37h1de35cc_0
send2trash 1.5.0 py_0 conda-forge
setuptools 46.1.1 py37_0
six 1.14.0 py37_0
soupsieve 2.0 py_0
sqlite 3.31.1 ha441bb4_0
tensorboard 2.0.0 pyhb38c66f_1
terminado 0.8.3 py37hc8dfbb8_1 conda-forge
testpath 0.4.4 py_0 conda-forge
tk 8.6.8 ha441bb4_0
torchvision 0.2.1 py_2 soumith
tornado 6.0.4 py37h9bfed18_1 conda-forge
tqdm 4.43.0 py_0
traitlets 4.3.3 py37hc8dfbb8_1 conda-forge
urllib3 1.25.8 py37_0
wcwidth 0.1.9 pyh9f0ad1d_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
werkzeug 1.0.0 py_0
wheel 0.34.2 py37_0
wrapt 1.12.1 py37h1de35cc_1
xz 5.2.4 h1de35cc_4
yaml 0.1.7 hc338f04_2
zeromq 4.3.2 h6de7cb9_2 conda-forge
zipp 3.1.0 py_0 conda-forge
zlib 1.2.11 h1de35cc_3
zstd 1.3.7 h5bba6e5_0
utils or something like that should be there but its not.
Update:
I've tried what this suggests and it does not work. I conda develop . my package and then when I run the python interpreter and try to import it the import fails. Why?
>>> import my_pkg
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'my_pkg'
I also tried to install it with pip pip install -e and it doesn't find my package either after I go to the interpreter and try to import it. Why is that?
If you look carefully after running conda develop . (where the setup.py file is) you will notice that the command outputs the following:
(automl) brandBrandoParetoopareto~/ultimate-utils/uutils $ conda develop .
added /Users/brandBrandoParetoopareto/ultimate-utils/uutils
completed operation for: /Users/brandBrandoParetoopareto/ultimate-utils/uutils
if you then check the sys.path python has then you can that new path has been added:
(automl) brandBrandoParetoopareto~/ultimate-utils/uutils $ python
Python 3.7.7 (default, Mar 26 2020, 10:32:53)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for p in sys.path:
... print(p)
...
/Users/brandBrandoParetoopareto/anaconda3/envs/automl/lib/python37.zip
/Users/brandBrandoParetoopareto/anaconda3/envs/automl/lib/python3.7
/Users/brandBrandoParetoopareto/anaconda3/envs/automl/lib/python3.7/lib-dynload
/Users/brandBrandoParetoopareto/anaconda3/envs/automl/lib/python3.7/site-packages
/Users/brandBrandoParetoopareto/automl-meta-learning/automl
/Users/brandBrandoParetoopareto/higher
/Users/brandBrandoParetoopareto/ultimate-utils/uutils
you can confirm this by removing it and printing the contents of sys.path:
(automl) brandBrandoParetoopareto~/ultimate-utils/uutils $ conda develop -u .
uninstalled: /Users/brandBrandoParetoopareto/ultimate-utils/uutils
check sys path again:
(automl) brandBrandoParetoopareto~/ultimate-utils/uutils $ python
Python 3.7.7 (default, Mar 26 2020, 10:32:53)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for p in sys.path:
... print(p)
...
/Users/brandBrandoParetoopareto/anaconda3/envs/automl/lib/python37.zip
/Users/brandBrandoParetoopareto/anaconda3/envs/automl/lib/python3.7
/Users/brandBrandoParetoopareto/anaconda3/envs/automl/lib/python3.7/lib-dynload
/Users/brandBrandoParetoopareto/anaconda3/envs/automl/lib/python3.7/site-packages
/Users/brandBrandoParetoopareto/automl-meta-learning/automl
/Users/brandBrandoParetoopareto/higher
now it's gone!
Also, note that conda or python (not sure which) also adds the current path always. This is important to note because when I did import tests outside of the repo I was working it wasn't working but it was inside the repo which seemed very mysterious which I can only attribute to that.
Addendum
to check python sys.path from terminal run that command as a string with python cmd:
python -c "import sys; print(sys.path)"
even better:
python -c "import sys; [print(p) for p in sys.path]"
Original answer: https://stackoverflow.com/a/59903590/1601580
I'm trying to import some packages with spyder (OS x64), Anaconda and pyton 3.x
The error is pretty famous in the internet. The solution proposed is to match the version of the library 1.10.5 with the HDF5 (mine is 1.10.4)
The question is that I can't find HDF5 version 1.10.5
and, the other hand, cannot understand what I could downgrade.
At this link: https://anaconda.org/conda-forge/hdf5 seems exist version 1.10.5 but when I type in the prompt of anaconda conda install -c conda-forge hdf5
the version remain 1.10.4.
Here the warning:
Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
You can, at your own risk, disable this warning by setting the environment
variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.10.4, library is 1.10.5
SUMMARY OF THE HDF5 CONFIGURATION
=================================
General Information:
‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑
HDF5 Version: 1.10.5
Configured on: 2019
Configured by: Visual Studio 15 2017 Win64
Host system: Windows.0.17763
Uname information: Windows
Byte sex: little‑endian
Installation point: C:/Program Files/HDF5
Compiling Options:
‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑
Build Mode:
Debugging Symbols:
Asserts:
Profiling:
Optimization Level:
Linking Options:
‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑
Libraries:
Statically Linked Executables: OFF
LDFLAGS: /machine:x64
H5_LDFLAGS:
AM_LDFLAGS:
Extra libraries:
Archiver:
Ranlib:
Languages:
‑‑‑‑‑‑‑‑‑‑
C: yes
C Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe 19.16.27027.1
CPPFLAGS:
H5_CPPFLAGS:
AM_CPPFLAGS:
CFLAGS: /DWIN32 /D_WINDOWS /W3
H5_CFLAGS:
AM_CFLAGS:
Shared C Library: YES
Static C Library: YES
Fortran: OFF
Fortran Compiler:
Fortran Flags:
H5 Fortran Flags:
AM Fortran Flags:
Shared Fortran Library: YES
Static Fortran Library: YES
C++: ON
C++ Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe 19.16.27027.1
C++ Flags: /DWIN32 /D_WINDOWS /W3 /GR /EHsc
H5 C++ Flags:
AM C++ Flags:
Shared C++ Library: YES
Static C++ Library: YES
JAVA: OFF
JAVA Compiler:
Features:
‑‑‑‑‑‑‑‑‑
Parallel HDF5: OFF
Parallel Filtered Dataset Writes:
Large Parallel I/O:
High‑level library: ON
Threadsafety: OFF
Default API mapping: v110
With deprecated public symbols: ON
I/O filters (external): DEFLATE DECODE ENCODE
MPE:
Direct VFD:
dmalloc:
Packages w/ extra debug output:
API Tracing: OFF
Using memory checker: OFF
Memory allocation sanity checks: OFF
Function Stack Tracing: OFF
Strict File Format Checks: OFF
Optimization Instrumentation:
Bye...
Here all the packages installed:
# packages in environment at C:\Users\Megaport\Anaconda3\envs\venv:
#
# Name Version Build Channel
_py-xgboost-mutex 2.0 cpu_0
_tflow_select 2.3.0 mkl
absl-py 0.8.0 pypi_0 pypi
alabaster 0.7.12 py37_0
asn1crypto 0.24.0 py37_0
astor 0.8.0 pypi_0 pypi
astroid 2.2.5 py37_0
atomicwrites 1.3.0 py37_1
attrs 19.1.0 py37_1
babel 2.7.0 py_0
backcall 0.1.0 py37_0
blas 1.0 mkl
bleach 3.1.0 py37_0
ca-certificates 2019.5.15 1
certifi 2019.6.16 py37_1
cffi 1.12.3 py37h7a1dbc1_0
chardet 3.0.4 py37_1003
cloudpickle 1.2.1 py_0
colorama 0.4.1 py37_0
cryptography 2.7 py37h7a1dbc1_0
cycler 0.10.0 py37_0
decorator 4.4.0 py37_1
defusedxml 0.6.0 py_0
docutils 0.15.2 py37_0
entrypoints 0.3 py37_0
fastcache 1.1.0 py37he774522_0
freetype 2.9.1 ha9979f8_1
gast 0.2.2 pypi_0 pypi
google-pasta 0.1.7 pypi_0 pypi
grpcio 1.23.0 pypi_0 pypi
h5py 2.10.0 pypi_0 pypi
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha66f8fd_1
idna 2.8 py37_0
imagesize 1.1.0 py37_0
importlib_metadata 0.19 py37_0
intel-openmp 2019.4 245
ipykernel 5.1.2 py37h39e3cac_0
ipython 7.8.0 py37h39e3cac_0
ipython_genutils 0.2.0 py37_0
isort 4.3.21 py37_0
jedi 0.15.1 py37_0
jinja2 2.10.1 py37_0
joblib 0.13.2 py37_0
jpeg 9b hb83a4c4_2
jsonschema 3.0.2 py37_0
jupyter_client 5.3.1 py_0
jupyter_core 4.5.0 py_0
keras 2.2.4 0
keras-applications 1.0.8 py_0
keras-base 2.2.4 py37_0
keras-preprocessing 1.1.0 py_1
keyring 18.0.0 py37_0
kiwisolver 1.1.0 py37ha925a31_0
lazy-object-proxy 1.4.2 py37he774522_0
libmklml 2019.0.5 0
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.8.0 h7bd577a_0
libsodium 1.0.16 h9d3ae62_0
libxgboost 0.90 0
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
markdown 3.1.1 py37_0
markupsafe 1.1.1 py37he774522_0
matplotlib 3.1.1 py37hc8f65d3_0
mccabe 0.6.1 py37_1
mistune 0.8.4 py37he774522_0
mkl 2019.4 245
mkl-service 2.0.2 py37he774522_0
mkl_fft 1.0.14 py37h14836fe_0
mkl_random 1.0.2 py37h343c172_0
more-itertools 7.2.0 py37_0
mpmath 1.1.0 py37_0
msys2-conda-epoch 20160418 1
nbconvert 5.5.0 py_0
nbformat 4.4.0 py37_0
numpy 1.17.2 pypi_0 pypi
numpy-base 1.16.4 py37hc3f5095_0
numpydoc 0.9.1 py_0
openssl 1.1.1c he774522_1
opt-einsum 3.0.1 pypi_0 pypi
packaging 19.1 py37_0
pandas 0.25.1 py37ha925a31_0
pandoc 2.2.3.2 0
pandocfilters 1.4.2 py37_1
parso 0.5.1 py_0
pickleshare 0.7.5 py37_0
pip 19.2.2 py37_0
pluggy 0.12.0 py_0
prompt_toolkit 2.0.9 py37_0
protobuf 3.9.1 pypi_0 pypi
psutil 5.6.3 py37he774522_0
py 1.8.0 py37_0
py-xgboost 0.90 py37_0
py-xgboost-cpu 0.90 py37_0
pycodestyle 2.5.0 py37_0
pycparser 2.19 py37_0
pyflakes 2.1.1 py37_0
pygments 2.4.2 py_0
pylint 2.3.1 py37_0
pyopenssl 19.0.0 py37_0
pyparsing 2.4.2 py_0
pyqt 5.9.2 py37h6538335_2
pyreadline 2.1 py37_1
pyrsistent 0.14.11 py37he774522_0
pysocks 1.7.0 py37_0
pytest 5.0.1 py37_0
python 3.7.4 h5263a28_0
python-dateutil 2.8.0 py37_0
pytz 2019.2 py_0
pywin32 223 py37hfa6e2cd_1
pyyaml 5.1.2 py37he774522_0
pyzmq 18.1.0 py37ha925a31_0
qt 5.9.7 vc14h73c81de_0
qtawesome 0.5.7 py37_1
qtconsole 4.5.4 py_0
qtpy 1.9.0 py_0
requests 2.22.0 py37_0
rope 0.14.0 py_0
scikit-learn 0.21.2 py37h6288b17_0
scipy 1.3.1 py37h29ff71c_0
setuptools 41.2.0 pypi_0 pypi
sip 4.19.8 py37h6538335_0
six 1.12.0 pypi_0 pypi
snowballstemmer 1.9.0 py_0
sphinx 2.1.2 py_0
sphinxcontrib-applehelp 1.0.1 py_0
sphinxcontrib-devhelp 1.0.1 py_0
sphinxcontrib-htmlhelp 1.0.2 py_0
sphinxcontrib-jsmath 1.0.1 py_0
sphinxcontrib-qthelp 1.0.2 py_0
sphinxcontrib-serializinghtml 1.1.3 py_0
spyder 3.3.6 py37_0
spyder-kernels 0.5.1 py37_0
sqlite 3.29.0 he774522_0
sympy 1.4 py37_0
tb-nightly 1.15.0a20190806 pypi_0 pypi
tensorboard 1.14.0 py37he3c9ec2_0
tensorflow 1.14.0 mkl_py37h7908ca0_0
tensorflow-base 1.14.0 mkl_py37ha978198_0
tensorflow-estimator 1.14.0 py_0
termcolor 1.1.0 pypi_0 pypi
testpath 0.4.2 py37_0
tornado 6.0.3 py37he774522_0
traitlets 4.3.2 py37_0
urllib3 1.24.2 py37_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_0
wcwidth 0.1.7 py37_0
webencodings 0.5.1 py37_1
werkzeug 0.15.6 pypi_0 pypi
wheel 0.33.6 pypi_0 pypi
win_inet_pton 1.1.0 py37_0
wincertstore 0.2 py37_0
wrapt 1.11.2 py37he774522_0
yaml 0.1.7 hc54c509_2
zeromq 4.3.1 h33f27b4_3
zipp 0.5.2 py_0
zlib 1.2.11 h62dcd97_3
Anyway, I don't understand why in the prompt HDF5 is version 1.10.4 and in the warning, version of HDF5 is 1.10.5
Maybe I am late, but I resolved this problem by upgrading hdf5 to 1.10.5.
On Windows 10, with anaconda you can do this:
conda install -c conda-forge hdf5=1.10.5
I'll leave this here, since it's a top stack thread for me without clear answer.
pip uninstall h5py
pip install h5py
If you are using conda to install tensorflow it installs h5py with 1.10.5 version and on top installs hdf5 1.10.4. Creating conflict that resolves after pip "juggling" since 1.10.4 satisfies the latest h5py.
I have the same problem with Windows 10. Here is what I did
Install some requirements for TensorFlow > 2.0
https://www.tensorflow.org/install/pip?lang
Create conda virtual environment:
conda create -n ai python==3.7.6
conda activate ai
conda install pandas matplotlib scikit-learn scrapy seaborn
conda install -c anaconda tensorflow
I had following same issue.
Warning! HDF5 library version mismatched error
Headers are 1.10.4, library is 1.10.6
My solution is making another conda environment and do every conda w/ 'conda-forge'.
Since hdf5 1.10.4 was installed with following command on my win10 PC w/ no GPU. Python is 3.7.10.
conda install tensorflow
By above command, 1.10.4 came with.
So, I should have done following.
conda install conda-forge tensorflow
Then, 1.10.6 was installed.
'conda-forge' w/ conda install is highly recommended.
I had the same problem as you. It came about because the tensorflow was installed by conda. And the error disappears when using channel anaconda.
conda install -c anaconda tensorflow
I actually solved this problem when I realized (on Mac OSX Mojave) that I had used Homebrew to install Octave, which was built to work with HDF5 1.10.5. I first ran in to this issue trying to install and run TensorFlow from iPython. I'm not actively using Octave, so I uninstalled Octave as well as HDF5 with
brew uninstall --force octave
brew uninstall hdf5
Then upon re-running
conda install h5py
and subsequently importing TensorFlow from iPython, everything seems to be working.
I have simple tensorflow code sum.py:
import tensorflow as tf
a = tf.Variable(1, name="a")
b = tf.Variable(2, name="b")
f = a + b
tf.print("The sum of a and b is", f)
===
I am window 10 user with Anaconda 3, tensorflow 2.0, jupyter, and pyhton 3.
I have similiar issues and i resovle the following iusses:
Error:
UserWarning: h5py is running against HDF5 1.10.5 when it was built against 1.10.4, this may cause problems
My environment was messad up with lots of pip install.
The following video resolve my problem.
https://www.youtube.com/watch?v=RgO8BBNGB8w&t=376s
It uses the tensorflow.yml with a list of clean environment:
https://github.com/jeffheaton/t81_558_deep_learning
Under the window prompt:
conda env create -v -f tensorflow
Then open anaconda prompt
conda acticate tensorflow
python sum.py
or in jupyter notebook. run with OK.
This happened to me when I installed tensorflow via
conda install -c conda-forge tensorflow
I resolved it as follows:
I uninstalled h5py and tensorflow by:
pip uninstall h5py
conda uninstall h5py
conda uninstall tensorflow
and reinstalled tensorflow by:
conda install -c anaconda tensorflow
Damn I had the same error shown in the anaconda prompt and the reason believe me is really silly.
I was multi-tasking and I forgot to activate the environment which resulted in two different versions of HDF5.
Please make sure to conda activate environment_name before launching the jupyter notebook.