We Keep Coding

Python not recognizing pyarrow version properly?

When trying to load some parquet files into pandas DataFrame, I get an error claiming that Pyarrow version should be at least 0.16.
If I check the version installed, I confirm that:
>>> import pyarrow
>>> pyarrow.__version__
'0.11.0'
However, when trying to install version 0.16, with
conda install -c conda-forge pyarrow=0.16
I see the following:
The following packages will be downloaded:
package | build
---------------------------|-----------------
arrow-cpp-0.16.0 | py37hf731a26_1 19.8 MB conda-forge
aws-sdk-cpp-1.7.164 | hba45d7a_2 2.1 MB conda-forge
boost-cpp-1.72.0 | h7b93d67_2 16.3 MB conda-forge
numpy-1.19.5 | py37haa41c4c_1 5.3 MB conda-forge
pyarrow-0.16.0 | py37h110162e_2 4.1 MB conda-forge
re2-2020.04.01 | he1b5a44_0 438 KB conda-forge
thrift-cpp-0.13.0 | h5aa387f_5 19.2 MB conda-forge
zstd-1.4.4 | h3b9ef0a_2 982 KB conda-forge
------------------------------------------------------------
Total: 68.1 MB
The following NEW packages will be INSTALLED:
boost-cpp conda-forge/linux-64::boost-cpp-1.72.0-h7b93d67_2
curl conda-forge/linux-64::curl-7.77.0-hea6ffbf_0
icu conda-forge/linux-64::icu-67.1-he1b5a44_0
pandas conda-forge/linux-64::pandas-1.2.5-py37h219a48f_0
python-dateutil conda-forge/noarch::python-dateutil-2.8.1-py_0
pytz conda-forge/noarch::pytz-2021.1-pyhd8ed1ab_0
thrift-cpp conda-forge/linux-64::thrift-cpp-0.13.0-h5aa387f_5
The following packages will be REMOVED:
orc-1.6.8-h58a87f1_0
The following packages will be DOWNGRADED:
arrow-cpp 4.0.1-py37hac2aefd_2_cpu --> 0.16.0-py37hf731a26_1
aws-sdk-cpp 1.8.186-hb4091e7_3 --> 1.7.164-hba45d7a_2
grpc-cpp 1.38.1-h36ce80c_0 --> 1.25.0-h213be95_2
libprotobuf 3.16.0-h780b84a_0 --> 3.11.0-h8b12597_0
lz4-c 1.9.3-h9c3ff4c_0 --> 1.8.3-he1b5a44_1001
numpy 1.21.0-py37h038b26d_0 --> 1.19.5-py37haa41c4c_1
pyarrow 4.0.1-py37he2832ee_2_cpu --> 0.16.0-py37h110162e_2
re2 2021.06.01-h9c3ff4c_0 --> 2020.04.01-he1b5a44_0
zstd 1.5.0-ha95c52a_0 --> 1.4.4-h3b9ef0a_2
Why pyarrow.__version__ returns 0.11 and the terminal ask for a DOWNGRADE from 4.0.1?
Your help is much appreciated.
EDIT to provide more information:
Doing: conda list pyarrow after activating my virtual environment returns:
# Name Version Build Channel
pyarrow 4.0.1 py37he2832ee_2_cpu conda-forge
However, if I start python, I check the library is not being load from my vritual environment but from outside:
>>> import pyarrow
>>> pyarrow.__file__
'/home/myuser/.local/lib/python3.7/site-packages/pyarrow/__init__.py'
And if I check my sys.path, what I see is the following:
>>> import sys
>>> sys.path
['', '/home/myuser/.conda/envs/myenv/lib/python37.zip', '/home/myuser/.conda/envs/myenv/lib/python3.7', '/home/myuser/.conda/envs/myenv/lib/python3.7/lib-dynload', '/home/myuser/.local/lib/python3.7/site-packages', '/home/myuser/.conda/envs/myenv/lib/python3.7/site-packages']
Is it relevant that '/home/myuser/.local/lib/python3.7/site-packages' appears before '/home/myuser/.conda/envs/myenv/lib/python3.7/site-packages'?

Conda update python wants to update other packages, not python

I have a conda environment running Python 3.7.5. I tried updating it to the latest version (3.8.0) but it wants to update other packages (even downgrade scipy), and says nothing about upgrading Python.
Why is this happening?
(workenv) $ conda update python
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/gabriel/miniconda3/envs/workenv
added / updated specs:
- python
The following packages will be downloaded:
package | build
---------------------------|-----------------
blas-1.0 | openblas 46 KB
certifi-2019.9.11 | py37_0 154 KB
libopenblas-0.3.6 | h5a2b251_2 7.7 MB
numpy-1.17.4 | py37hd5be1e1_0 4 KB
numpy-base-1.17.4 | py37h2f8d375_0 4.1 MB
scipy-1.3.1 | py37he2b7bc3_0 14.0 MB
------------------------------------------------------------
Total: 26.0 MB
The following NEW packages will be INSTALLED:
libopenblas pkgs/main/linux-64::libopenblas-0.3.6-h5a2b251_2
The following packages will be REMOVED:
intel-openmp-2019.4-243
mkl-2019.4-243
mkl-service-2.3.0-py37he904b0f_0
mkl_fft-1.0.15-py37ha843d7b_0
mkl_random-1.1.0-py37hd6b4f25_0
The following packages will be UPDATED:
blas 1.0-mkl --> 1.0-openblas
ca-certificates conda-forge::ca-certificates-2019.9.1~ --> pkgs/main::ca-certificates-2019.10.16-0
numpy 1.17.3-py37hd14ec0e_0 --> 1.17.4-py37hd5be1e1_0
numpy-base 1.17.3-py37hde5b4d6_0 --> 1.17.4-py37h2f8d375_0
openssl conda-forge::openssl-1.1.1d-h516909a_0 --> pkgs/main::openssl-1.1.1d-h7b6447c_3
The following packages will be SUPERSEDED by a higher-priority channel:
certifi conda-forge --> pkgs/main
The following packages will be DOWNGRADED:
scipy 1.3.1-py37h7c811a0_0 --> 1.3.1-py37he2b7bc3_0
Proceed ([y]/n)?

It's might check and install the dependencies of python before installing the python update itself. (You could check it by comparing the dependencies list of the different python versions.)
If the newest version of python is incompatible with the newest version of scipy, it will install an older version of scipy to make sure it will run stable.

Installing Python package will download, install, and remove other packages

I want to install cvxopt in Anaconda Python. Here's the version of my software:
$ python --version
Python 3.6.5 :: Anaconda, Inc.
$ conda --version
conda 4.5.11
$ anaconda --version
anaconda Command line client (version 1.6.14)
I try to install by typing:
conda install cvxopt
I was prompted that about 40 packages will be downloaded, about 10 NEW packages will be installed, the package anaconda will be removed, and about 25 packages will be updated?
Are the NEW packages dependencies?
Why will conda remove the anaconda package? Isn't conda a part of anaconda?
Here is the full output:
Solving environment: done
## Package Plan ##
environment location: /usr/local/anaconda3
added / updated specs:
- cvxopt
The following packages will be downloaded:
package | build
---------------------------|-----------------
suitesparse-5.2.0 | h171a5a3_0 2.4 MB
openssl-1.1.1a | h7b6447c_0 5.0 MB
freetype-2.9.1 | h8a8886c_1 822 KB
harfbuzz-1.8.8 | hffaf4a1_0 863 KB
libssh2-1.8.0 | h1ba5d50_4 233 KB
cairo-1.14.12 | h8948797_3 1.3 MB
libstdcxx-ng-8.2.0 | hdf63c60_1 2.9 MB
conda-4.6.1 | py36_0 1.7 MB
glpk-4.65 | h3ceedfd_2 1.1 MB
expat-2.2.6 | he6710b0_0 187 KB
krb5-1.16.1 | h173b8e3_7 1.4 MB
tbb-2018.0.5 | h6bb024c_0 1.4 MB
sqlite-3.26.0 | h7b6447c_0 1.9 MB
cvxopt-1.2.0 | py36h9e0dedd_0 537 KB
glib-2.56.2 | hd408876_0 5.0 MB
python-3.6.8 | h0371630_0 34.4 MB
certifi-2018.11.29 | py36_0 146 KB
gsl-2.4 | h14c3975_4 2.6 MB
cryptography-2.4.2 | py36h1ba5d50_0 618 KB
libuuid-1.0.3 | h1bed415_2 16 KB
curl-7.63.0 | hbc83047_1000 145 KB
libgcc-ng-8.2.0 | hdf63c60_1 7.6 MB
fribidi-1.0.5 | h7b6447c_0 112 KB
ca-certificates-2018.12.5 | 0 123 KB
pango-1.42.3 | h8589676_0 522 KB
libpng-1.6.36 | hbc83047_0 346 KB
pycurl-7.43.0.2 | py36h1ba5d50_0 185 KB
tk-8.6.8 | hbc83047_0 3.1 MB
qt-5.9.7 | h5867ecd_1 85.9 MB
mkl-2018.0.3 | 1 198.7 MB
libcurl-7.63.0 | h20c2e04_1000 550 KB
metis-5.1.0 | hf484d3e_4 4.1 MB
fontconfig-2.13.0 | h9420a91_0 291 KB
matplotlib-3.0.2 | py36h5429711_0 6.5 MB
pillow-5.4.1 | py36h34e0f95_0 627 KB
------------------------------------------------------------
Total: 373.2 MB
The following NEW packages will be INSTALLED:
cvxopt: 1.2.0-py36h9e0dedd_0
fribidi: 1.0.5-h7b6447c_0
glpk: 4.65-h3ceedfd_2
gsl: 2.4-h14c3975_4
krb5: 1.16.1-h173b8e3_7
libuuid: 1.0.3-h1bed415_2
metis: 5.1.0-hf484d3e_4
suitesparse: 5.2.0-h171a5a3_0
tbb: 2018.0.5-h6bb024c_0
The following packages will be REMOVED:
anaconda: 5.2.0-py36_3
The following packages will be UPDATED:
ca-certificates: 2018.03.07-0 --> 2018.12.5-0
cairo: 1.14.12-h7636065_2 --> 1.14.12-h8948797_3
certifi: 2018.4.16-py36_0 --> 2018.11.29-py36_0
conda: 4.5.11-py36_0 --> 4.6.1-py36_0
cryptography: 2.2.2-py36h14c3975_0 --> 2.4.2-py36h1ba5d50_0
curl: 7.60.0-h84994c4_0 --> 7.63.0-hbc83047_1000
expat: 2.2.5-he0dffb1_0 --> 2.2.6-he6710b0_0
fontconfig: 2.12.6-h49f89f6_0 --> 2.13.0-h9420a91_0
freetype: 2.8-hab7d2ae_1 --> 2.9.1-h8a8886c_1
glib: 2.56.1-h000015b_0 --> 2.56.2-hd408876_0
harfbuzz: 1.7.6-h5f0a787_1 --> 1.8.8-hffaf4a1_0
libcurl: 7.60.0-h1ad7b7a_0 --> 7.63.0-h20c2e04_1000
libgcc-ng: 7.2.0-hdf63c60_3 --> 8.2.0-hdf63c60_1
libpng: 1.6.34-hb9fc6fc_0 --> 1.6.36-hbc83047_0
libssh2: 1.8.0-h9cfc8f7_4 --> 1.8.0-h1ba5d50_4
libstdcxx-ng: 7.2.0-hdf63c60_3 --> 8.2.0-hdf63c60_1
matplotlib: 2.2.2-py36h0e671d2_1 --> 3.0.2-py36h5429711_0
mkl: 2018.0.2-1 --> 2018.0.3-1
openssl: 1.0.2o-h20670df_0 --> 1.1.1a-h7b6447c_0
pango: 1.41.0-hd475d92_0 --> 1.42.3-h8589676_0
pillow: 5.1.0-py36h3deb7b8_0 --> 5.4.1-py36h34e0f95_0
pycurl: 7.43.0.1-py36hb7f436b_0 --> 7.43.0.2-py36h1ba5d50_0
python: 3.6.5-hc3d631a_2 --> 3.6.8-h0371630_0
qt: 5.9.5-h7e424d6_0 --> 5.9.7-h5867ecd_1
sqlite: 3.23.1-he433501_0 --> 3.26.0-h7b6447c_0
tk: 8.6.7-hc745277_3 --> 8.6.8-hbc83047_0
Proceed ([y]/n)? n
CondaSystemExit: Exiting.

The anaconda package is called a "meta-package" because it does not contain any code itself, but instead specifies dependencies of other packages that should be installed. The package cvoptx is not one of the packages specified by the anaconda package, so if you try to install cvoptx, there is a conflict that conda resolves by removing the anaconda package (since you give cvoptx higher priority by specifying as required to be installed in the environment).
conda and anaconda are separate, and have separate versions. conda is the command line interface that allows you to install, update, and remove packages and environments. anaconda is described above.
Finally (and confusingly), the anaconda command on the command line is provided by the anaconda-client package, not the anaconda package. If you want to see the version of Anaconda that is installed, use conda list anaconda. If you want the version of the anaconda-client command line tool, use anaconda --version. The anaconda-client is used to upload packages and environments to https://anaconda.org

The best way is to install it in a separate environment. It is recommended to have different environments for each project. I tend to avoid mixing project requirements together. For the project using cvxopt, I would do:
conda create --name cvx python=3.6
So here I create an environment called cvx that has Python 3.6
This will download all needed packages for Python36. Since it is a clean environment, there would be no upgrading or downgrading.
After that you can activate your environment as:
conda activate cvx
conda install -c conda-forge cvxopt
In these environment you have cvxopt. You can the build your project and run it in these environment. If you need extra packages, you can do:
conda install -n cvxopt PACKAGESNAME
Here is everything you need to know to work with environments https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Can not install VTK with Anaconda

I can not install VTK by anaconda in Mac OS X. I use
conda install -c anaconda vtk=6.3.0
Python 2.7 but get the following error:
UnsatisfiableError: The following specifications were found to be in conflict: - python 3.6* - vtk 6.3.0* -> python 2.7* -> openssl 1.0.1* Use "conda info " to see the dependencies for each package.
But I've updated Python to 3.6, and the same error.

Conda Search
$ conda search vtk
Fetching package metadata ...........
vtk 5.10.1 py27_0 defaults
5.10.1 py27_1 defaults
6.3.0 py27_0 defaults
6.3.0 py27_1 defaults
Create the choicest env
$ conda create -n vtk python=2.7 vtk=6.3.0
Fetching package metadata ...........
Solving package specifications: .
Package plan for installation in environment /home/sidon/opt/anaconda3/envs/vtk:
The following NEW packages will be INSTALLED:
openssl: 1.0.2k-1
pip: 9.0.1-py27_1
python: 2.7.13-0
readline: 6.2-2
setuptools: 27.2.0-py27_0
sqlite: 3.13.0-0
tk: 8.5.18-0
vtk: 6.3.0-py27_1
wheel: 0.29.0-py27_0
zlib: 1.2.8-3
Proceed ([y]/n)? y
vtk-6.3.0-py27 100% |#############################################################################################| Time: 0:00:15 1.97 MB/s
#
# To activate this environment, use:
# > source activate vtk
#
# To deactivate this environment, use:
# > source deactivate vtk
#
Active the new env
$ source activate vtk
Test
$ python -c "import vtk; print vtk.vtkSphereSource()"
vtkSphereSource (0xcfdcc0)
Debug: Off
Modified Time: 39
Reference Count: 2
Registered Events: (none)
Executive: 0xae5e00
ErrorCode: Success
Information: 0xf6e210
AbortExecute: Off
Progress: 0
Progress Text: (None)
Theta Resolution: 8
Phi Resolution: 8
Theta Start: 0
Phi Start: 0
Theta End: 360
Phi End: 180
Radius: 0.5
Center: (0, 0, 0)
LatLong Tessellation: 0
Output Points Precision: 0

I would suggest installing directly from the wheel file. You can get the VTK wheel file from here (wheel file directory site). This site has pretty much every wheel file so it is your friend! just search for it and you will see a bunch of .whl VTK files.
just download the wheel file that matches your anaconda python root version (either 3.5, 3.6, 2.7 etc.) and either 32 or 64 bit (depending on your python version).
To install from a wheel file, open a terminal in the directory the wheel is in and type
pip install "your_wheel_file_name"
If you want to install it for a conda environment, just activate the environment first (activate "env_name") and then install from the wheel file.

Try this out for me:
conda create --name vtk_test python=2.7 vtk
This will create a new environment with VTK, Python and their dependencies inside.

ImportError when using Python Anaconda package grib_api

I am trying to use the ECMWF GRIB_API to access GRIB files from Python. I get this error
Traceback (most recent call last):
File "/home/martin/markj/JustImportGRIB.py", line 1, in <module>
from gribapi import *
ImportError: No module named gribapi
One line of code :)
from gribapi import *
I am using Anaconda Python 2.7.12, Linux Centos 64-bit, installed as user and not added Anaconda to the path if that's relevant - the system Python (2.6) is on the path. I installed GRIB_API from the Pingu Carsti channel which was the top Google hit. The conda install did not report any errors.
[martin#bonnie ~]$ anaconda2/bin/conda install -c pingucarsti grib_api=1.10.4
Fetching package metadata .........
Solving package specifications: ..........
Package plan for installation in environment /home/martin/anaconda2:
The following packages will be downloaded:
package | build
---------------------------|-----------------
grib_api-1.10.4 | 3 1.7 MB pingucarsti
The following NEW packages will be INSTALLED:
grib_api: 1.10.4-3 pingucarsti
Proceed ([y]/n)? y
Fetching packages ...
grib_api-1.10. 100% |################################| Time: 0:00:01 1.24 MB/s
Extracting packages ...
[ COMPLETE ]|###################################################| 100%
Linking packages ...
[ COMPLETE ]|###################################################| 100%
I found some bug reports about this error being caused by a bug in the ECMWF GRIB_API, but I don't know enough about Anaconda (or perhaps Python) to figure out how to apply the workarounds.

I read GRIB files in Python with pygrib. I have successfully installed it inside a conda virtual environment (Miniconda2 on Ubuntu 16.04). Here is what has worked for me:
Install system dependencies for pygrib:
sudo apt-get install libjpeg9
sudo apt-get install libgrib-api-dev
Install pygrib and ecmwf_grib from conda-forge channel.
conda install -c conda-forge pygrib ecmwf_grib

Try conda-forge's build at python-eccodes (note: that's not eccodes which is the C library).
How it is built: https://github.com/conda-forge/python-eccodes-feedstock/blob/master/recipe/meta.yaml
The ECMWF's ecCodes library is the next version of their GRIB API, and is extremely similar to the old GRIB API.
pelson> conda create -n gribby -c conda-forge python-eccodes
Fetching package metadata .............
Solving package specifications: .
Package plan for installation in environment /Users/pelson/miniconda/envs/gribby:
The following NEW packages will be INSTALLED:
ca-certificates: 2017.7.27.1-0 conda-forge
curl: 7.54.1-0 conda-forge
eccodes: 2.4.0-0 conda-forge
hdf4: 4.2.12-0 conda-forge
hdf5: 1.8.18-1 conda-forge
intel-openmp: 2018.0.0-h68bdfb3_7 defaults
jasper: 1.900.1-4 conda-forge
jpeg: 9b-1 conda-forge
krb5: 1.14.2-0 conda-forge
libgfortran: 3.0.1-h93005f0_2 defaults
libnetcdf: 4.4.1.1-8 conda-forge
libpng: 1.6.28-1 conda-forge
libssh2: 1.8.0-1 conda-forge
mkl: 2018.0.0-h5ef208c_6 defaults
ncurses: 5.9-10 conda-forge
numpy: 1.13.3-py27h62f9060_0 defaults
openssl: 1.0.2l-0 conda-forge
python: 2.7.14-0 conda-forge
python-eccodes: 2.4.0-py27_1 conda-forge
readline: 6.2-0 conda-forge
sqlite: 3.13.0-1 conda-forge
tk: 8.5.19-2 conda-forge
zlib: 1.2.8-3 conda-forge
#
# To activate this environment, use:
# > source activate gribby
#
# To deactivate an active environment, use:
# > source deactivate
#
pelson> source activate gribby
python(gribby) pelson> python
Python 2.7.14 | packaged by conda-forge | (default, Oct 5 2017, 23:08:53)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import gribapi
>>> gribapi.__version__
'2.4.0'