Which package index did pip just install from? - python

I am trying to install numpy and some other python packages in their intel optimized versions. They are not distributed as wheels via PyPI, but only via anaconda cloud. But that index doesn't contain all the dependencies, so PyPI must also be used. The command to run is
python -m pip install -i https://pypi.anaconda.org/intel/simple --extra-index-url https://pypi.org/simple numpy
Unfortunately, there seems to be a version race, and if PyPI has a newer version of NumPy, I won't get the intel-version of NumPy. To check if I got the right NumPy version, I run python -c "import numpy; numpy.show_config()" and verify it has MKL support. For SciPy, I don't know of a corresponding command. I have tried this a few times now, and it is using a cached scipy version, so I don't get any information from the pip output specifying which repository it is using.
How can I see after the installation has been completed which repository was used?

Related

I can't install the "numba" package via "pip"

enter image description here
I can't install the "numba" package via "pip". I want to install the "numba" package, but an error appears every time
please check your python and numpy version . Numba is compatible with Python 3.6 or later, and Numpy versions 1.15 or later. Please check the documentation
https://numba.pydata.org/numba-doc/latest/user/installing.html
Installing using pip on x86/x86_64 Platforms
Binary wheels for Windows, Mac, and Linux are also available from PyPI. You can install Numba using pip:
$ pip install numba

NumPy 1.21.2 may not yet support Python 3.10

Python 3.10 is released and when I try to install NumPy it gives me this: NumPy 1.21.2 may not yet support Python 3.10.. what should I do?
If on Windows, numpy has not yet released a precompiled wheel for Python 3.10. However you can try the unofficial wheels available at https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy .
Specifically look for
numpy‑1.21.2+mkl‑cp310‑cp310‑win_amd64.whl or
numpy‑1.21.2+mkl‑cp310‑cp310‑win32.whl
depending on you system architecture.
After downloading the file go to the download directory and run pip install "<filename>.whl".)
(I have personally installed numpy‑1.21.2+mkl‑cp310‑cp310‑win_amd64.whl and it worked for me.)
Since you are on MS-Windows you can also make use of pipwin - this windows only utility is pip installable and can download and install a number of "unofficial" builds (provided by the excellent Christoph Gohlke) of scientific from the https://www.lfd.uci.edu/~gohlke/pythonlibs/ but takes the guesswork out of which file(s) to download and install.
A session might run:
pip install pipwin
pipwin install numpy
Alternatively you could use the py launcher as in:
py -3.10 -mpip install pipwin
py -3.10 -mpipwin refresh
py -3.10 -mpipwin install numpy
The middle step tells pipwin to populate its list of what is currently available.
If you don't mind using Docker and Debian, the official python:3.10 (==python:3.10-bullseye) docker container has pip pre-installed. And some packages like numpy can be installed using pip install and run under python 3.10 (good luck with other packages though:).
Here's some PoC and proof that numpy really works there:
$ docker run -it --rm --name python310 -u 0 python:3.10 bash -c 'pip --version; pip install numpy --user --no-cache; pip show numpy; python -c "import numpy as np; print(np.ones(5))"'
..which should output:
pip 21.2.4 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)
Collecting numpy
Downloading numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
|████████████████████████████████| 15.9 MB 36.9 MB/s
Installing collected packages: numpy
WARNING: The scripts f2py, f2py3 and f2py3.10 are installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.21.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Name: numpy
Version: 1.21.4
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email:
License: BSD
Location: /root/.local/lib/python3.10/site-packages
Requires:
Required-by:
[1. 1. 1. 1. 1.]
I use Ubuntu and I had the same issue but when changed the version from 1.19.5 to 1.22.1 my issue fixed

Can all/most python pip packages be installed using pypy3?

I am able to install the python3 package of forexconnect using:
python3 -m pip install forexconnect
but when I try to install it for pypy using:
pypy3 -m pip install forexconnect
I get the following error,
ERROR: Could not find a version that satisfies the requirement forexconnect (from versions: none)
ERROR: No matching distribution found for forexconnect
I have looked through the pypy documentation and can't work out what I need to do now. Perhaps I need to re-compile/rebuild the library but unfortunately my knowledge of python isn't good enough to understand what is probably quite straight-forward.
Please, does anyone know what is going on here and what needs to be done?
forexeconnect does not release a source tarball, so they must build a binary version for each python version they wish to support and upload that to PyPI. Most packages will release a source tarball and upload that to PyPI, then pip install can build the binary package from source. In this case, the package provider (assuming they do not want to release the source) would have to build a version for PyPy, there is nothing pip nor PyPy can do.
To see supported versions by pip try
pip install forexconnect==0
Then you can set specifical version with
pip install forexconnect==versionnumber

Different ways to install numpy, scipy, and matplotlib on macOS via homebrew

Today I decided to install python and the scipy stack manually, instead of using Anaconda (or Canopy) as I had previously done. I use homebrew on my mac and have python2 and python3 (2.7 and 3.6) installed via homebrew. But reading through the documentation, there are multiple ways to install the scipy stack and I want to know what the differences are. I have tested they independently and they all work.
From the Homebrew documentation:
python2 -m pip install numpy scipy matplotlib
python3 -m pip install numpy scipy matplotlib
These are the same two commands that the Matplotlib installation docuentation lists for how to install matplotlib through homebrew. Why does this use pip (the system Python 2.7.x's pip) instead of pip2 and pip3 respectively? Is it because you call python2/python3 first?
However, the SciPy documentation for installing these modules when using homebrew is different:
brew tap homebrew/science && brew install numpy scipy matplotlib
(NOTE: the matplotlib formula is located in the homebrew/science repository, which is why you need to use brew tap.)
Finally, from the command line readout when installing python2 and python3 via homebrew:
pip2 install numpy scipy matplotlib
pip3 install numpy scipy matplotlib
which are based on the following readouts:
Pip and setuptools have been installed. To update them
pip2 install --upgrade pip setuptools
You can install Python packages with
pip2 install <package>
They will install into the site-package directory
/usr/local/lib/python2.7/site-packages
See: https://docs.brew.sh/Homebrew-and-Python.html
...
Pip, setuptools, and wheel have been installed. To update them
pip3 install --upgrade pip setuptools wheel
You can install Python packages with
pip3 install <package>
They will install into the site-package directory
/usr/local/lib/python3.6/site-packages
See: https://docs.brew.sh/Homebrew-and-Python.html
So between four sources of documentation, there are three different ways to install scipy when using homebrew and they all work; but how is each different and should one be preferred?
From what I can tell, the first and third methods, which both invoke pip (pip2/pip3), are functionally equivalent - both invoke Homebrew's Python X.X.X's pip - but one implicitly, the other explicitly. I assume this means both methods install the pre-built binary packages from pip in the form of wheels. For the second method, I think it installs homebrew's own formulae for these packages (i.e. maintained separately by homebrew in it's repository).
If this is true, then I assume one should use the second method if you are using a version of python which is maintained by homebrew (i.e. installed via brew install python or python3). My reasoning is that if you later decide to install another formula via homebrew that has any of the scipy stack as a dependency, it will install those modules again from homebrew's repository if you installed them using pip previously.
As mentioned, I am not sure if my understanding is correct and I have not been able to find any answers, so any insights or confirmations would be appreciated.
Your analysis seems correct: variants 1 and 3 will install numpy/scipy from the python package index (PyPI) and will use pre-built wheels (if available for your platform, which they most likely are).
Variant 2 installs the brew formula.
As mentioned by #Evhz, the conda packages for numpy and scipy use the Intel Math Kernel library, which can provide significant speedups (not just on Intel processors) versus the packages installed from PyPI or brew, both of which are linked against OpenBLAS.
Concerning which method to prefer: it's not entirely straightforward.
Yes, on the surface, using brew to manage both the python interpreter and the python packages would seem consistent.
However, homebrew only provides formulae for a handful of python packages, so you'll end up needing to mix with pip in any case.
If you want performance, you go with conda, which will be managing both the interpreter and python packages.
However, also anaconda / conda-forge still have some catching up to do with PyPI, so you'll likely need to mix with pip again.
In the end, there is no perfect solution but as long as you knowingly decide for one, you're unlikely to run into issues.

Install Scipy with MKL through PIP

I am using PIP to install Scipy with MKL to accelerate the performance. My OS is Ubuntu 64 bit. Using the solution from this question, I create a file .numpy-site.cfg
[mkl]
library_dirs=/opt/intel/composer_xe_2013_sp1/mkl/lib/intel64/
include_dirs=/opt/intel/mkl/include/
mkl_libs=mkl_intel_lp64,mkl_intel_thread,mkl_core,mkl_rt
lapack_libs=
This file helps me to install Numpy with MKL successfully. However, using the same above file, installing Scipy prompts the error
ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory
I also use
export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1/mkl/lib/intel64
but the problem is still the same.
Anyone know how to fix this problem? I don't want to install Scipy manually so anyone give me some hints to fix it.
Intel has been publishing wheels of packages like Numpy, Scipy and Scikit-learn to PyPI. These wheels have been built while linking against Intel MKL, and include various optimizations.
If you want Scipy built with Intel MKL:
#Remove existing Numpy and/or Scipy:
pip uninstall numpy scipy -y
#Install scipy built with Intel MKL:
pip install intel-scipy
More information available here
2 years have passed since this question was asked.
There are now numpy/scipy wheels for linux that use a openblas compiled for avx2, so you can get much better performance without building packages. You may need to upgrade pip to get it to install the wheel:
pip install --upgrade pip
pip install numpy scipy
If you want MKL, then you can install Anaconda or Intel Distribution for Python. They use conda instead of pip to manage packages, but they are free and distribute packages that contain all the dependences, including MKL.
I have Win10 64Bit with Python 3.6.2 i have installed scipy through http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy
I followed following steps :
Step 1: Uninstall if you have any previous version of numpy,
pip uninstall numpy
Step 2 : Download numpy‑1.13.1+mkl‑cp36‑cp36m‑win_amd64.whl with
MKL(Math Kernel Library) from below link,
http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy
Step 3: copy downloaded file into another location and launch command
prompt from that location.
Step 4: run this command,
pip install -U numpy-1.13.1+mkl-cp36-cp36m-win_amd64.whl
Step 5: Now Download scipy library from,
http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy
Step 6: Copy downloaded file into same location in which numpy is
copied.
Step 7: In CMD prompt run this cmd,
pip install scipy-0.19.1-cp36-cp36m-win_amd64.whl
Done!
If you are having trouble installing or running with specific version then first uninstall and then install
Step 1:
pip uninstall -v numpy
Step 2: download the wheel file and install
pip install -U numpy-1.13.0+mkl-cp36-cp36m-win_amd64.whl
In this example wheel file name is "numpy-1.13.0+mkl-cp36-cp36m-win_amd64.whl
"
I have been facing this problem the past few weeks on:
Windows 10 64 bit Python 3.5.2
My workaround:
First: pip install wheel
Next: Download Numpy and Scipy form Gholke's repo
Numpy and SciPy
Then:
pip install numpy_package.whl
pip install scipy_package.whl
Since the actual question itself was not answered, let me give it a shot...
I think the problem here basically is that the BLAS/LAPACK libraries being used are spread out across multiple location, and numpy doesn't handle this well.
We have fixed this in EasyBuild, where we have been building numpy/scipy on top of Intel MKL for a while now, with this patch: https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/n/numpy/numpy-1.8.1-mkl.patch
For me, MKL came with Scipy library by conda install scipy
#rscohn2 solution gave me a hint for using packages.
For the reference, my environment is included Ubuntu, Anaconda, Python 3.6, Scipy 1.1 and MKL 2018.0.

Categories