I am using PIP to install Scipy with MKL to accelerate the performance. My OS is Ubuntu 64 bit. Using the solution from this question, I create a file .numpy-site.cfg
[mkl]
library_dirs=/opt/intel/composer_xe_2013_sp1/mkl/lib/intel64/
include_dirs=/opt/intel/mkl/include/
mkl_libs=mkl_intel_lp64,mkl_intel_thread,mkl_core,mkl_rt
lapack_libs=
This file helps me to install Numpy with MKL successfully. However, using the same above file, installing Scipy prompts the error
ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory
I also use
export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1/mkl/lib/intel64
but the problem is still the same.
Anyone know how to fix this problem? I don't want to install Scipy manually so anyone give me some hints to fix it.
Intel has been publishing wheels of packages like Numpy, Scipy and Scikit-learn to PyPI. These wheels have been built while linking against Intel MKL, and include various optimizations.
If you want Scipy built with Intel MKL:
#Remove existing Numpy and/or Scipy:
pip uninstall numpy scipy -y
#Install scipy built with Intel MKL:
pip install intel-scipy
More information available here
2 years have passed since this question was asked.
There are now numpy/scipy wheels for linux that use a openblas compiled for avx2, so you can get much better performance without building packages. You may need to upgrade pip to get it to install the wheel:
pip install --upgrade pip
pip install numpy scipy
If you want MKL, then you can install Anaconda or Intel Distribution for Python. They use conda instead of pip to manage packages, but they are free and distribute packages that contain all the dependences, including MKL.
I have Win10 64Bit with Python 3.6.2 i have installed scipy through http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy
I followed following steps :
Step 1: Uninstall if you have any previous version of numpy,
pip uninstall numpy
Step 2 : Download numpy‑1.13.1+mkl‑cp36‑cp36m‑win_amd64.whl with
MKL(Math Kernel Library) from below link,
http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy
Step 3: copy downloaded file into another location and launch command
prompt from that location.
Step 4: run this command,
pip install -U numpy-1.13.1+mkl-cp36-cp36m-win_amd64.whl
Step 5: Now Download scipy library from,
http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy
Step 6: Copy downloaded file into same location in which numpy is
copied.
Step 7: In CMD prompt run this cmd,
pip install scipy-0.19.1-cp36-cp36m-win_amd64.whl
Done!
If you are having trouble installing or running with specific version then first uninstall and then install
Step 1:
pip uninstall -v numpy
Step 2: download the wheel file and install
pip install -U numpy-1.13.0+mkl-cp36-cp36m-win_amd64.whl
In this example wheel file name is "numpy-1.13.0+mkl-cp36-cp36m-win_amd64.whl
"
I have been facing this problem the past few weeks on:
Windows 10 64 bit Python 3.5.2
My workaround:
First: pip install wheel
Next: Download Numpy and Scipy form Gholke's repo
Numpy and SciPy
Then:
pip install numpy_package.whl
pip install scipy_package.whl
Since the actual question itself was not answered, let me give it a shot...
I think the problem here basically is that the BLAS/LAPACK libraries being used are spread out across multiple location, and numpy doesn't handle this well.
We have fixed this in EasyBuild, where we have been building numpy/scipy on top of Intel MKL for a while now, with this patch: https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/n/numpy/numpy-1.8.1-mkl.patch
For me, MKL came with Scipy library by conda install scipy
#rscohn2 solution gave me a hint for using packages.
For the reference, my environment is included Ubuntu, Anaconda, Python 3.6, Scipy 1.1 and MKL 2018.0.
Related
I am trying to install numpy and some other python packages in their intel optimized versions. They are not distributed as wheels via PyPI, but only via anaconda cloud. But that index doesn't contain all the dependencies, so PyPI must also be used. The command to run is
python -m pip install -i https://pypi.anaconda.org/intel/simple --extra-index-url https://pypi.org/simple numpy
Unfortunately, there seems to be a version race, and if PyPI has a newer version of NumPy, I won't get the intel-version of NumPy. To check if I got the right NumPy version, I run python -c "import numpy; numpy.show_config()" and verify it has MKL support. For SciPy, I don't know of a corresponding command. I have tried this a few times now, and it is using a cached scipy version, so I don't get any information from the pip output specifying which repository it is using.
How can I see after the installation has been completed which repository was used?
When I try to install scipy for 3.4 it throws this error when I run this command:
sudo pip3.4 install scipy
https://www.pastiebin.com/5a7c65d7e44fe
I was able to get it to install with Python 3.6 but not 3.4.. which I need.
Installing Scipy or Numpy from scratch is usually hard work. At best try a distribution that already has it installed (WinPython, Anaconda). If you are on Linux install the packages that come with your distro. On Windows you can also grab the wheels at Gohlke.
Today I decided to install python and the scipy stack manually, instead of using Anaconda (or Canopy) as I had previously done. I use homebrew on my mac and have python2 and python3 (2.7 and 3.6) installed via homebrew. But reading through the documentation, there are multiple ways to install the scipy stack and I want to know what the differences are. I have tested they independently and they all work.
From the Homebrew documentation:
python2 -m pip install numpy scipy matplotlib
python3 -m pip install numpy scipy matplotlib
These are the same two commands that the Matplotlib installation docuentation lists for how to install matplotlib through homebrew. Why does this use pip (the system Python 2.7.x's pip) instead of pip2 and pip3 respectively? Is it because you call python2/python3 first?
However, the SciPy documentation for installing these modules when using homebrew is different:
brew tap homebrew/science && brew install numpy scipy matplotlib
(NOTE: the matplotlib formula is located in the homebrew/science repository, which is why you need to use brew tap.)
Finally, from the command line readout when installing python2 and python3 via homebrew:
pip2 install numpy scipy matplotlib
pip3 install numpy scipy matplotlib
which are based on the following readouts:
Pip and setuptools have been installed. To update them
pip2 install --upgrade pip setuptools
You can install Python packages with
pip2 install <package>
They will install into the site-package directory
/usr/local/lib/python2.7/site-packages
See: https://docs.brew.sh/Homebrew-and-Python.html
...
Pip, setuptools, and wheel have been installed. To update them
pip3 install --upgrade pip setuptools wheel
You can install Python packages with
pip3 install <package>
They will install into the site-package directory
/usr/local/lib/python3.6/site-packages
See: https://docs.brew.sh/Homebrew-and-Python.html
So between four sources of documentation, there are three different ways to install scipy when using homebrew and they all work; but how is each different and should one be preferred?
From what I can tell, the first and third methods, which both invoke pip (pip2/pip3), are functionally equivalent - both invoke Homebrew's Python X.X.X's pip - but one implicitly, the other explicitly. I assume this means both methods install the pre-built binary packages from pip in the form of wheels. For the second method, I think it installs homebrew's own formulae for these packages (i.e. maintained separately by homebrew in it's repository).
If this is true, then I assume one should use the second method if you are using a version of python which is maintained by homebrew (i.e. installed via brew install python or python3). My reasoning is that if you later decide to install another formula via homebrew that has any of the scipy stack as a dependency, it will install those modules again from homebrew's repository if you installed them using pip previously.
As mentioned, I am not sure if my understanding is correct and I have not been able to find any answers, so any insights or confirmations would be appreciated.
Your analysis seems correct: variants 1 and 3 will install numpy/scipy from the python package index (PyPI) and will use pre-built wheels (if available for your platform, which they most likely are).
Variant 2 installs the brew formula.
As mentioned by #Evhz, the conda packages for numpy and scipy use the Intel Math Kernel library, which can provide significant speedups (not just on Intel processors) versus the packages installed from PyPI or brew, both of which are linked against OpenBLAS.
Concerning which method to prefer: it's not entirely straightforward.
Yes, on the surface, using brew to manage both the python interpreter and the python packages would seem consistent.
However, homebrew only provides formulae for a handful of python packages, so you'll end up needing to mix with pip in any case.
If you want performance, you go with conda, which will be managing both the interpreter and python packages.
However, also anaconda / conda-forge still have some catching up to do with PyPI, so you'll likely need to mix with pip again.
In the end, there is no perfect solution but as long as you knowingly decide for one, you're unlikely to run into issues.
I have problems with installing SciPy on windows. I've already tried pip install (which worked for NumPy) but it didn't work out. Any idea how to solve this problem?
I recommend installing Anaconda which is a binary distribution system, which means that you don't need to compile yourself. Overall I find this to be a far easier way to handle packages on windows.
Using Anaconda, you can install scipy via:
conda install scipy
A good way with those kind of packages (scipy, numpy, ...) on Windows is to download (unoficial) binaries from this website and install it from here.
Once download, you can install it using pip :
pip install scipy-package.whl
I want to install NumPy using the pip install numpy command, but I get the following error:
RuntimeError: Broken toolchain: cannot link a simple C program
I'm using Windows 7 32 bit, Python 2.7.9, pip 6.1.1 and some MSVC compiler. I think it uses the compiler from Visual C++ 2010 Express, but actually I'm not sure which one, because I have several Visual Studio installations.
I know that there are prebuilt packages for Windows, but is there some way to do it just by typing pip install numpy?
I think that there could be other packages which must be compiled before usage, so it's not only about NumPy. I want to solve the problem with my compiler, so I could easily install any other similar package without necessity to search for prebuilt packages (and hope that there are some at all).
Check the installation of Python 2.7, and then install/reinstall pip which is described here. Then a open command line windows and write:
pip install numpy
Or
pip install scipy
If already installed, try this:
pip install -U numpy
Installing extension modules can be an issue with pip. This is why Conda exists. Conda is an open-source BSD-licensed cross-platform package manager. It can easily install NumPy.
Two options:
Install Anaconda here
Install Miniconda here and then go to a command line and type conda install numpy (make sure your PATH includes the location Conda was installed to).
Frustratingly, the NumPy package published to PyPI won't install on most Windows computers: Windows wheel package (.whl) on Pypi #5479
Instead:
Download the NumPy wheel for your Python version from Archived: Unofficial Windows Binaries for Python Extension Packages, NumPy
Install it from the command line:
pip install numpy-1.10.2+mkl-cp35-none-win_amd64.whl
As of March 2016, pip install numpy works on Windows without a Fortran compiler. See here.
pip install scipy still tries to use a compiler.
July 2018: mojoken reports pip install scipy working on Windows without a Fortran compiler.
py -m pip install numpy
Worked for me!
Install miniconda (here)
After installed, open Anaconda Prompt (search this in Start Menu)
Write:
pip install numpy
After installed, test:
import numpy as np
First go through page Download Python to download Python 3.6.1 or 2.7.13 either of your choice. I preferred to use Python 2.7 or 3.4.4.
Now after installation, go to the folder name python27 or python34, and click on the script. Now here open the command prompt by left clicking and Run as administrator.
After the command prompt appears, write "pip install numpy" there. This will install the latest version of NumPy and installing it will show a success comment. That's all.
Similarly, Matplotlib can be installed by just typing "pip install matplotlip". And now if you want to download SciPy, then just write "pip install scipy" and if it doesn't work then you need to download Python SciPy from SciPy: Scientific Library for Python and install it.
I had the same problem.
I decided in a very unexpected way. I just opened the command line as an administrator. And then typed:
pip install numpy