I have looked for an easy way to install/compile Numpy with OpenBLAS but didn't find an easy answer. All the documentation I have seen takes too much knowledge as granted for someone like me who is not used to compile software.
There are two packages in Ubuntu related to OpenBLAS : libopenblas-base and libopenblas-dev.
Once they are installed, what should I do to install Numpy again with them?
Thanks!
Note that when these OpenBLAS packages are installed, Numpy doesn't work anymore: it can't be imported: ImportError: /usr/lib/liblapack.so.3gf: undefined symbol: ATL_chemv.
The problem occurs as well when installing Theano with their website instructions for Ubuntu.
This was noticed here already.
Run sudo update-alternatives --all and set liblapack.so.3gf to /usr/lib/lapack/liblapack.so.3gf
To add to the accepted answer (of using update-alternatives), the reason for this is because OpenBlas is not compatible with the Atlas version of Lapack. For each of the Blas and Lapack versions:
Default Blas + Default Lapack => OK
OpenBlas + Default Lapack => OK
Atlas-Blas + Default Lapack => OK
Atlas-Blas + Atlas-Lapack => OK
OpenBlas + Atlas-Lapack => ERROR! (The following case here.)
This is from both personal experience (with the exact same issue) and realizing why such a combination wasn't mentioned in this comparison blog.
By the way, you can just find the necessary files in /etc/alternatives/, usually with a filename starting with lib*. For each one do sudo update-alternatives --config <filename>. For example, do to following:
sudo update-alternatives --config libblas.so
sudo update-alternatives --config libblas.so.3
to change the Blas version.
Consider using EasyBuild (http://hpcugent.github.io/easybuild/), an open-source framework for building and installing software.
It allows you to (very easily) build and install (scientific) software with various compiler, and using different BLAS libraries (ATLAS, OpenBLAS, ACML, Intel MKL, ...).
Once you install EasyBuild (pro tip: use the bootstrapping procedure described at https://github.com/hpcugent/easybuild/wiki/Bootstrapping-EasyBuild), it boils down to running a single command, something like:
eb numpy-1.6.2-goolf-1.4.10-Python-2.7.3.eb -ldr
This will first build and install of full compiler toolchain (goolf: GCC+OpenBLAS+OpenMPI+LAPACK+FFTW), and subsequently build Python and numpy with that toolchain. And all that while you're getting lunch. ;-)
Disclaimer: I'm one of the EasyBuild developers.
Related
I have a numpy version installed on Mac via pip install numpy, which by default is shipped (and is linked against) the OpenBLAS implementation of BLAS and LAPACK.
Is it possible to make numpy use MKL (which I manually installed) without using conda or compiling numpy from source? I know that there's a hack to make it work on Linux, but is this possible on Mac as well? if yes, is it a good idea (e.g. copying and renaming libmkl_core.dylib to libopenblas.0.dylib?)
I'm struggling installing GDAL on ubuntu 16.04 to work with GeoDjango (Django 2.1, python3), so I need to understand what I'm actually installing.
What is the rôle of each library/package/module ?
apt
gdal-bin (A 'C' library containing the actual functions ?)
python-gdal (The same in python, or just some kind of bridge ?)
python3-gdal (see above, but for python3. Does it need python-gdal ?)
pip
gdal
pygdal
What is the link between pip modules and apt packages here ?
Every piece of info is available, if one is willing to search for it.
DEBs (installed system-wide):
gdal-bin ([Ubtu]: Package: gdal-bin) - a collection of gdal related binaries (tools and utilities)
python3-gdal ([Ubtu]: Package: python3-gdal) - Python 3 bindings, extensions (.sos) and some wrapper scripts, which enable gdal usage from Python
python-gdal - the same thing, but for Python 2 (totally unrelated to previous item)
WHLs (installed as Python modules to the interpreter used to launch pip):
GDAL ([PyPI]: GDAL) - the sources (.tar.gz) for #2. (and / or #3.). During pip install phase, they are built and installed for current Python
pygdal ([PyPI]: pygdal) - same thing (but for VEnv?) as previous item. It seems to be a lighter version (it doesn't contain the scripts)
But, all of the above depend on libgdal ([Ubtu]: Package: libgdal1i), which is the gdal library.
Currently, I am trying to build Ipopt linking against openblas. I downloaded the openblas source and did make in the parent directory.
The configure script of Ipopt has several options to link against blas:
I tried ./configure --with-blas="-L/home/moritz/build/CoinIpopt_test/ThirdParty/OpenBLAS-0.2.14/libopenblas.so"
but I do get the error
checking whether user supplied BLASLIB="-L/home/moritz/build/CoinIpopt_test/ThirdParty/OpenBLAS-0.2.14/libopenblas.so" works... no
configure: error: user supplied BLAS library "-L/home/moritz/build/CoinIpopt_test/ThirdParty/OpenBLAS-0.2.14/libopenblas.so" does not work
Any tips how to achieve what I want ? Finally, I would like to make a conda package. I do have installed openblas with anaconda. But I do get the same error message if I link against the installed libopenblas.so
Managed to get it work. I had to install openblas to a directory of my choice by
make install PREFIX=/home/....../
aferwards I compiled Ipopt using
./configure --with-blas-incdir="-I/home/.../openblas/include/" --with-blas-lib="-L/home/.../openblas/lib/"
I tried to update the scipy library, but it failed. Then, I tried to uninstall and reinstall it. Removing the old version was simple, but the pip tool would not let me install the new version. The error message said that the blas and lapack packages were missing. I guess that this is required, but I was able to use scipy until now.
I checked the distributors' websites, but there is no simple way to install lapack or blas. I am not very experienced with manual setups, and I can't find enough information on installing it on my system. I am running 64-bit Windows 8 with Python 2.7 (from Anaconda).
P.S. There may be a simple solution or something obvious that I missed, so please be patient.
I'm trying to speed up matrix operations using NumPy in Ubuntu 14.04 LTS (64-bit). Instead of using ATLAS (actually when I use ATLAS, there is only 1 thread which is fully running, with 7 other opened threads doing nothing, even if I specify OMP_NUM_THREADS=8 for instance. Don't know why.), I decided to give OpenBLAS a try.
I've spent hours by following several tutorials to build the source code of OpenBLAS and NumPy, e.g. [1], [2], [3], [4], and [5]. However, none of them can generate _dotblas.so after compiling NumPy, which is a critical file to speed up dot operation between matrices.
May I know if anyone has successfully built NumPy and OpenBLAS under Ubuntu 14.04? If so, may you please let me know how to do?
Thank you.
Update:
Below is basically what I summarized from the above five posts and tried in my machine:
# OpenBLAS
git clone git://github.com/xianyi/OpenBLAS
cd OpenBLAS
make FC=gfortran
sudo make PREFIX=/opt/OpenBLAS/ install
cd ..
# let the system know
sudo sh -c 'echo "/opt/OpenBLAS/lib" > /etc/ld.so.conf.d/openblas.conf'
sudo ldconfig
# Numpy
git clone https://github.com/numpy/numpy
cd numpy
vim site.cfg # and put the following content within #### in site.cfg
####
[default]
library_dirs = /opt/OpenBLAS/lib
[atlas]
atlas_libs = openblas
library_dirs = /opt/OpenBLAS/lib
[lapack]
lapack_libs = openblas
library_dirs = /opt/OpenBLAS/lib
####
export BLAS=/opt/OpenBLAS/lib/libopenblas.a
export LAPACK=/opt/OpenBLAS/lib/libopenblas.a
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/OpenBLAS/lib/
python setup.py build
sudo python setup.py install
UPDATE: The above script has been tested to work well on both Ubuntu 12.04 and 14.04 64-bit.
For those who are also struggling with building NumPy with OpenBLAS, the _dotblas module is no longer available since NumPy 1.10.0, according to the Release Notes. Found from this post.