I'm trying to build a local version of Numpy from source against BLIS (for BLAS and CBLAS) and against OpenBLAS for LAPACK.
I started with building BLIS locally for zen3 with CBLAS enabled, like so:
./configure --enable-threading=openmp --enable-cblas --prefix=$HOME/blis zen3
then ran the tests (which all passed) and ran make install. I made sure all relevant files are in the $HOME/blis library (see attached screenshot).
I also built openBLAS locally, no special configs there.
Afterwards, I modified numpy's site.cfg to configure openBLAS and blis folders' accordingly:
[blis]
libraries = blis
library_dirs = /home/or/blis/lib/
include_dirs = /home/or/blis/include/blis
runtime_library_dirs = /home/or/blis/lib/
[openblas]
libraries = openblas
library_dirs = /opt/OpenBLAS/lib
include_dirs = /opt/OpenBLAS/include
runtime_library_dirs = /opt/OpenBLAS/lib
I continued by building and installing numpy with:
NPY_BLAS_ORDER=blis NPY_LAPACK_ORDER=openblas NPY_CBLAS_LIBS= python ./numpy/setup.py build -j 32
Note that NPY_CBLAS_LIBS is empty as numpy's build docs say to do so if CBLAS is included in the BLIS library, which it is.
Then, importing numpy resulting in:
Original error was: /home/or/.pyenv/versions/3.9.6/lib/python3.9/site-packages/numpy-1.24.0.dev0+998.g6a5086c9b-py3.9-linux-x86_64.egg/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so: undefined symbol: cblas_sgemm
I'm clueless at this point as I couldn't find anything online about this specific case.
Installing numpy from pip (which comes built with openblas) can be imported successfully.
Update 1:
While reading make install logs, I found out that it couldn't find my BLIS library files at the location, even though the files are in the specified path. I also tried to recompile and install BLIS in various paths and reconfigure numpy before compiling it, but got the same result.
When I downloaded a pre-compiled version of BLIS from AMD's website, numpy seems to get it, but this isn't the recommended way to go because I'm missing optimizations for Zen3.
Related
LINK : fatal error LNK1181: cannot open input file 'lapack.lib'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.34.31933\\bin\\HostX86\\x64\\link.exe' failed with exit code 1181
I'm getting this error while installing cvxopt. I've set the directory for lapack lib in the setup.py file inside the cvxopt folder. However running the command
python setup.py install
generates this error. Can anyone help me with this?
If you use pip install cvxopt it will use a pre-built binary wheel package already.
If you want to build from source (as you are currently doing) then you need to have BLAS and LAPACK installed on your system
https://cvxopt.org/install/#building-and-installing-from-source
Required and optional software
The package requires version 3.x of Python, and building from source requires core binaries and header files and libraries for Python.
The installation requires BLAS and LAPACK. Using an architecture optimized implementation such as ATLAS, OpenBLAS, or MKL is recommended and gives a large performance improvement over reference implementations of the BLAS and LAPACK libraries.
The installation also requires SuiteSparse. We recommend linking against a shared SuiteSparse library. It is also possible to build the required components of SuiteSparse when building CVXOPT, but this requires the SuiteSparse source which is no longer included with CVXOPT and must be downloaded separately.
The following software libraries are optional.
The GNU Scientific Library GSL.
FFTW is a C library for discrete Fourier transforms.
GLPK is a linear programming package.
MOSEK version 9 is a commercial library of convex optimization solvers.
DSDP5.8 is a semidefinite programming solver.
Currently, I am trying to build Ipopt linking against openblas. I downloaded the openblas source and did make in the parent directory.
The configure script of Ipopt has several options to link against blas:
I tried ./configure --with-blas="-L/home/moritz/build/CoinIpopt_test/ThirdParty/OpenBLAS-0.2.14/libopenblas.so"
but I do get the error
checking whether user supplied BLASLIB="-L/home/moritz/build/CoinIpopt_test/ThirdParty/OpenBLAS-0.2.14/libopenblas.so" works... no
configure: error: user supplied BLAS library "-L/home/moritz/build/CoinIpopt_test/ThirdParty/OpenBLAS-0.2.14/libopenblas.so" does not work
Any tips how to achieve what I want ? Finally, I would like to make a conda package. I do have installed openblas with anaconda. But I do get the same error message if I link against the installed libopenblas.so
Managed to get it work. I had to install openblas to a directory of my choice by
make install PREFIX=/home/....../
aferwards I compiled Ipopt using
./configure --with-blas-incdir="-I/home/.../openblas/include/" --with-blas-lib="-L/home/.../openblas/lib/"
I'm trying to speed up matrix operations using NumPy in Ubuntu 14.04 LTS (64-bit). Instead of using ATLAS (actually when I use ATLAS, there is only 1 thread which is fully running, with 7 other opened threads doing nothing, even if I specify OMP_NUM_THREADS=8 for instance. Don't know why.), I decided to give OpenBLAS a try.
I've spent hours by following several tutorials to build the source code of OpenBLAS and NumPy, e.g. [1], [2], [3], [4], and [5]. However, none of them can generate _dotblas.so after compiling NumPy, which is a critical file to speed up dot operation between matrices.
May I know if anyone has successfully built NumPy and OpenBLAS under Ubuntu 14.04? If so, may you please let me know how to do?
Thank you.
Update:
Below is basically what I summarized from the above five posts and tried in my machine:
# OpenBLAS
git clone git://github.com/xianyi/OpenBLAS
cd OpenBLAS
make FC=gfortran
sudo make PREFIX=/opt/OpenBLAS/ install
cd ..
# let the system know
sudo sh -c 'echo "/opt/OpenBLAS/lib" > /etc/ld.so.conf.d/openblas.conf'
sudo ldconfig
# Numpy
git clone https://github.com/numpy/numpy
cd numpy
vim site.cfg # and put the following content within #### in site.cfg
####
[default]
library_dirs = /opt/OpenBLAS/lib
[atlas]
atlas_libs = openblas
library_dirs = /opt/OpenBLAS/lib
[lapack]
lapack_libs = openblas
library_dirs = /opt/OpenBLAS/lib
####
export BLAS=/opt/OpenBLAS/lib/libopenblas.a
export LAPACK=/opt/OpenBLAS/lib/libopenblas.a
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/OpenBLAS/lib/
python setup.py build
sudo python setup.py install
UPDATE: The above script has been tested to work well on both Ubuntu 12.04 and 14.04 64-bit.
For those who are also struggling with building NumPy with OpenBLAS, the _dotblas module is no longer available since NumPy 1.10.0, according to the Release Notes. Found from this post.
I want to install scipy on a machine where I am not root following the instruction I found on this page but I meeting a BLAS not found error despite the fact that BLAS is install and "properly defined".
So I started installing the last python release (2.7.6) in a custom directoty ~/Tools/Python) which works fine. I compile then BLAST and LAPACK (3..5.0) following the instruction on this page. The compilation works fine and I put the respective .a files in some custom folders also (~Tools/BLAS with blas_LINUX.a and ~/TOOLS/LAPACK with liblapack.a and libtmglib.a). I then define two environment variables BLAS and LAPACK poiting to the directory above where the .a files are.
I then install numpy (1.8.1) using
python setup.py install --prefix=~/TOOLS/Python
and update my PYTHONPATH accordingly. Numpy works fine. However when shifting to the install of scipy (0.13.3), I get the following error:
numpy.distutils.system_info.BlasNotFoundError:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
I try various combinaisons like specifying BLAS just before the call to the setup function but still it doesn't work. However I don't understand as when I type
echo $BLAS
the BLAS path is outputed. So any idea of what went wrong?
I have looked for an easy way to install/compile Numpy with OpenBLAS but didn't find an easy answer. All the documentation I have seen takes too much knowledge as granted for someone like me who is not used to compile software.
There are two packages in Ubuntu related to OpenBLAS : libopenblas-base and libopenblas-dev.
Once they are installed, what should I do to install Numpy again with them?
Thanks!
Note that when these OpenBLAS packages are installed, Numpy doesn't work anymore: it can't be imported: ImportError: /usr/lib/liblapack.so.3gf: undefined symbol: ATL_chemv.
The problem occurs as well when installing Theano with their website instructions for Ubuntu.
This was noticed here already.
Run sudo update-alternatives --all and set liblapack.so.3gf to /usr/lib/lapack/liblapack.so.3gf
To add to the accepted answer (of using update-alternatives), the reason for this is because OpenBlas is not compatible with the Atlas version of Lapack. For each of the Blas and Lapack versions:
Default Blas + Default Lapack => OK
OpenBlas + Default Lapack => OK
Atlas-Blas + Default Lapack => OK
Atlas-Blas + Atlas-Lapack => OK
OpenBlas + Atlas-Lapack => ERROR! (The following case here.)
This is from both personal experience (with the exact same issue) and realizing why such a combination wasn't mentioned in this comparison blog.
By the way, you can just find the necessary files in /etc/alternatives/, usually with a filename starting with lib*. For each one do sudo update-alternatives --config <filename>. For example, do to following:
sudo update-alternatives --config libblas.so
sudo update-alternatives --config libblas.so.3
to change the Blas version.
Consider using EasyBuild (http://hpcugent.github.io/easybuild/), an open-source framework for building and installing software.
It allows you to (very easily) build and install (scientific) software with various compiler, and using different BLAS libraries (ATLAS, OpenBLAS, ACML, Intel MKL, ...).
Once you install EasyBuild (pro tip: use the bootstrapping procedure described at https://github.com/hpcugent/easybuild/wiki/Bootstrapping-EasyBuild), it boils down to running a single command, something like:
eb numpy-1.6.2-goolf-1.4.10-Python-2.7.3.eb -ldr
This will first build and install of full compiler toolchain (goolf: GCC+OpenBLAS+OpenMPI+LAPACK+FFTW), and subsequently build Python and numpy with that toolchain. And all that while you're getting lunch. ;-)
Disclaimer: I'm one of the EasyBuild developers.