We Keep Coding

Importing Numpy fails after building from source against amd blis

I'm trying to build a local version of Numpy from source against BLIS (for BLAS and CBLAS) and against OpenBLAS for LAPACK.
I started with building BLIS locally for zen3 with CBLAS enabled, like so:
./configure --enable-threading=openmp --enable-cblas --prefix=$HOME/blis zen3
then ran the tests (which all passed) and ran make install. I made sure all relevant files are in the $HOME/blis library (see attached screenshot).
I also built openBLAS locally, no special configs there.
Afterwards, I modified numpy's site.cfg to configure openBLAS and blis folders' accordingly:
[blis]
libraries = blis
library_dirs = /home/or/blis/lib/
include_dirs = /home/or/blis/include/blis
runtime_library_dirs = /home/or/blis/lib/
[openblas]
libraries = openblas
library_dirs = /opt/OpenBLAS/lib
include_dirs = /opt/OpenBLAS/include
runtime_library_dirs = /opt/OpenBLAS/lib
I continued by building and installing numpy with:
NPY_BLAS_ORDER=blis NPY_LAPACK_ORDER=openblas NPY_CBLAS_LIBS= python ./numpy/setup.py build -j 32
Note that NPY_CBLAS_LIBS is empty as numpy's build docs say to do so if CBLAS is included in the BLIS library, which it is.
Then, importing numpy resulting in:
Original error was: /home/or/.pyenv/versions/3.9.6/lib/python3.9/site-packages/numpy-1.24.0.dev0+998.g6a5086c9b-py3.9-linux-x86_64.egg/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so: undefined symbol: cblas_sgemm
I'm clueless at this point as I couldn't find anything online about this specific case.
Installing numpy from pip (which comes built with openblas) can be imported successfully.
Update 1:
While reading make install logs, I found out that it couldn't find my BLIS library files at the location, even though the files are in the specified path. I also tried to recompile and install BLIS in various paths and reconfigure numpy before compiling it, but got the same result.
When I downloaded a pre-compiled version of BLIS from AMD's website, numpy seems to get it, but this isn't the recommended way to go because I'm missing optimizations for Zen3.

linking ipopt against openblas

Currently, I am trying to build Ipopt linking against openblas. I downloaded the openblas source and did make in the parent directory.
The configure script of Ipopt has several options to link against blas:
I tried ./configure --with-blas="-L/home/moritz/build/CoinIpopt_test/ThirdParty/OpenBLAS-0.2.14/libopenblas.so"
but I do get the error
checking whether user supplied BLASLIB="-L/home/moritz/build/CoinIpopt_test/ThirdParty/OpenBLAS-0.2.14/libopenblas.so" works... no
configure: error: user supplied BLAS library "-L/home/moritz/build/CoinIpopt_test/ThirdParty/OpenBLAS-0.2.14/libopenblas.so" does not work
Any tips how to achieve what I want ? Finally, I would like to make a conda package. I do have installed openblas with anaconda. But I do get the same error message if I link against the installed libopenblas.so

Managed to get it work. I had to install openblas to a directory of my choice by
make install PREFIX=/home/....../
aferwards I compiled Ipopt using
./configure --with-blas-incdir="-I/home/.../openblas/include/" --with-blas-lib="-L/home/.../openblas/lib/"

Installing numpy RPM with older Python version

I'm trying to install numpy 1.7 via an RPM on an older Linux machine with Python 2.4. The numpy release notes and the RPM page say it is supposed to be compatible with 2.4 (or <= 2.7), but when I try to install it on the machine with the command
rpm -i /tmp/python-numpy-1.7.0-2.1.i586.rpm
I get a number of missing dependency notes, including:
libc.so.6(GLIBC_2.11) is needed by python-numpy-1.7.0-2.1.i586
libc.so.6(GLIBC_2.4) is needed by python-numpy-1.7.0-2.1.i586
liblapack.so.3 is needed by python-numpy-1.7.0-2.1.i586
libpython2.7.so.1.0 is needed by python-numpy-1.7.0-2.1.i586
python >= 2.7 is needed by python-numpy-1.7.0-2.1.i586
python = 2.7 is needed by python-numpy-1.7.0-2.1.i586
python(abi) = 2.7 is needed by python-numpy-1.7.0-2.1.i586
rpmlib(PayloadIsLzma) <= 4.4.6-1 is needed by python-numpy-1.7.0-2.1.i586
So now at least Python 2.7 is needed, rather than up to 2.7. Is this a real discrepancy or am I using rpm incorrectly? I'm used to higher-level Linux package managers that report dependencies correctly and install them automatically, so I'm unsure how to proceed here.

Are you sure your distribution does not provide numpy already? It looks like numpy is part of epel.
If for some reason you are unwilling to use the version in the distribution, you're likely going to have to build the RPM yourself. I was able to build 1.7.1 on CentOS 5.7 like so:
sudo yum install rpm-build gcc python-devel
wget 'https://pypi.python.org/packages/source/n/numpy/numpy-1.7.1.tar.gz'
tar -xf numpy-1.7.1.tar.gz
cd numpy-1.7.1/
python setup.py bdist_rpm
sudo yum localinstall dist/numpy-1.7.1-1.i386.rpm
the generated RPM (in ./dist) should be useable without rebuilding on all of the machines with similar hardware and OS.

If all the machines have identical versions of Python, glibc, etc., then it would probably be easier to get the numpy source and build it yourself, assuming you have gcc (and perhaps gfortran) installed, along with dependencies like BLAS and LAPACK. Once it's installed on one machine, you can copy the numpy folder (and any .egg file) from /usr/lib/python2.4/site-packages (or whichever directory) and distribute that around the world. Make sure to create static libraries when you build so you don't need all the dependencies everywhere.
I'd also get numpy 1.7.1, as it fixes some issues with 1.7.0.

Supplying NumPy site.cfg arguments to pip

I'm using NumPy built against Intel's Math Kernel Library. I use virtualenv, and typically use pip to install packages.
However, in order for NumPy to find the MKL libraries, it's necessary to create a site.cfg file in the NumPy source directory prior to compiling it, then manually build and install. I could script this whole process, but I was hoping for a simpler solution.
I have a standard site.cfg file that can be used for this purpose under version control. Are there any pip command line options that will tell it to copy a particular file to the source directory before building a package?
Alternatively, are there any environment variables that can be set instead of supplying the library paths in a site.cfg file? Here is the site.cfg file that I use. It was taken almost verbatim from Intel's site.
[mkl]
library_dirs = /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64
include_dirs = /opt/intel/composer_xe_2013.1.117/mkl/include
mkl_libs = mkl_rt
lapack_libs =
For reference, I'm running Ubuntu, Python 2.7, and NumPy 1.6.

From the source (https://github.com/numpy/numpy/blob/master/site.cfg.example):
To assist automatic installation like easy_install, the user's home directory
will also be checked for the file ~/.numpy-site.cfg .
Is that a workable solution? You'd still need to preload the home directories with the global .numpy-site.cfg, but you wouldn't have to muck with the build or installation after that.

I ended up putting together a script to automate this. Here it is, in case it can help someone else. I've tested it in Python 2.7, but it should work elsewhere without significant modifications.
from __future__ import unicode_literals
import io
import os.path
import re
import subprocess
import urllib2
# This downloads, builds, and installs NumPy against the MKL in the
# currently active virtualenv
file_name = 'numpy-1.6.2.tar.gz'
url = ('http://sourceforge.net/projects/numpy/files/NumPy/1.6.2/'
'numpy-1.6.2.tar.gz/download')
def main():
# download NumPy and unpack it
file_data = urllib2.urlopen(url).read()
with io.open(file_name, 'wb') as fobj:
fobj.write(file_data)
subprocess.check_call('tar -xvf {0}'.format(file_name), shell=True)
base_name = re.search(r'(.*)\.tar\.gz$', file_name).group(1)
os.chdir(base_name)
# write out a site.cfg file in the build directory
site_cfg = (
'[mkl]\n'
'library_dirs = /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64\n'
'include_dirs = /opt/intel/composer_xe_2013.1.117/mkl/include\n'
'mkl_libs = mkl_rt\n'
'lapack_libs =\n')
with io.open('site.cfg', 'wt', encoding='UTF-8') as fobj:
fobj.write(site_cfg)
# build and install NumPy
subprocess.check_call('python setup.py build', shell=True)
subprocess.check_call('python setup.py install', shell=True)
if __name__ == '__main__':
main()

Your goal of installing NumPy to use Intel's Math Kernel Library is now much easier since Intel created pips to install MKL + NumPy:
pip uninstall numpy -y # if the standard numpy is present
pip install intel-numpy
as well as intel-scipy, intel-scikit-learn, pydaal, tbb4py, mkl_fft, mkl_random, and the lower level packages if you need just them. Again, you must first uninstall the standard packages if they're already installed in your virtualenv.
NOTE:
If standard NumPy, SciPy and Scikit-Learn packages are already installed, the packages must be uninstalled before installing the Intel® variants of these packages(intel-numpy etc) to avoid any conflicts. As mentioned earlier, pydaal uses intel-numpy, hence it is important to first remove the standard Numpy library (if installed) and then install pydaal.

Alternatively, are there any environment variables that can be set instead of supplying the library paths in a site.cfg file?
NumPy 1.21 introduces environment variables for this purpose.
E.g.
NPY_BLAS_ORDER=MKL NPY_LAPACK_ORDER=MKL pip install numpy --no-binary numpy
to auto-detect the MKL library when installing NumPy from source code. If needed, you can set the environment variables NPY_BLAS_LIBS, NPY_CBLAS_LIBS, and NPY_LAPACK_LIBS to linker CLI options which put your chosen libraries on the linker path.
This is easier for a script to do than creating a ~/.numpy-site.cfg file,
[openblas]
libraries = openblas
library_dirs = /usr/local/opt/openblas/lib
include_dirs = /usr/local/opt/openblas/include
runtime_library_dirs = /usr/local/opt/openblas/lib
then running
pip install numpy --no-binary numpy
BTW, the file ~/.numpy-site.cfg also works when installing scipy from source code:
pip install scipy --no-binary scipy
NOTE: If you're still using Python 2.7, install numpy then install scipy. Attempting to install them together will:
invoke a SciPy easy_install installer that requests NumPy,
load the latest NumPy installer (even if you specifically asked pip to install numpy==1.14.6 scipy==1.0.1 --no-binary numpy,scipy), then
fail with RuntimeError: Python version >= 3.5 required because the latest NumPy does not support Python 2.7.

How to use numpy with OpenBLAS instead of Atlas in Ubuntu?

I have looked for an easy way to install/compile Numpy with OpenBLAS but didn't find an easy answer. All the documentation I have seen takes too much knowledge as granted for someone like me who is not used to compile software.
There are two packages in Ubuntu related to OpenBLAS : libopenblas-base and libopenblas-dev.
Once they are installed, what should I do to install Numpy again with them?
Thanks!
Note that when these OpenBLAS packages are installed, Numpy doesn't work anymore: it can't be imported: ImportError: /usr/lib/liblapack.so.3gf: undefined symbol: ATL_chemv.
The problem occurs as well when installing Theano with their website instructions for Ubuntu.
This was noticed here already.

Run sudo update-alternatives --all and set liblapack.so.3gf to /usr/lib/lapack/liblapack.so.3gf

To add to the accepted answer (of using update-alternatives), the reason for this is because OpenBlas is not compatible with the Atlas version of Lapack. For each of the Blas and Lapack versions:
Default Blas + Default Lapack => OK
OpenBlas + Default Lapack => OK
Atlas-Blas + Default Lapack => OK
Atlas-Blas + Atlas-Lapack => OK
OpenBlas + Atlas-Lapack => ERROR! (The following case here.)
This is from both personal experience (with the exact same issue) and realizing why such a combination wasn't mentioned in this comparison blog.
By the way, you can just find the necessary files in /etc/alternatives/, usually with a filename starting with lib*. For each one do sudo update-alternatives --config <filename>. For example, do to following:
sudo update-alternatives --config libblas.so
sudo update-alternatives --config libblas.so.3
to change the Blas version.

Consider using EasyBuild (http://hpcugent.github.io/easybuild/), an open-source framework for building and installing software.
It allows you to (very easily) build and install (scientific) software with various compiler, and using different BLAS libraries (ATLAS, OpenBLAS, ACML, Intel MKL, ...).
Once you install EasyBuild (pro tip: use the bootstrapping procedure described at https://github.com/hpcugent/easybuild/wiki/Bootstrapping-EasyBuild), it boils down to running a single command, something like:
eb numpy-1.6.2-goolf-1.4.10-Python-2.7.3.eb -ldr
This will first build and install of full compiler toolchain (goolf: GCC+OpenBLAS+OpenMPI+LAPACK+FFTW), and subsequently build Python and numpy with that toolchain. And all that while you're getting lunch. ;-)
Disclaimer: I'm one of the EasyBuild developers.