We Keep Coding

How to build NumPy from source linked to Apple Accelerate framework?

It is my understanding that NumPy dropped support for using the Accelerate BLAS and LAPACK at version 1.20.0. According to the release notes for NumPy 1.21.1, these bugs have been resolved and building NumPy from source using the Accelerate framework on MacOS >= 11.3 is now possible again: https://numpy.org/doc/stable/release/1.21.0-notes.html, but I cannot find any documentation on how to do so. This seems like it would be an interesting thing to try and do because the Accelerate framework is supposed to be highly-optimized for M-series processors. I imagine the process is something like this:
Download numpy source code folder and navigate to this folder.
Make a site.cfg file that looks something like:
[DEFAULT]
library_dirs = /some/directory/
include_dirs = /some/other/directory/
[accelerate]
libraries = Accelerate, vecLib
Run python setup.py build
The problem is I do not know 1. what the variables library_dirs and include_dirs should be so that NumPy knows to use Accelerate BLAS and LAPACK and 2. if there are any other additional steps that need to be taken. If anyone knows how to do this or can provide any insight, it would be greatly appreciated.

I actually attempted this earlier today and these are the steps I used:
In the site.cfg file, put
[accelerate]
libraries = Accelerate, vecLib
Build with NPY_LAPACK_ORDER=accelerate python3 setup.py build
Install with pip3 install .
Afterwards, np.show_config() returned the following
blas_mkl_info:
NOT AVAILABLE
blis_info:
NOT AVAILABLE
openblas_info:
NOT AVAILABLE
accelerate_info:
extra_compile_args = ['-I/System/Library/Frameworks/vecLib.framework/Headers']
extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
blas_opt_info:
extra_compile_args = ['-I/System/Library/Frameworks/vecLib.framework/Headers']
extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
lapack_mkl_info:
NOT AVAILABLE
openblas_lapack_info:
NOT AVAILABLE
openblas_clapack_info:
NOT AVAILABLE
flame_info:
NOT AVAILABLE
lapack_opt_info:
extra_compile_args = ['-I/System/Library/Frameworks/vecLib.framework/Headers']
extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
Supported SIMD extensions in this NumPy install:
baseline = NEON,NEON_FP16,NEON_VFPV4,ASIMD
found = ASIMDHP,ASIMDDP
not found =
and my quick test suggest significant performance boost relative to OpenBlas.

No it doesn't have to be that complicated. I used these two commands and was able to install numpy with Apple Accelerate on Mac M1.
pip install cython pybind11
pip install --no-binary :all: --no-use-pep517 numpy
Reference: How to install SciPy on Apple Silicon (ARM / M1)

How to change numpy mkl BLAS libraries?

So I just installed anaconda, and when I type numpy.__config__.show(), I get the following output:
blas_mkl_info:
libraries = ['mkl_rt']
library_dirs = ['C:/Anaconda3\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/Anaconda3\\Library\\include']
I have another python build (on separate computer) where the libraries = ['mkl_core_dll', 'mkl_intel_lp64_dll', 'mkl_intel_thread_dll']
My question is how do I change the anaconda build to include these different mkl libraries? I've tried messing with the site.cfg file in numpy, but no luck.
Also, does using different mkl libraries lead to different end results? I am really trying to get reproducible results between these two builds, and right now these different mkl settings seem to be messing it up.
Thanks in advance.

ATLAS on a python virtualenv in Fedora for numpy/scipy/scikit-learn

I am struggeling to let python find and use the installed ATLAS libraries from my distribution when using virtualenv.
This is on Fedora 21, atlas, atlas-devel, blas, blas-devel are installed. Outside of a virtualenv, the command python -c 'import numpy; numpy.show_config()' shows that I have ATLAS:
atlas_3_10_blas_threads_info:
libraries = ['tatlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('HAVE_CBLAS', None), ('ATLAS_INFO', '"\\"3.10.1\\""')]
language = c
include_dirs = ['/usr/include']
lapack_opt_info:
libraries = ['tatlas', 'tatlas', 'tatlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('ATLAS_INFO', '"\\"3.10.1\\""')]
language = f77
include_dirs = ['/usr/include']
blas_opt_info:
libraries = ['tatlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('HAVE_CBLAS', None), ('ATLAS_INFO', '"\\"3.10.1\\""')]
language = c
include_dirs = ['/usr/include']
openblas_info:
NOT AVAILABLE
openblas_lapack_info:
NOT AVAILABLE
[...]
And ls /usr/lib64/atlas/ gives:
libatlas.a libsatlas.so libsatlas.so.3.10 libtatlas.so.3
libcblas.so libsatlas.so.3 libtatlas.so libtatlas.so.3.10
I also setup an environment variable, so that echo $ATLAS gives
/usr/lib64/atlas/libsatlas.so
But when I do the following:
virtualenv venv
source venv/bin/activate
pip install --upgrade pip
pip install numpy
pip install scipy
pip install scikit-learn
python -c 'import numpy; numpy.show_config()'
I get:
lapack_opt_info:
libraries = ['openblas', 'openblas']
library_dirs = ['/usr/local/lib']
define_macros = [('HAVE_CBLAS', None)]
language = c
blas_opt_info:
libraries = ['openblas', 'openblas']
library_dirs = ['/usr/local/lib']
define_macros = [('HAVE_CBLAS', None)]
language = c
openblas_info:
libraries = ['openblas', 'openblas']
library_dirs = ['/usr/local/lib']
define_macros = [('HAVE_CBLAS', None)]
language = c
openblas_lapack_info:
libraries = ['openblas', 'openblas']
library_dirs = ['/usr/local/lib']
define_macros = [('HAVE_CBLAS', None)]
language = c
blas_mkl_info:
NOT AVAILABLE
I would really appreciate help with how to get numpy and scipy simply find and use ATLAS from my distribution in a virtualenv.

You need to tell numpy's setup.py where to find your ATLAS libraries.
Try creating a .numpy-site.cfg file in your home folder before running pip install. Here's the template for this file.
You probably need to add the lines
[atlas]
library_dirs = /usr/lib64/atlas
include_dirs = /usr/include
Although, this has required a little bit of trial-and-error in my experience.
Using this file in conjunction with pip install seems to work reliably on RHEL and Ubuntu, at least within virtual environments where you don't need sudo.
If this fails for one or more of numpy, scipy and scikit-learn, consider installing from source instead, and modify the site.cfg file inside the source dir before running python setup.py install.

Link MKL to an installed Numpy in Anaconda?

>>> numpy.__config__.show()
atlas_threads_info:
NOT AVAILABLE
blas_opt_info:
libraries = ['f77blas', 'cblas', 'atlas']
library_dirs = ['/home/admin/anaconda/lib']
define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
language = c
atlas_blas_threads_info:
NOT AVAILABLE
openblas_info:
NOT AVAILABLE
lapack_opt_info:
libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
library_dirs = ['/home/admin/anaconda/lib']
define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
language = f77
openblas_lapack_info:
NOT AVAILABLE
atlas_info:
libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
library_dirs = ['/home/admin/anaconda/lib']
define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
language = f77
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
atlas_blas_info:
libraries = ['f77blas', 'cblas', 'atlas']
library_dirs = ['/home/admin/anaconda/lib']
define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
language = c
mkl_info:
NOT AVAILABLE
This is numpy.config.show()
And this numpy is based on Anaconda.
in /home/admin/anaconda/lib/python2.7/site-packages/numpy
And meanwhile, I have installed MKL in /opt/intel/mkl.
Also it seems that, I can use MKL, when I used Theano with this Anaconda and this numpy.
Thank you~

To do this, you'll have to get the Numpy source code and create a site.cfg file in the main folder that tells Numpy where to look for the MKL libraries. Use the site.cfg.example file as a guide; there's a section in there on MKL that should tell you the exact lines you need.
Once you have a site.cfg file, run python setup.py config. This will spit out the configuration info (it looks like the output of numpy.config.show()). You should see the paths to your MKL libraries in there (if not, you'll have to play around with the site.cfg file until you do).
Then, you can build and install with python setup.py build and python setup.py install. You may want to uninstall your old Numpy first.
I have a blog post on installing Numpy with OpenBLAS. While your situation is a little different, a lot of steps are similar I think.

Windows + virtualenv + pip + NumPy (problems when installing NumPy)

On Windows, I normally just use the binary installer, but I would like to install NumPy only in a virtualenv this time, so I created a virtual env:
virtualenv --no-site-packages --distribute summary_python
cd summary_python/Scripts
activate.bat
Then I tried to install NumPy
pip install numpy
And I get an error. My pip.log is pasted below:
Downloading/unpacking numpy
Running setup.py egg_info for package numpy
non-existing path in 'numpy\\distutils': 'site.cfg'
F2PY Version 2
blas_opt_info:
blas_mkl_info:
libraries mkl,vml,guide not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries mkl,vml,guide not found in C:\
NOT AVAILABLE
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries ptf77blas,ptcblas,atlas not found in C:\
NOT AVAILABLE
atlas_blas_info:
libraries f77blas,cblas,atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries f77blas,cblas,atlas not found in C:\
NOT AVAILABLE
blas_info:
libraries blas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries blas not found in C:\
NOT AVAILABLE
blas_src_info:
NOT AVAILABLE
NOT AVAILABLE
lapack_opt_info:
lapack_mkl_info:
mkl_info:
libraries mkl,vml,guide not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries mkl,vml,guide not found in C:\
NOT AVAILABLE
NOT AVAILABLE
atlas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries lapack_atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries ptf77blas,ptcblas,atlas not found in C:\
libraries lapack_atlas not found in C:\
numpy.distutils.system_info.atlas_threads_info
NOT AVAILABLE
atlas_info:
libraries f77blas,cblas,atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries lapack_atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries f77blas,cblas,atlas not found in C:\
libraries lapack_atlas not found in C:\
numpy.distutils.system_info.atlas_info
NOT AVAILABLE
lapack_info:
libraries lapack not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries lapack not found in C:\
NOT AVAILABLE
lapack_src_info:
NOT AVAILABLE
NOT AVAILABLE
running egg_info
running build_src
build_src
building py_modules sources
building library "npymath" sources
No module named msvccompiler in numpy.distutils; trying from distutils
Running from numpy source directory.c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:531: UserWarning: Specified path is invalid.
warnings.warn('Specified path %s is invalid.' % d)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1417: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1426: UserWarning:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
warnings.warn(BlasNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1429: UserWarning:
Blas (http://www.netlib.org/blas/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [blas_src]) or by setting
the BLAS_SRC environment variable.
warnings.warn(BlasSrcNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1333: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1344: UserWarning:
Lapack (http://www.netlib.org/lapack/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [lapack]) or by setting
the LAPACK environment variable.
warnings.warn(LapackNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1347: UserWarning:
Lapack (http://www.netlib.org/lapack/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [lapack_src]) or by setting
the LAPACK_SRC environment variable.
warnings.warn(LapackSrcNotFoundError.__doc__)
error: Unable to find vcvarsall.bat
Complete output from command python setup.py egg_info:
non-existing path in 'numpy\\distutils': 'site.cfg'
F2PY Version 2
blas_opt_info:
blas_mkl_info:
libraries mkl,vml,guide not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries mkl,vml,guide not found in C:\
NOT AVAILABLE
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries ptf77blas,ptcblas,atlas not found in C:\
NOT AVAILABLE
atlas_blas_info:
libraries f77blas,cblas,atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries f77blas,cblas,atlas not found in C:\
NOT AVAILABLE
blas_info:
libraries blas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries blas not found in C:\
NOT AVAILABLE
blas_src_info:
NOT AVAILABLE
NOT AVAILABLE
lapack_opt_info:
lapack_mkl_info:
mkl_info:
libraries mkl,vml,guide not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries mkl,vml,guide not found in C:\
NOT AVAILABLE
NOT AVAILABLE
atlas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries lapack_atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries ptf77blas,ptcblas,atlas not found in C:\
libraries lapack_atlas not found in C:\
numpy.distutils.system_info.atlas_threads_info
NOT AVAILABLE
atlas_info:
libraries f77blas,cblas,atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries lapack_atlas not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries f77blas,cblas,atlas not found in C:\
libraries lapack_atlas not found in C:\
numpy.distutils.system_info.atlas_info
NOT AVAILABLE
lapack_info:
libraries lapack not found in c:\Users\fname.lname\Documents\summary_python\lib
libraries lapack not found in C:\
NOT AVAILABLE
lapack_src_info:
NOT AVAILABLE
NOT AVAILABLE
running egg_info
running build_src
build_src
building py_modules sources
building library "npymath" sources
No module named msvccompiler in numpy.distutils; trying from distutils
Running from numpy source directory.c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:531: UserWarning: Specified path is invalid.
warnings.warn('Specified path %s is invalid.' % d)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1417: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1426: UserWarning:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
warnings.warn(BlasNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1429: UserWarning:
Blas (http://www.netlib.org/blas/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [blas_src]) or by setting
the BLAS_SRC environment variable.
warnings.warn(BlasSrcNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1333: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1344: UserWarning:
Lapack (http://www.netlib.org/lapack/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [lapack]) or by setting
the LAPACK environment variable.
warnings.warn(LapackNotFoundError.__doc__)
c:\Users\fname.lname\Documents\summary_python\build\numpy\numpy\distutils\system_info.py:1347: UserWarning:
Lapack (http://www.netlib.org/lapack/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [lapack_src]) or by setting
the LAPACK_SRC environment variable.
warnings.warn(LapackSrcNotFoundError.__doc__)
error: Unable to find vcvarsall.bat
----------------------------------------
Command python setup.py egg_info failed with error code 1
Exception information:
Traceback (most recent call last):
File "c:\Users\fname.lname\Documents\summary_python\lib\site-packages\pip-1.0.1-py2.7.egg\pip\basecommand.py", line 126, in main
self.run(options, args)
File "c:\Users\fname.lname\Documents\summary_python\lib\site-packages\pip-1.0.1-py2.7.egg\pip\commands\install.py", line 223, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "c:\Users\fname.lname\Documents\summary_python\lib\site-packages\pip-1.0.1-py2.7.egg\pip\req.py", line 986, in prepare_files
req_to_install.run_egg_info()
File "c:\Users\fname.lname\Documents\summary_python\lib\site-packages\pip-1.0.1-py2.7.egg\pip\req.py", line 222, in run_egg_info
command_desc='python setup.py egg_info')
File "c:\Users\fname.lname\Documents\summary_python\lib\site-packages\pip-1.0.1-py2.7.egg\pip\__init__.py", line 255, in call_subprocess
% (command_desc, proc.returncode))
InstallationError: Command python setup.py egg_info failed with error code 1

I've had success installing NumPy binaries into a virtualenv with good 'ol easy_install and a little bit of un-archiving magic.
The numpy-1.x.x-win32-superpack-python2.x.exe release you download from SourceForge is really just a thin wrapper around three separate binary distributions (with SSE3, SSE2, or no SSE enabled, depending on the capabilities of your CPU). If you open up the superpack EXE file in 7-Zip (or another archive utility), you can extract those individual setup files somewhere to use separately.
Then, activate your virtual environment, and run
easy_install c:\path\to\extracted\numpy-1.x.x-sse3.exe
to install the SSE3-optimized binaries, for example. easy_install is smart enough to find everything it needs inside that wininst bundle and will extract the compiled egg into your virtualenv's site-packages folder. I can also confirm that pip is still able to recognize and/or uninstall NumPy when you do this, and that using pip to install other packages which depend on NumPy works just fine.
The only catch is knowing which optimization level to use (SSE3, SSE2, or no-SSE). If you have a reasonably modern processor (newer than, say, a Pentium 4 or Athlon 64), it's probably safe to go with the full SSE3. You can probably also run the test suite to confirm everything works as expected.
I've found the easy_install "trick" to be really useful for installing all sorts of binary packages into a virtualenv. Even though I have all the requisite compilers set up on my machine, it's usually easier/faster/safer to stick with the official release when one is provided.

Since pip can only install from source, you need to have the setup to build and compile NumPy to be able to use pip. I never tried this.
What I did recently to get NumPy, SciPy and Matplotlib into a virtual environment on Windows 7 is to copy the directories and egg files from my regular Python install directly into the virtual environment. For Matplotlib I also managed to unzip the binary installer directly into the virtualenv (I only tried this for the Gohlke installer for Python 3.2).
I did manage in the past to easy_install NumPy into a virtualenv from a binary distribution, but I didn't manage this time (NumPy 1.5.1).
Someone recommended http://pypi.python.org/pypi/pyg which should allow to install from binary packages to a virtualenv, but I haven't tried it yet.

I didn't have to do the trick of unpacking the .exe file. I just activated my virtual environment, and then:
easy_install numpy