I'm trying to install sklearn on top of a Docker image (FROM astronomerinc/ap-airflow:master-1.10.5-onbuild). Environment coming with the source image:
Alpine Linux v3.10 (kernel 4.9.93-linuxkit-aufs)
Python 3.7.3
numpy==1.17.2
pandas==0.25.1
pandas-gbq==0.11.0
...
I had scipy==1.3.1 in my requirements.txt and had no issues installing it with pip, however when I added scikit-learn to requirements.txt and rebuilt again, I got this error saying a numpy header is missing:
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/sklearn
creating build/temp.linux-x86_64-3.7/sklearn/svm
creating build/temp.linux-x86_64-3.7/sklearn/svm/src
creating build/temp.linux-x86_64-3.7/sklearn/svm/src/libsvm
compile options: '-I/usr/lib/python3.7/site-packages/numpy/core/include -c'
g++: sklearn/svm/src/libsvm/libsvm_template.cpp
ar: adding 1 object files to build/temp.linux-x86_64-3.7/liblibsvm-skl.a
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
resetting extension 'sklearn.svm.liblinear' language from 'c' to 'c++'.
customize UnixCCompiler
customize UnixCCompiler using build_ext
building 'sklearn.__check_build._check_build' extension
compiling C sources
C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
creating build/temp.linux-x86_64-3.7/sklearn/__check_build
compile options: '-I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/include/python3.7m -c'
gcc: sklearn/__check_build/_check_build.c
gcc -shared -Wl,--as-needed -Wl,--as-needed build/temp.linux-x86_64-3.7/sklearn/__check_build/_check_build.o -L/usr/lib -Lbuild/temp.linux-x86_64-3.7 -lpython3.7m -o build/lib.linux-x86_64-3.7/sklearn/__check_build/_check_build.cpython-37m-x86_64-linux-gnu.so
building 'sklearn.cluster._dbscan_inner' extension
compiling C++ sources
C compiler: g++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
creating build/temp.linux-x86_64-3.7/sklearn/cluster
compile options: '-I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/include/python3.7m -c'
g++: sklearn/cluster/_dbscan_inner.cpp
sklearn/cluster/_dbscan_inner.cpp:652:10: fatal error: numpy/arrayobject.h: No such file or directory
#include "numpy/arrayobject.h"
^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: Command "g++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/include/python3.7m -c sklearn/cluster/_dbscan_inner.cpp -o build/temp.linux-x86_64-3.7/sklearn/cluster/_dbscan_inner.o -MMD -MF build/temp.linux-x86_64-3.7/sklearn/cluster/_dbscan_inner.o.d" failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-o8ktwf40/scikit-learn/setup.py'"'"'; __file__='"'"'/tmp/pip-install-o8ktwf40/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-p6ejlhi_/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.
WARNING: You are using pip version 19.2.1, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c pip install --no-cache-dir -q -r requirements.txt' returned a non-zero code: 1
Several things I've tried:
upgrading pip
specifying an older version of scikit-learn
"explicitly" installing py3-numpy
None of them worked unfortunately. This post recommends setting the path manually but that just wasn't the answer I was looking for.
Insights? Any help is appreciated!
I suggest you to install py-numpy-dev in your Dockerfile:
RUN apk add py-numpy-dev
Related
I am trying to install pyamg in my virtual environment. However, I am getting the following error. I am using mac OS.
c++: pyamg/amg_core/amg_core_wrap.cxx
clang: error: no such file or directory: '“-I/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0”'
clang: error: no such file or directory: '“-I/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0”'
error: Command "c++ -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE “-I/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0” -arch x86_64 -arch i386 -pipe -D__STDC_FORMAT_MACROS=1 -I/Users/mas/PycharmProjects/Whale/Zahraa5/lib/python2.7/site-packages/numpy/core/include -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c pyamg/amg_core/amg_core_wrap.cxx -o build/temp.macosx-10.10-intel-2.7/pyamg/amg_core/amg_core_wrap.o" failed with exit status 1
Use Anaconda or Miniconda
conda install pyamg
takes only a few seconds.
You can create an environment with:
conda create --name my_env python=2.7
Change into it:
source activate my_env
and install pyamg:
conda install pyamg
You can still use pip to install packages conda cannot find.
Life is too short to spent time on compilation issues. ;)
I'm pretty sure that the -I at the beginning of the paths is what's screwing everything up. I bet if you tried adding those export statements back to .bashrc but took out the -I and -L prefixes then your original command would start working.
Perhaps you had those there for a reason, I've never seen that, but removing those prefixes would probably work.
Actually, I think #oarfish called it correctly in the comments. The problem seems to be the funny “ and ” characters in those paths, which are distinct from the usual " double quote symbol.
The following reproduces the error for me:
~$ CPPFLAGS=“-I/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0” \
LIBS=“-L/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0/stage/lib” \
pip install pyamg
Collecting pyamg
Downloading pyamg-3.0.1.tar.gz (759kB)
100% |████████████████████████████████| 761kB 33.2MB/s
Installing collected packages: pyamg
Running setup.py install for pyamg
...
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/pyamg
creating build/temp.linux-x86_64-2.7/pyamg/amg_core
compile options: '-D__STDC_FORMAT_MACROS=1 -I/home/alistair/.venvs/pyamg/local/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c'
c++: pyamg/amg_core/amg_core_wrap.cxx
g++: error: “-I/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0”: No such file or directory
g++: error: “-I/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0”: No such file or directory
error: Command "c++ -pthread -DNDEBUG -g -fwrapv -O2 -Wall -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security “-I/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0” -fPIC -D__STDC_FO
RMAT_MACROS=1 -I/home/alistair/.venvs/pyamg/local/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c pyamg/amg_core/amg_core_wrap.cxx -o build/temp.linux-x86_64-2.7/pyamg/amg_core/amg_core_wrap.o" failed with exit
status 1
----------------------------------------
Command "/home/alistair/.venvs/pyamg/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-Cl5_2g/pyamg/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" insta
ll --record /tmp/pip-kkjcoa-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/alistair/.venvs/pyamg/include/site/python2.7/pyamg" failed with error code 1 in /tmp/pip-build-Cl5_2g/pyamg
Whereas with " symbols the installation succeeds:
~$ CPPFLAGS="-I/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0" \
LIBS="-L/Users/mas/PycharmProjects/kaggle-ndsb/boost_1_59_0/stage/lib" \
pip install pyamg
Collecting pyamg
Using cached pyamg-3.0.1.tar.gz
Installing collected packages: pyamg
Running setup.py install for pyamg
Successfully installed pyamg-3.0.1
The paths themselves are irrelevant - the compilation succeeds in spite of the fact that those directories don't actually exist on my machine.
I'm trying to install PyLucene, and when I do "pushd jcc" and "python setup.py build", I get this error:
ld: internal error: atom not found in symbolIndex(__ZN7JNIEnv_13CallIntMethodEP8_jobjectP10_jmethodIDz) for architecture x86_64
What could be the problem? I have the Java 1.7 SDK installed. I installed Python using "brew install Python" and I added /usr/local/bin before /urs/bin in my PATH so that this version of Python 2.7.8 is used and not the one shipped by Apple (2.7).
Here are some lines that get outputted on the "python setup.py build" command:
found JAVAHOME = /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
found JAVAFRAMEWORKS = /System/Library/Frameworks/JavaVM.framework
and
writing /Users/administrator/Downloads/pylucene-4.9.0-0/jcc/jcc/config.py
copying jcc/config.py -> build/lib.macosx-10.9-x86_64-2.7/jcc
copying jcc/classes/org/apache/jcc/PythonVM.class -> build/lib.macosx-10.9-x86_64-2.7/jcc/classes/org/apache/jcc
copying jcc/classes/org/apache/jcc/PythonException.class -> build/lib.macosx-10.9-x86_64-2.7/jcc/classes/org/apache/jcc
running build_ext
building 'jcc' extension
clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Qunused-arguments -Qunused-arguments -dynamiclib -D_jcc_lib -DJCC_VER="2.20" -I/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/include/darwin -I_jcc -Ijcc/sources -I/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c jcc/sources/jcc.cpp -o build/temp.macosx-10.9-x86_64-2.7/jcc/sources/jcc.o -DPYTHON -fno-strict-aliasing -Wno-write-strings
clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Qunused-arguments -Qunused-arguments -dynamiclib -D_jcc_lib -DJCC_VER="2.20" -I/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/include/darwin -I_jcc -Ijcc/sources -I/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c jcc/sources/JCCEnv.cpp -o build/temp.macosx-10.9-x86_64-2.7/jcc/sources/JCCEnv.o -DPYTHON -fno-strict-aliasing -Wno-write-strings
clang++ -Wl,-x -dynamiclib -undefined dynamic_lookup -Qunused-arguments -Qunused-arguments build/temp.macosx-10.9-x86_64-2.7/jcc/sources/jcc.o build/temp.macosx-10.9-x86_64-2.7/jcc/sources/JCCEnv.o -o build/lib.macosx-10.9-x86_64-2.7/libjcc.dylib -L/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib -ljava -L/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/server -ljvm -Wl,-rpath -Wl,/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib -Wl,-rpath -Wl,/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/server -Wl,-S -install_name #rpath/libjcc.dylib -current_version 2.20 -compatibility_version 2.20
ld: internal error: atom not found in symbolIndex(__ZN7JNIEnv_13CallIntMethodEP8_jobjectP10_jmethodIDz) for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang++' failed with exit status 1
You either need to build Python from source with the same compiler, as described here, or alternatively build it in NO_SHARED mode. The latter is the default for the pylucene homebrew formula.
I cannot get pillow installed.
(env)noah:cupalensic2 broinjc$ which cc
/usr/bin/cc
(env)noah:cupalensic2 broinjc$ cc --version
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.1.0
Thread model: posix
I ran pip install mezzanine and when it got to satisfying pillow it screwed up...
cc -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -pipe -arch x86_64 -DHAVE_LIBZ -I/System/Library/Frameworks/Tcl.framework/Versions/8.5/Headers -I/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/opt/X11/include/freetype2 -I/Users/broinjc/cupalensic2/env/build/pillow/libImaging -I/Users/broinjc/cupalensic2/env/include -I/usr/local/include -I/opt/X11/include -I/usr/include -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c _imaging.c -o build/temp.macosx-10.9-intel-2.7/_imaging.o
clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
error: command 'cc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /Users/broinjc/cupalensic2/env/bin/python -c "import setuptools;__file__='/Users/broinjc/cupalensic2/env/build/pillow/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/lg/y_cwnrnd3rn4x_j1sbd1mg_h0000gn/T/pip-o0HQHe-record/install-record.txt --single-version-externally-managed --install-headers /Users/broinjc/cupalensic2/env/include/site/python2.7 failed with error code 1 in /Users/broinjc/cupalensic2/env/build/pillow
Storing complete log in /Users/broinjc/.pip/pip.log
Any idea what might be going on here?
This is a known issue with OSX and recent Xcode upgrade.
Instructions how to avoid the issue are here:
Ansible Installation -clang: error: unknown argument: '-mno-fused-madd'
This should do the trick::
ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install Pillow
here is my error when I try to install gevent-socketio
Installing collected packages: gevent, greenlet
Running setup.py install for gevent
building 'gevent.core' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes >-fPIC -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CHECK_ENABLE=0 -DEV_CLEANUP_ENABLE=0 >-DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -Ibuild/temp.linux-x86_64-2.7/libev -Ilibev >-I/usr/include/python2.7 -c gevent/gevent.core.c -o build/temp.linux-x86_64-2.7/gevent/gevent.core.o
gevent/gevent.core.c:17:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
Complete output from command /usr/bin/python -c "import setuptools;file='/var/www/bleu/build/gevent/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --single-version-externally-managed --record /tmp/pip-_kv6Fy-record/install-record.txt:
running install
running build
running build_py
running build_ext
building 'gevent.core' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC > -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CHECK_ENABLE=0 -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -Ibuild/temp.linux-x86_64-2.7/libev -Ilibev -I/usr/include/python2.7 -c gevent/gevent.core.c -o build/temp.linux-x86_64-2.7/gevent/gevent.core.o
gevent/gevent.core.c:17:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
Have you an idea how i can fix this?
Install the Development Package(s):
CentOS/RHEL::
yum install python-devel
Debian/Ubuntu:
apt-get install python-dev
I am trying to install the fabric library to an old machine. There are some legacy libraries in /usr/lib, such as libgmp.
(py27)[qrtt1#hcservice app]$ ls /usr/lib|grep gmp
libgmp.a
libgmp.so
libgmp.so.3
libgmp.so.3.3.3
libgmpxx.a
libgmpxx.so
libgmpxx.so.3
libgmpxx.so.3.0.5
I have compiled the libgmp 5.x in my $HOME/app, and then am trying to install pycrypto (it is the dependency of fab):
CFLGAS=-I/home/qrtt1/app/include LDFLGAS=-L/home/qrtt1/app/lib pip install pycrypto
I observed that none of my include or lib directories are in the in the compilation / linking options:
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/usr/include/ -I/home/qrtt1/app/include/python2.7 -c src/_fastmath.c -o build/temp.linux-i686-2.7/src/_fastmath.o
gcc -pthread -shared build/temp.linux-i686-2.7/src/_fastmath.o -lgmp -o build/lib.linux-i686-2.7/Crypto/PublicKey/_fastmath.so
building 'Crypto.Hash._MD2' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/home/qrtt1/app/include/python2.7 -c src/MD2.c -o build/temp.linux-i686-2.7/src/MD2.o
gcc -pthread -shared build/temp.linux-i686-2.7/src/MD2.o -o build/lib.linux-i686-2.7/Crypto/Hash/_MD2.so
building 'Crypto.Hash._MD4' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/home/qrtt1/app/include/python2.7 -c src/MD4.c -o build/temp.linux-i686-2.7/src/MD4.o
gcc -pthread -shared build/temp.linux-i686-2.7/src/MD4.o -o build/lib.linux-i686-2.
How do I assign the CFLAGS and LDFLAGS correctly for building pycrypto ?
I try to download pycrypto-2.5 and install it:
(py27)[qrtt1#hcservice pycrypto-2.5]$ CFLGAS=-I/home/qrtt1/app/include LDFLGAS=-L/home/qrtt1/app/lib python setup.py install
No CFLAGS or LDFLAGS set up with it. May be the pycrypto-2.5 going wrong ?
Please check what you have typed :
CFLAGS=-I/home/qrtt1/app/include LDFLAGS=-L/home/qrtt1/app/lib pip install pycrypto
it should be CFLAGS