Disabling Compiler Optimizations when Building CPython - python

I've been following along with Philip Guo's excellent series about CPython's internals and have been walking through the interpreter's code with GDB. Unfortunately, GDB seems to skip around quite a bit when I step through the code, executing instructions out of order (a series of assignment operations for example, nothing fancy).
From what I understand, this is because of compiler optimizations. Indeed, CPython's configure file seems to default to -O2. This is shown to be the case when compiling Python 2.7.15 with ./configure and make:
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c
...
I tried fixing this by configuring without optimizations via ./configure CFLAGS="-g -O0" and compiling with make again, which works (note the change from -fno-strict-aliasing -g -O2 to -fno-strict-aliasing -g -O0 on each line):
gcc -pthread -c -fno-strict-aliasing -g -O0 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c
gcc -pthread -c -fno-strict-aliasing -g -O0 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c
gcc -pthread -c -fno-strict-aliasing -g -O0 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c
gcc -pthread -c -fno-strict-aliasing -g -O0 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c
...
Unfortunately, GDB still jumps around when I step through the code. For completeness, here's the command I'm running with GDB:
gdb --args ./python test.py
The contents of test.py are just some basic arithmetic and print functions, which isn't really relevant.
My understanding of at least one of these steps must be incomplete. Am I setting the optimization flag incorrectly? Is GDB jumping between instructions due to some other reason? Any advice would be greatly appreciated. Thanks!


tl;dr: You want the configure setting --with-pydebug.
You've got an -O3 on the command line that comes after the -O2, so changing the -O2 to -O0 doesn't do any good; the -O3 still overrides it. You need to figure out where the -O3 is coming from and remove it.
If you look at the generated Makefile, you should see something like this:
# Compiler options
OPT= -DNDEBUG -g -fwrapv -O3 -Wall
# ...
# Avoid assigning CFLAGS, LDFLAGS, etc. so users can use them on the
# command line to append to these values without stomping the pre-set
# values.
PY_CFLAGS= $(BASECFLAGS) $(OPT) $(CONFIGURE_CFLAGS) $(CFLAGS) $(EXTRA_CFLAGS)
You can, of course, just edit the generated Makefile to change the -O3 to -O0. But if you look inside the configure script and Makefile.pre, you can see where this comes from: it's the default value, only replaced in this case:
case $ac_cv_prog_cc_g in
yes)
if test "$Py_DEBUG" = 'true' ; then
# Optimization messes up debuggers, so turn it off for
# debug builds.
if "$CC" -v --help 2>/dev/null |grep -- -Og > /dev/null; then
OPT="-g -Og -Wall"
else
OPT="-g -O0 -Wall"
fi
else
OPT="-g $WRAP -O3 -Wall"
fi
;;
This also sets the right flags for other compilers besides gcc.
However, the main effect of --with-pydebug is to enable Py_DEBUG—which you probably want if you're source-debugging CPython, but it is a separate thing from -O0, which is what you were actually asking about. So, if you want only -O0, as far as I know, the only thing you can do is edit the Makefile (or come up with some complicated set of env variables and configure arguments that tricks it into doing a debug build but then not enabling Py_DEBUG).

Related

g++ doesn't support "-ffast-math -march=native" arguments when I try to run python2 file in conda environment

I am trying to run a GitHub repository that is about SNNs. But, when I run the python script, I get a compiler error from the weave package.
Link: https://github.com/djsaunde/lm-snn/blob/master/code/train/csnn_pc_mnist.py
The code generates neuron groups and after that tried to apply STDP backpropagation algorith to them but brian.STDP object couldn't be generated beacuse of the compiler error.
Error
weave.build_tools.CompileError: error: Command "/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/bin/x86_64-conda_cos6-linux-gnu-c++ -fno-strict-aliasing -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -DNDEBUG -fwrapv -O3 -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/include -fPIC -I/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave -I/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave/scxx -I/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/numpy/core/include -I/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/include/python2.7 -c /home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave/scxx/weave_imp.cpp -o /tmp/weave-mzp7-UMTCud/python27_intermediate/compiler_7d6a1f80b8499f1aba77652aa8d9d6d7/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave/scxx/weave_imp.o -MMD -MF /tmp/weave-mzp7-UMTCud/python27_intermediate/compiler_7d6a1f80b8499f1aba77652aa8d9d6d7/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave/scxx/weave_imp.o.d -O3 -ffast-math -march=native" failed with exit status 1
x86_64-conda_cos6-linux-gnu-c++: error: unrecognized command line option '-ffast-math -march=native'; did you mean '-fsso-struct=native'?
I use the miniconda2 environment.
Conda package: libgcc
Pip packages:
numpy
matplotlib
scipy
brian
weave
sympy
networkx
pandas
sklearn
brian2
Thank you.

set compile flags when installing Python + C++ project

When pip-installing a Python project with C++ parts,
pip3 install . --verbose --user
the typical compile line will be something like
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/eigen3/ -I/usr/local/include/python3.7 -I/home/nschloe/.local/include/python3.7m -I/usr/include/python3.7m -c src/generate.cpp -o build/temp.linux-x86_64-3.7/src/generate.o
Explicitly setting CFLAGS like
OPT="" CFLAGS="" pip3 install . --verbose --user
results in
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/eigen3/ -I/usr/local/include/python3.7 -I/home/nschloe/.local/include/python3.7m -I/usr/include/python3.7m -c src/generate.cpp -o build/temp.linux-x86_64-3.7/src/generate.o
retaining most of the flags.
Question: Where are all the compiler options set? More specifically: How can I remove -g? (It's eating too much memory.)

No rule to make target `libpython2.6.a'

I am trying to compile Python2.6 from its source codes, and took steps as follow, and encountered an error message:
./configure
...
[root#KuGouIdc Python-2.6.6]# ls
config.log Demo Makefile.pre Objects Python
config.status Doc Makefile.pre.in Parser README
configure Grammar Makefile.pre.in.cflags PC RISCOS
configure.in Include Makefile.pre.in.fix-parallel-make PCbuild setup.py
configure.in.check-for-XML_SetHashSalt install-sh Makefile.pre.in.lib64 pyconfig.h setup.py.add-RPATH-to-pyexpat
configure.in.disable-pymalloc-on-valgrind Lib Makefile.pre.in.no-static-lib pyconfig.h.in setup.py.expat
configure.in.expat LICENSE Makefile.pre.in.systemtap pyconfig.h.in.disable-pymalloc-on-valgrind setup.py.lib64
configure.in.rpath Mac Misc pyconfig.h.in.systemtap systemtap-example.stp
configure.in.systemtap Makefile Modules pyfuntop.stp Tools
[root#KuGouIdc Python-2.6.6]# make -n all
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c
make: *** No rule to make target `libpython2.6.a', needed by `python'. Stop.
I reviewed the Makefile detail, and really could not find any rule to make the target 'libpythonXXX.a'. I googled around, and it seemed that no one had encountered the same issue as me?
In the Makefile, it really has a rule for making the target 'libpythonXXX.so'.

Works for me
$ docker run --rm -it debian:stretch
# mkdir /src
# cd /src
# apt update
# apt install wget build-essential
# wget https://www.python.org/ftp/python/2.6/Python-2.6.tgz
# tar xzf Python-2.6.tgz
# cd Python-2.6
# ./configure
...
# make -n all
gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c
gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c
gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c
gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c
gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/node.o Parser/node.c
...

Really appriciating replies from #Jean-François Fabre and #Messa.
I repeated the steps of #Messa, and it really worked fine. So I tried to diff the 'Makefile.pre.in' from https://www.python.org/ftp/python/2.6/Python-2.6.tgz with the one I downloaded and rpminstalled from http://vault.centos.org/6.9/os/Source/SPackages/python-2.6.6-66.el6_8.src.rpm, and I found that the latter one had missing following pieces:
# Build static library
# avoid long command lines, same as LIBRARY_OBJS
$(LIBRARY): $(LIBRARY_OBJS)
-rm -f $#
$(AR) cr $# Modules/getbuildinfo.o
$(AR) cr $# $(PARSER_OBJS)
$(AR) cr $# $(OBJECT_OBJS)
$(AR) cr $# $(PYTHON_OBJS)
$(AR) cr $# $(MODULE_OBJS) $(SIGNAL_OBJS)
$(AR) cr $# $(MODOBJS)
$(RANLIB) $#
That's the direct cause of the Make error message. Since I took following steps to attain the Python2.6 source code:
wget http://vault.centos.org/6.9/os/Source/SPackages/python-2.6.6-66.el6_8.src.rpm
rpm -i python-2.6.6-66.el6_8.src.rpm
cd ~/rpmbuild/SPECS
rpmbuild -bp --nodeps python.spec
...
Patch #111 (python-2.6.2-no-static-lib.patch):
...
It was the 'python-2.6.2-no-static-lib.patch' that had removed those pieces!

aarch64-poky-linux-gcc: error: : No such file or directory

I am trying to include mlpack to my yocto build. Without enabling the python bindings my recipe bakes successfully.
But while building the python bindings I get the following error:
aarch64-poky-linux-gcc -march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -O2 -pipe -g -feliminate-unused-debug-types -fPIC
-I/builds/build-h3/tmp/sysroots/x86_64-linux/usr/lib/python2.7/site-packages/numpy/core/include
-I/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/
-I/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/git/src
-I/builds/build-h3/tmp/sysroots/h3ulcb/usr/include
-I/builds/build-h3/tmp/sysroots/h3ulcb/usr/include
-I/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/..
-I/builds/build-h3/tmp/sysroots/h3ulcb/usr/include/python2.7 -c /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/sparse_coding.cpp
-o build/temp.linux-x86_64-2.7/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/sparse_coding.o
-DBINDING_TYPE=BINDING_TYPE_PYX -std=c++11 -march=armv8-a
-mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wall
-Wextra -ftemplate-depth=1000 -O3 -fopenmp aarch64-poky-linux-gcc: error: : No such file or directory aarch64-poky-linux-gcc: error: : No such file or directory error: command 'aarch64-poky-linux-gcc' failed with exit status 1 src/mlpack/methods/sparse_coding/CMakeFiles/build_pyx_sparse_coding.dir/build.make:60: recipe for target 'src/mlpack/methods/sparse_coding/CMakeFiles/build_pyx_sparse_coding' failed make[2]: *** [src/mlpack/methods/sparse_coding/CMakeFiles/build_pyx_sparse_coding] Error 1 make[2]: Leaving directory '/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build' CMakeFiles/Makefile2:6651: recipe for target 'src/mlpack/methods/sparse_coding/CMakeFiles/build_pyx_sparse_coding.dir/all' failed make[1]: *** [src/mlpack/methods/sparse_coding/CMakeFiles/build_pyx_sparse_coding.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... /builds/build-h3/tmp/sysroots/x86_64-linux/usr/lib/python2.7/site-packages/setuptools-36.2.7-py2.7.egg/setuptools/dist.py:350: UserWarning: The version specified ('3.0.x') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details. running build_ext cythoning /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/preprocess_split.pyx to /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/preprocess_split.cpp building 'mlpack.preprocess_split' extension aarch64-poky-linux-gcc
-march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -O2 -pipe -g -feliminate-unused-debug-types -fPIC
-I/builds/build-h3/tmp/sysroots/x86_64-linux/usr/lib/python2.7/site-packages/numpy/core/include
-I/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/
-I/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/git/src
-I/builds/build-h3/tmp/sysroots/h3ulcb/usr/include
-I/builds/build-h3/tmp/sysroots/h3ulcb/usr/include
-I/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/..
-I/builds/build-h3/tmp/sysroots/h3ulcb/usr/include/python2.7 -c /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/preprocess_split.cpp
-o build/temp.linux-x86_64-2.7/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/preprocess_split.o
-DBINDING_TYPE=BINDING_TYPE_PYX -std=c++11 -march=armv8-a
-mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wall
-Wextra -ftemplate-depth=1000 -O3 -fopenmp aarch64-poky-linux-gcc: error: : No such file or directory aarch64-poky-linux-gcc: error: : No such file or directory error: command 'aarch64-poky-linux-gcc' failed with exit status 1 src/mlpack/methods/preprocess/CMakeFiles/build_pyx_preprocess_split.dir/build.make:60: recipe for target 'src/mlpack/methods/preprocess/CMakeFiles/build_pyx_preprocess_split' failed make[2]: *** [src/mlpack/methods/preprocess/CMakeFiles/build_pyx_preprocess_split] Error 1 make[2]: Leaving directory '/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build' CMakeFiles/Makefile2:1387: recipe for target 'src/mlpack/methods/preprocess/CMakeFiles/build_pyx_preprocess_split.dir/all' failed make[1]: *** [src/mlpack/methods/preprocess/CMakeFiles/build_pyx_preprocess_split.dir/all] Error 2 [ 89%] Linking CXX executable ../../../../bin/generate_pyx_softmax_regression cd /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/methods/softmax_regression && /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/cmake -E cmake_link_script CMakeFiles/generate_pyx_softmax_regression.dir/link.txt --verbose=1 /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/aarch64-poky-linux/aarch64-poky-linux-g++
-march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wall
-Wextra -ftemplate-depth=1000 -O3 -fopenmp -O2 -pipe -g
-feliminate-unused-debug-types -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -DNDEBUG
-march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wl,-O1
-Wl,--hash-style=gnu -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu
-Wl,--as-needed CMakeFiles/generate_pyx_softmax_regression.dir/__/__/bindings/python/generate_pyx_softmax_regression.cpp.o CMakeFiles/generate_pyx_softmax_regression.dir/__/__/bindings/python/print_pyx.cpp.o
-o ../../../../bin/generate_pyx_softmax_regression -rdynamic ../../../../lib/libmlpack.so.2.0 -lrt -larmadillo
-lboost_program_options-mt -lboost_unit_test_framework
-lboost_serialization-mt cd /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/methods/softmax_regression && /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/cmake
-DGENERATE_PYX_PROGRAM=/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_softmax_regression
-DPYX_OUTPUT_FILE=/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/softmax_regression.pyx
-P /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/git/CMake/GeneratePYX.cmake /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_softmax_regression: /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_softmax_regression: cannot execute binary file make[2]: Leaving directory '/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build' [ 89%] Built target generate_pyx_softmax_regression [ 89%] Linking CXX executable ../../../../bin/generate_pyx_knn cd /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/methods/neighbor_search && /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/cmake -E cmake_link_script CMakeFiles/generate_pyx_knn.dir/link.txt --verbose=1 /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/aarch64-poky-linux/aarch64-poky-linux-g++
-march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wall
-Wextra -ftemplate-depth=1000 -O3 -fopenmp -O2 -pipe -g
-feliminate-unused-debug-types -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -DNDEBUG
-march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wl,-O1
-Wl,--hash-style=gnu -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu
-Wl,--as-needed CMakeFiles/generate_pyx_knn.dir/__/__/bindings/python/generate_pyx_knn.cpp.o CMakeFiles/generate_pyx_knn.dir/__/__/bindings/python/print_pyx.cpp.o
-o ../../../../bin/generate_pyx_knn -rdynamic ../../../../lib/libmlpack.so.2.0 -lrt -larmadillo
-lboost_program_options-mt -lboost_unit_test_framework
-lboost_serialization-mt cd /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/methods/neighbor_search && /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/cmake
-DGENERATE_PYX_PROGRAM=/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_knn
-DPYX_OUTPUT_FILE=/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/knn.pyx
-P /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/git/CMake/GeneratePYX.cmake /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_knn: /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_knn: cannot execute binary file make[2]: Leaving directory '/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build' [ 89%] Built target generate_pyx_knn [ 89%] Linking CXX executable ../../../../bin/generate_pyx_kfn cd /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/methods/neighbor_search && /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/cmake -E cmake_link_script CMakeFiles/generate_pyx_kfn.dir/link.txt --verbose=1 /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/aarch64-poky-linux/aarch64-poky-linux-g++
-march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wall
-Wextra -ftemplate-depth=1000 -O3 -fopenmp -O2 -pipe -g
-feliminate-unused-debug-types -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -DNDEBUG
-march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wl,-O1
-Wl,--hash-style=gnu -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu
-Wl,--as-needed CMakeFiles/generate_pyx_kfn.dir/__/__/bindings/python/generate_pyx_kfn.cpp.o CMakeFiles/generate_pyx_kfn.dir/__/__/bindings/python/print_pyx.cpp.o
-o ../../../../bin/generate_pyx_kfn -rdynamic ../../../../lib/libmlpack.so.2.0 -lrt -larmadillo
-lboost_program_options-mt -lboost_unit_test_framework
-lboost_serialization-mt cd /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/methods/neighbor_search && /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/cmake
-DGENERATE_PYX_PROGRAM=/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_kfn
-DPYX_OUTPUT_FILE=/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/kfn.pyx
-P /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/git/CMake/GeneratePYX.cmake /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_kfn: /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_kfn: cannot execute binary file make[2]: Leaving directory '/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build' [ 89%] Built target generate_pyx_kfn [ 89%] Linking CXX executable ../../../../bin/generate_pyx_krann cd /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/methods/rann && /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/cmake -E cmake_link_script CMakeFiles/generate_pyx_krann.dir/link.txt
--verbose=1 /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/aarch64-poky-linux/aarch64-poky-linux-g++
-march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wall
-Wextra -ftemplate-depth=1000 -O3 -fopenmp -O2 -pipe -g
-feliminate-unused-debug-types -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -DNDEBUG
-march=armv8-a -mtune=cortex-a57.cortex-a53
--sysroot=/builds/build-h3/tmp/sysroots/h3ulcb -O2 -pipe -g
-feliminate-unused-debug-types -fvisibility-inlines-hidden -Wl,-O1
-Wl,--hash-style=gnu -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu
-Wl,--as-needed CMakeFiles/generate_pyx_krann.dir/__/__/bindings/python/generate_pyx_krann.cpp.o CMakeFiles/generate_pyx_krann.dir/__/__/bindings/python/print_pyx.cpp.o
-o ../../../../bin/generate_pyx_krann -rdynamic ../../../../lib/libmlpack.so.2.0 -lrt -larmadillo
-lboost_program_options-mt -lboost_unit_test_framework
-lboost_serialization-mt cd /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/methods/rann && /builds/build-h3/tmp/sysroots/x86_64-linux/usr/bin/cmake
-DGENERATE_PYX_PROGRAM=/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_krann
-DPYX_OUTPUT_FILE=/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/src/mlpack/bindings/python/mlpack/krann.pyx
-P /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/git/CMake/GeneratePYX.cmake /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_krann: /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build/bin/generate_pyx_krann: cannot execute binary file make[2]: Leaving directory '/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build' [ 89%] Built target generate_pyx_krann make[1]: Leaving directory '/builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/build' Makefile:141: recipe for target 'all' failed make: *** [all] Error 2 ERROR: oe_runmake failed ERROR: Function failed: do_compile (log file is located at /builds/build-h3/tmp/work/aarch64-poky-linux/mlpack/git-r0/temp/log.do_compile.17361)
My recipe looks like this:
DEPENDS = "boost armadillo python-native python-setuptools-native
python-pandas-native python-numpy-native python-cython-native
python-pytest-runner-native doxygen-native"
LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=3cee6555b8c975c14d29687fe7e78038"
SRC_URI = "git://github.com/mlpack/mlpack.git;protocol=https \
"
SRCREV = "b8ee319818e6213da4b92cb69dab94b956795401"
S = "${WORKDIR}/git"
inherit cmake pythonnative
EXTRA_OECMAKE = "-DDEBUG:BOOL=OFF \
-DPROFILE:BOOL=OFF \
-DBUILD_TESTS:BOOL=OFF \
-DCMAKE_BUILD_TYPE:STRING=Release \
-DCMAKE_SKIP_INSTALL_RPATH:BOOL=YES \
-DCMAKE_SKIP_RPATH:BOOL=YES \
-DPY_PANDAS:PATH=${STAGING_LIBDIR_NATIVE}/python2.7/site-packages/pandas \
"
do_compile_prepend() {
export STAGING_LIBDIR="${STAGING_LIBDIR}"
export STAGING_INCDIR="${STAGING_INCDIR}"
}
Is there something else that needs to be included in my recipe?
Or is the build environment missing something?
Any pointers would really help.

mlpack builds the Python bindings in multiple steps:
build a generate_pyx_<name> program to generate the Cython .pyx file that will be compiled
run the generate_pyx_<name> program and save the output .pyx file
use Cython and setuptools to generate a .cpp file from the .pyx file then compile it
It's a bit of a roundabout process, but I believe what is happening here is that the generate_pyx_<name> programs are being compiled for the device instead of the host, and therefore can't be run to generate the .pyx files. There won't be a particularly easy way to fix this, unfortunately, because CMake will be compiling the generate_pyx_<name> target with the same options you are compiling the rest of the library with (which in your case targets the device).
In order to make this work, you'll need to modify src/mlpack/bindings/python/CMakeLists.txt, specifically the line
add_executable(generate_pyx_${name}
${CMAKE_BINARY_DIR}/src/mlpack/bindings/python/generate_pyx_${name}.cpp
${CMAKE_SOURCE_DIR}/src/mlpack/bindings/python/print_pyx.hpp
${CMAKE_SOURCE_DIR}/src/mlpack/bindings/python/print_pyx.cpp)
and related lines. In essence you will need to make sure that the CXXFLAGS like -march=armv8-a -mtune=cortex-a57.cortex-a53 do not get used for that particular target.

Run "ldd aarch64-poky-linux-gcc" first, check if all the libraries it linked with were existed in your machine, if not the system will tell you "No such file or directory".

How to set CFLAGS and LDFLAGS to compile pycrypto

I am trying to install the fabric library to an old machine. There are some legacy libraries in /usr/lib, such as libgmp.
(py27)[qrtt1#hcservice app]$ ls /usr/lib|grep gmp
libgmp.a
libgmp.so
libgmp.so.3
libgmp.so.3.3.3
libgmpxx.a
libgmpxx.so
libgmpxx.so.3
libgmpxx.so.3.0.5
I have compiled the libgmp 5.x in my $HOME/app, and then am trying to install pycrypto (it is the dependency of fab):
CFLGAS=-I/home/qrtt1/app/include LDFLGAS=-L/home/qrtt1/app/lib pip install pycrypto
I observed that none of my include or lib directories are in the in the compilation / linking options:
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/usr/include/ -I/home/qrtt1/app/include/python2.7 -c src/_fastmath.c -o build/temp.linux-i686-2.7/src/_fastmath.o
gcc -pthread -shared build/temp.linux-i686-2.7/src/_fastmath.o -lgmp -o build/lib.linux-i686-2.7/Crypto/PublicKey/_fastmath.so
building 'Crypto.Hash._MD2' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/home/qrtt1/app/include/python2.7 -c src/MD2.c -o build/temp.linux-i686-2.7/src/MD2.o
gcc -pthread -shared build/temp.linux-i686-2.7/src/MD2.o -o build/lib.linux-i686-2.7/Crypto/Hash/_MD2.so
building 'Crypto.Hash._MD4' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/home/qrtt1/app/include/python2.7 -c src/MD4.c -o build/temp.linux-i686-2.7/src/MD4.o
gcc -pthread -shared build/temp.linux-i686-2.7/src/MD4.o -o build/lib.linux-i686-2.
How do I assign the CFLAGS and LDFLAGS correctly for building pycrypto ?
I try to download pycrypto-2.5 and install it:
(py27)[qrtt1#hcservice pycrypto-2.5]$ CFLGAS=-I/home/qrtt1/app/include LDFLGAS=-L/home/qrtt1/app/lib python setup.py install
No CFLAGS or LDFLAGS set up with it. May be the pycrypto-2.5 going wrong ?

Please check what you have typed :
CFLAGS=-I/home/qrtt1/app/include LDFLAGS=-L/home/qrtt1/app/lib pip install pycrypto
it should be CFLAGS

Categories