When pip-installing a Python project with C++ parts,
pip3 install . --verbose --user
the typical compile line will be something like
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/eigen3/ -I/usr/local/include/python3.7 -I/home/nschloe/.local/include/python3.7m -I/usr/include/python3.7m -c src/generate.cpp -o build/temp.linux-x86_64-3.7/src/generate.o
Explicitly setting CFLAGS like
OPT="" CFLAGS="" pip3 install . --verbose --user
results in
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/eigen3/ -I/usr/local/include/python3.7 -I/home/nschloe/.local/include/python3.7m -I/usr/include/python3.7m -c src/generate.cpp -o build/temp.linux-x86_64-3.7/src/generate.o
retaining most of the flags.
Question: Where are all the compiler options set? More specifically: How can I remove -g? (It's eating too much memory.)
Related
Ubuntu 20.04 on WSL 2.0 under windows 10 ARM64 (Surface Pro X with WSL2.0).
pip3 install jupyterlab // failure
Building wheels for collected packages: argon2-cffi
Building wheel for argon2-cffi (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3.8 /usr/local/lib/python3.8/dist-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp5k32hshp
cwd: /tmp/pip-install-jo5pljfy/argon2-cffi_f65af512bd344c9b956794545e86c839
Complete output (39 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-aarch64-3.8
creating build/lib.linux-aarch64-3.8/argon2
copying src/argon2/__main__.py -> build/lib.linux-aarch64-3.8/argon2
copying src/argon2/exceptions.py -> build/lib.linux-aarch64-3.8/argon2
copying src/argon2/_password_hasher.py -> build/lib.linux-aarch64-3.8/argon2
copying src/argon2/_ffi_build.py -> build/lib.linux-aarch64-3.8/argon2
copying src/argon2/low_level.py -> build/lib.linux-aarch64-3.8/argon2
copying src/argon2/_utils.py -> build/lib.linux-aarch64-3.8/argon2
copying src/argon2/_legacy.py -> build/lib.linux-aarch64-3.8/argon2
copying src/argon2/__init__.py -> build/lib.linux-aarch64-3.8/argon2
running build_clib
building 'argon2' library
creating build/temp.linux-aarch64-3.8
creating build/temp.linux-aarch64-3.8/extras
creating build/temp.linux-aarch64-3.8/extras/libargon2
creating build/temp.linux-aarch64-3.8/extras/libargon2/src
creating build/temp.linux-aarch64-3.8/extras/libargon2/src/blake2
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iextras/libargon2/src/../include -Iextras/libargon2/src/blake2 -c extras/libargon2/src/argon2.c -o build/temp.linux-aarch64-3.8/extras/libargon2/src/argon2.o
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iextras/libargon2/src/../include -Iextras/libargon2/src/blake2 -c extras/libargon2/src/blake2/blake2b.c -o build/temp.linux-aarch64-3.8/extras/libargon2/src/blake2/blake2b.o
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iextras/libargon2/src/../include -Iextras/libargon2/src/blake2 -c extras/libargon2/src/core.c -o build/temp.linux-aarch64-3.8/extras/libargon2/src/core.o
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iextras/libargon2/src/../include -Iextras/libargon2/src/blake2 -c extras/libargon2/src/encoding.c -o build/temp.linux-aarch64-3.8/extras/libargon2/src/encoding.o
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iextras/libargon2/src/../include -Iextras/libargon2/src/blake2 -c extras/libargon2/src/ref.c -o build/temp.linux-aarch64-3.8/extras/libargon2/src/ref.o
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iextras/libargon2/src/../include -Iextras/libargon2/src/blake2 -c extras/libargon2/src/thread.c -o build/temp.linux-aarch64-3.8/extras/libargon2/src/thread.o
aarch64-linux-gnu-gcc-ar rcs build/temp.linux-aarch64-3.8/libargon2.a build/temp.linux-aarch64-3.8/extras/libargon2/src/argon2.o build/temp.linux-aarch64-3.8/extras/libargon2/src/blake2/blake2b.o build/temp.linux-aarch64-3.8/extras/libargon2/src/core.o build/temp.linux-aarch64-3.8/extras/libargon2/src/encoding.o build/temp.linux-aar aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iextras/libargon2/include -I/usr/include/python3.8 -c build/temp.linux-aarch64-3.8/_ffi.c -o build/temp.linux-aarch64-3.8/build/temp.linux-aarch64-3.8/_ffi.o
build/temp.linux-aarch64-3.8/_ffi.c:50:14: fatal error: pyconfig.h: No such file or directory
50 | # include <pyconfig.h>
| ^~~~~~~~~~~~
compilation terminated.
error: command 'aarch64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for argon2-cffi
Failed to build argon2-cffi
ERROR: Could not build wheels for argon2-cffi which use PEP 517 and cannot be installed directly
How can I build argon2-cffi successfully?
I solved it following these steps:
sudo apt install -y build-essential python3.8-dev // ubuntu 20.04
pip3 install pep517 // or python3.8 -m pip install pep517
pip3 install jupyterlab // successfully
I am trying to run a GitHub repository that is about SNNs. But, when I run the python script, I get a compiler error from the weave package.
Link: https://github.com/djsaunde/lm-snn/blob/master/code/train/csnn_pc_mnist.py
The code generates neuron groups and after that tried to apply STDP backpropagation algorith to them but brian.STDP object couldn't be generated beacuse of the compiler error.
Error
weave.build_tools.CompileError: error: Command "/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/bin/x86_64-conda_cos6-linux-gnu-c++ -fno-strict-aliasing -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -DNDEBUG -fwrapv -O3 -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/include -fPIC -I/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave -I/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave/scxx -I/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/numpy/core/include -I/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/include/python2.7 -c /home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave/scxx/weave_imp.cpp -o /tmp/weave-mzp7-UMTCud/python27_intermediate/compiler_7d6a1f80b8499f1aba77652aa8d9d6d7/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave/scxx/weave_imp.o -MMD -MF /tmp/weave-mzp7-UMTCud/python27_intermediate/compiler_7d6a1f80b8499f1aba77652aa8d9d6d7/home/mzp7/workspace/DSNN/Examples/lm-snn/miniconda2/lib/python2.7/site-packages/weave/scxx/weave_imp.o.d -O3 -ffast-math -march=native" failed with exit status 1
x86_64-conda_cos6-linux-gnu-c++: error: unrecognized command line option '-ffast-math -march=native'; did you mean '-fsso-struct=native'?
I use the miniconda2 environment.
Conda package: libgcc
Pip packages:
numpy
matplotlib
scipy
brian
weave
sympy
networkx
pandas
sklearn
brian2
Thank you.
I've been following along with Philip Guo's excellent series about CPython's internals and have been walking through the interpreter's code with GDB. Unfortunately, GDB seems to skip around quite a bit when I step through the code, executing instructions out of order (a series of assignment operations for example, nothing fancy).
From what I understand, this is because of compiler optimizations. Indeed, CPython's configure file seems to default to -O2. This is shown to be the case when compiling Python 2.7.15 with ./configure and make:
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c
...
I tried fixing this by configuring without optimizations via ./configure CFLAGS="-g -O0" and compiling with make again, which works (note the change from -fno-strict-aliasing -g -O2 to -fno-strict-aliasing -g -O0 on each line):
gcc -pthread -c -fno-strict-aliasing -g -O0 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c
gcc -pthread -c -fno-strict-aliasing -g -O0 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c
gcc -pthread -c -fno-strict-aliasing -g -O0 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c
gcc -pthread -c -fno-strict-aliasing -g -O0 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c
...
Unfortunately, GDB still jumps around when I step through the code. For completeness, here's the command I'm running with GDB:
gdb --args ./python test.py
The contents of test.py are just some basic arithmetic and print functions, which isn't really relevant.
My understanding of at least one of these steps must be incomplete. Am I setting the optimization flag incorrectly? Is GDB jumping between instructions due to some other reason? Any advice would be greatly appreciated. Thanks!
tl;dr: You want the configure setting --with-pydebug.
You've got an -O3 on the command line that comes after the -O2, so changing the -O2 to -O0 doesn't do any good; the -O3 still overrides it. You need to figure out where the -O3 is coming from and remove it.
If you look at the generated Makefile, you should see something like this:
# Compiler options
OPT= -DNDEBUG -g -fwrapv -O3 -Wall
# ...
# Avoid assigning CFLAGS, LDFLAGS, etc. so users can use them on the
# command line to append to these values without stomping the pre-set
# values.
PY_CFLAGS= $(BASECFLAGS) $(OPT) $(CONFIGURE_CFLAGS) $(CFLAGS) $(EXTRA_CFLAGS)
You can, of course, just edit the generated Makefile to change the -O3 to -O0. But if you look inside the configure script and Makefile.pre, you can see where this comes from: it's the default value, only replaced in this case:
case $ac_cv_prog_cc_g in
yes)
if test "$Py_DEBUG" = 'true' ; then
# Optimization messes up debuggers, so turn it off for
# debug builds.
if "$CC" -v --help 2>/dev/null |grep -- -Og > /dev/null; then
OPT="-g -Og -Wall"
else
OPT="-g -O0 -Wall"
fi
else
OPT="-g $WRAP -O3 -Wall"
fi
;;
This also sets the right flags for other compilers besides gcc.
However, the main effect of --with-pydebug is to enable Py_DEBUG—which you probably want if you're source-debugging CPython, but it is a separate thing from -O0, which is what you were actually asking about. So, if you want only -O0, as far as I know, the only thing you can do is edit the Makefile (or come up with some complicated set of env variables and configure arguments that tricks it into doing a debug build but then not enabling Py_DEBUG).
I was trying to build c through python distutils. I want to replace CC with gcc and follow this page
CC=gcc python setup.py build
Then I got
gcc -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c hello.c -o build/temp.linux-x86_64-2.7/hello.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/hello.o -o build/lib.linux-x86_64-2.7/hello.so
CC only changed to gcc on first stage, and it would become original x86_64-linux-gnu-gcc, I am not sure is there anything missing, thanks.
You must set "CC" with
os.environ["CC"] = "GCC"
However, this sometimes does not work in windows and involves some changes in the config files.
Create file "C:\Python27\Lib\distutils\distutils.cfg" and write this inside
Then insert the following code:
[build]
compiler = gcc
Remove all instances of "-mno-cygwin" gcc option from file "C:\Python27\Lib\distutils\cygwinccompiler.py":
In the same file convert this:
self.set_executables(compiler='gcc -mno-cygwin -O -Wall',
compiler_so='gcc -mno-cygwin -mdll -O -Wall',
compiler_cxx='g++ -mno-cygwin -O -Wall',
linker_exe='gcc -mno-cygwin',
linker_so='%s -mno-cygwin %s %s'
% (self.linker_dll, shared_option,
entry_point))
to this:
self.set_executables(compiler='gcc -O -Wall',
compiler_so='gcc -mdll -O -Wall',
compiler_cxx='g++ -O -Wall',
linker_exe='gcc',
linker_so='%s %s %s'
% (self.linker_dll, shared_option,
entry_point))
I found the answer here, LDSHARED is the variable I need.
The full command would be like this
CC=gcc LDSHARED=gcc -pthread -shared python setup.py build
I am trying to install the fabric library to an old machine. There are some legacy libraries in /usr/lib, such as libgmp.
(py27)[qrtt1#hcservice app]$ ls /usr/lib|grep gmp
libgmp.a
libgmp.so
libgmp.so.3
libgmp.so.3.3.3
libgmpxx.a
libgmpxx.so
libgmpxx.so.3
libgmpxx.so.3.0.5
I have compiled the libgmp 5.x in my $HOME/app, and then am trying to install pycrypto (it is the dependency of fab):
CFLGAS=-I/home/qrtt1/app/include LDFLGAS=-L/home/qrtt1/app/lib pip install pycrypto
I observed that none of my include or lib directories are in the in the compilation / linking options:
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/usr/include/ -I/home/qrtt1/app/include/python2.7 -c src/_fastmath.c -o build/temp.linux-i686-2.7/src/_fastmath.o
gcc -pthread -shared build/temp.linux-i686-2.7/src/_fastmath.o -lgmp -o build/lib.linux-i686-2.7/Crypto/PublicKey/_fastmath.so
building 'Crypto.Hash._MD2' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/home/qrtt1/app/include/python2.7 -c src/MD2.c -o build/temp.linux-i686-2.7/src/MD2.o
gcc -pthread -shared build/temp.linux-i686-2.7/src/MD2.o -o build/lib.linux-i686-2.7/Crypto/Hash/_MD2.so
building 'Crypto.Hash._MD4' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/home/qrtt1/app/include/python2.7 -c src/MD4.c -o build/temp.linux-i686-2.7/src/MD4.o
gcc -pthread -shared build/temp.linux-i686-2.7/src/MD4.o -o build/lib.linux-i686-2.
How do I assign the CFLAGS and LDFLAGS correctly for building pycrypto ?
I try to download pycrypto-2.5 and install it:
(py27)[qrtt1#hcservice pycrypto-2.5]$ CFLGAS=-I/home/qrtt1/app/include LDFLGAS=-L/home/qrtt1/app/lib python setup.py install
No CFLAGS or LDFLAGS set up with it. May be the pycrypto-2.5 going wrong ?
Please check what you have typed :
CFLAGS=-I/home/qrtt1/app/include LDFLAGS=-L/home/qrtt1/app/lib pip install pycrypto
it should be CFLAGS