I want to wrap a test project containing C++ and OpenMP code with Cython, and build it with distutils via a setup.py file. The content of my file looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
modules = [Extension("Interface",
["Interface.pyx", "Parallel.cpp"],
language = "c++",
extra_compile_args=["-fopenmp"],
extra_link_args=["-fopenmp"])]
for e in modules:
e.cython_directives = {"embedsignature" : True}
setup(name="Interface",
cmdclass={"build_ext": build_ext},
ext_modules=modules)
The -fopenmp flag is used with gcc to compile and link against OpenMP. However, if I just invoke
cls ~/workspace/CythonOpenMP/src $ python3 setup.py build
this flag is not recognized, because the compiler is clang:
running build
running build_ext
skipping 'Interface.cpp' Cython extension (up-to-date)
building 'Interface' extension
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c Interface.cpp -o build/temp.macosx-10.8-x86_64-3.3/Interface.o -fopenmp
clang: warning: argument unused during compilation: '-fopenmp'
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c Parallel.cpp -o build/temp.macosx-10.8-x86_64-3.3/Parallel.o -fopenmp
clang: warning: argument unused during compilation: '-fopenmp'
Parallel.cpp:24:10: warning: unknown pragma ignored [-Wunknown-pragmas]
#pragma omp parallel for
^
1 warning generated.
c++ -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib build/temp.macosx-10.8-x86_64-3.3/Interface.o build/temp.macosx-10.8-x86_64-3.3/Parallel.o -o build/lib.macosx-10.8-x86_64-3.3/Interface.so -fopenmp
ld: library not found for -lgomp
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'c++' failed with exit status 1
I've unsucessfully tried to specify gcc:
cls ~/workspace/CythonOpenMP/src $ python3 setup.py build --compiler=g++-4.7
running build
running build_ext
error: don't know how to compile C/C++ code on platform 'posix' with 'g++-4.7' compiler
How can I tell distutils to use gcc?
Try setting the "CC" environment variable from inside the setup.py with os.environ.
I just took a look at the distutils source, and the --compiler option expects "unix", "msvc", "cygwin", "mingw32", "bcpp", or "emx". It checks the compiler name you want by checking the CC environment variable. Try calling build like this:
CC=gcc python setup.py build
You don't need to set CXX, it doesn't check for that.
Just in case some others are facing the same problem under Windows (where CC environment variable wouldn't have any effect) :
Create file "C:\Python27\Lib\distutils\distutils.cfg" and write this inside :
Code :
[build]
compiler = mingw32
Remove all instances of "-mno-cygwin" gcc option from file "C:\Python27\Lib\distutils\cygwinccompiler.py" :
This :
self.set_executables(compiler='gcc -mno-cygwin -O -Wall',
compiler_so='gcc -mno-cygwin -mdll -O -Wall',
compiler_cxx='g++ -mno-cygwin -O -Wall',
linker_exe='gcc -mno-cygwin',
linker_so='%s -mno-cygwin %s %s'
% (self.linker_dll, shared_option,
entry_point))
Becomes this :
self.set_executables(compiler='gcc -O -Wall',
compiler_so='gcc -mdll -O -Wall',
compiler_cxx='g++ -O -Wall',
linker_exe='gcc',
linker_so='%s %s %s'
% (self.linker_dll, shared_option,
entry_point))
The second point can be necessary in case you are using a recent version of gcc, where the deprecated option -mno-cygwin has been removed.
Hope this will help even if it is not directly related to the OP real needs (but still related to the question's title...)
According to this wiki, Python versions after 3.4 do not support MinGW anymore.
CPython 3.7 for Windows is compiled with MSC v.1916. When I try to use above-mentioned method with distutils.cfg, I then get an error from distutils: Unknown MS Compiler Version 1916. Looks like it has a hardcoded table of msvcr libraries in its cygwincompiler.py file (which is also responsible for MinGW), and last version known to that file is 1600 from VS2010 / MSVC 10.0.
Try this:
http://mail.python.org/pipermail/distutils-sig/2002-August/002944.html
In short, it appears that you should try: python setup.py build --compiler=g++ first.
On linux while using distutils.ccompiler do
os.environ('CC')='gcc' and then call distutils.sysconfig.customize_compiler(compiler)
It will do the job.
Related
I am trying to figure out how to use C/C++ code in python using Cython. I can get the following example working as C code:
sum.h:
#ifndef MY_SUM_H_
#define MY_SUM_H_
int my_sum(int a, int b);
#endif
sum.c:
int my_sum(int a, int b){
return a + b;
}
test.pyx:
cdef extern from "my_sum.h":
cdef int my_sum(int a, int b)
cpdef sum_wrap(int a, int b):
return my_sum(a, b)
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("test", ["test.pyx", "my_sum.c"], language = "c")]
setup(cmdclass = {'build_ext': build_ext}, ext_modules = ext_modules)
However, if I try to test it as C++ code it fails. I rename sum.c to sum.cpp and change language to c++ in setup.py. After that, it looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("test", ["test.pyx", "my_sum.cpp"], language = "c++")]
setup(cmdclass = {'build_ext': build_ext}, ext_modules = ext_modules)
That should be enough right? It produces the following error:
$ python setup.py build_ext --inplace
running build_ext
cythoning test.pyx to test.cpp
/Users/jensrenders/miniconda3/lib/python3.7/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /Users/jensrenders/Dropbox/cython_demo/test.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
building 'test' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/jensrenders/miniconda3/include -arch x86_64 -I/Users/jensrenders/miniconda3/include -arch x86_64 -I/Users/jensrenders/miniconda3/include/python3.7m -c test.cpp -o build/temp.macosx-10.7-x86_64-3.7/test.o
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard
library instead [-Wstdlibcxx-not-found]
1 warning generated.
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/jensrenders/miniconda3/include -arch x86_64 -I/Users/jensrenders/miniconda3/include -arch x86_64 -I/Users/jensrenders/miniconda3/include/python3.7m -c my_sum.cpp -o build/temp.macosx-10.7-x86_64-3.7/my_sum.o
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard
library instead [-Wstdlibcxx-not-found]
1 warning generated.
g++ -bundle -undefined dynamic_lookup -L/Users/jensrenders/miniconda3/lib -arch x86_64 -L/Users/jensrenders/miniconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.7/test.o build/temp.macosx-10.7-x86_64-3.7/my_sum.o -o /Users/jensrenders/Dropbox/cython_demo/test.cpython-37m-darwin.so
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'g++' failed with exit status 1
I do get a test.cpp file as output but no shared object.
What causes this, and how can it be solved?
Thanks for your help!
EDIT:
As #MaximEgorushkin points out, it is strange that cython tries to compile the C++ files with gcc. I can force it to use g++ by adding
os.environ["CC"] = "g++" to setup.py, but this does not solve the problem:
$ python setup.py build_ext --inplace
running build_ext
building 'test' extension
g++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/jensrenders/miniconda3/include -arch x86_64 -I/Users/jensrenders/miniconda3/include -arch x86_64 -I/Users/jensrenders/miniconda3/include/python3.7m -c test.cpp -o build/temp.macosx-10.7-x86_64-3.7/test.o
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard
library instead [-Wstdlibcxx-not-found]
1 warning generated.
g++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/jensrenders/miniconda3/include -arch x86_64 -I/Users/jensrenders/miniconda3/include -arch x86_64 -I/Users/jensrenders/miniconda3/include/python3.7m -c my_sum.cpp -o build/temp.macosx-10.7-x86_64-3.7/my_sum.o
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard
library instead [-Wstdlibcxx-not-found]
1 warning generated.
g++ -bundle -undefined dynamic_lookup -L/Users/jensrenders/miniconda3/lib -arch x86_64 -L/Users/jensrenders/miniconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.7/test.o build/temp.macosx-10.7-x86_64-3.7/my_sum.o -o /Users/jensrenders/Dropbox/cython_demo/test.cpython-37m-darwin.so
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'g++' failed with exit status 1
Here is a similar problem:
https://github.com/pandas-dev/pandas/issues/23424
As they suggest, and as indicated by the line in the output
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
adding extra_link_args=["-stdlib=libc++", "-mmacosx-version-min=10.9"] solves the problem. setup.py then looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("test",
sources=["test.pyx", "my_sum.cpp"],
language="c++",
extra_link_args=["-stdlib=libc++", "-mmacosx-version-min=10.9"])]
setup(cmdclass = {'build_ext': build_ext}, ext_modules = ext_modules)
I'm trying to do a malware analysis by using cuckoo sandbox and a VM Machine (WinXP) running on VirtualBox. But however, I can't get cuckoo to run because I haven't installed cybox and maec correctly.
Thus leads me to this problem where I can't get myself to install lxml.
It just won't let me install either by using pip or manual installation with the setup.py file.
Here is what I got:
Building lxml version 3.4.1.
Building without Cython.
Using build configuration of libxslt 1.1.26
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
warnings.warn(msg)
running install
running bdist_egg
running egg_info
writing requirements to src/lxml.egg-info/requires.txt
writing src/lxml.egg-info/PKG-INFO
writing top-level names to src/lxml.egg-info/top_level.txt
writing dependency_links to src/lxml.egg-info/dependency_links.txt
reading manifest file 'src/lxml.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'src/lxml.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.8-intel/egg
running install_lib
running build_py
copying src/lxml/includes/lxml-version.h -> build/lib.macosx-10.8-intel-2.7/lxml/includes
running build_ext
building 'lxml.etree' extension
clang -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -I/usr/include/libxml2 -I/Users/ajprameswari/Downloads/lxml-3.4.1/src/lxml/includes -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.8-intel-2.7/src/lxml/lxml.etree.o -w -flat_namespace
clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
error: command 'clang' failed with exit status 1
I tried both lxml-3.4.0 and lxml-3.4.1 version but they gave the same result.
I'm using Python 2.7 and OS X 10.8.5. Is there anything that could be pointed to help me solve this issue? I'm a newbie in using OS X, I used to work on my Ubuntu, but due to lack spec of my Ubuntu machine I need to work here.
The Apple LLVM compiler in Xcode 5.1 treats unrecognized command-line options as errors. This issue has been seen when building Python native extensions, where some invalid compiler options are specified.
It seems that the newer version of the llvm compiler shipping is a little more restrictive when it comes to warnings.
Fix:
There is a temporary solution to tell the compiler not to raise this error by setting the following environment variables :
sudo -E export CFLAGS=-Qunused-arguments
sudo -E export CPPFLAGS=-Qunused-arguments
I'm trying to compile a Python package which has some Cython-generated C code, and I'm running into:
gcc: error: x86_64: No such file or directory
which indicates that the gcc compiler is too recent, so doesn't support -arch anymore. I tried setting CC=/usr/bin/gcc prior to python setup.py install, and this works for the main compile command, but not for the command to make the shared object library:
% setenv CC /usr/bin/gcc
% python setup.py install
running install
running build
running build_py
running build_ext
skipping 'hyperion/util/integrate_core.c' Cython extension (up-to-date)
building 'hyperion.util.integrate_core' extension
/usr/bin/gcc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.5.sdk -DNDEBUG -g -O3 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.5.sdk -I/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/numpy/core/include -I/Library/Frameworks/EPD64.framework/Versions/7.2/include/python2.7 -c hyperion/util/integrate_core.c -o build/temp.macosx-10.5-x86_64-2.7/hyperion/util/integrate_core.o
gcc -bundle -undefined dynamic_lookup -g -arch x86_64 -arch x86_64 build/temp.macosx-10.5-x86_64-2.7/hyperion/util/integrate_core.o -o build/lib.macosx-10.5-x86_64-2.7/hyperion/util/integrate_core.so
gcc: error: x86_64: No such file or directory
gcc: error: x86_64: No such file or directory
gcc: error: unrecognized option ‘-arch’
gcc: error: unrecognized option ‘-arch’
error: command 'gcc' failed with exit status 1
Is there a way to specify the absolute path of the compiler to use for linking?
Use LDSHARED variable: LDSHARED='/usr/bin/gcc -shared'.
It's buried in sysconfig module. Try this:
>>> from sysconfig import get_config_vars
>>> get_config_vars('CC', 'CXX', 'LDSHARED')
I've done it like so:
CXX="/Developer/usr/bin/g++-4.0" CC="/Developer/usr/bin/gcc-4.0" python setup.py build
but I'm not sure that's "right"; it's just part of a long command I need to use when installing scipy/numpy.
(full command I end up using is
LD_LIBRARY_PATH=/Developer/SDKs/MacOSX10.6.sdk/usr/lib/ FFLAGS="-m64" CFLAGS="-arch x86_64 -I/Developer/SDKs/MacOSX10.6.sdk/usr/lib/gcc/i686-apple-darwin10/4.0.1/ -I/usr/local/include/freetype2 -I/usr/X11/include -L/usr/X11/lib" LDFLAGS="-Wall -undefined dynamic_lookup -bundle -lpng -arch x86_64" CXX="/Developer/usr/bin/g++-4.0" CC="/Developer/usr/bin/gcc-4.0" python setup.py build)
I run Ubuntu 11.10, gcc version 4.6.1.
I've been trying to install psyco, but keep getting this error:
PROCESSOR = 'i386'
running build
running build_py
running build_ext
building 'psyco._psyco' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DALL_STATIC=1 -Ic/i386 -I/usr/include/python2.7 -c c/psyco.c -o build/temp.linux-i686-2.7/c/psyco.o
In file included from c/initialize.h:55:0,
from c/psyco.c:14:
c/mergepoints.c:242:3: error: ‘JUMP_IF_FALSE’ undeclared here (not in a function)
c/mergepoints.c:242:3: error: ‘JUMP_IF_TRUE’ undeclared here (not in a function)
c/codegen.c:127:19: warning: ‘psyco_source_condition’ defined but not used [-Wunused-function]
c/codegen.c:747:10: warning: ‘integer_lshift’ defined but not used [-Wunused-function]
c/Objects/plistobject.c:115:10: warning: ‘PsycoList_SingletonNew’ defined but not used [-Wunused-function]
error: command 'gcc' failed with exit status 1
I've installed python-setuptools but nothing worked so far. Thanks for a ny suggestions.
Google is your friend: psyco does not build and does not work with Python 2.7, as documented in the corresponding Ubuntu bug report.
Why do you require Psyco? If you need to optimize tight loops, possibly Cython can come to your rescue.
maybe you should get "https://github.com/develersrl/gccwinbinaries" install path :C:/mingw and Installation path ：C:/mingw/bin .Finally, find distutils.cfg at the python installation directory D:/Python26/Lib/distutils
First of all I am very new to all this.
I recently upgraded to Snow Leopard and installed the Xcode + iPhone dev package, 3.1.2.
I went on to install the Django framework + MYSQLDB handler. During the build stage, the terminal shows me the gcc exit status 1 error. But I have the Xcode already installed?
Where am I going wrong?
Also while trying to fix things up, I installed the Xcode that came with the Snow Leopard DVD. still the same error.
Now I think I should remove the Xcode completely and do a fresh install?
Is the ver.3.1.2 specific to 32-bit? Please help me out.
Here is the complete error:
Amit-Vermas-MacBook:mysql-python-1.2.2 amitverma$ gcc-4.0
i686-apple-darwin10-gcc-4.0.1: no input files
Amit-Vermas-MacBook:mysql-python-1.2.2 amitverma$ python setup.py build
running build
running build_py
copying MySQLdb/release.py -> build/lib.macosx-10.3-i386-2.5/MySQLdb
running build_ext
building '_mysql' extension
gcc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Dversion_info=(1,2,2,'final',0) -D__version__=1.2.2 -I/usr/local/mysql/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c _mysql.c -o build/temp.macosx-10.3-i386-2.5/_mysql.o -g -Os -arch x86_64 -fno-common -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT -DDONT_DECLARE_CXA_PURE_VIRTUAL
cc1: error: unrecognized command line option "-Wno-long-double"
error: command 'gcc' failed with exit status 1
I am not sure if you solved your problem. I got the exact same issue after I upgraded from Mac OS 10.4 to 10.6.
Following some blog posts, on a hunch, I ran setup.py pointing to an older gcc version (gcc-4.0).
CC='/usr/bin/gcc-4.0' python setup.py build
The build encountered few warnings.
running build
running build_py
copying MySQLdb/release.py -> build/lib.macosx-10.3-i386-2.5/MySQLdb
running build_ext
building '_mysql' extension
/usr/bin/gcc-4.0 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Dversion_info=(1,2,3,'gamma',1) -D__version__=1.2.3c1 -I/usr/local/mysql/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c _mysql.c -o build/temp.macosx-10.3-i386-2.5/_mysql.o -g -Os -arch i386 -fno-common -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT -DDONT_DECLARE_CXA_PURE_VIRTUAL
In file included from _mysql.c:36:
/usr/local/mysql/include/my_config.h:1050:1: warning: "HAVE_WCSCOLL" redefined
In file included from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:8,
from pymemcompat.h:10,
from _mysql.c:29:
/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyconfig.h:724:1: warning: this is the location of the previous definition
gcc -arch i386 -arch ppc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g -bundle -undefined dynamic_lookup build/temp.macosx-10.3-i386-2.5/_mysql.o -L/usr/local/mysql/lib -lmysqlclient_r -lz -lm -lmygcc -o build/lib.macosx-10.3-i386-2.5/_mysql.so
ld: warning: in build/temp.macosx-10.3-i386-2.5/_mysql.o, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmysqlclient_r.dylib, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmygcc.a, file is not of required architecture
Ran the same command again.
Everything works now magically. The install also went through. The module works as expected.
It normally also shows why a compilation fails. Could you give the text preceding "gcc exited with code 1" or something like that? And perhaps the source file also.
Also look for other threads on this subject, like this one. Good search terms on google or stack overflow are leopard and MySQLdb.
If it helps any, I solved this issue with sym links, and I think it will work for you. I wrote this with my version of gcc in mind, which is 4.2:
cd /usr/bin
rm cc gcc c++ g++
ln -s gcc-4.2 cc
ln -s gcc-4.2 gcc
ln -s c++-4.2 c++
ln -s g++-4.2 g++
ln -s gcc-4.2 gcc-4.0
There ya go!