I have exactly copied the example code given in the Cython documentation for wrapping C++ classes. I can successfully build and import the rect.so extension using distutils and the cythonize() method, i.e. by:
Putting the following directives at the top of rect.pyx:
# distutils: language = c++
# distutils: sources = Rectangle.cpp
Writing a setup.py file that contains this:
from distutils.core import setup
from Cython.Build import cythonize
setup(
name = "rectangleapp",
ext_modules = cythonize('*.pyx'),
)
Calling
$ python setup.py build_ext --inplace
However, when I'm wrapping C code in Cython I often find it more convenient to compile individual extensions manually from the command line, i.e.:
Generate the .c code using the command line Cython compiler
$ cython foo.pyx
Manually compile it using gcc:
$ gcc -shared -fPIC -O3 -I /usr/lib/python2.7 -L /usr/lib/python2.7 \
foo.c -lpython2.7 -o foo.so
I've tried applying the same process to build the rect.so example above:
$ cython --cplus rect.pyx
$ g++ -shared -fPIC -O3 -I /usr/lib/python2.7 -L /usr/lib/python2.7 \
rect.cpp -lpython2.7 -o rect.so
Both the Cython and g++ compilation steps seem to succeed - I don't get any command line output, and at the end I have a rect.so built. However, when I then try to import the module I get an undefined symbol error:
In [1]: import rect
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-ba16f97c2145> in <module>()
----> 1 import rect
ImportError: ./rect.so: undefined symbol: _ZN6shapes9Rectangle9getLengthEv
What's the correct procedure for manually compiling Cython code that wraps C++ classes?
The problem here is that you said that somewhere you will provide the definition of a class called Rectangle -- where the example code states
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass Rectangle:
...
However, when you compiled the library you didn't provide the code for Rectangle, or a library that contained it, so rect.so has no idea where to find this Rectangle class.
To run your code you must first create the Rectangle object file.
gcc -c Rectangle.cpp # creates a file called Rectangle.o
Now you can either create a library to dynamically link against, or statically link the object file into rect.so. I'll cover statically linking first as it's simplest.
gcc -shared -fPIC -I /usr/include/python2.7 rect.cpp Rectangle.o -o rect.so
Note that I haven't included the library for python. This is because you expect your library to be loaded by the python interpreter, thus the python libraries will already be loaded by the process when your library is loaded. In addition to providing rect.cpp as a source I also provide Rectangle.o. So lets try running a program using your module.
run.py
import rect
print(rect.PyRectangle(0, 0, 1, 2).getLength())
Unfortunately, this produces another error:
ImportError: /home/user/rectangle/rect.so undefined symbol: _ZTINSt8ios_base7failureE
This is because cython needs the c++ standard library, but python hasn't loaded it. You can fix this by adding the c++ standard library to the required libraries for rect.so
gcc -shared -fPIC -I/usr/include/python2.7 rect.cpp Rectangle.o -lstdc++ \
-o rect.so
Run run.py again and all should work. However, the code for rect.so is larger than it needs to be, especially if you produce multiple libraries that depend on the same code. You can dynamically link the Rectangle code, by making it a library as well.
gcc -shared -fPIC Rectangle.o -o libRectangle.so
gcc -shared -fPIC -I/usr/include/python2.7 -L. rect.cpp -lRectangle -lstdc++ \
-o rect.so
We compile the Rectangle code into a shared library in the current directory and provide -L. so gcc knows to look for libraries in the current directory and -lRectangle so gcc knows to look for the Rectangle library. Finally, to be able to run your code you must tell python where the Rectangle library lives. Before running python enter
export LD_LIBRARY_PATH="/home/user/rectangle" # where libRectangle.so lives
You can use a shell script to make sure this is done every time before you run your program, but it makes things messier. Best to just stick with statically linking Rectangle.
Related
This question is related to a couple of questions in SO like this one and this other one. Unfortunately the solution there is not working for me so far. I have a module.pxc file which I am compiling by a setup.py file, such as the following one:
# setup.py
module_extension = Extension(
name="iolif",
sources=["/home/maurizio/Ongoing.Projects/c_libraries/dcomplex_libc.c",
"/home/maurizio/Ongoing.Projects/c_libraries/special_functions_libc.c",
"/home/maurizio/Ongoing.Projects/c_libraries/models/freq_cv_libc.c",
"module.pyx"],
libraries=['gsl', 'gslcblas', 'm'],
# library_dirs=["lib"],
include_dirs=["/home/maurizio/Ongoing.Projects/pycustommodules",
"/home/maurizio/Ongoing.Projects/c_libraries",
"/home/maurizio/Ongoing.Projects/c_libraries/models"]
)
setup(
name="iolif",
ext_modules=cythonize([module_extension])
)
From command line, in the same directory of module.pxc, when writing python setup.py build_ext --inplace compilation works fine, and the iolif.so library is produced. The issue is that I can only import this library if I use Python2.7, whereas if I attempt to import it in Python3.x I get the known ImportError: dynamic module does not define module export function (PyInit_iolif).
Googling around, and as pointed out in the two questions linked above, it seems that this is due to the fact that cython is looking at Python2.7 rather than Python3.x (which is the one I work with instead). Accordingly, I attempted asking cythonize in my setup.py to use Python3.x by:
...
setup(
name="iolif",
ext_modules=cythonize([module_extension],
compiler_directives={'language_level': "3"})
)
but it still does not work. The last compilation message indeed produces:
gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -
specs=/usr/lib/rpm/redhat/redhat-hardened-ld build/temp.linux-x86_64-2.7/pylif_io.o
build/temp.linux-x86_64-2.7/home/maurizio/Ongoing.Projects/c_libraries/dcomplex_libc.o
build/temp.linux-x86_64-2.7/home/maurizio/Ongoing.Projects/c_libraries/special_functions_libc.o build/temp.linux-x86_64-2.7/home/maurizio/Ongoing.Projects/c_libraries/models/freq_cv_libc.o -L/usr/lib64
-lgsl -lgslcblas -lm -lpython2.7 -o /home/maurizio/Ongoing.Projects/DePitta.PNAS/Software/LIF.Analysis/iolif.so
where you can see that it is still linking with the -lpython2.7 library (whereas it should use for example -lpython3.7m). How do I solve it? What am I missing?
Easy solution. My python command was still associated with python2.7 (I recently moved to Python3.x). Sorry about it. Hence:
python3 setup.py build_ext --inplace
will make the trick. Indeed compilation now reads:
gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g build/temp.linux-x86_64-3.7/pylif_io.o build/temp.linux-x86_64-3.7/home/maurizio/Ongoing.Projects/c_libraries/dcomplex_libc.o build/temp.linux-x86_64-3.7/home/maurizio/Ongoing.Projects/c_libraries/special_functions_libc.o build/temp.linux-x86_64-3.7/home/maurizio/Ongoing.Projects/c_libraries/models/freq_cv_libc.o -L/usr/lib64 -lgsl -lgslcblas -lm -lpython3.7m -o /home/maurizio/Ongoing.Projects/DePitta.PNAS/Software/LIF.Analysis/iolif.cpython-37m-x86_64-linux-gnu.so
as required.
I am trying to create .so file of a c++ file(twocams.cpp) which includes main() and another C++ file (say abc.h). abc.c includes opencv. while creating an object using ctypes,
g++ -fPIC -shared twocams.cpp -o twocams.so
Test = ctypes.cdll.LoadLibrary('/home/administrator/Desktop/project/twocams/twocams.so')
i am getting error as:-
undefined symbol: _ZN2cv12VideoCapturersERNS_3MatE
How to solve this problem? I want to use the c++ code in python. Any alternative is acceptable.
When you create a shared object and want to use it, you have to link your artifact against all dependencies of this shared object. E.g. I create a libtwocams.so of
#include <opencv2/videoio.hpp>
void test() {
cv::VideoCapture v;
cv::Mat m;
v >> m;
}
To use this shared object I have to link against libopencv_core, libopencv_videoio, libopencv_imgproc, libopencv_imgcodecs, libz and some more. I compile my program with
g++ main.cpp -o main -ltwocams -lopencv_core -lopencv_imgproc -lopencv_imgcodecs -lopencv_videoio -lz -lwebp -lpthread -ltiff -lpng
Another solution is to link the shared object against the dependent libraries. E.g.
g++ -fPIC -shared twocams.cpp -o -ltwocams -lopencv_core -lopencv_imgproc -lopencv_imgcodecs -lopencv_videoio -lz -lwebp -lpthread -ltiff -lpng
creates a shared library that makes the loader load all dependencies. You can check the dependencies with ldd.
The order of the libraries is important. If libA has a dependency to libB then you have to link against libA and then libB.
If you use opencv's shared library then you don't need to link all other dependencies.
Here is a step by step manual:
Install conan
Install cmake
Add repository bincrafters to conan
conan remote add bincrafters https://api.bintray.com/conan/bincrafters/public-conan
Create:
conanfile.txt
CMakeLists.txt
src/twocams.cpp
build/
conanfile.txt:
[requires]
opencv/3.4.2#bincrafters/stable
[generators]
cmake
[options]
opencv:shared=True
CMakeLists.txt:
cmake_minimum_required(VERSION 3.12)
project(twocams)
include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup()
add_library(twocams SHARED src/twocams.cpp src/twocams.h)
target_link_libraries(twocams ${CONAN_LIBS})
Go to build and install dependencies:
cd build
conan install .. --build missing
Build project with cmake:
cmake ..
cmake --build .
I had a similar issue, giving me an
OSError: /lib64/libarmadillo.so.9: undefined symbol: H5Ovisit
in python, when calling
libCustCv = ctypes.cdll.LoadLibrary('./../so/opencv_cust.so')
To resolve this, I did the following:
In the cpp files, I replaced the general import #include <opencv2/opencv.hpp> by the specific ones: #include <opencv2/core/core.hpp> and #include <opencv2/imgproc/imgproc.hpp>
Instead of giving the bash-output of pkg-config --libs --cflags opencv as arguments of the compilation command (which gave quite a list), I just used the specific flags:
g++ -fPIC -shared -I/usr/include/opencv4 -lopencv_imgproc -lopencv_core -Wall -Wl,-soname,opencv_cust.so -o opencv_cust.so opencv_cust.cpp
Cython code can be compiled as
cython hello.pyx -o hello.c
gcc -shared -fPIC hello.c -o hello.so `pkg-config python --libs --cflags`
But if you change hello.so above to hello2.so, the module will fail to import in python due to assumptions that a function inithello2 will exist in a python module hello2.so. With distutils, you can change the name using a keyword to setup(name='hello2', ...). But I need to get this functionality without using distutils. Can it be done?
I am trying to get cython to realize I have a c compiler in MinGW 32-bit and I've tried everything I can find on the web but it's still not working. I am running Windows 7 Professional 64-bit. Here is what I have tried:
(1) I have Python 2.7 and I just installed MinGW with options gcc and g++ and some other options
(2) I edited the PATH environmental variable so it includes
C:\MinGW\bin;C:\MinGW\MSYS\1.0\local\bin;C:\MinGW\MSYS\1.0\bin
(3) I told Python to use MinGW as the default compiler by creating a file named
C:\Python27\Lib\distutils\distutils.cfg, containing
[build]
compiler = mingw32
(I do have MinGW32 by the way)
(4) I removed all instances of -mno-cygwin from the file C:\Python27\Lib\distutils\cygwincompiler.py
(5) I have a file called setup.py and a module called tryingcython.pyx that is written in python. My setup.py says
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext':build_ext},
ext_modules=[Extension("tryingcython",["tryingcython.pyx"])]
)
So then I open Command Prompt and get into the directory that contains setup.py and tryingcython.pyx, and I type
python setup.py build_ext --inplace --compiler=mingw32
Then it tells me:
running build_ext
skipping 'tryingcython.c' Cython extension (up-to-date)
building 'tryingcython.c' extension
gcc -mdll -O -Wall -IC:\Python27\include -IC:\Python27\PC -c tryingcython.c -o build\
temp.win32-2.7\Release\tryingcython.o
error: command 'gcc' failed: No such file or directory
So I guess Cython can't tell that I have gcc and it can't find it or what, even though I've tried about every single piece of advice I can find online for making it realize that I have MinGW which has gcc included.
Any help/additional ideas on how I can get cython to actually work would be much appreciated.
You are using exactly the same operational system and versions than me.
Try to cal gcc using:
SET input=intput.c
SET output=output.pyd
gcc -shared -IC:\Python27\include -LC:\Python27\libs -O2 -o %output% %input% -lpython27
Usually I put this call in a cythongcc.bat file, in a directory recognized by the PATH environment variable as:
gcc -shared -IC:\Python27\include -LC:\Python27\libs -O3 -mtune=native -o %1.pyd %2.c -lpython27
So that I can , from where my cython .pyx files are, just do:
cython input.pyx
cythongcc input input
To get the compiled .pyd working!
I was trying to build this package I wrote (which I know to be working), first in the usual way through distutils:
# python2.7 setup.py build
running build
running build_py
running build_ext
building 'uptime._posix' extension
gcc -fno-strict-aliasing -Wno-error -march=i586 -DHAVE_DB_185_H -I/usr/pkg/include -I/usr/include -DNDEBUG -Wno-error -march=i586 -DHAVE_DB_185_H -I/usr/pkg/include -I/usr/include -I/usr/pkg/include/python2.7 -c src/_posix.c -o build/temp.minix-3-i686-2.7/src/_posix.o
ld -L/usr/tmp/work/lang/python27/work/Python-2.7.2 -lcompat_minix -minlib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/lib -Wl,-R/usr/lib build/temp.minix-3- i686-2.7/src/_posix.o -o build/lib.minix-3-i686-2.7/uptime/_posix.so
ld: unrecognized option '-Wl,-R/usr/pkg/lib'
ld: use the --help option for usage information
build failed: uptime._posix (no big deal)
Well, fine; distutils is one of those modules that often break on less popular platforms. So I tried compiling the extension part manually:
# gcc -fno-strict-aliasing -march=i586 -DNDEBUG -I/usr/pkg/include/python2.7 -fPIC -shared -o _posix.so src/_posix.c
This worked—that is, it produced _posix.so without complaining—but, trying to import the resulting module in Python didn't:
>>> import _posix
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: Service unavailable
Google suggests this is related to the ability of the OS to load shared libraries (which Python extensions are). I went looking, and it turns out MINIX didn't support shared libraries at all until last year, but now it should. In fact, /usr/pkg/include/python2.7/pyconfig.h defines both HAVE_DLOPEN and HAVE_DYNAMIC_LOADING, so it clearly does.
What's going on?
MINIX does support shared libraries now, but the binary Python packages provided by pkgin are statically linked, which prevents them from being able to load shared libraries anyway. The only way around this is to compile Python yourself, ideally through pkgsrc. All pkgsrc packages are built dynamically by default.
(You'll still have to build the extensions manually, too.)