We Keep Coding

PyBind11: simple invoke task, standard C++ library not found

i am trying to wrap the C++ library PyrochloreAFM.hpp which itself uses the lib boost/randoom.hpp so that i can import it as a python module. The C++ part itself works fine and i can succesfully import and use all of this from my main.
#include "PyrochloreAFM.hpp"
int main (int argc, const char *argv[]) {
PyrochloreAFM pyrochloreAFM (&parameters, &statistics, &rng);
}
Now following a tutorial i set up my c++ wrapper:
// pybind11_wrapper.cpp
#include <pybind11/pybind11.h>
#include <PyrochloreAFM.hpp>
PYBIND11_MODULE(pybind11_example, m) {
m.doc() = "pybind11 example plugin"; // Optional module docstring
m.def("cpp_function", &PyrochloreAFM, "A function that multiplies two numbers");
}
and my tasks.py file
# tasks.py
import invoke
invoke.run(
"g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC PyrochloreAFM.cpp "
"-o libpyro.so "
)
However now $invoke build-PyrochloreAFM or even $invoke --list seem to have lost the track of the standard C++ library.
In file included from PyrochloreAFM.cpp:1:
./Parameters.hpp:16:10: fatal error: 'boost/random.hpp' file not found
#include "boost/random.hpp" // tested with boost 1.53
^~~~~~~~~~~~~~~~~~
1 error generated.
This might be a simple PATH problem so i would be very glad for any tips!
Thank you, Andres!

"Of course" the tasks.py file which compiles the .cpp file through the python module invoke has to instruct the compiler g++ to use the libraries we desire. In my case it is the flag -I /opt/homebrew/Cellar/boost/1.80.0/include i also use in my makefile or directly in terminal.

Problem with interface Python and Cpp with OpenCV and Cuda

I'm facing an issue while interfacing python and cpp. I need to compute an optical flow with the GPU. To do so, I'm using the tvl1 algorithm with opencv on with cpp and cuda.
My cpp code is working well with GPU and I now want to call the function in my python code because there isn't a way to compute optical flow on GPU with this algo on Python.
I'm using ctypes to make the link beetwin cpp and python. Here are my codes :
python
from ctypes import cdll
opencvfunc = cdll.LoadLibrary('opt_flow_lib/test.so')
cpp
//test.cpp
#define DLLEXPORT extern "C"
#include "opencv2/xfeatures2d.hpp"
#include "opencv2/cudaarithm.hpp"
#include "opencv2/cudaoptflow.hpp"
#include "opencv2/cudacodec.hpp"
#include "opencv2/video/tracking.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/core/mat.hpp"
using namespace cv::cuda;
using namespace cv;
DLLEXPORT int tvl(int a, int b) {
int bound = 20;
Size new_size(224, 224);
Mat curr_gray, prev_gray, prev_image, curr_image;
Mat flow_x, flow_y;
GpuMat d_frame_0, d_frame_1;
GpuMat d_flow;
cv::Ptr<cuda::OpticalFlowDual_TVL1> alg_tvl1 =
cuda::OpticalFlowDual_TVL1::create();
return 3;
}
Here is how I build my cpp file :
gcc -o test.so -shared -fPIC test.cpp `pkg-config opencv --cflags --libs`
Here is what I have when I launch my python code :
OSError: opt_flow_lib/test.so: undefined symbol: _ZN2cv4cuda20OpticalFlowDual_TVL16createEdddiididdb
When I remove this line from cpp :
cv::Ptr<cuda::OpticalFlowDual_TVL1> alg_tvl1 =
cuda::OpticalFlowDual_TVL1::create();
Everything is going ok, I have my 3 int returned. But it seems that cuda doesn't want to do its job
Is someone who had ever turned opencv with cuda with ctypes and cpp/python could hepl me ?
Thanks you
EDIT :
Maybe it could be interesting to have my $ pkg-config opencv --cflags --libs :
-I/usr/local/include/opencv4/opencv2 -I/usr/local/include/opencv4 -L/usr/local/lib -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_ml -lopencv_video -lopencv_xfeatures2d -lopencv_calib3d -lopencv_objdetect

BoostPython and CMake

I have successfully followed this example for how to connect C++ and python. It works fine when I use the given Makefile. When I try to use cmake instead, it does not go as well.
C++ Code:
#include <boost/python.hpp>
#include <iostream>
extern "C"
char const* greet()
{
return "hello, world";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
int main(){
std::cout<<greet()<<std::endl;
return 0;
}
Makefile:
# location of the Python header files
PYTHON_VERSION = 27
PYTHON_DOT_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_DOT_VERSION)
# location of the Boost Python include files and library
BOOST_INC = /usr/include
BOOST_LIB = /usr/lib/x86_64-linux-gnu/
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python-py$(PYTHON_VERSION) -L/usr/lib/python$(PYTHON_DOT_VERSION)/config-x86_64-linux-gnu -lpython$(PYTHON_DOT_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).cpp
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).cpp
When I compile this I get a .so file that can be included in the script
import sys
sys.path.append('/home/myname/Code/Trunk/TestBoostPython/build/')
import libhello_ext_lib as hello_ext
print(hello_ext.greet())
I really want to use cmake instead of a manually written Makefile so I wrote this:
cmake_minimum_required(VERSION 3.6)
PROJECT(hello_ext)
# Find Boost
find_package(Boost REQUIRED COMPONENTS python-py27)
set(PYTHON_DOT_VERSION 2.7)
set(PYTHON_INCLUDE /usr/include/python2.7)
set(PYTHON_LIBRARY /usr/lib/python2.7/config-x86_64-linux-gnu)
include_directories(${PYTHON_INCLUDE} ${Boost_INCLUDE_DIRS})
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -lrt -O3")
SET(EXECUTABLE_OUTPUT_PATH ${PROJECT_SOURCE_DIR}/bin)
SET(LIBNAME hello_ext_lib)
add_library(${LIBNAME} SHARED src/hello_ext.cpp)
add_executable(${PROJECT_NAME} src/hello_ext.cpp)
TARGET_LINK_LIBRARIES(${PROJECT_NAME} ${Boost_LIBRARIES} -lpython2.7 -fPIC)
TARGET_LINK_LIBRARIES(${LIBNAME} ${Boost_LIBRARIES} -lpython2.7 -fPIC -shared)
Here I currently type the Python-paths by hand but I have also tried using fin_package(PythonLibs) without success.
The program compiles fine and executes when I run the executable file in ../bin/. However, when I run the python script I get always:
ImportError: dynamic module does not define init function (initlibhello_ext_lib)
I found this which says this can happen if the lib and the executable have different names. Which indeed is the case, but how can I obtain the .so with correct name?
I also tried to not compile the executable but only the library. That did also not work.

BOOST_PYTHON_MODULE(hello_ext) creates an init function "inithello_ext", which should correspond to a module "hello_ext". But you are trying to import "libhello_ext_lib".
Give the module the same name as the filename. E.g. BOOST_PYTHON_MODULE(libhello_ext_lib).

SWIG Python undefined symbol error

I'm trying to create a *.so file for further use in Python using SWIG, but something isn't working.
I have two files:
DataGatherer.h
#include <iostream>
#include <stdlib.h>
#include <time.h>
#include "gnublin.h"
#include <pthread.h>
class dataGatherer
{
private:
int threshold;
int timeThreshold;
int data[4096];
bool running;
gnublin_spi* spiDevice;
pthread_t spiThread;
void *params;
public:
dataGatherer(void);
dataGatherer(int, int);
void initData();
int getThreshold(void);
int* getData(void);
int getPeak(void);
void initSPI(void);
void gatherData();
void * run(void * arg);
void stop(void);
// for use of thread we have to implement some methods from C
static void * start_static(void * params)
{
dataGatherer * thread_this = static_cast<dataGatherer*>(params);
return thread_this->run(thread_this->params);
}
void start(void * params)
{
this->params = params;
pthread_create(&spiThread, 0, &dataGatherer::start_static, this);
}
};
and spiController.h
#include "dataGatherer.h"
class spiController
{
private:
bool runGather;
dataGatherer* gatherer;
int data[4096];
public:
spiController(void);
spiController(int, int);
void initData();
bool checkStop();
void stop();
void start();
};
My spiController.i interface file looks like this:
/* spiController.i */
%module spiController
%{
#include "dataGatherer.h"
#include "spiController.h"
#include "gnublin.h"
#include <stdlib.h>
#include <time.h>
#include <pthread.h>
extern void initData();
extern bool checkStop();
extern void stop();
extern void start();
%}
extern void initData();
extern bool checkStop();
extern void stop();
extern void start();
At the end I try to create the *.so file using commands in the terminal like in the example on the SWIG page with:
swig -python -c++ spiController.i
c++ -c spiController_wrap.c -I/usr/include/python2.7
c++ -shared spiController_wrap.o -o _spiController.so
*.cxx, *.o and *.so file are created with no error, but when I import the spiController into the python code I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "spiController.py", line 26, in <module>
_spiController = swig_import_helper()
File "spiController.py", line 22, in swig_import_helper
_mod = imp.load_module('_spiController', fp, pathname, description)
ImportError: ./_spiController.so: undefined symbol: _Z9checkStopv
It's my first try using SWIG and I'm already stuck at this point. How can I resolve this?

I just got the same error and finally figured out why. As above people said, when it says unfound symbol like yours and gives the undefined function name '_Z9checkStopv', always check for the implementation of this function in .cpp file as well as any function declaration of the same name!!
For my case, my cpp does define my 'unfound symbol' constructor function, but in my .h file, i have an overloaded operator= (for the constructor) which is undefined in .cpp file. So swig wraps both default constructor(implemented in .cpp) and operator= (not implemented). Therefore when import, this unimplemented operator= produces the error. Hope this helps!

You must link the library that defines your C++ functions that you have declared like checkStop etc. You would add -L<path to your C++ DLL> -l<name of your C++ DLL> on 3rd line of your example's compile steps.
Like this:
c++ -L<path to DLL> -l<name of your dll> -shared spiController_wrap.o -o _spiController.so

Just as Adam's comment and my experience, you should first compile your XXX.cpp file into XXX.o, the whole command lines maybe like following:
swig -python -c++ XXX.i
g++ -c -fpic XXX.cpp* (this command will generate XXX.o file)
g++ -c -fpic XXX_wrap.cxx -I/usr/include/python2.7* (this command will generate XXX_wrap.o file)
g++ -shared XXX.o XXX_wrap.o -o XXX.so

Although there may be many causes for this problem, I got the exact same error when I compiled the shared library with the python v3.5 headers, e.g.
swig -python example.i
gcc -fPIC -c example.c example_wrap.c -I/usr/include/python3.5 # <-- ISSUE HERE
gcc -shared example.o example_wrap.o -o _example.so
But then later tried to use the example library using python test.py, which ran python v2.7 on system (so it was a python version mismatch issue).

In my case I was also getting that error, and spent some time trying out stuff without success. My problem was that although my source file was plain C I had it named with the .cpp extension, assuming it wouldn't matter. Changing the extension to .c solved automatically the issue.
Another way of solving it was to add the line #include "example.cpp" to the header section of SWIG's .i file.
So, summarizing:
example.c
int fact(int n) {
if (n <= 1) return 1;
else return n*fact(n-1);
}
example.i
%module example
%{
extern int fact(int n);
%}
extern int fact(int n);
Then the following worked for me (in Ubuntu 17.10):
swig -c++ -python example.i
gcc -fPIC -c example.c example_wrap.c -I /usr/include/python2.7
gcc -shared example.o example_wrap.o -o _example.so
python -c "import example; print example.fact(5)"
Hope this helps someone!
Cheers

I know the solution
At the end of make up the share,you should usr g++ -shared -fpic *.o *.o -o _***.so

Cython: Calling Python code from C program

I am trying to make a Cython wrapper so I can call Python code from C. I am having issues with import as I would like the wrapper to be separate from original code.
Code below ends in segfault when calling imported function. If the code is written as a python module and imported via import the program says that the name ... is not defined. The problem does not exhibit itself when everything is in one file and there's no import involved (indeed code generated by Cython fails when cimporting). The code works fine as well when libcimpy.pyx is imported from other python script (either compiled to .so or live)
I have prepared a minimal example. This is far from actual code but covers the principle.
cimpPy.pyx: Sample python code (converted to Cython)
cdef sum(a, b):
return a + b
cimpPy.pxd
cdef sum(a, b)
libcimpy.pyx (glue Cython code)
cimport cimpPy
cdef public int cSum(int a, int b):
return cimpPy.sum(a, b)
ci.c (c code from which we want to call cimpPy)
#include <stdio.h>
#include <stdlib.h>
#include <Python.h>
#include "libcimp.h"
int main(int argc, char **argv) {
Py_Initialize();
initlibcimp();
int a = 2;
int b = 3;
int c = cSum(a, b);
printf("sum of %d and %d is %d\n", a, b, c);
Py_Finalize();
return 0;
}
Makefile
EXECUTABLE = ci
OBJS = ci.o
CC = gcc
CFLAGS = -g -I/usr/include/python2.7 -I$(shell pwd)
LINKER = g++
LDFLAGS = -L$(shell pwd) $(shell python-config --ldflags) -lcimp
.PHONY: clean cython
all: cython $(EXECUTABLE)
cython:
python setup.py build_ext --inplace
.c.o:
$(CC) $(CFLAGS) -c $<
$(EXECUTABLE) : $(OBJS)
$(LINKER) -o $(EXECUTABLE) $(OBJS) $(LDFLAGS)
clean:
rm -rf *.o *.so libcimp.c libcimp.h core build $(EXECUTABLE)
setup.py
from distutils.core import setup, Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
extensions = [
Extension("libcimp", ["libcimp.pyx"])
]
setup(
name = "CIMP",
cmdclass = {"build_ext": build_ext},
ext_modules = cythonize(extensions)
)
What I intend to achieve is being able to plug Python code into larger C system. The assumption is that users will be able to write Python themselves. The C code is a simulation engine which can operate on agents in a environment. The idea is that the behaviour of agents and environment can be specified in python and passed to the engine for evaluation when necessary. The best analogy would be a map reduce system where Python scripts are mappers. In this sense I want to call Python from C and not the other way round.
Converting everything to Cython, while compelling would be to large undertaking.
Is this the right approach? Why import works only under python interpreter and not when embedded externally? Any suggestions and reference articles or documentation are appreciated.

In this code, the initlibcimp() actually fails, but you don't see it right away because the error is reported by setting a python exception. I'm not 100% sure this is the correct way to do this, but I could see the error by adding the following code below that call:
if (PyErr_Occurred())
{
PyErr_Print();
exit(-1);
}
Then, the program will output:
Traceback (most recent call last):
File "libcimpy.pyx", line 1, in init libcimpy (libcimpy.c:814)
cimport cimpPy
ImportError: No module named cimpPy
The reason that the cimpPy module is not yet defined, is that you need to do a call to initcimpPy() before calling initlibcimp.