We Keep Coding

PyBind11: simple invoke task, standard C++ library not found

i am trying to wrap the C++ library PyrochloreAFM.hpp which itself uses the lib boost/randoom.hpp so that i can import it as a python module. The C++ part itself works fine and i can succesfully import and use all of this from my main.
#include "PyrochloreAFM.hpp"
int main (int argc, const char *argv[]) {
PyrochloreAFM pyrochloreAFM (&parameters, &statistics, &rng);
}
Now following a tutorial i set up my c++ wrapper:
// pybind11_wrapper.cpp
#include <pybind11/pybind11.h>
#include <PyrochloreAFM.hpp>
PYBIND11_MODULE(pybind11_example, m) {
m.doc() = "pybind11 example plugin"; // Optional module docstring
m.def("cpp_function", &PyrochloreAFM, "A function that multiplies two numbers");
}
and my tasks.py file
# tasks.py
import invoke
invoke.run(
"g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC PyrochloreAFM.cpp "
"-o libpyro.so "
)
However now $invoke build-PyrochloreAFM or even $invoke --list seem to have lost the track of the standard C++ library.
In file included from PyrochloreAFM.cpp:1:
./Parameters.hpp:16:10: fatal error: 'boost/random.hpp' file not found
#include "boost/random.hpp" // tested with boost 1.53
^~~~~~~~~~~~~~~~~~
1 error generated.
This might be a simple PATH problem so i would be very glad for any tips!
Thank you, Andres!

"Of course" the tasks.py file which compiles the .cpp file through the python module invoke has to instruct the compiler g++ to use the libraries we desire. In my case it is the flag -I /opt/homebrew/Cellar/boost/1.80.0/include i also use in my makefile or directly in terminal.

How to wrap C++ VTK custom function for Python

Is there any place I can find a working example to wrap a C++ function (using VTK) to be used by
Python?
Searched and found this page
mix VTK and SWIG Python
but there is no compiling details, and this simple example on this page
https://public.kitware.com/pipermail/vtkusers/2002-October/013980.html
But for my system Ubuntu 16.04, vtk 8.2.0, Python 3.6.8, I could not make the example work.
I have to modify them to let the compiling done without error.
Details:
#============simple.cpp===================
// simple.c
//based on https://public.kitware.com/pipermail/vtkusers/2002-October/013980.html
#include "Python.h"
#include "vtkPythonUtil.h"
#include "vtkObject.h"
#include "vtkObjectBase.h"
#include <stdio.h>
#include "vtkRenderer.h"
extern "C"
{
static PyObject* simple_vtk_function(PyObject *self, PyObject *args);
}
static PyObject* simple_vtk_function(PyObject *self, PyObject *args)
{
PyObject *vtkpy;
if (!PyArg_ParseTuple(args, "O", &vtkpy))
return NULL;
vtkObjectBase *obj = vtkPythonUtil::GetPointerFromObject(vtkpy,"PyObject");
/* Do what you want to do here. */
//printf("%s\n", obj->GetClassName());
printf("inside module simple_vtk, function simple_vtk_function\n");
Py_RETURN_NONE;
}
static char simple_docs[] =
"simple( ): Any message you want to put here!!\n";
static PyMethodDef module_methods[] =
{
{
"simple_vtk_function", (PyCFunction) simple_vtk_function,
METH_VARARGS, simple_docs
},
{NULL}
};
static struct PyModuleDef simple_vtk =
{
PyModuleDef_HEAD_INIT,
"simple_vtk",/* name of module */
"",/* module documentation, may be NULL */
-1,
module_methods
};
PyMODINIT_FUNC PyInit_simple_vtk(void)
{
return PyModule_Create(&simple_vtk);
}
#============setup.py===================
#!/usr/bin/env python3
import distutils.core as dc
module = dc.Extension('simple_vtk',
sources = ['simple.cpp'],
include_dirs=['/usr/local/include/vtk-8.2'],
libraries_dirs = ['/usr/local/include/vtk-8.2'],
#libraries=['vtkPythonUtil']
)
dc.setup(name = 'simple_vtk',
version = '1.0',
ext_modules = [module])
compile command
CFLAGS="-std=c++11" python setup.py build
and get this printing out
/home/jun/anaconda3/lib/python3.6/distutils/extension.py:131: UserWarning: Unknown Extension options: 'libraries_dirs'
warnings.warn(msg)
running build
running build_ext
building 'simple_vtk' extension
gcc -pthread -B /home/jun/anaconda3/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c++11 -fPIC -I/usr/local/include/vtk-8.2 -I/home/jun/anaconda3/include/python3.6m -c simple.cpp -o build/temp.linux-x86_64-3.6/simple.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
simple.cpp: In function ‘PyObject* simple_vtk_function(PyObject*, PyObject*)’:
simple.cpp:22:19: warning: unused variable ‘obj’ [-Wunused-variable]
vtkObjectBase *obj = vtkPythonUtil::GetPointerFromObject(vtkpy,"PyObject");
^
g++ -pthread -shared -B /home/jun/anaconda3/compiler_compat -L/home/jun/anaconda3/lib -Wl,-rpath=/home/jun/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ -std=c++11 build/temp.linux-x86_64-3.6/simple.o -o build/lib.linux-x86_64-3.6/simple_vtk.cpython-36m-x86_64-linux-gnu.so
to test in python
>>> import simple_vtk
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /home/jun/project/play/swig/for_vtk_v2/simple_vtk.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN13vtkPythonUtil20GetPointerFromObjectEP7_objectPKc
The C++ function is so simple, it just to convert a PyObject and convert to vtkObjectBase. I didn't even remove the simple print
printf("%s\n", obj->GetClassName());
But I could never make it work. I think it is a common need for people to wrap a custom C++ vtk function for Python, a little bit surpring there is almost no woring example on internet.
Could you please help me on this? Thanks!

BoostPython and CMake

I have successfully followed this example for how to connect C++ and python. It works fine when I use the given Makefile. When I try to use cmake instead, it does not go as well.
C++ Code:
#include <boost/python.hpp>
#include <iostream>
extern "C"
char const* greet()
{
return "hello, world";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
int main(){
std::cout<<greet()<<std::endl;
return 0;
}
Makefile:
# location of the Python header files
PYTHON_VERSION = 27
PYTHON_DOT_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_DOT_VERSION)
# location of the Boost Python include files and library
BOOST_INC = /usr/include
BOOST_LIB = /usr/lib/x86_64-linux-gnu/
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python-py$(PYTHON_VERSION) -L/usr/lib/python$(PYTHON_DOT_VERSION)/config-x86_64-linux-gnu -lpython$(PYTHON_DOT_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).cpp
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).cpp
When I compile this I get a .so file that can be included in the script
import sys
sys.path.append('/home/myname/Code/Trunk/TestBoostPython/build/')
import libhello_ext_lib as hello_ext
print(hello_ext.greet())
I really want to use cmake instead of a manually written Makefile so I wrote this:
cmake_minimum_required(VERSION 3.6)
PROJECT(hello_ext)
# Find Boost
find_package(Boost REQUIRED COMPONENTS python-py27)
set(PYTHON_DOT_VERSION 2.7)
set(PYTHON_INCLUDE /usr/include/python2.7)
set(PYTHON_LIBRARY /usr/lib/python2.7/config-x86_64-linux-gnu)
include_directories(${PYTHON_INCLUDE} ${Boost_INCLUDE_DIRS})
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -lrt -O3")
SET(EXECUTABLE_OUTPUT_PATH ${PROJECT_SOURCE_DIR}/bin)
SET(LIBNAME hello_ext_lib)
add_library(${LIBNAME} SHARED src/hello_ext.cpp)
add_executable(${PROJECT_NAME} src/hello_ext.cpp)
TARGET_LINK_LIBRARIES(${PROJECT_NAME} ${Boost_LIBRARIES} -lpython2.7 -fPIC)
TARGET_LINK_LIBRARIES(${LIBNAME} ${Boost_LIBRARIES} -lpython2.7 -fPIC -shared)
Here I currently type the Python-paths by hand but I have also tried using fin_package(PythonLibs) without success.
The program compiles fine and executes when I run the executable file in ../bin/. However, when I run the python script I get always:
ImportError: dynamic module does not define init function (initlibhello_ext_lib)
I found this which says this can happen if the lib and the executable have different names. Which indeed is the case, but how can I obtain the .so with correct name?
I also tried to not compile the executable but only the library. That did also not work.

BOOST_PYTHON_MODULE(hello_ext) creates an init function "inithello_ext", which should correspond to a module "hello_ext". But you are trying to import "libhello_ext_lib".
Give the module the same name as the filename. E.g. BOOST_PYTHON_MODULE(libhello_ext_lib).

failing to import a python module generated by swig into C++

I used swig to generate a python module that is wrapping up some C++ code:
my .i file :
%module module_test
%{
#define SWIG_FILE_WITH_INIT
#include "headers.h"
%}
%include "headers.h"
I ran these commands :
swig -c++ -python swig.i
swig -Wall -c++ -python -external-runtime runtime_swig.h
After building, this generated a module_test.py and a _module_test.pyd files.
The wrapper works fine in python.
Now, from another C++ project I'm trying to load this new python module, using the following code:
Py_Initialize();
PySys_SetPath(L"path_to_my_files");
PyObject * pName = PyString_FromString("module_test.py");
PyObject *module = PyImport_Import(pName);
if(module == NULL)
{
PyErr_Print();
std::cout << "module == NULL !!";
exit(-1);
}
However the import fails with the error :
import imp
ImportError: No module named 'imp'
Can someone please tell me what's going on ? The module_test.py file generated by swig is doing an "import imp" in the code, but this import never failed when I run the file from python...
Do I need to specify something else so that the C++ code knows what "imp" is ?
thanks!

Solved it by using instead :
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
PyList_Append(path, PyString_FromString("path_to_file"));

Compile c++ Programm with CMake that uses Boost/python.hpp

I'm currently trying to compile a c++ file using CMake.
But since I'm using Boost::python it won't compile.
I set up a little test file to figure out what I need to do but I just can't get it to work.
Any Help would be greatly appreciated!!
The test file:
#include <Python.h>
#include <boost/python.hpp>
#include <iostream>
using std::cout;
using std::endl;
int main()
{
namespace py = boost::python;
Py_Initialize();
// Retrieve the main module's namespace
py::object global(py::import("__main__").attr("__dict__"));
py::exec("print 'Hello from Python!' \n", global, global);
return 0;
}
It will compile if I just use,
clang++ -I/usr/include/python2.7 -lpython2.7 -lboost_python -std=c++11 boosttest.cpp -o boosttest
I tried this CMakeLists.txt to get it to work.
cmake_minimum_required(VERSION 3.2)
FIND_PACKAGE(PythonLibs)
FIND_PACKAGE(Boost)
INCLUDE_DIRECTORIES(${Boost_INCLUDE_DIRS} ${PYTHON_INCLUDE_DIRS})
LINK_LIBRARIES(${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
add_executable(Test1 boosttest.cpp)
target_link_libraries(Test1 ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
and what I get is
undefined reference to `boost::python::import(boost::python::str)'
and a couple more of the same category.

Thanks for your help Mark, thanks to the new errors after I included
find_package(Boost REQUIRED python)
I was able to figure out that the problem was that CMake selected the libs for python 3.4 but Boost was build against 2.7.
So the Solution was to include the version as so:
FIND_PACKAGE(PythonLibs 2.7 REQUIRED)

Did you try
find_package(Boost REQUIRED python)
also run with verbosity to see what is going on
cmake . --debug-output
make VERBOSE=1