We Keep Coding

How to wrap a C++ class/struct with attributes that are vectors for fluid use in python

I'm wrapping a C++ API for use in Python, so I'd like the functionality of the Python wrapper classes to pretty closely mirror that of the C++ classes. In one case, I have two objects that are actually nested structs:
// myheader.hpp
#include <vector>
namespace mynames{
struct Data{
struct Piece{
double piece1;
int piece2;
std::vector<double> piece3;
};
std::vector<Piece> pieces;
};
}
I'd like to interact with this object fluidly in Python as if it were a typical Python class using numpy and extension types. So I began by declaring two extension types:
# mydeclarations.pxd
from libcpp.vector cimport vector
cdef extern from "myheader.hpp" namespace "mynames":
cdef cppclass Data:
vector[Piece] pieces
cdef extern from "myheader.hpp" namespace "mynames::Data":
cdef cppclass Piece:
double piece1
int piece2
vector[double] piece3
Then wrapping in Python:
# mytypes.pyx
cimport mydeclarations as cpp
from cython cimport NULL
from libcpp.vector cimport vector
import numpy as np
cdef class Piece:
cdef cpp.Piece *_cppPiece
def __cinit__(self):
self._cppPiece = new cpp.Piece()
def __dealloc__(self):
if self._cppPiece is not NULL:
del self._cppPiece
#property
def piece1(self):
return self._cppPiece.piece1
#piece1.setter
def piece1(self, double d):
self._cppPiece.piece1 = d
#property
def piece2(self):
return self._cppPiece.piece2
#piece2.setter
def piece2(self, int i):
self._cppPiece.piece2 = i
# Use cython's automatic type conversion: (cpp)vector <---> (py)list (COPIES)
#property
def piece3(self):
return np.asarray(self._cppPiece.piece3, dtype=np.double)
#piece3.setter
def piece3(self, arr):
self._cppPiece.piece3 = <vector[double]>np.asarray(arr, dtype=np.double)
#----------------------------------------------------------------------
cdef class Data:
cdef cpp.Data *_cppData
def __cinit__(self):
self._cppData = new cpp.Data()
def __dealloc__(self):
if self._cppData is not NULL:
del self._cppData
#property
def pieces(self):
# Create a list of Python objects that hold copies of the C++ data
cdef Piece pyPiece
cdef int i
pyPieces = []
for i in range(self._cppData.pieces.size()):
pyPiece = Piece()
pyPiece._cppPiece[0] = self._cppData.pieces.at(i)
pyPieces.append(pyPiece)
return np.asarray(pyPieces, dtype=Piece)
#pieces.setter
def pieces(self, arr):
# Clear the existing vector and create a new one containing copies of the data in arr
cdef Piece pyPiece
self._cppData.pieces.clear()
for pyPiece in arr:
self._cppData.pieces.push_back(deref(pyPiece._cppPiece))
This is a simple implementation and as far as I can tell it works, but there are some issues:
Since we use copies, there's no in-place functionality that you might expect if, for example, Piece().piece3 was a python class attribute holding a numpy array. For example,
a = Piece()
a.piece3 = [1,2,3]
a.piece3[0] = 55 # No effect, need to do b = a.piece3; b[0]=55; a.piece3=b
There's a lot of iterating over data and copying. This is probably an issue when the size of Data.pieces is very large.
Can anyone suggest some better alternatives to address these issues? Although Data is more complicated that Pieces, I think they are related and ultimately boil down to wrapping C++ classes with vector attributes for use in Python.

If you want to avoid data copying then it probably involves creating a wrapper class.
cdef class DoubleVector:
cdef vector[double] *vec
cdef owner
def __dealloc__(self):
if owner is not None:
del self.vec
#staticmethod
cdef create_from_existing_Piece(Piece obj):
out = DoubleVector()
out.owner = obj
out.vec = obj._cppData.piece3
return out
# create len/__getindex__/__setindex__ functions
# You could also have this class expose the buffer protocol
Here I've assumed that DoubleVector doesn't own its own data the majority of the time. Therefore it keeps a reference to the Python class that owns the C++ class that owns that data (thus ensuring that object lifetimes are preserved).
Some details (mainly creating a nice sequence interface) is left for you to fill in.
Exposing vector[Piece] is more difficult, largely because any changes to the vector (including resizing it) would invalidate any pointers into the vector. Therefore I'd give serious thought to having a different Python interface to C++ interface.
Could you make Data immutable (so that you simply can't change it from Python and so can safely return pointers into it)?
Could you avoid returning Piece from data and have functions like get_ith_piece1, get_ith_piece2, get_ith_piece3 (i.e. remove a layer from your Python wrapping)?
Alternatively you could do something like
cdef class BasePiece:
cdef cpp.Piece* get_piece(self):
raise NotImplementedError
# most of the implementation of your Piece class goes here
cdef class Piece(BasePiece):
# wrapper that owns its own data.
# largely as before but with
cdef cpp.Piece* get_piece(self):
return self._cppPiece
# ...
cdef class UnownedPiece(BasePiece):
cdef Data d
cdef int index
cdef cpp.Piece* get_piece(self):
return self.d._cppClass.pieces[index]
This is at least safe if the contents of the vector changes (it doesn't point to an existing Piece, but just to the indexed position). You obviously need to be careful about changing the size.
Your getter function for Data.pieces might be something like
#property
def pieces(self):
l = []
for i in range(self.pieces.size()):
l.append(UnownedPiece(self.pieces[i], self))
return tuple(l) # convert to tuple so it's immutable and people
# won't be tempted to try to append to it.
There's obviously a number of other approaches that you could take, but you can create a reasonably nice interface with this kind of approach.
The main thing is: restrict the Python interface as much as is possible.

Cython throws an error when pass function with void * argument to dict

I have a function with void * parameter and I want to store it in the dict.
What I do is:
%%cython
cdef void foo(void * bar):
pass
cdef dict foobar = {'foo': foo}
But this code raises an Error: Cannot convert 'void (void *)' to Python object Any ways to overcome this problem?

The easiest solution is to create a cdef class that can wrap this function. Since the cdef class is a Python object it can be stored in a dictionary like any other Python object.
ctypedef void (*void_func_ptr)(void*)
cdef class VoidFuncWrapper:
cdef void_func_ptr func
def __cinit__(self):
self.func = NULL
#staticmethod
cdef VoidFuncWrapper make_from_ptr(void_func_ptr f):
cdef VoidFuncWrapper out = VoidFuncWrapper()
out.func = f
return out
Then you can simply do:
cdef dict foobar = {'foo': VoidFuncWrapper.make_from_ptr(foo)}

Your problem here is that void* has no defined converter from a Python object, so a Python wrapper function (that accepts Python objects, converts them, and passes them to the underlying C function) can't be defined, either explicitly with cpdef or implicitly by putting the cdef function in a Python object (the dict in this case).
You could make this work by defining the argument to be something Cython knows how to convert to, e.g.:
cpdef void foo(const char * bar):
pass
cdef dict foobar = {'foo': foo}
Try it online!
But that may not work for your scenario if the function needs to accept an arbitrary pointer. If that's the case, you may want to switch to using C++ containers that can hold your function pointer type directly:
from libcpp.unordered_map cimport unordered_map
from libcpp.string cimport string
cdef void foo(void *bar):
pass
ctypedef void (*f_type)(void *)
cdef unordered_map[string, f_type] foobar
foobar['foo'.encode('ascii')] = foo

cython wrap cpp structs and function with parameter an array of structs

I am using Cython to wrap a c++ library in python. Unfortunately, I have no access to the c++ library. Therefore, I have to find somehow a way to wrap the structures and functions as exposed by the lib API.
My question regards the best way to wrap a c++ structure and; subsequently, how to create a memory view in python and pass its pointer (first element address) to a c++ function with parameter an array of cpp structures.
For example, let say I have the following h file:
//test.h
struct cxxTestData
{
int m_id;
double m_value;
};
void processData(cxxTestData* array_of_test_data, int isizeArr)
My pyx file will look like the following
cdef extern from "Test.h":
cdef struct cxxTestData:
int m_id
double m_value
cdef class pyTestData:
cdef cxxTestData cstr
def __init__(self, id, value):
self.cstr.m_id = id
self.cstr.m_value = value
#property
def ID(self):
return self.cstr.m_id
#property
def Value(self):
return self.cstr.m_value
Now, I want to create a number of pyTestData and store them in an array of dtype object. Then I want to pass this array as memory view in a cython/python function.
The wrapping function will have the following signature
cpdef void pyProcessData(pyTestData[::1] test_data_arr)
I have tested the above and it compiles successfully. I managed also to modify the members of each structure. However, this is not what I am trying to achieve. My question is how from this point I can pass an array with the c++ structures encapsulated in each pyTestData object (via the self.cstr).
As an example please have a look to the following listing:
cpdef void pyProcessData(pyTestData[::1] test_data_arr):
cdef int isize test_data_arr.shape[0]
# here I want to create/insert an array of type cxxTestData to pass it
# to the cpp function
# In other words, I want to create an array of [test_data_arr.cstr]
# I guess I can use cxxTestData[::1] or cxxTestData* via malloc and
# copy each test_data_arr[i].cstr to this new array
cdef cxxTestData* testarray = <cxxTestData*>malloc(isize*sizeof(cxxTestData))
cdef int i
for i in range(isize):
testarray[i] = test_data_arr[i].cstr
processData(&testarray[0], isize)
for i in range(isize):
arrcntrs[i].pystr = testarray[i]
free(testarray)
Has anyone come across with such a case? Is there any better way to pass my python objects in the above function without having to copy over the cxx structures internally?
Many thanks in advance and apologies if I do something fundamentally wrong.

Since you want an array of cxxTestData to pass to your C++ functions, the best thing to do is to allocate it as an array. Some untested code that illustrates the approach:
cdef class TestDataArray:
cdef cxxTestData* array:
def __init__(self, int length):
self.array = <cxxTestData*>calloc(length,sizeof(cxxTestData))
def __dealloc__(self):
free(self.array)
def __getitem__(self, int idx):
return PyTestData.from_pointer(&self.array[idx],self) # see later
def __setitem__(self, int idx, PyTestData pyobj): # this is optional
self.array[idx] = deref(pyobj.cstr)
You then want to slightly modify your PyTestData class so that it holds a pointer rather than holding the class directly. It should also have a field representing the ultimate owner of the data (e.g. the array). This ensures the array is kept alive, and can also allow for the case where the PyTestData owns its own data:
cdef class PyTestData:
cdef cxxTestData* cstr
cdef object owner
def __init__(self, id, value):
self.owner = None
self.cstr = <cxxTestData*>malloc(sizeof(cxxTestData))
self.cstr.m_id = id
self.cstr.m_value = value
def __dealloc__(self):
if self.owner is None: # i.e. this class owns it
free(self.cstr)
#staticmethod
cdef PyTestData from_pointer(cxxTestData* ptr, owner):
# calling __new__ avoids calling the constructor
cdef PyTestData x = PyTestData.__new__(PyTestData)
x.owner = owner
x.cstr = ptr
return x
There is a little extra effort in creating the TestDataArray class, but it stores the data in a format directly usable from C++, and so I think it's the best solution.

Cython: overloaded constructor initialization using raw pointer [duplicate]

This question already has answers here:
Wrapping a pre-initialized pointer in a cython class
(3 answers)
Closed 1 year ago.
I'm trying to wrap two C++ classes: Cluster and ClusterTree. ClusterTree has a method get_current_cluster() that instantiates a Cluster object, and returns a reference to it. ClusterTree owns the Cluster object, and manages its creation and deletion in C++.
I've wrapped Cluster with cython, resulting in PyCluster.
PyCluster should have two ways of creation:
1) By passing in two arrays, which then implies that Python should then automatically handle deletion (via __dealloc__)
2) By directly passing in a raw C++ pointer (created by ClusterTree's get_current_cluster()). In this case, ClusterTree then assumes responsibility of deleting the underlying pointer.
from libcpp cimport bool
from libcpp.vector cimport vector
cdef extern from "../include/Cluster.h" namespace "Terran":
cdef cppclass Cluster:
Cluster(vector[vector[double]],vector[int]) except +
cdef class PyCluster:
cdef Cluster* __thisptr
__autoDelete = True
def __cinit__(self, vector[vector[double]] data, vector[int] period):
self.__thisptr = new Cluster(data, period)
#classmethod
def __constructFromRawPointer(self, raw_ptr):
self.__thisptr = raw_ptr
self.__autoDelete = False
def __dealloc__(self):
if self.__autoDelete:
del self.__thisptr
cdef extern from "../include/ClusterTree.h" namespace "Terran":
cdef cppclass ClusterTree:
ClusterTree(vector[vector[double]],vector[int]) except +
Cluster& getCurrentCluster()
cdef class PyClusterTree:
cdef ClusterTree *__thisptr
def __cinit__(self, vector[vector[double]] data, vector[int] period):
self.__thisptr = new ClusterTree(data,period)
def __dealloc__(self):
del self.__thisptr
def get_current_cluster(self):
cdef Cluster* ptr = &(self.__thisptr.getCurrentCluster())
return PyCluster.__constructFromRawPointer(ptr)
This results in:
Error compiling Cython file:
------------------------------------------------------------
...
def get_current_cluster(self):
cdef Cluster* ptr = &(self.__thisptr.getCurrentCluster())
return PyCluster.__constructFromRawPointer(ptr)
^
------------------------------------------------------------
terran.pyx:111:54: Cannot convert 'Cluster *' to Python object
Note I cannot cdef __init__ or #classmethods.

Pointers can only be passed to cdef'd functions as arguments, and cinit has to be def'd. But providing a classmethod is almost the way to go!
cdef Cluster* __thisptr
cdef bool __wrapped ## defaults to False
#staticmethod
cdef PyCluster wrap(Cluster* ptr):
cdef PyCluster pc = PyCluster([], []) ## Initialize as cheaply as possible
del pc.__thisptr ## delete the old pointer to avoid memory leaks!
pc.__thisptr = ptr
pc.__wrapped = True
return pc

I know this is an old question, but after my own recent struggles with Cython I thought I'd post an answer for the sake of posterity.
It seems to me you could use a copy constructor to create a new PyCluster object from an existing Cluster object.
Define the copy constructor in your C code, then call the copy constructor in the Python class definition (in this case, when a pointer is passed) using new. This will work, although it may not be the best or most performant solution.

Passing C++ pointer as argument into Cython function

cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
def __cinit__(self, Bar *b)
bar = b
This will always give me something like:
Cannot convert Python object argument to type 'Bar *'
Is there a way to accomplish this, or do I need to extract everything from a Bar object, create a Python equivalent, pass it in, then reconstruct it in PyClass?

I came across this problem trying to wrap C code with structs as python classes. The issue seems to be that "special" function including __init__ and __cinit__ must be declared as def rather than cdef. This means that they can be called from normal python, so the type parameters are effectively ignored and everything is treated as object.
In J.F. Sebastian's answer the fix is not the wrapping - a double is a basic numeric type and so there is a default conversion between the C/C++ type and the Python object. Czarek's answer is basically correct - you need to use a fake constructor idiom, using a global function. It is not possible to use a #staticmethod decorator as they cannot be applied to cdef functions. The answer looks simpler on the original example provided.
cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
cdef PyClass_Init(Bar *b):
result = PyClass()
result.bar = b
return result

As of Cython 0.21 it has been possible to declare cdef methods with the #staticmethod decorator. This allows static creator methods that take non-Python arguments:
cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
#staticmethod
cdef create(Bar *bar):
cdef PyClass pc = PyClass()
pc.bar = bar
return pc

For each cdef class create a global cdef function that acts as a constructor, CefResponse is a C++ object, PyResponse a python equivalent of a c++ object:
cdef object CreatePyResponse(CefRefPtr[CefResponse] cefResponse):
pyResponse = PyResponse()
pyResponse.cefResponse = cefResponse
return pyResponse
cdef class PyResponse:
cdef CefRefPtr[CefResponse] cefResponse
def GetStatus(self):
return (<CefResponse*>(self.cefResponse.get())).GetStatus()
So instead of resp = PyResponse(cppObject) call resp = CreatePyResponse(cppObject).
Example taken from CEF Python:
https://code.google.com/p/cefpython/source/browse/cefpython/response.pyx?r=0250b65e046a

Python class accepts Python arguments. To pass a C++ argument you need to wrap it:
# distutils: language = c++
cdef extern from "Foo.h" namespace "baz":
cdef cppclass Bar:
Bar(double d)
double get()
cdef class PyBar: # wrap Bar class
cdef Bar *thisptr
def __cinit__(self, double d):
self.thisptr = new Bar(d)
def __dealloc__(self):
del self.thisptr
property d:
def __get__(self):
return self.thisptr.get()
PyBar instances can be used as any other Python objects both from Cython and pure Python:
class PyClass:
def __init__(self, PyBar bar):
self.bar = bar
print(PyClass(PyBar(1)).bar.d)