We Keep Coding

How to wrap a C++ class/struct with attributes that are vectors for fluid use in python

I'm wrapping a C++ API for use in Python, so I'd like the functionality of the Python wrapper classes to pretty closely mirror that of the C++ classes. In one case, I have two objects that are actually nested structs:
// myheader.hpp
#include <vector>
namespace mynames{
struct Data{
struct Piece{
double piece1;
int piece2;
std::vector<double> piece3;
};
std::vector<Piece> pieces;
};
}
I'd like to interact with this object fluidly in Python as if it were a typical Python class using numpy and extension types. So I began by declaring two extension types:
# mydeclarations.pxd
from libcpp.vector cimport vector
cdef extern from "myheader.hpp" namespace "mynames":
cdef cppclass Data:
vector[Piece] pieces
cdef extern from "myheader.hpp" namespace "mynames::Data":
cdef cppclass Piece:
double piece1
int piece2
vector[double] piece3
Then wrapping in Python:
# mytypes.pyx
cimport mydeclarations as cpp
from cython cimport NULL
from libcpp.vector cimport vector
import numpy as np
cdef class Piece:
cdef cpp.Piece *_cppPiece
def __cinit__(self):
self._cppPiece = new cpp.Piece()
def __dealloc__(self):
if self._cppPiece is not NULL:
del self._cppPiece
#property
def piece1(self):
return self._cppPiece.piece1
#piece1.setter
def piece1(self, double d):
self._cppPiece.piece1 = d
#property
def piece2(self):
return self._cppPiece.piece2
#piece2.setter
def piece2(self, int i):
self._cppPiece.piece2 = i
# Use cython's automatic type conversion: (cpp)vector <---> (py)list (COPIES)
#property
def piece3(self):
return np.asarray(self._cppPiece.piece3, dtype=np.double)
#piece3.setter
def piece3(self, arr):
self._cppPiece.piece3 = <vector[double]>np.asarray(arr, dtype=np.double)
#----------------------------------------------------------------------
cdef class Data:
cdef cpp.Data *_cppData
def __cinit__(self):
self._cppData = new cpp.Data()
def __dealloc__(self):
if self._cppData is not NULL:
del self._cppData
#property
def pieces(self):
# Create a list of Python objects that hold copies of the C++ data
cdef Piece pyPiece
cdef int i
pyPieces = []
for i in range(self._cppData.pieces.size()):
pyPiece = Piece()
pyPiece._cppPiece[0] = self._cppData.pieces.at(i)
pyPieces.append(pyPiece)
return np.asarray(pyPieces, dtype=Piece)
#pieces.setter
def pieces(self, arr):
# Clear the existing vector and create a new one containing copies of the data in arr
cdef Piece pyPiece
self._cppData.pieces.clear()
for pyPiece in arr:
self._cppData.pieces.push_back(deref(pyPiece._cppPiece))
This is a simple implementation and as far as I can tell it works, but there are some issues:
Since we use copies, there's no in-place functionality that you might expect if, for example, Piece().piece3 was a python class attribute holding a numpy array. For example,
a = Piece()
a.piece3 = [1,2,3]
a.piece3[0] = 55 # No effect, need to do b = a.piece3; b[0]=55; a.piece3=b
There's a lot of iterating over data and copying. This is probably an issue when the size of Data.pieces is very large.
Can anyone suggest some better alternatives to address these issues? Although Data is more complicated that Pieces, I think they are related and ultimately boil down to wrapping C++ classes with vector attributes for use in Python.

If you want to avoid data copying then it probably involves creating a wrapper class.
cdef class DoubleVector:
cdef vector[double] *vec
cdef owner
def __dealloc__(self):
if owner is not None:
del self.vec
#staticmethod
cdef create_from_existing_Piece(Piece obj):
out = DoubleVector()
out.owner = obj
out.vec = obj._cppData.piece3
return out
# create len/__getindex__/__setindex__ functions
# You could also have this class expose the buffer protocol
Here I've assumed that DoubleVector doesn't own its own data the majority of the time. Therefore it keeps a reference to the Python class that owns the C++ class that owns that data (thus ensuring that object lifetimes are preserved).
Some details (mainly creating a nice sequence interface) is left for you to fill in.
Exposing vector[Piece] is more difficult, largely because any changes to the vector (including resizing it) would invalidate any pointers into the vector. Therefore I'd give serious thought to having a different Python interface to C++ interface.
Could you make Data immutable (so that you simply can't change it from Python and so can safely return pointers into it)?
Could you avoid returning Piece from data and have functions like get_ith_piece1, get_ith_piece2, get_ith_piece3 (i.e. remove a layer from your Python wrapping)?
Alternatively you could do something like
cdef class BasePiece:
cdef cpp.Piece* get_piece(self):
raise NotImplementedError
# most of the implementation of your Piece class goes here
cdef class Piece(BasePiece):
# wrapper that owns its own data.
# largely as before but with
cdef cpp.Piece* get_piece(self):
return self._cppPiece
# ...
cdef class UnownedPiece(BasePiece):
cdef Data d
cdef int index
cdef cpp.Piece* get_piece(self):
return self.d._cppClass.pieces[index]
This is at least safe if the contents of the vector changes (it doesn't point to an existing Piece, but just to the indexed position). You obviously need to be careful about changing the size.
Your getter function for Data.pieces might be something like
#property
def pieces(self):
l = []
for i in range(self.pieces.size()):
l.append(UnownedPiece(self.pieces[i], self))
return tuple(l) # convert to tuple so it's immutable and people
# won't be tempted to try to append to it.
There's obviously a number of other approaches that you could take, but you can create a reasonably nice interface with this kind of approach.
The main thing is: restrict the Python interface as much as is possible.

cython wrap cpp structs and function with parameter an array of structs

I am using Cython to wrap a c++ library in python. Unfortunately, I have no access to the c++ library. Therefore, I have to find somehow a way to wrap the structures and functions as exposed by the lib API.
My question regards the best way to wrap a c++ structure and; subsequently, how to create a memory view in python and pass its pointer (first element address) to a c++ function with parameter an array of cpp structures.
For example, let say I have the following h file:
//test.h
struct cxxTestData
{
int m_id;
double m_value;
};
void processData(cxxTestData* array_of_test_data, int isizeArr)
My pyx file will look like the following
cdef extern from "Test.h":
cdef struct cxxTestData:
int m_id
double m_value
cdef class pyTestData:
cdef cxxTestData cstr
def __init__(self, id, value):
self.cstr.m_id = id
self.cstr.m_value = value
#property
def ID(self):
return self.cstr.m_id
#property
def Value(self):
return self.cstr.m_value
Now, I want to create a number of pyTestData and store them in an array of dtype object. Then I want to pass this array as memory view in a cython/python function.
The wrapping function will have the following signature
cpdef void pyProcessData(pyTestData[::1] test_data_arr)
I have tested the above and it compiles successfully. I managed also to modify the members of each structure. However, this is not what I am trying to achieve. My question is how from this point I can pass an array with the c++ structures encapsulated in each pyTestData object (via the self.cstr).
As an example please have a look to the following listing:
cpdef void pyProcessData(pyTestData[::1] test_data_arr):
cdef int isize test_data_arr.shape[0]
# here I want to create/insert an array of type cxxTestData to pass it
# to the cpp function
# In other words, I want to create an array of [test_data_arr.cstr]
# I guess I can use cxxTestData[::1] or cxxTestData* via malloc and
# copy each test_data_arr[i].cstr to this new array
cdef cxxTestData* testarray = <cxxTestData*>malloc(isize*sizeof(cxxTestData))
cdef int i
for i in range(isize):
testarray[i] = test_data_arr[i].cstr
processData(&testarray[0], isize)
for i in range(isize):
arrcntrs[i].pystr = testarray[i]
free(testarray)
Has anyone come across with such a case? Is there any better way to pass my python objects in the above function without having to copy over the cxx structures internally?
Many thanks in advance and apologies if I do something fundamentally wrong.

Since you want an array of cxxTestData to pass to your C++ functions, the best thing to do is to allocate it as an array. Some untested code that illustrates the approach:
cdef class TestDataArray:
cdef cxxTestData* array:
def __init__(self, int length):
self.array = <cxxTestData*>calloc(length,sizeof(cxxTestData))
def __dealloc__(self):
free(self.array)
def __getitem__(self, int idx):
return PyTestData.from_pointer(&self.array[idx],self) # see later
def __setitem__(self, int idx, PyTestData pyobj): # this is optional
self.array[idx] = deref(pyobj.cstr)
You then want to slightly modify your PyTestData class so that it holds a pointer rather than holding the class directly. It should also have a field representing the ultimate owner of the data (e.g. the array). This ensures the array is kept alive, and can also allow for the case where the PyTestData owns its own data:
cdef class PyTestData:
cdef cxxTestData* cstr
cdef object owner
def __init__(self, id, value):
self.owner = None
self.cstr = <cxxTestData*>malloc(sizeof(cxxTestData))
self.cstr.m_id = id
self.cstr.m_value = value
def __dealloc__(self):
if self.owner is None: # i.e. this class owns it
free(self.cstr)
#staticmethod
cdef PyTestData from_pointer(cxxTestData* ptr, owner):
# calling __new__ avoids calling the constructor
cdef PyTestData x = PyTestData.__new__(PyTestData)
x.owner = owner
x.cstr = ptr
return x
There is a little extra effort in creating the TestDataArray class, but it stores the data in a format directly usable from C++, and so I think it's the best solution.

How do I return wrapped C++ Objects in Cython from another wrapped Object?

I've been wrapping C++ Classes with Cython. Now I have the following problem: Class A has a method foo() which returns an instance of Class B. Both are C++ classes. Both are wrapped in Cython to Class PyA and PyB. I want to make the method foo() accessible for users in a way, that the user calls PyA_object.foo() and gets a PyB_object. How do I implement the method?
cimport cppClassA
cdef class A:
cdef cppClassA.A *ptr
...
def foo():
return self.ptr.foo()
This way I return the direct C++ Class B, but I want to return a PyB. Therefore I need to create a PyB Object, but how do I construct the cinit() method of PyB? How can I give the returned object from self.ptr.foo() to a new PyB object and return it instead?

It isn't entirely clear from your question whether foo returns by pointer or value (i.e. whether it has the signature B* foo() or B foo()). What you have to do depends on this - I've shown both.
The problem that I think you're run into is that you the __init__ or __cinit__ method of PyB can only take Python objects as arguments, so you've got no way of passing the C++ B object in. The solution is to create a PyB that is either empty or has the default C++ object as ptr and then replace it.
The wrapping of the C++ code looks like this
cdef extern from "somefile.hpp":
cdef cppclass A:
B foo1()
B* foo2()
cdef cppclass B:
# only need to provide these for foo1_alternative
B()
B(const B&)
Note that I've provided two versions of foo - foo1 which returns by value and foo2 which returns a pointer.
The Python B holder (PyB) looks like
cdef class PyB:
cdef B* ptr
def __cinit__(self,make_ptr=True):
if make_ptr:
self.ptr = new B()
else:
self.ptr = NULL
def __dealloc__(self):
del self.ptr
I've provided an optional make_ptr argument which you can set to False to avoid setting self.ptr to anything.
The Python A holder (PyA) is then
cdef class PyA:
cdef A* ptr
# code goes here
def foo1(self):
cdef PyB val = PyB()
val.ptr[0] = self.ptr.foo1() # use move (or copy) assignment operator
def foo1_alternative(self):
cdef PyB val = PyB(make_ptr=False)
val.ptr = new B(self.ptr.foo1()) # use move (or copy) constructor
def foo2(self):
cdef PyB val = PyB(make_ptr=False)
val.ptr = self.ptr.foo2()
For foo2, which returns a pointer, you just create an empty PyB and assign the Foo* you get back to self.ptr. For foo1 you have two options, one of which involves creating a PyB with something in then using move (or copy) assignment operator to copy the data across. The second option involves creating an empty PyB and then using Bs move (or copy) constructor to create a B*.
You're better off using the move constructor rather than the copy constructor if you can, but you don't need to tell Cython about either (C++ will pick the right one automatically).

Cython - how to properly call operator with references

I have the following code in my Cython wrapper to a C++ code:
# distutils: language = c++
# distutils: sources = symbolic.cpp
from libcpp.vector cimport vector
from libcpp.pair cimport pair
from libcpp.string cimport string
from libcpp cimport bool
cdef extern from "symbolic.h" namespace "metadiff::symbolic":
cdef cppclass SymbolicMonomial:
vector[pair[int, int]] powers
long long coefficient;
SymbolicMonomial()
SymbolicMonomial(long)
SymbolicMonomial(const SymbolicMonomial&)
bool is_constant()
long long int eval(vector[int]&)
long long int eval()
string to_string()
string to_string_with_star() const
cdef SymbolicMonomial mul_mm"operator*"(const SymbolicMonomial&, const SymbolicMonomial&)
# SymbolicMonomial operator*(long long, const SymbolicMonomial&)
# SymbolicMonomial operator*(const SymbolicMonomial&, long long)
cdef class SymMonomial:
cdef SymbolicMonomial* thisptr # hold a C++ instance which we're wrapping
def __cinit__(self):
self.thisptr = new SymbolicMonomial()
def __cinit__(self, int value):
self.thisptr = new SymbolicMonomial(value)
def __dealloc__(self):
del self.thisptr
def is_constant(self):
return self.thisptr.is_constant()
def eval(self):
return self.thisptr.eval()
def __str__(self):
return self.to_string_with_star()
def to_string(self):
return self.thisptr.to_string().decode('UTF-8')
def to_string_with_star(self):
return self.thisptr.to_string_with_star().decode('UTF-8')
def __mul__(self, other):
return mul_mm(self.thisptr, other)
def variable(variable_id):
monomial = SymMonomial()
monomial.thisptr.powers.push_back((variable_id, 1))
return monomial
However, I never figured it out how to call the mul_mm method correctly. It keeps saying Cannot convert 'SymbolicMonomial' to Python object or vice versa. The thing is I need to be able to multiply two SymMonomials in this way. However for some reason I can not get the hang of it of how to do it properly. Any advices?

You have a number of issues:
You can't return C++ objects directly to Python - you need to return your wrapper type (assign to thisptr of the wrapper)
You can't guarantee either self or other is of the correct type at the point the function is called (see the note in http://docs.cython.org/src/userguide/special_methods.html#arithmetic-methods about how the methods can be called with the operands in either order). To use the C/C++ members of a Cython class you need to ensure Cython knows that object is indeed of that class. I recommend using the <Classname?> style cast (note the question mark) which throws an exception if it doesn't match.
You need to get thisptr from other too, rather than just passing the Python wrapper class to your C++ function.
The following should work.
def __mul__(self,other):
cdef SymMonomial tmp = SymMonomial()
cdef SymMonomial self2, other2
try:
self2 = <SymMonomial?>self
other2 = <SymMonomial?>other
except TypeError:
return NotImplemented # this is what Python expects for operators
# that don't know what to do
tmp.thisptr[0] = mul_mm(self2.thisptr[0],other2.thisptr[0])
return tmp

Passing C++ pointer as argument into Cython function

cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
def __cinit__(self, Bar *b)
bar = b
This will always give me something like:
Cannot convert Python object argument to type 'Bar *'
Is there a way to accomplish this, or do I need to extract everything from a Bar object, create a Python equivalent, pass it in, then reconstruct it in PyClass?

I came across this problem trying to wrap C code with structs as python classes. The issue seems to be that "special" function including __init__ and __cinit__ must be declared as def rather than cdef. This means that they can be called from normal python, so the type parameters are effectively ignored and everything is treated as object.
In J.F. Sebastian's answer the fix is not the wrapping - a double is a basic numeric type and so there is a default conversion between the C/C++ type and the Python object. Czarek's answer is basically correct - you need to use a fake constructor idiom, using a global function. It is not possible to use a #staticmethod decorator as they cannot be applied to cdef functions. The answer looks simpler on the original example provided.
cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
cdef PyClass_Init(Bar *b):
result = PyClass()
result.bar = b
return result

As of Cython 0.21 it has been possible to declare cdef methods with the #staticmethod decorator. This allows static creator methods that take non-Python arguments:
cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
#staticmethod
cdef create(Bar *bar):
cdef PyClass pc = PyClass()
pc.bar = bar
return pc

For each cdef class create a global cdef function that acts as a constructor, CefResponse is a C++ object, PyResponse a python equivalent of a c++ object:
cdef object CreatePyResponse(CefRefPtr[CefResponse] cefResponse):
pyResponse = PyResponse()
pyResponse.cefResponse = cefResponse
return pyResponse
cdef class PyResponse:
cdef CefRefPtr[CefResponse] cefResponse
def GetStatus(self):
return (<CefResponse*>(self.cefResponse.get())).GetStatus()
So instead of resp = PyResponse(cppObject) call resp = CreatePyResponse(cppObject).
Example taken from CEF Python:
https://code.google.com/p/cefpython/source/browse/cefpython/response.pyx?r=0250b65e046a

Python class accepts Python arguments. To pass a C++ argument you need to wrap it:
# distutils: language = c++
cdef extern from "Foo.h" namespace "baz":
cdef cppclass Bar:
Bar(double d)
double get()
cdef class PyBar: # wrap Bar class
cdef Bar *thisptr
def __cinit__(self, double d):
self.thisptr = new Bar(d)
def __dealloc__(self):
del self.thisptr
property d:
def __get__(self):
return self.thisptr.get()
PyBar instances can be used as any other Python objects both from Cython and pure Python:
class PyClass:
def __init__(self, PyBar bar):
self.bar = bar
print(PyClass(PyBar(1)).bar.d)