We Keep Coding

pybind11 - std::vector member of boost::optional struct is emptied if any members of the struct are accessed

I have a hierarchical data structure of 3 structs in C++, which I renamed in my minimum working example (mwe) House, Room, and Objects. Room is a boost::optional type and has a std::vector<Object> member containing all Objects in this room. The Objects are just a container for some numbers.
I am aware that this is overly complex for such information, but it is necessary in the original code and cannot be changed easily. I tried to change it into a std::experimental::optional since we do not use c++17, but this broke some parts in our c++ code and I don't know whether it would actually solve the problem.
In C++, I have no issues at all and the boost::optional and all member variables work perfectly. But after binding everything I run into the weird problem that the std::vector objects is emptied as soon as I access any member variables of the Room. This can be either the length, width, area or the objects itself as shown in the Python example. If the objects are accessed for the first time, they are actually returned normally, but when trying the second access they are gone as if the had been moved. The same behavior applies if you perform x = myHouse.kitchen.objects.copy(). The list is in x and can even be accessed multiple times, but the information in kitchen is immediately lost. Weirdly, this also only applies to the objects and all other double members can be accessed indefinitely.
For the convenience of compiling everything for the mwe and debugging, I stuffed everything into a single cpp file, which is obviously not the case in the original code. For the mwe, the c++ code is compiled and included via cppimport, but also manual compilation does not change anything.
Here is the mwe:
mwe.cpp:
#include <boost/optional.hpp>
#include <iostream>
#include <map>
#include <string>
#include <vector>
#include <pybind11/complex.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
/* implementing structs */
struct Object {
Object() = default;
Object(double price, double height) : price(price), height(height) {
};
double price = 0.0;
double height = 0.0;
};
struct Room {
Room() = default;
double length;
double width;
std::vector<Object> objects; // this is the buggy vector
double area() const {
return length * width;
};
};
struct House {
House() = default;
boost::optional<Room> bedroom;
boost::optional<Room> kitchen;
boost::optional<Room> livingroom;
std::map<std::string, std::vector<Object>> getObjects() {
std::map<std::string, std::vector<Object>> out;
if (bedroom) {
out.insert(std::make_pair("bedroom", bedroom->objects));
}
if (kitchen) {
out.insert(std::make_pair("kitchen", kitchen->objects));
}
if (livingroom) {
out.insert(std::make_pair("livingroom", livingroom->objects));
}
return out;
};
};
/* everything works fine in C++ -> get data this way to have complete object map */
House initSomethingInCpp() {
auto myHouse = House();
myHouse.bedroom = Room();
myHouse.kitchen = Room();
myHouse.bedroom->length = 10.0;
myHouse.bedroom->width = 2.0;
myHouse.kitchen->length = 5.0;
myHouse.kitchen->width = 3.0;
std::vector<Object> bedroomObjects;
std::vector<Object> kitchenObjects;
Object closet = Object(100.0, 2.5);
Object bed = Object(200.0, 1.0);
Object oven = Object(500.0, 1.5);
Object table = Object(50.0, 1.5);
bedroomObjects.push_back(closet);
bedroomObjects.push_back(bed);
kitchenObjects.push_back(oven);
kitchenObjects.push_back(table);
myHouse.bedroom->objects = bedroomObjects;
myHouse.kitchen->objects = kitchenObjects;
return myHouse;
};
namespace pybind11 {
/* taking care of boost type */
namespace detail {
/* boost::optional */
template<typename T>
struct type_caster<boost::optional<T>> : optional_caster<boost::optional<T>> {};
} // namespace detail
} // namespace pybind11
/* binding structs */
void init_house(pybind11::module& main) {
pybind11::class_<House> house(main, "House");
house.def(pybind11::init<>());
house.def_readwrite("bedroom", &House::bedroom);
house.def_readwrite("kitchen", &House::kitchen);
house.def_readwrite("livingroom", &House::livingroom);
house.def("get_objects", &House::getObjects);
};
void init_room(pybind11::module& main) {
pybind11::class_<Room> room(main, "Room");
room.def(pybind11::init<>());
room.def_readwrite("length", &Room::length);
room.def_readwrite("width", &Room::width);
room.def_readwrite("objects", &Room::objects);
room.def_property_readonly("area", &Room::area);
};
void init_objects(pybind11::module& main) {
pybind11::class_<Object> object(main, "Object");
object.def(pybind11::init<>());
object.def(pybind11::init<double, double>());
object.def_readonly("price", &Object::price);
object.def_readonly("heigth", &Object::height);
};
/* define module and bind init_in_cpp function */
PYBIND11_MODULE(mwe, m) {
init_house(m);
init_room(m);
init_objects(m);
m.def("init_something_in_cpp", &initSomethingInCpp);
};
execute.py:
import cppimport
#cppimport.set_quiet(False)
#cppimport.force_rebuild()
mod = cppimport.imp('mwe')
# get data
myHouse = mod.init_something_in_cpp()
print("\n")
print("all data is here")
objs = myHouse.get_objects()
print(objs)
print(myHouse.kitchen.area) # by accessing kitchen members, the objects list is emptied
print("\n")
print("kitchen objects are now missing")
objs = myHouse.get_objects()
print(objs)
print("but area still works:")
print(myHouse.kitchen.area) # everything but objects still works
print("\n")
print("also works directly with same variable")
print("bedroom objects are accessed:")
print(myHouse.bedroom.objects)
print("bedroom objects are accessed again:")
print(myHouse.bedroom.objects)
The execution gives the following output:
all data is here
{'bedroom': [mwe.Object object at 0x7fbc9c2a43f0, mwe.Object object at 0x7fbc9c2a4670], 'kitchen': [mwe.Object object at 0x7fbc9c2a4c30, mwe.Object object at 0x7fbc9c2a4df0]}
15.0
kitchen objects are now missing
{'bedroom': [mwe.Object object at 0x7fbc9c2a4e70, mwe.Object object at 0x7fbc9c2a4eb0], 'kitchen': []}
but area still works:
15.0
also works directly with same variable
bedroom objects are accessed:
[mwe.Object object at 0x7fbc9c2a4c30, mwe.Object object at 0x7fbc9c2a4df0]
bedroom objects are accessed again:
[]

Turns out this was actually a bug in pybind11 release 2.5 https://github.com/pybind/pybind11/issues/1919
It is fixed in the current master branch and future releases.

Python/C API - The result is not displayed

I would like to integrate C modules in the Python, so my choice fell on the interface Python.h. Everything compiled without errors and warnings, so I can not understand what the problem is.
C side:
#include <python3.5m/Python.h>
...
#define PyInt_AsLong(x) (PyLong_AsLong((x)))
typedef PyObject* Py;
static Py getSumma(Py self, Py args){
Py nums;
if (!PyArg_ParseTuple(args, "O", &nums)){
return NULL;
}
size_t numsAmount = PyList_Size(args);
int32_t summa = 0;
for (size_t i = 0; i < numsAmount; i++){
Py temp = PyList_GetItem(nums, i);
int32_t num = PyInt_AsLong(temp);
summa += num;
}
return Py_BuildValue("l", summa);
}
static PyMethodDef moduleMethods[] = {
{"getSumma", (PyCFunction)getSumma, METH_VARARGS, NULL},
{NULL, NULL, 0, NULL}
};
static PyModuleDef SummaLogic = {
PyModuleDef_HEAD_INIT,
"SummaLogic",
"",
-1,
moduleMethods
};
PyMODINIT_FUNC PyInit_SummaLogic(void){
return PyModule_Create(&SummaLogic);
}
setup.py:
from distutils.core import setup, Extension
SummaLogic = Extension("SummaLogic", sources=['SummaLogic.c'])
setup(ext_modules=[SummaLogic])
Python side:
from SummaLogic import getSumma
if __name__ == "__main__":
a = [1, 2, 3]
b = getSumma(a)
print(b)
It seems right, but when I start it in terminal - nothing happens, just hanging without any activity. What could I miss?

It boils down to PyList_Size and that you don't check for errors there.
You probably wanted to use it on nums, not args as argument. However you used on args and a very interesting thing happened:
args is a tuple,
therefore PyList_Size failed and returned -1
that -1 which was cast to an unsigned size_t which probably resulted in a very huge number, probably 2**64-1
therefore your iteration runs a "very long time" because it takes quite a while to iterate over 2**64-1 items (apart from all the out-of-bound memory accesses).
The quick fix would be to use:
Py_ssize_t listlength = PyList_Size(nums); /* nums instead of args */
if (listlength == -1) { /* check for errors */
return NULL;
}
size_t numsAmount = (size_t)listlength /* cast to size_t only after you checked for errors */
However you should check what the error conditions are and test for them after every python C API function call otherwise you'll get a lot of undefined behaviours. Also I probably would stick to the defined return types instead of int32_t (PyInt_AsLong returns long so you might get weird casting errors there as well!), size_t, ... and the typedef PyObject* Py; makes things really tricky for someone who regularly writes C extensions.

boost python returning tuple containing custom types

I have a C++ myObject class that I expose via boost python using a wrapper structure:
struct myObjectWrapper{
static tuple compute(myObject& o,const Container& x0, const double& t0, Container& x){
double t;
int stat = o.evaluate(x0,t0,x,t);
return make_tuple(stat,t);
}
}
BOOST_PYTHON_MODULE(myModule)
{
// not shown here is code to expose Container class
class_<myObject>("MyObject")
.def("compute",&myObjectWrapper::compute)
;
}
Container is currently defined as:
typedef std::valarray<double> Container
and is exposed to python.
Now in python I can do.
x = Container()
(status,t) = obj.compute(Container([0.,0.,0.]),0.0,x)
print status, t, x[0]
This is not very pythonic. I would prefer to do:
(status,t,x) = obj.compute(Container([0.,0.,0.]),0.0)
print status, t, x[0]
I could write an additional wrapper in python, but I would prefer to avoid adding more wrappers.
The following code does't compile:
struct myObjectWrapper{
static tuple compute(myObject& o,const Container& x0, const double& t0){
double t;
Container x;
int stat = o.evaluate(x0,t0,x,t);
return make_tuple(stat,t,x);
}
}
Also I would prefer to steal the content of the local variable x and have python manage it rather than copy it:
return make_tuple(stat,t,std::move(x));
How do I achieve this?

In short, allocate the wrapper on the free store and use the manage_new_object result convert to transfer ownership to a Python object. This will cause Boost.Python to copy the pointer when constructing the Python object, rather than copying the pointee. For more details, see this answer.
Here is an auxiliary function that will transfer ownership to a Python object:
/// #brief Transfer ownership to a Python object. If the transfer fails,
/// then object will be destroyed and an exception is thrown.
template <typename T>
boost::python::object transfer_to_python(T* t)
{
// Transfer ownership to a smart pointer, allowing for proper cleanup
// incase Boost.Python throws.
std::unique_ptr<T> ptr(t);
// Use the manage_new_object generator to transfer ownership to Python.
namespace python = boost::python;
typename python::manage_new_object::apply<T*>::type converter;
// Transfer ownership to the Python handler and release ownership
// from C++.
python::handle<> handle(converter(*ptr));
ptr.release();
return python::object(handle);
}
And one could use it as follows:
boost::python::tuple myObjectWrapper::compute(
myObject& o, const Container& x0, const double& t0)
{
auto x1 = std::make_unique<container>();
double t1 = 0;
int stat = self.evaluate(x0, t0, *x1, t1);
return boost::python::make_tuple(stat, t1, transfer_to_python(x1.release()));
}
Here is a complete example based on the original question that demonstrates using the transfer_to_python auxiliary function.
#include <boost/python.hpp>
#include <cassert>
#include <memory> // std::unique_ptr
// Mock legacy API.
struct container
{
container() {}
container(boost::python::object) {}
container(const container&)
{
// For this example, guarantee copy is not made.
assert(false);
}
};
struct my_object
{
int evaluate(container&, double&, container&, double&) { return 42; }
};
/// #brief Transfer ownership to a Python object. If the transfer fails,
/// then object will be destroyed and an exception is thrown.
template <typename T>
boost::python::object transfer_to_python(T* t)
{
// Transfer ownership to a smart pointer, allowing for proper cleanup
// incase Boost.Python throws.
std::unique_ptr<T> ptr(t);
// Use the manage_new_object generator to transfer ownership to Python.
namespace python = boost::python;
typename python::manage_new_object::apply<T*>::type converter;
// Transfer ownership to the Python handler and release ownership
// from C++.
python::handle<> handle(converter(*ptr));
ptr.release();
return python::object(handle);
}
// API wrapper.
boost::python::tuple my_object_compute(
my_object& self, container& x0, double t0)
{
auto x1 = std::make_unique<container>();
double t1 = 21;
int stat = self.evaluate(x0, t0, *x1, t1);
return boost::python::make_tuple(stat, t1, transfer_to_python(x1.release()));
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<container>("Container")
.def(python::init<python::object>())
;
python::class_<my_object>("MyObject")
.def("compute", &my_object_compute)
;
}
Interactive usage:
>>> import example
>>> my_object = example.MyObject()
>>> status, t, x = my_object.compute(example.Container([1, 2, 3]), 4)
>>> assert(status == 42)
>>> assert(t == 21)
>>> assert(isinstance(x, example.Container))

How to pass Python instance to C++ via Python/C API

I'm extending my library with Python (2.7) by wrapping interfaces with SWIG 2.0, and have a graph object in which I want to create a visitor. In C++, the interface looks like this:
struct Visitor
{
virtual void OnStateBegin() = 0;
virtual void OnNode(Node* n) = 0;
virtual void OnStateEnd() = 0;
};
I would like to define a class in Python that does the equivalent, all defined in python, that will allow for the definition of a visitor:
class GraphVisitor:
def __init__(self, label):
self._label = label
print("__init__(self,{0})".format(self._label))
def OnStateBegin(self):
print("OnStateBegin()" + self._label)
def OnNode(self, i_node):
print("OnNode()" + self._label)
def OnStateEnd(self):
print("OnStateEnd()" + self._label)
And what I'm trying to do is create an instance of a GraphVisitor in python script, and call the methods OnStateBegin(), OnNode(), and OnStateEnd() for a given instance from C++. Here's what I'd like to do in Python:
#model is a SWIG wrapped class
mvis = GraphVisitor("This is a test")
model.Visit("mvis") # I'm not sure how to pass the instance 'mvis' to C++?
And in my C++ wrapped by Swig, I'm not sure how to get at the instance 'mvis'? I can call functions defined in Python no problem, but instances has me stumped!

In order to solve this problem, I retrieved the class from the module given it's module name and class name (the code below assumes the module hasn't already been loaded):
void Model::Visit(const char* mod_name, const char* class_name)
{
PyErr_Clear();
PyObject* mod_name_obj = PyString_FromString(mod_name);
PyObject* class_name_obj = PyString_FromString(class_name);
PyObject* py_module = PyImport_Import(mod_name_obj);
PyObject* err_1 = PyErr_Occurred();
if(err_1)
PyErr_Print();
Once I had the module, I looked up the class from it's dictionary:
if(py_module)
{
PyObject* py_module_dict = PyModule_GetDict(py_module);
PyObject* py_class = PyDict_GetItem(py_module_dict, class_name_obj);
I simplified my problem a bit by instantiating the python class in C++, then created my visitor, and finally visited it:
if(py_class && PyClass_Check(py_class) && PyCallable_Check(py_class))
{
PyObject* inst = PyInstance_New(py_class, 0, 0);
if(inst && PyInstance_Check(inst))
{
IModel::IVisitorPtr py_visitor = new PyModelVisitor(inst);
_model->Visit(py_visitor);
}
}
}
}
The visitor had 3 functions OnStateBegin(), OnNode(), and OnStateEnd(). I added to my SWIG python binding generator an option to generate a header file for external access to the SWIG runtime with the -external-runtime option, so I could create a class in C++ (INode* below) and pass it to Python as the argument to the python OnNode() member function as follows (error checking removed for brevity):
VisitorCtrl OnNode(INode* node)
{
Node* node_impl = new NodeImpl(node);
PyObject* pynode = SWIG_NewPointerObj(node_impl, SWIG_TypeQuery("Node *"), 0);
PyObject* result = PyObject_CallMethodObjArgs(_inst, PyString_FromString("OnNode"), pynode, 0);
long rivis = PyInt_AsLong(result);
return(static_cast<VisitorCtrl>(rivis));
}

I don't know if that's possible with SWIG, but you can do it with SIP.
sip_vector_test.h:
class EXPORT Node {
public:
explicit Node(int n) : n_(n) {};
int getN() const { return n_; }
private:
int n_;
};
struct EXPORT NodeVisitor {
virtual void OnNode(Node* n) = 0;
};
struct EXPORT Graph {
public:
void addNode(int num);
void accept(NodeVisitor *nv);
private:
std::vector< std::shared_ptr<Node> > nodes_;
};
visitor.sip:
%Module pyvisit
%ModuleHeaderCode
#include "sip_visitor_test.h"
%End
class Node {
public:
explicit Node(int n);
int getN() const;
};
struct NodeVisitor {
virtual void OnNode(Node* n) = 0;
};
struct Graph {
public:
void addNode(int num);
void accept(NodeVisitor *nv);
};
Using it from Python:
>>> import pyvisit
>>> g = pyvisit.Graph()
>>> g.addNode(3)
>>> g.addNode(5)
>>> class PyNodeVisitor(pyvisit.NodeVisitor):
>>> def OnNode(self, node):
>>> print(node.getN())
>>> pnv = PyNodeVisitor()
>>> g.accept(pnv)
3
5
I've put a zip file containing the source code of this test project on my homepage.

C equivalent to python pickle (object serialization)?

What would be the C equivalent to this python code?
Thanks.
data = gather_me_some_data()
# where data = [ (metic, datapoints), ... ]
# and datapoints = [ (timestamp, value), ... ]
serialized_data = cPickle.dumps(data, protocol=-1)
length_prefix = struct.pack("!L", len(serialized_data))
message = length_prefix + serialized_data

C doesn't supports direct serialization mechanism because in C you can't get type information at run-time. You must yourself inject some type info at run-time and then construct required object by that type info. So define all your possible structs:
typedef struct {
int myInt;
float myFloat;
unsigned char myData[MY_DATA_SIZE];
} MyStruct_1;
typedef struct {
unsigned char myUnsignedChar;
double myDouble;
} MyStruct_2;
Then define enum which collects info about what structs in total you have:
typedef enum {
ST_MYSTRUCT_1,
ST_MYSTRUCT_2
} MyStructType;
Define helper function which lets to determine any struct size:
int GetStructSize(MyStructType structType) {
switch (structType) {
case ST_MYSTRUCT_1:
return sizeof(MyStruct_1);
case ST_MYSTRUCT_2:
return sizeof(MyStruct_2);
default:
// OOPS no such struct in our pocket
return 0;
}
}
Then define serialize function:
void BinarySerialize(
MyStructType structType,
void * structPointer,
unsigned char * serializedData) {
int structSize = GetStructSize(structType);
if (structSize != 0) {
// copy struct metadata to serialized bytes
memcpy(serializedData, &structType, sizeof(structType));
// copy struct itself
memcpy(serializedData+sizeof(structType), structPointer, structSize);
}
}
And de-serialization function:
void BinaryDeserialize(
MyStructType structTypeDestination,
void ** structPointer,
unsigned char * serializedData)
{
// get source struct type
MyStructType structTypeSource;
memcpy(&structTypeSource, serializedData, sizeof(structTypeSource));
// get source struct size
int structSize = GetStructSize(structTypeSource);
if (structTypeSource == structTypeDestination && structSize != 0) {
*structPointer = malloc(structSize);
memcpy(*structPointer, serializedData+sizeof(structTypeSource), structSize);
}
}
Serialization usage example:
MyStruct_2 structInput = {0x69, 0.1};
MyStruct_1 * structOutput_1 = NULL;
MyStruct_2 * structOutput_2 = NULL;
unsigned char testSerializedData[SERIALIZED_DATA_MAX_SIZE] = {0};
// serialize structInput
BinarySerialize(ST_MYSTRUCT_2, &structInput, testSerializedData);
// try to de-serialize to something
BinaryDeserialize(ST_MYSTRUCT_1, &structOutput_1, testSerializedData);
BinaryDeserialize(ST_MYSTRUCT_2, &structOutput_2, testSerializedData);
// determine which object was de-serialized
// (plus you will get code-completion support about object members from IDE)
if (structOutput_1 != NULL) {
// do something with structOutput_1
free(structOutput_1);
}
else if (structOutput_2 != NULL) {
// do something with structOutput_2
free(structOutput_2);
}
I think this is most simple serialization approach in C. But it has some problems:
struct must not have pointers, because you will never know how much memory one needs to allocate when serializing pointers and from where/how to serialize data into pointers.
this example has issues with system endianess - you need to be careful about how data is stored in memory - in big-endian or little-endian fashion and reverse bytes if needed [when casting char * to integal type such as enum] (...or refactor code to be more portable).

If you can use C++, there is the PicklingTools library