We Keep Coding

PyBind11: How to implement Vector of Shared Pointers of Classes with Inheritence? Unable to load a custom holder type from a default-holder instance

I have a base class in c++ that looks like:
class EdgeAttributePipelineStep {
public:
virtual void setAttributes(Edge& edge) = 0;
};
And then some classes that inherit from this class:
class FuelCostStep : public EdgeAttributePipelineStep {
public:
FuelCostStep(float fuelCostPerTonne)
: m_FuelCostPerTonne(fuelCostPerTonne) {};
void setAttributes(Edge& edge);
private:
float m_FuelCostPerTonne;
};
class HireCostStep : public EdgeAttributePipelineStep {
public:
HireCostStep(float hireCostPerDay) : m_HireCostPerHour(hireCostPerDay/24) {};
void setAttributes(Edge &edge);
private:
float m_HireCostPerHour;
};
And then finally, a class that accepts a vector of the base class in its constructor:
class EdgeAttributePipeline {
public:
EdgeAttributePipeline(std::vector<std::shared_ptr<EdgeAttributePipelineStep>> steps);
private:
std::vector<std::shared_ptr<EdgeAttributePipelineStep>> m_steps;
};
I want to be able to create the pipeline class in python like:
pipeline = EdgeAttributePipeline([
FuelCostStep(10),
HireCostStep(10)
])
And I attempted some python bindings using PyBind11 to do this:
#include "edge_attributes.h"
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
PYBIND11_MODULE(WeatherRouting, m)
{
py::class_<EdgeAttributePipelineStep>(m, "EdgeAttribtuePipelineStep");
// I tried both using std::shared_ptr
py::class_<FuelCostStep, std::shared_ptr<FuelCostStep>>(m, "FuelCostStep")
.def(py::init<double>());
// And not using std::shared_ptr
py::class_<HireCostStep, EdgeAttributePipelineStep>(m, "HireCostStep")
.def(py::init<double>());
py::class_<EdgeAttributePipeline>(m, "EdgeAttributePipeline")
.def(py::init<std::vector<std::shared_ptr<EdgeAttributePipelineStep>>>());
}
However both of these attempts resulted in the same error:
>>> EdgeAttributePipeline([FuelCostStep(10)])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: Unable to load a custom holder type from a default-holder instance
>>> EdgeAttributePipeline([HireCostStep(10)])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: Unable to load a custom holder type from a default-holder instance
I also had a look at making a trampoline class:
class PyEdgeAttributePipelineStep : public EdgeAttributePipelineStep {
public:
using EdgeAttributePipelineStep::EdgeAttributePipelineStep;
};
And modifying the pybind code:
py::class_<EdgeAttributePipelineStep, PyEdgeAttributePipelineStep, std::shared_ptr<EdgeAttributePipelineStep>>(m, "EdgeAttribtuePipelineStep");
py::class_<FuelCostStep, EdgeAttributePipelineStep>(m, "FuelCostStep")
.def(py::init<double>());
However this now makes a new error when ran in python:
ImportError: generic_type: type "FuelCostStep" does not have a non-default holder type while its base "EdgeAttributePipelineStep" does
I am a bit stuck trying to implement this. How do I get this code to work?

You need to declare the same holder for all related types (so the base, child, grandchild, etc). You forgot to set a shared_ptr holder for HireCostStep.
FWIW, I use the smart_holder branch of pybind11 and it removes all of this holder related pain. I don't understand why it isn't in master by now.

Swig exception in Python director class

I'm wrapping some classes using Swig directors in Python.
Each of my class methods returns a MyError class. What happens is, once I derive a python class from one of my C++ and I forget to return the MyError() object but I return None of "pass" or I forget to return anything, my software crashes in the default constructor of the MyError class through a read access violation, and I'm not able to track this exception through a try/catch block.
What is the correct way in Swig to handle such kind of situations ?
Thanks!

Section 36.5.4 Exception unrolling in the SWIG documentation:
With directors routing method calls to Python, and proxies routing them to C++, the handling of exceptions is an important concern. By default, the directors ignore exceptions that occur during method calls that are resolved in Python. To handle such exceptions correctly, it is necessary to temporarily translate them into C++ exceptions. This can be done with the %feature("director:except") directive. The following code should suffice in most cases:
%feature("director:except") {
if ($error != NULL) {
throw Swig::DirectorMethodException();
}
}
This code will check the Python error state after each method call from a director into Python, and throw a C++ exception if an error occurred. This exception can be caught in C++ to implement an error handler. Currently no information about the Python error is stored in the Swig::DirectorMethodException object, but this will likely change in the future.
It may be the case that a method call originates in Python, travels up to C++ through a proxy class, and then back into Python via a director method. If an exception occurs in Python at this point, it would be nice for that exception to find its way back to the original caller. This can be done by combining a normal %exception directive with the director:except handler shown above. Here is an example of a suitable exception handler:
%exception {
try { $action }
catch (Swig::DirectorException &e) { SWIG_fail; }
}
The class Swig::DirectorException used in this example is actually a base class of Swig::DirectorMethodException, so it will trap this exception. Because the Python error state is still set when Swig::DirectorMethodException is thrown, Python will register the exception as soon as the C wrapper function returns.
Example
Here's an example test.i that illustrates the technique:
%module test
%module(directors="1") test
%feature("director");
%feature("director:except") {
if ($error != NULL) {
throw Swig::DirectorMethodException();
}
}
%exception {
try { $action }
catch (Swig::DirectorException &e) { SWIG_fail; }
}
%inline %{
class MyError {
int m_n;
public:
MyError(int n = 0) : m_n(n) {}
~MyError() {}
int get() const { return m_n; }
};
class Demo {
public:
Demo() {}
virtual ~Demo() {}
virtual MyError test() { return MyError(5); }
};
int func(Demo* d) { return d->test().get(); }
%}
After swigging and compiling, a demo:
>>> import test
>>> d=test.Demo() # default class implementation
>>> test.func(d) # Call virtual method in a C++ test function.
5
The above worked normally. Below overrides incorrectly:
>>> class Demo2(test.Demo): # New class
... def test(self): # Override virtual function
... return 7 # But don't return a MyError object.
...
>>> d=Demo2()
>>> test.func(d)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: SWIG director type mismatch in output value of type 'MyError'
That caught the exception and returned a useful exception. Below overrides correctly:
>>> class Demo2(test.Demo):
... def test(self):
... return test.MyError(7)
...
>>> d=Demo2()
>>> test.func(d)
7

How do I use SWIG to return specific exceptions based on C function return type?

Here are examples of functions in my C API:
int do_something(struct object *with_this)
struct object *get_something(struct object *from_this)
do_something will return 0 for success or -1 and set errno for failure. get_something will return a valid pointer on success or NULL and set errno for failure.
I'm using to SWIG 2.0 to generate python bindings. For the do_something binding I'd like it to return an exception based on errno if it fails and return the Python None object if it succeeds. For the get_something binding I'd like it to return an exception based on errno if it fails and the opaque object if it succeeds.
The question is how do I get SWIG to do all of this magic for me?
Currently I'm using SWIG 2.0.
I can use %exception and define different exceptions per symbol, but I'd like it to be based on return type. I'll have a lot of the API functions and I don't want to list them out. So I can do this for each symbol:
%exception do_something {
$action
if (result < 0) {
return PyErr_SetFromErrno(PyExc_RuntimeError);
}
}
%exception get_something {
$action
if (result == NULL) {
return PyErr_SetFromErrno(PyExc_RuntimeError);
}
}
It would be a lot better if I could do something like this (like you can do with %typemap):
%exception int {
$action
if (result < 0) {
return PyErr_SetFromErrno(PyExc_RuntimeError);
}
}
%exception struct object * {
$action
if (result == NULL) {
return PyErr_SetFromErrno(PyExc_RuntimeError);
}
}
Can I do something like this? If I had something like this, then that would take care of the exceptions. And I believe my get_something binding would be perfect.
My do_something binding would still need something like this to get it to return the Python None object if I use %typemap:
%typemap(out) int {
$result = Py_BuildValue("");
}
Does the magic I'm looking for exist or am I going about this all wrong? Is there a better way to do this?

To help answer this question I took your functions and created the following test.h file that included enough real code to actually illustrate the problem:
struct object { int bar; };
static int do_something(struct object *with_this) {
errno = EACCES;
return -1;
}
static struct object *get_something(struct object *from_this) {
errno = EAGAIN;
return NULL;
}
I'm fairly sure that %typemap(out) is the place to write this code in, not %exception, because the behaviour you want is determined by the return type, not the name of the function called. The fact that you also care about the value of the return further backs this up. You can to some degree avoid repetition using $typemap to reference other typemaps within yours.
So I wrote a little SWIG interface to illustrate this:
%module test
%{
#include <errno.h>
#include "test.h"
%}
%typemap(out) int do_something %{
if ($1 < 0) {
PyErr_SetFromErrno(PyExc_RuntimeError);
SWIG_fail;
}
$result = Py_None; // Return only controls exception
Py_INCREF($result);
%}
%typemap(out) struct object *get_something %{
if (!$1) {
PyErr_SetFromErrno(PyExc_RuntimeError);
SWIG_fail;
}
// Return passed through unless NULL
$typemap(out,$1_ltype);
%}
%include "test.h"
Which worked when tested:
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> test.do_something(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: (13, 'Permission denied')
>>> test.get_something(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: (11, 'Resource temporarily unavailable')
>>>
This only worked though because I explicitly named do_something when matching the typemap, so the fallback without the named function is fine. If you just try to remove that restriction you'll get an error about recursive typemaps. You can work around that using %apply, e.g.
%typemap(out) struct object * MY_NULL_OR_ERRNO %{
if (!$1) {
PyErr_SetFromErrno(PyExc_RuntimeError);
SWIG_fail;
}
$typemap(out,$1_ltype);
%}
%apply struct object * MY_NULL_OR_ERRNO { struct object *get_something, struct object *some_other_function };
If that's too painful you can of course just write the typemap by hand:
%typemap(out) struct object * %{
if (!$1) {
PyErr_SetFromErrno(PyExc_RuntimeError);
SWIG_fail;
}
$result = SWIG_NewPointerObj(SWIG_as_voidptr($1), $1_descriptor, $owner);
%}
Which seems the simplest option if you really want that everywhere a struct object* gets returned.
There are other possible solutions using %pythoncode or %pythonappend, but neither of them are really improvements over the above in my view.

TypeError when iterating over a boost::python vector

I get a very similar issue as in:
unexpected result iterating over a boost::python vector_indexing_suite
I have the following C++ code:
#include <vector>
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
using namespace boost::python;
class Composite {
public:
std::string name;
std::vector<Composite*>& getChildren() {
return children;
};
std::vector<Composite*> children;
};
typedef std::vector<Composite*> CompositeArray;
BOOST_PYTHON_MODULE(coin)
{
class_<CompositeArray>("CompositeArray")
.def(vector_indexing_suite<CompositeArray, true>());
class_<Composite>("Composite", "Composite*", init<>())
.def("getChildren", &Composite::getChildren, return_internal_reference<>())
.def_readwrite("name", &Composite::name, "str")
;
}
And the following Python code:
import coin
gna = coin.Composite()
gna.name = "test"
gna2 = coin.Composite()
gna2.name = "test2"
gna.getChildren().append(gna2)
for slip in gna.getChildren():
print(slip.name)
produces the error:
Traceback (most recent call last):
File "gna.py", line 34, in <module>
for prout in gna.getChildren():
TypeError: No to_python (by-value) converter found for C++ type: class Composite * __ptr64
This was working fine with Boost 1.59, but not anymore with Boost 1.60.
Any idea ?
Edit:
I tried by updating (see proposal below) :
class_<Composite>("Composite", "Composite*", init<>())
to:
class_<Composite, Composite*>("Composite", "Composite*", init<>())
And I confirm that it is working with boost 1.59 but not boost 1.60.

Changing
class_<Composite>("Composite", "Composite*", init<>())
to
class_<Composite, Composite*>("Composite", "Composite*", init<>())
helped with boost 1.55.

I could find any "nice" solution for this issue and migrated all my raw pointers to std::shared_ptr to make it work. This issue is discussed here:
https://github.com/boostorg/python/issues/29
and here:
https://github.com/boostorg/python/issues/56

Strange TypeError occured when calling method function exposed by boost python

I write a python wrapper for box2d, everything work perfect but a strange TypeError error occurs sometimes when calling method function exposed by boost python. It's a random behavior, not happened every time.
The problem python code:
try:
world = body.GetWorld() # world is b2World instance, body is b2Body instance
world.DestroyBody(body) # raise TypeError: 'NoneType' object is not callable
except TypeError: # catch it, and print some infomation
print "xxxxx", world # I got a b2World instance here
print "xxxxx", sys.getrefcount(world) # I got a value of 66 here
print "xxxxx", world.DestroyBody # I got a bound method object here
raise
It seems all okay. How does that happened?
And part of my wrapper code:
// [file]: https://github.com/layzerar/box2d-py/blob/master/python/world.cpp
// [project]: https://github.com/layzerar/box2d-py
template<class T>
inline void b2Func_ClearUserData(T& self)
{
using namespace boost::python;
xdecref((PyObject*)self.GetUserData());
self.SetUserData(NULL);
}
inline void b2World_DestroyBody(b2World& self, b2Body* body)
{
b2Assert(self.GetBodyCount() > 0);
b2Assert(self.IsLocked() == false);
b2Func_ClearUserData(*body);
self.DestroyBody(body);
}
class_<b2World, b2World_W , boost::noncopyable>("b2World")
//...
.def("CreateBody", b2World_CreateBody, return_internal_reference<>())
.def("DestroyBody", b2World_DestroyBody)
//...
;
Did I make an obvious mistake?

After a few days exploring I found the answer.
I make a mistake about the question, the exception is really not thrown by DestroyBody. It's thrown by a virtual function which is rewrite in python and called in DestroyBody function.