We Keep Coding

SWIG/Python: Unable to access referenced parameter in Python callback

In SWIG, I've got a wrapped C++ base class that is inherited in Python. The C++ side invokes callbacks on the Python derived class, which works fine. Unfortunately, one of the parameters of the callback is passed as reference. Accessing this reference yields:
Fatal unhandled exception: SWIG director method error. Error detected
when calling 'Base.Start'
test.i
%module(directors="1", allprotected="1") test
%feature("director") Base;
class Base
{
public:
virtual bool Start(const Property& config) = 0; // PASSED AS REFERENCE
};
enum PropertyType{PT_REAL, PT_STRING};
class Property
{
public:
PropertyType GetType() const;
int GetSize() const;
};
In Python, Start is correctly callback/invoked. Unfortunately, invoking GetType() or GetSize() yields the error above. Invoking functions of parameters passed as pointer is going fine.
import test
class PyClient(test.Base):
def Start(self, config):
config_size = config.GetSize() // YIELDS ERROR
config_type = config.GetType() // YIELDS ERROR
return True
I guess I need to convert the parameter Property from a reference to a pointer, but it is unclear to me how this works in SWIG.
UPDATE
It seems the argument in the invoked callback has a different underlying type than when created on the Python side.
def Start(self, config):
prop_test = test.Property()
type_test = prop_test.GetType() #FINE
type_test2 = config.GetType() # ERROR
When a Property (prop_test) is created on the Python side in Start(), its type is
<mds.Property; proxy of <Swig Object of type 'Property *' at 0x000001CC24DA3D80> >
Whereas the passed property has a type
<Swig Object of type 'Base::Property *' at 0x000001CC24CBE1E0>
I wonder whether this is expected, or this might lead to the underlying issue.
Swig 4.0.2
Any help would be really appreciated.
Ben

You've not provided a reproducible example, so I just filled in the blanks of missing code and it worked. Here's my working example if it helps:
test.i:
%module(directors="1", allprotected="1") test
%feature("director") Base;
%inline %{
enum PropertyType{PT_REAL, PT_STRING};
class Property
{
PropertyType m_prop;
int m_size;
public:
Property(PropertyType prop, int size) : m_prop(prop), m_size(size) {}
PropertyType GetType() const { return m_prop; }
int GetSize() const { return m_size; }
};
class Base
{
public:
virtual bool Start(const Property& config) = 0;
virtual ~Base() {}
};
void demo(Base* obj) {
Property prop(PT_STRING,2);
obj->Start(prop);
}
%}
ex.py:
import test
class PyClient(test.Base):
def Start(self, config):
config_size = config.GetSize()
config_type = config.GetType()
print(f'{config_size=} {config_type=}')
return True
p = PyClient()
test.demo(p)
Output:
config_size=2 config_type=1

Ah, I finally figured it out, turned out that the Property class was in a different namespace in the C++ code, but not in the SWIG interface file.
By setting the right namespace, a proxy was generated and everything worked fine.

Increase ref_count of pybind11 handle in C++

I have a virtual object that I subclass and implement in python using pybind11, and that object then is used in C++ code but the python object goes out of scope when the python script finished and python garbage collector destroys my object before I can use it in C++ side.
To avoid the destruction I have added a "keep" method to the binding code that increases the ref_count, so that python garbage collector doesn't destroy it.
C++ code for the bindings
VNCS::Real VNCS::py::BlendingField::blending(const VNCS::Point_3 &p) const
{
std::array<double, 3> pyP{p[0], p[1], p[2]};
PYBIND11_OVERLOAD_PURE(VNCS::Real, /* Return type */
VNCS::BlendingField, /* Parent class */
blending, /* Name of function in C++ (must match Python name) */
pyP /* Argument(s) */
);
}
void VNCS::py::module::blendingField(pybind11::module &m)
{
pybind11::class_<VNCS::BlendingField, VNCS::py::BlendingField, std::shared_ptr<VNCS::BlendingField>>(
m, "BlendingField")
.def(pybind11::init(
[](pybind11::args &args, pybind11::kwargs &kwargs) { return std::make_shared<VNCS::py::BlendingField>(); }))
.def("keep", [](pybind11::handle handle) { handle.inc_ref(); });
}
Python code
class BlendingField(PyVNCS.BlendingField):
def __init__(self, *args, **kwargs):
PyVNCS.BlendingField.__init__(self, *args, **kwargs)
def __del__(self):
print("dtor from python")
def blending(self, point):
return point[0]
blendingField = BlendingField()
blendingField.keep()
simCreator = initObject.addObject("SimCreator",
name="simCreator",
coarseMesh="#lowResTopology",
detailMesh="#highResTopology")
simCreator.setBlendingField(blendingField)
This doesn't look very nice, as the keep method there feels like a terrible hack. What would be the correct way to implement this?

I don't think messing with the ref count is the best solution. If you want to use an object in C++-land, use C++ lifetime management tools.
Pybind integrates lifetime management into the language bindings, you just need to change it's behavior. To quote the docs:
The binding generator for classes, class_, can be passed a template type that denotes a special holder type that is used to manage references to the object. If no such holder type template argument is given, the default for a type named Type is std::unique_ptr<Type>, which means that the object is deallocated when Python’s reference count goes to zero.
It is possible to switch to other types of reference counting wrappers or smart pointers, which is useful in codebases that rely on them. For instance, the following snippet causes std::shared_ptr to be used instead.
Here's an example lifted from the tests:
// Object managed by a std::shared_ptr<>
class MyObject2 {
public:
MyObject2(const MyObject2 &) = default;
MyObject2(int value) : value(value) { print_created(this, toString()); }
std::string toString() const { return "MyObject2[" + std::to_string(value) + "]"; }
virtual ~MyObject2() { print_destroyed(this); }
private:
int value;
};
py::class_<MyObject2, std::shared_ptr<MyObject2>>(m, "MyObject2")
.def(py::init<int>());
m.def("make_myobject2_1", []() { return new MyObject2(6); });
m.def("make_myobject2_2", []() { return std::make_shared<MyObject2>(7); });
m.def("print_myobject2_1", [](const MyObject2 *obj) { py::print(obj->toString()); });
m.def("print_myobject2_2", [](std::shared_ptr<MyObject2> obj) { py::print(obj->toString()); });
m.def("print_myobject2_3", [](const std::shared_ptr<MyObject2> &obj) { py::print(obj->toString()); });
m.def("print_myobject2_4", [](const std::shared_ptr<MyObject2> *obj) { py::print((*obj)->toString()); });
If you do go that route, which I would recommend, this question would be a duplicate of this one.
Note that there are some minor pitfalls here especially if youre codebase uses raw pointers around python code. The docs point to a possible double-free if a C++ method returning a raw pointer to an shared object is called from python. Again stealing from the docs:
class Child { };
class Parent {
public:
Parent() : child(std::make_shared<Child>()) { }
Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */
private:
std::shared_ptr<Child> child;
};
PYBIND11_MODULE(example, m) {
py::class_<Child, std::shared_ptr<Child>>(m, "Child");
py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent")
.def(py::init<>())
.def("get_child", &Parent::get_child);
}
from example import Parent
print(Parent().get_child()) // UB or Segfault
Though these issue don't seem to me to be unique to shared_ptr over unique_ptr.

pybind11 how to use custom type caster for simple example class

Motivation
I am currently trying to use a custom class both in python and in c++ with pybind11. The motivation behind this is to use a python-trained classifier within c++.
There are some working examples, e.g. in the official documentation https://pybind11.readthedocs.io/en/stable/advanced/cast/custom.html or by the nice examples from tdegeus https://github.com/tdegeus/pybind11_examples/blob/master/09_numpy_cpp-custom-matrix/pybind_matrix.h
However, i still have problems to transfer this to a simple example with a custom class.
Below is a minimal working example with a wrapper function for pybind11 in c++ and the use of it in python.
#include <vector>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
namespace py = pybind11;
//A custom data class
class class_DATA{
public:
std::vector<int> a;
};
//the actual function
class_DATA func_test(class_DATA data){
std::vector<int> a = {1,2};
data.a = a;
return data;
}
//the wrapper function that communicates with python
py::object wrapper(py::object data_py){
class_DATA data_cpp = class_DATA();
py::object a = data_py.attr("a");
data_cpp.a = a.cast<std::vector<int>>();
class_DATA data_result_cpp = func_test(data_cpp);
py::object data_result_py = data_py;
data_result_py.attr("a") = py::cast(data_result_cpp.a);
return data_result_py;
}
//defining the modules
PYBIND11_MODULE(TEST,m){
m.doc() = "pybind11 example plugin";
//the test function
m.def("func_test", &wrapper);
//the custom class
py::class_<class_DATA>(m, "class_DATA", py::dynamic_attr())
.def(py::init<>())
.def_readwrite("a", &class_DATA::a);
}
from build.Debug.TEST import func_test
//the custom data class in python
class class_DATA_py:
def __init__(self):
self.a = [0]
test_py = class_DATA_py()
data_return = func_test(test_py)
The output input is data.a = 0 and the output data_return.a = [1,2] as desired.
Question
How can I replace the wrapper function call with a custom caster? Most likely with the general shape outlined below (?)
namespace pybind11 { namespace detail {
template <> struct type_caster<class_DATA> : public type_caster_base<class_DATA> {
using base = type_caster_base<class_DATA>;
public:
PYBIND11_TYPE_CASTER(class_DATA, _("class_DATA"));
// Conversion part 1 (Python->C++):
bool load(py::handle src, bool convert){
PyObject *source = src.ptr();
//what to do here?
return true;
}
// Conversion part 2 (C++ -> Python):
static py::handle cast(class_DATA src, py::return_value_policy policy, py::handle parent){
//what to do here?
return base::cast(src, policy, parent);
}
};
}}
c++ is not my forte and python crashes repeatedly without any error messages when I meddle with this part, so any help would be really appreciated.

Create a property with a call policy - boost::python

I have the following C++ classes that I would like expose to python.
class PlainOldData
{
...
};
class FancyClass
{
public:
const PlainOldData& GetMyPOD() {return myPOD;}
private:
PlainOldData myPOD;
};
Because I want my python classes to be pythonic, I would like expose myPOD as a property. However, when I try to do the following:
// expose with boost::python
BOOST_PYTHON_MODULE(mymod)
{
class_<PlainOldData>("PlainOldData", init<>());
// this fails
class_<FancyClass>("FancyClass", init<>())
.add_property("myPOD", &FancyClass::GetMyPOD);
}
I get the following error: error C2027: use of undefined type 'boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<T>'
But, if I try to specify a call policy, such as:
class_<FancyClass>("FancyClass", init<>())
.add_property("myPOD", &FancyClass::GetMyPOD, return_value_policy<copy_const_reference>());
I get an incredibly long error message.
Is it possible to expose this function as a property; am I doing something wrong?

Similar to how Python's property() is passed Python callable objects, the boost::python::class_::add_property() function can accept Python callable objects that can be created with CallPolicies, such as those returned from boost::python::make_function().
For example, the property in the original code could be exposed like:
class_<FancyClass>("FancyClass", init<>())
.add_property("myPOD", make_function(&FancyClass::GetMyPOD,
return_value_policy<copy_const_reference>()));
Here is a complete minimal example:
#include <boost/python.hpp>
class egg {};
class spam
{
public:
const egg& get_egg() { return egg_; }
private:
egg egg_;
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<egg>("Egg");
python::class_<spam>("Spam")
.add_property("egg", python::make_function(&spam::get_egg,
python::return_value_policy<python::copy_const_reference>()))
;
}
Interactive usage:
>>> import example
>>> spam = example.Spam()
>>> assert(spam.egg is not spam.egg) # unique identities as spam.egg
# returns a copy
>>> egg1 = spam.egg
>>> assert(egg1 is not spam.egg)
>>> egg2 = spam.egg
>>> assert(egg1 is not egg2)

Pass a type object (class, not an instance) from python to c++

I would like to have a boost::python-wrapped c++ function which is able to receive type (rather than an instance), a boost::python-wrapped c++ class. I can declare the wrapped function taking an object, but how to extract the type? I tried something like this, but type objects don't seem to be extract-able:
#include<boost/python.hpp>
namespace py=boost::python;
struct A {};
struct B: public A {};
int func(py::object klass) {
py::extract<std::type_info> T(klass);
if(!T.check()) throw std::runtime_error("Unable to extract std::type_info");
if(T()==typeid(A)) return 0;
if(T()==typeid(B)) return 1;
return -1;
}
BOOST_PYTHON_MODULE(deadbeef)
{
py::def("func",func);
py::class_<A>("A");
py::class_<B,py::bases<A>>("B");
}
Compiled with
clang++ -lboost_python -fPIC `pkg-config python --cflags` a.cc -std=c++11 -shared -o deadbeef.so
I run
PYTHONPATH=. python
>>> import deadbeef
>>> deadbeef.func(deadbeef.A) ## I want this to return 0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: Unable to extract std::type_info
Thanks for any ideas.

To pass a Python type object, one needs to create a C++ type and register a custom a custom converter. As a Python type object is a python object, creating a type that derives from boost::python::object is appropriate:
/// #brief boost::python::object that refers to a type.
struct type_object:
public boost::python::object
{
/// #brief If the object is a type, then refer to it. Otherwise,
/// refer to the instance's type.
explicit
type_object(boost::python::object object):
boost::python::object(get_type(object))
{}
private:
/// #brief Get a type object from the given borrowed PyObject.
static boost::python::object get_type(boost::python::object object)
{
return PyType_Check(object.ptr())
? object
: object.attr("__class__");
}
};
// ... register custom converter for type_object.
However, the example code presents an additional problem. One cannot directly perform comparisons between a Python type object and a C++ type. Furthermore, The Python type object has no direct association with the C++ type. To perform comparisons, one needs to compare the Python type objects.
Boost.Python uses an internal registry to associate C++ type identity, in the form of boost::python::type_info, to a Python class object. This association is one-way, in that one can only lookup a Python class object. Lets expand the type_object class to allow to provide auxiliaries functions for checking against C++ types:
/// #brief boost::python::object that refers to a type.
struct type_object:
public boost::python::object
{
...
/// #brief Type identity check. Returns true if this is the object returned
/// returned from type() when passed an instance of an object created
/// from a C++ object with type T.
template <typename T>
bool is() const
{
// Perform an identity check that registartion for type T and type_object
// are the same Python type object.
return get_class_object<T>() == static_cast<void*>(ptr());
}
/// #brief Type identity check. Returns true if this is the object is a
/// subclass of the type returned returned from type() when passed
/// an instance of an object created from a C++ object with type T.
template <typename T>
bool is_subclass() const
{
return PyType_IsSubtype(reinterpret_cast<PyTypeObject*>(ptr()),
get_class_object<T>());
}
private:
...
/// #brief Get the Python class object for C++ type T.
template <typename T>
static PyTypeObject* get_class_object()
{
namespace python = boost::python;
// Locate registration based on the C++ type.
const python::converter::registration* registration =
python::converter::registry::query(python::type_id<T>());
// If registration exists, then return the class object. Otherwise,
// return NULL.
return (registration) ? registration->get_class_object()
: NULL;
}
};
Now, if type is an instance of type_object, one could check:
If type is the Python type associated with the C++ Spam type with type.is<Spam>().
If type is a subclass of the Python type associated with the C++ Spam type with type.is_subclass<Spam>().
Here is a complete example based on the original code that demonstrates receiving type objects to functions, checking for type identity and subclasses:
#include <boost/python.hpp>
/// #brief boost::python::object that refers to a type.
struct type_object:
public boost::python::object
{
/// #brief If the object is a type, then refer to it. Otherwise,
/// refer to the instance's type.
explicit
type_object(boost::python::object object):
boost::python::object(get_type(object))
{}
/// #brief Type identity check. Returns true if this is the object returned
/// returned from type() when passed an instance of an object created
/// from a C++ object with type T.
template <typename T>
bool is() const
{
// Perform an identity check that registartion for type T and type_object
// are the same Python type object.
return get_class_object<T>() == static_cast<void*>(ptr());
}
/// #brief Type identity check. Returns true if this is the object is a
/// subclass of the type returned returned from type() when passed
/// an instance of an object created from a C++ object with type T.
template <typename T>
bool is_subclass() const
{
return PyType_IsSubtype(reinterpret_cast<PyTypeObject*>(ptr()),
get_class_object<T>());
}
private:
/// #brief Get a type object from the given borrowed PyObject.
static boost::python::object get_type(boost::python::object object)
{
return PyType_Check(object.ptr())
? object
: object.attr("__class__");
}
/// #brief Get the Python class object for C++ type T.
template <typename T>
static PyTypeObject* get_class_object()
{
namespace python = boost::python;
// Locate registration based on the C++ type.
const python::converter::registration* registration =
python::converter::registry::query(python::type_id<T>());
// If registration exists, then return the class object. Otherwise,
// return NULL.
return (registration) ? registration->get_class_object()
: NULL;
}
};
/// #brief Enable automatic conversions to type_object.
struct enable_type_object
{
enable_type_object()
{
boost::python::converter::registry::push_back(
&convertible,
&construct,
boost::python::type_id<type_object>());
}
static void* convertible(PyObject* object)
{
return (PyType_Check(object) || Py_TYPE(object)) ? object : NULL;
}
static void construct(
PyObject* object,
boost::python::converter::rvalue_from_python_stage1_data* data)
{
// Obtain a handle to the memory block that the converter has allocated
// for the C++ type.
namespace python = boost::python;
typedef python::converter::rvalue_from_python_storage<type_object>
storage_type;
void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;
// Construct the type object within the storage. Object is a borrowed
// reference, so create a handle indicting it is borrowed for proper
// reference counting.
python::handle<> handle(python::borrowed(object));
new (storage) type_object(python::object(handle));
// Set convertible to indicate success.
data->convertible = storage;
}
};
// Mockup types.
struct A {};
struct B: public A {};
struct C {};
/// Mockup function that receives an object's type.
int func(type_object type)
{
if (type.is<A>()) return 0;
if (type.is<B>()) return 1;
return -1;
}
/// Mockup function that returns true if the provided object type is a
/// subclass of A.
bool isSubclassA(type_object type)
{
return type.is_subclass<A>();
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Enable receiving type_object as arguments.
enable_type_object();
python::class_<A>("A");
python::class_<B, python::bases<A> >("B");
python::class_<C>("C");
python::def("func", &func);
python::def("isSubclassA", &isSubclassA);
}
Interactive usage:
>>> import example
>>> assert(example.func(type("test")) == -1)
>>> assert(example.func(example.A) == 0)
>>> assert(example.func(example.B) == 1)
>>> assert(example.isSubclassA(example.A))
>>> assert(example.isSubclassA(example.B))
>>> assert(not example.isSubclassA(example.C))
>>> assert(example.func("test") == -1)
>>> assert(example.func(example.A()) == 0)
>>> assert(example.func(example.B()) == 1)
>>> assert(example.isSubclassA(example.A()))
>>> assert(example.isSubclassA(example.B()))
>>> assert(not example.isSubclassA(example.C()))