We Keep Coding

SWIG/Python: Unable to access referenced parameter in Python callback

In SWIG, I've got a wrapped C++ base class that is inherited in Python. The C++ side invokes callbacks on the Python derived class, which works fine. Unfortunately, one of the parameters of the callback is passed as reference. Accessing this reference yields:
Fatal unhandled exception: SWIG director method error. Error detected
when calling 'Base.Start'
test.i
%module(directors="1", allprotected="1") test
%feature("director") Base;
class Base
{
public:
virtual bool Start(const Property& config) = 0; // PASSED AS REFERENCE
};
enum PropertyType{PT_REAL, PT_STRING};
class Property
{
public:
PropertyType GetType() const;
int GetSize() const;
};
In Python, Start is correctly callback/invoked. Unfortunately, invoking GetType() or GetSize() yields the error above. Invoking functions of parameters passed as pointer is going fine.
import test
class PyClient(test.Base):
def Start(self, config):
config_size = config.GetSize() // YIELDS ERROR
config_type = config.GetType() // YIELDS ERROR
return True
I guess I need to convert the parameter Property from a reference to a pointer, but it is unclear to me how this works in SWIG.
UPDATE
It seems the argument in the invoked callback has a different underlying type than when created on the Python side.
def Start(self, config):
prop_test = test.Property()
type_test = prop_test.GetType() #FINE
type_test2 = config.GetType() # ERROR
When a Property (prop_test) is created on the Python side in Start(), its type is
<mds.Property; proxy of <Swig Object of type 'Property *' at 0x000001CC24DA3D80> >
Whereas the passed property has a type
<Swig Object of type 'Base::Property *' at 0x000001CC24CBE1E0>
I wonder whether this is expected, or this might lead to the underlying issue.
Swig 4.0.2
Any help would be really appreciated.
Ben

You've not provided a reproducible example, so I just filled in the blanks of missing code and it worked. Here's my working example if it helps:
test.i:
%module(directors="1", allprotected="1") test
%feature("director") Base;
%inline %{
enum PropertyType{PT_REAL, PT_STRING};
class Property
{
PropertyType m_prop;
int m_size;
public:
Property(PropertyType prop, int size) : m_prop(prop), m_size(size) {}
PropertyType GetType() const { return m_prop; }
int GetSize() const { return m_size; }
};
class Base
{
public:
virtual bool Start(const Property& config) = 0;
virtual ~Base() {}
};
void demo(Base* obj) {
Property prop(PT_STRING,2);
obj->Start(prop);
}
%}
ex.py:
import test
class PyClient(test.Base):
def Start(self, config):
config_size = config.GetSize()
config_type = config.GetType()
print(f'{config_size=} {config_type=}')
return True
p = PyClient()
test.demo(p)
Output:
config_size=2 config_type=1

Ah, I finally figured it out, turned out that the Property class was in a different namespace in the C++ code, but not in the SWIG interface file.
By setting the right namespace, a proxy was generated and everything worked fine.

Eigen::Matrix contained inside a struct is marked as non-writeable by pybind11

I'm using pybind11 to create a Python binding for my C++ library that uses Eigen types. However, when I try to bind a struct that contains an Eigen::Matrix, the corresponding generated Python class has all the ndarray (the Python destination class for Eigen::Matrix members) elements with the Writeable flag set to false, which are therefore non-modifiable. I've managed to reduce my problem to the following MWE:
// binding.cpp
#include "Eigen/Dense"
#include "pybind11/pybind11.h"
#include "pybind11/eigen.h"
struct Example {
Eigen::Matrix<double, 3, 1> m;
};
namespace py = pybind11;
PYBIND11_MODULE(MyBinding, m)
{
py::class_<Example>(m, "Example")
.def(py::init<>())
.def_readwrite("m", &Example::m);
}
# script.py
import MyBinding
ex = MyBinding.Example()
# ex.m has Writeable flag equal to false!
On the contrary, when binding the following code, everything works as expected, i.e. the value returned by f, when called in Python, is modifiable (has the Writeable flag set to true):
Eigen::Matrix<double,3,1> f() {
return {};
}
PYBIND11_MODULE(MyBinding, m)
{
m.def("f", &f);
}
What am I missing here? Thank you for any help!

def_readwrite is a shortcut for def_property (source link) where getter takes the class instance by const reference. Therefore resulting numpy array is not modifiable.
All you need to do is to write custom getter and setter functions:
namespace py = pybind11;
PYBIND11_MODULE(MyBinding, m)
{
py::class_<Example>(m, "Example")
.def(py::init<>())
.def_property("m",
[](Example& example){ return example.m; } // getter
[](Example& example, const Eigen::Matrix<double, 3, 1>& value ){ // setter
example.m = value;
}
);
}
Caution: untested code, you may need pass extra return value policy argument

pybind11 how to use custom type caster for simple example class

Motivation
I am currently trying to use a custom class both in python and in c++ with pybind11. The motivation behind this is to use a python-trained classifier within c++.
There are some working examples, e.g. in the official documentation https://pybind11.readthedocs.io/en/stable/advanced/cast/custom.html or by the nice examples from tdegeus https://github.com/tdegeus/pybind11_examples/blob/master/09_numpy_cpp-custom-matrix/pybind_matrix.h
However, i still have problems to transfer this to a simple example with a custom class.
Below is a minimal working example with a wrapper function for pybind11 in c++ and the use of it in python.
#include <vector>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
namespace py = pybind11;
//A custom data class
class class_DATA{
public:
std::vector<int> a;
};
//the actual function
class_DATA func_test(class_DATA data){
std::vector<int> a = {1,2};
data.a = a;
return data;
}
//the wrapper function that communicates with python
py::object wrapper(py::object data_py){
class_DATA data_cpp = class_DATA();
py::object a = data_py.attr("a");
data_cpp.a = a.cast<std::vector<int>>();
class_DATA data_result_cpp = func_test(data_cpp);
py::object data_result_py = data_py;
data_result_py.attr("a") = py::cast(data_result_cpp.a);
return data_result_py;
}
//defining the modules
PYBIND11_MODULE(TEST,m){
m.doc() = "pybind11 example plugin";
//the test function
m.def("func_test", &wrapper);
//the custom class
py::class_<class_DATA>(m, "class_DATA", py::dynamic_attr())
.def(py::init<>())
.def_readwrite("a", &class_DATA::a);
}
from build.Debug.TEST import func_test
//the custom data class in python
class class_DATA_py:
def __init__(self):
self.a = [0]
test_py = class_DATA_py()
data_return = func_test(test_py)
The output input is data.a = 0 and the output data_return.a = [1,2] as desired.
Question
How can I replace the wrapper function call with a custom caster? Most likely with the general shape outlined below (?)
namespace pybind11 { namespace detail {
template <> struct type_caster<class_DATA> : public type_caster_base<class_DATA> {
using base = type_caster_base<class_DATA>;
public:
PYBIND11_TYPE_CASTER(class_DATA, _("class_DATA"));
// Conversion part 1 (Python->C++):
bool load(py::handle src, bool convert){
PyObject *source = src.ptr();
//what to do here?
return true;
}
// Conversion part 2 (C++ -> Python):
static py::handle cast(class_DATA src, py::return_value_policy policy, py::handle parent){
//what to do here?
return base::cast(src, policy, parent);
}
};
}}
c++ is not my forte and python crashes repeatedly without any error messages when I meddle with this part, so any help would be really appreciated.

Exposing a class with a constructor containing a nested private class in constructor using Boost Python

I'm new to Boost Python and I'm looking to expose a class that looks like this:
///Header File structure
class A
{ public:
A();
~A();
void B();
private:
class Impl;
std::unique_ptr Impl impl_;
};
///Class Implementation
class A::Impl
{
public:
void C();
}
A::A():impl_(new Impl)
{
}
A::~A()
{
}
void A::B()
{
void C();
}
Can someone suggest how to do it since the current methods I've tried gives errors since Impl is private and also an accessing already deleted function error:
BOOST_PYTHON_MODULE(libA)
{
class_<A::Impl>("Impl")
.def("C", &A::Impl::C)
class_<A>("A",init<std::unique_ptr>)
.def("B", &A::B)
}

The whole point of the pimpl idiom is that it's private and completely transparent to the users of the class. You don't expose it.
What you do need to do is make it clear that A isn't copyable:
class_<A, noncopyable>("A", init<>())
.def("B", &A::B)
;

Boost.Python create handle from type

In some places in exposing C++ code to python, I need to use a PyObject*. If I have an instance of a boost::python::class_ object, I can invoke ptr() on it. But what if I just have the type?
Basically, given the type list boost::python::bases<A, B, C>, I want to convert this to a boost::python::tuple of instances that I can pass into something like PyErr_NewExceptionWithDoc(). Is this possible?

Given a C++ type T, one can create a boost::python::type_id object, and then query into the Boost.Python registry for registration information. If an entry is found in the registry, then one can use it to obtain a handle to the Python class created for type T:
/// #brief Get the class object for a wrapped type that has been exposed
/// through Boost.Python.
template <typename T>
boost::python::object get_instance_class()
{
// Query into the registry for type T.
namespace python = boost::python;
python::type_info type = python::type_id<T>();
const python::converter::registration* registration =
python::converter::registry::query(type);
// If the class is not registered, return None.
if (!registration) return python::object();
python::handle<PyTypeObject> handle(python::borrowed(
registration->get_class_object()));
return python::object(handle);
}
Here is a complete example demonstrating locating a Python class object in the Boost.Python registry:
#include <boost/python.hpp>
#include <iostream>
/// #brief Get the class object for a wrapped type that has been exposed
/// through Boost.Python.
template <typename T>
boost::python::object get_instance_class()
{
// Query into the registry for type T.
namespace python = boost::python;
python::type_info type = python::type_id<T>();
const python::converter::registration* registration =
python::converter::registry::query(type);
// If the class is not registered, return None.
if (!registration) return python::object();
python::handle<PyTypeObject> handle(python::borrowed(
registration->get_class_object()));
return python::object(handle);
}
struct spam {};
int main()
{
Py_Initialize();
namespace python = boost::python;
try
{
// Create the __main__ module.
python::object main_module = python::import("__main__");
python::object main_namespace = main_module.attr("__dict__");
// Create `Spam` class.
// >>> class Spam: pass
auto spam_class_object = python::class_<spam>("Spam", python::no_init);
// >>> print Spam
main_module.attr("__builtins__").attr("print")(get_instance_class<spam>());
// >>> assert(spam is spam)
assert(spam_class_object.ptr() == get_instance_class<spam>().ptr());
}
catch (python::error_already_set&)
{
PyErr_Print();
return 1;
}
}
Output:
<class 'Spam'>
For more type related functionality, such as accepting type objects, is, and issubclass, see this answer.