I have a virtual object that I subclass and implement in python using pybind11, and that object then is used in C++ code but the python object goes out of scope when the python script finished and python garbage collector destroys my object before I can use it in C++ side.
To avoid the destruction I have added a "keep" method to the binding code that increases the ref_count, so that python garbage collector doesn't destroy it.
C++ code for the bindings
VNCS::Real VNCS::py::BlendingField::blending(const VNCS::Point_3 &p) const
{
std::array<double, 3> pyP{p[0], p[1], p[2]};
PYBIND11_OVERLOAD_PURE(VNCS::Real, /* Return type */
VNCS::BlendingField, /* Parent class */
blending, /* Name of function in C++ (must match Python name) */
pyP /* Argument(s) */
);
}
void VNCS::py::module::blendingField(pybind11::module &m)
{
pybind11::class_<VNCS::BlendingField, VNCS::py::BlendingField, std::shared_ptr<VNCS::BlendingField>>(
m, "BlendingField")
.def(pybind11::init(
[](pybind11::args &args, pybind11::kwargs &kwargs) { return std::make_shared<VNCS::py::BlendingField>(); }))
.def("keep", [](pybind11::handle handle) { handle.inc_ref(); });
}
Python code
class BlendingField(PyVNCS.BlendingField):
def __init__(self, *args, **kwargs):
PyVNCS.BlendingField.__init__(self, *args, **kwargs)
def __del__(self):
print("dtor from python")
def blending(self, point):
return point[0]
blendingField = BlendingField()
blendingField.keep()
simCreator = initObject.addObject("SimCreator",
name="simCreator",
coarseMesh="#lowResTopology",
detailMesh="#highResTopology")
simCreator.setBlendingField(blendingField)
This doesn't look very nice, as the keep method there feels like a terrible hack. What would be the correct way to implement this?
I don't think messing with the ref count is the best solution. If you want to use an object in C++-land, use C++ lifetime management tools.
Pybind integrates lifetime management into the language bindings, you just need to change it's behavior. To quote the docs:
The binding generator for classes, class_, can be passed a template type that denotes a special holder type that is used to manage references to the object. If no such holder type template argument is given, the default for a type named Type is std::unique_ptr<Type>, which means that the object is deallocated when Python’s reference count goes to zero.
It is possible to switch to other types of reference counting wrappers or smart pointers, which is useful in codebases that rely on them. For instance, the following snippet causes std::shared_ptr to be used instead.
Here's an example lifted from the tests:
// Object managed by a std::shared_ptr<>
class MyObject2 {
public:
MyObject2(const MyObject2 &) = default;
MyObject2(int value) : value(value) { print_created(this, toString()); }
std::string toString() const { return "MyObject2[" + std::to_string(value) + "]"; }
virtual ~MyObject2() { print_destroyed(this); }
private:
int value;
};
py::class_<MyObject2, std::shared_ptr<MyObject2>>(m, "MyObject2")
.def(py::init<int>());
m.def("make_myobject2_1", []() { return new MyObject2(6); });
m.def("make_myobject2_2", []() { return std::make_shared<MyObject2>(7); });
m.def("print_myobject2_1", [](const MyObject2 *obj) { py::print(obj->toString()); });
m.def("print_myobject2_2", [](std::shared_ptr<MyObject2> obj) { py::print(obj->toString()); });
m.def("print_myobject2_3", [](const std::shared_ptr<MyObject2> &obj) { py::print(obj->toString()); });
m.def("print_myobject2_4", [](const std::shared_ptr<MyObject2> *obj) { py::print((*obj)->toString()); });
If you do go that route, which I would recommend, this question would be a duplicate of this one.
Note that there are some minor pitfalls here especially if youre codebase uses raw pointers around python code. The docs point to a possible double-free if a C++ method returning a raw pointer to an shared object is called from python. Again stealing from the docs:
class Child { };
class Parent {
public:
Parent() : child(std::make_shared<Child>()) { }
Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */
private:
std::shared_ptr<Child> child;
};
PYBIND11_MODULE(example, m) {
py::class_<Child, std::shared_ptr<Child>>(m, "Child");
py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent")
.def(py::init<>())
.def("get_child", &Parent::get_child);
}
from example import Parent
print(Parent().get_child()) // UB or Segfault
Though these issue don't seem to me to be unique to shared_ptr over unique_ptr.
This appears to be a known limitation in pybind11. I read through all the docs, whatever bug reports seemed applicable, and everything I could find in the pybind11 gitter. I have a custom exception class in c++ that contains custom constructors and fields. A very basic example of such a class, trimmed for space is here:
class BadData : public std::exception
{
public:
// Constructors
BadData()
: msg(),
stack(),
_name("BadData")
{}
BadData(std::string _msg, std::string _stack)
: msg(_msg),
stack(_stack),
_name("BadData")
{}
const std::string&
getMsg() const
{
return msg;
}
void
setMsg(const std::string& arg)
{
msg = arg;
}
// Member stack
const std::string&
getStack() const
{
return stack;
}
void
setStack(const std::string& arg)
{
stack = arg;
}
private:
std::string msg;
std::string stack;
std::string _name;
I currently have python binding code that binds this into python, but it is custom generated and we'd much rather use pybind11 due to its simplicity and compile speed.
The default mechanism for binding an exception into pybind11 would look like
py::register_exception<BadData>(module, "BadData");
That will create an automatic translation between the C++ exception and the python exception, with the what() value of the c++ exception translating into the message of the python exception. However, all the extra data from the c++ exception is lost and if you're trying to throw the exception in python and catch it in c++, you cannot throw it with any of the extra data.
You can bind extra data onto the python object using the attr and I even went somewhat down the path of trying to extend the pybind11:exception class to make it easier to add custom fields to exceptions.
template <typename type>
class exception11 : public ::py::exception<type>
{
public:
exception11(::py::handle scope, const char *name, PyObject *base = PyExc_Exception)
: ::py::exception<type>(scope, name, base)
{}
template <typename Func, typename... Extra>
exception11 &def(const char *name_, Func&& f, const Extra&... extra) {
::py::cpp_function cf(::py::method_adaptor<type>(std::forward<Func>(f)),
::py::name(name_),
::py::is_method(*this),
::py::sibling(getattr(*this, name_, ::py::none())),
extra...);
this->attr(cf.name()) = cf;
return *this;
}
};
This adds a def function to exceptions similar to what is done with class_. The naive approach to using this doesn't work
exception11< ::example::data::BadData>(module, "BadData")
.def("getStack", &::example::data::BadData::getStack);
Because there is no automatic translation between BadData in c++ and in python. You can try to work around this by binding in a lambda:
.def("getStack", [](py::object& obj) {
::example::data::BadData *cls = obj.cast< ::example::data::BadData* >();
return cls->getStack();
});
The obj.cast there also fails because there is no automatic conversion. Basically, with no place to store the c++ instance, there isn't really a workable solution for this approach that I could find. In addition I couldn't find a way to bind in custom constructors at all, which made usability on python very weak.
The next attempt was based on a suggestion in the pybind11 that you could use the python exception type as a metaclass a normal class_ and have python recognize it as a valid exception. I tried a plethora of variations on this approach.
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), py::reinterpret_borrow<py::object>(PyExc_Exception))
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), py::cast(PyExc_Exception))
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), py::cast(PyExc_Exception->ob_type))
py::class_< ::example::data::BadData>(module, "BadData", py::metaclass((PyObject *) &PyExc_Exception->ob_type))
There were more that I don't have saved. But the overall results was either 1) It was ignored completely or 2) it failed to compile or 3) It compiled and then immediately segfaulted or ImportError'd when trying to make an instance. There might have been one that segfaulted on module import too. It all blurs together. Maybe there is some magic formula that would make such a thing work, but I was unable to find it. From my reading of the pybind11 internals, I do not believe that such a thing is actually possible. Inheriting from a raw python type does not seem to be something it is setup to let you do.
The last thing I tried seemed really clever. I made a python exception type
static py::exception<::example::data::BadData> exc_BadData(module, "BadDataBase");
and then had my pybind11 class_ inherit from that.
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), exc_BadData)
But that also segfaulted on import too. So I'm basically back to square one with this.
So I figured out a way to actually do this but it involves 1) doing some hacking of the pybind11 code itself and 2) introducing some size inefficiencies to the bound python types. From my point of view, the size issues are fairly immaterial. Yes it would be better to have everything perfectly sized but I'll take some extra bytes of memory for ease of use. Given this inefficiency, though, I'm not submitting this as a PR to the pybind11 project. While I think the trade-off is worth it, I doubt that making this the default for most people would be desired. It would be possible, I guess to hide this functionality behind a #define in c++ but that seems like it would be super messy long-term. There is probably a better long-term answer that would involve a degree of template meta-programming (parameterizing on the python container type for class_) that I'm just not up to.
I'm providing my changes here as diffs against the current master branch in git when this was written (hash a54eab92d265337996b8e4b4149d9176c2d428a6).
The basic approach was
Modify pybind11 to allow the specification of an exception base class for a class_ instance.
Modify pybind11's internal container to have the extra fields needed for a python exception type
Write a small amount of custom binding code to handle setting the error correctly in python.
For the first part, I added a new attribute to type_record to specify if a class is an exception and added the associated process_attribute call for parsing it.
diff --git a/src/pybind11/include/pybind11/attr.h b/src/pybind11/include/pybind11/attr.h
index 58390239..b5535558 100644
--- a/src/pybind11/include/pybind11/attr.h
+++ b/src/pybind11/include/pybind11/attr.h
## -73,6 +73,9 ## struct module_local { const bool value; constexpr module_local(bool v = true) :
/// Annotation to mark enums as an arithmetic type
struct arithmetic { };
+// Annotation that marks a class as needing an exception base type.
+struct is_except {};
+
/** \rst
A call policy which places one or more guard variables (``Ts...``) around the function call.
## -211,7 +214,8 ## struct function_record {
struct type_record {
PYBIND11_NOINLINE type_record()
: multiple_inheritance(false), dynamic_attr(false), buffer_protocol(false),
- default_holder(true), module_local(false), is_final(false) { }
- default_holder(true), module_local(false), is_final(false),
- is_except(false) { }
/// Handle to the parent scope
handle scope;
## -267,6 +271,9 ## struct type_record {
/// Is the class inheritable from python classes?
bool is_final : 1;
- // Does the class need an exception base type?
- bool is_except : 1;
- PYBIND11_NOINLINE void add_base(const std::type_info &base, void *(*caster)(void *)) {
auto base_info = detail::get_type_info(base, false);
if (!base_info) {
## -451,6 +458,11 ## struct process_attribute<is_final> : process_attribute_default<is_final> {
static void init(const is_final &, type_record *r) { r->is_final = true; }
};
+template <>
+struct process_attribute<is_except> : process_attribute_default<is_except> {
- static void init(const is_except &, type_record *r) { r->is_except = true; }
+};
I modified the internals.h file to add a separate base class for exception types. I also added an extra bool argument to make_object_base_type.
diff --git a/src/pybind11/include/pybind11/detail/internals.h b/src/pybind11/include/pybind11/detail/internals.h
index 6224dfb2..d84df4f5 100644
--- a/src/pybind11/include/pybind11/detail/internals.h
+++ b/src/pybind11/include/pybind11/detail/internals.h
## -16,7 +16,7 ## NAMESPACE_BEGIN(detail)
// Forward declarations
inline PyTypeObject *make_static_property_type();
inline PyTypeObject *make_default_metaclass();
-inline PyObject *make_object_base_type(PyTypeObject *metaclass);
+inline PyObject *make_object_base_type(PyTypeObject *metaclass, bool is_except);
// The old Python Thread Local Storage (TLS) API is deprecated in Python 3.7 in favor of the new
// Thread Specific Storage (TSS) API.
## -107,6 +107,7 ## struct internals {
PyTypeObject *static_property_type;
PyTypeObject *default_metaclass;
PyObject *instance_base;
+ PyObject *exception_base;
#if defined(WITH_THREAD)
PYBIND11_TLS_KEY_INIT(tstate);
PyInterpreterState *istate = nullptr;
## -292,7 +293,8 ## PYBIND11_NOINLINE inline internals &get_internals() {
internals_ptr->registered_exception_translators.push_front(&translate_exception);
internals_ptr->static_property_type = make_static_property_type();
internals_ptr->default_metaclass = make_default_metaclass();
- internals_ptr->instance_base = make_object_base_type(internals_ptr->default_metaclass);
+ internals_ptr->instance_base = make_object_base_type(internals_ptr->default_metaclass, false);
+ internals_ptr->exception_base = make_object_base_type(internals_ptr->default_metaclass, true);
And then in class.h I added the necessary code to generate the exception base type. The first caveat is here. Since PyExc_Exception is a garbage collected type, I had to scope the assert call that checked the GC flag on the type. I have not currently seen any bad behavior from this change, but this is definitely voiding the warranty right here. I would highly, highly recommend always passing the py:dynamic_attr() flag to any classes you are using py:except on, since that turns on all the necessary bells and whistles to handle GC correctly (I think). A better solution might be to turn all those things on in make_object_base_type without having to invoke py::dynamic_attr.
diff --git a/src/pybind11/include/pybind11/detail/class.h b/src/pybind11/include/pybind11/detail/class.h
index a05edeb4..bbb9e772 100644
--- a/src/pybind11/include/pybind11/detail/class.h
+++ b/src/pybind11/include/pybind11/detail/class.h
## -368,7 +368,7 ## extern "C" inline void pybind11_object_dealloc(PyObject *self) {
/** Create the type which can be used as a common base for all classes. This is
needed in order to satisfy Python's requirements for multiple inheritance.
Return value: New reference. */
-inline PyObject *make_object_base_type(PyTypeObject *metaclass) {
+inline PyObject *make_object_base_type(PyTypeObject *metaclass, bool is_except=false) {
constexpr auto *name = "pybind11_object";
auto name_obj = reinterpret_steal<object>(PYBIND11_FROM_STRING(name));
## -387,7 +387,12 ## inline PyObject *make_object_base_type(PyTypeObject *metaclass) {
auto type = &heap_type->ht_type;
type->tp_name = name;
- type->tp_base = type_incref(&PyBaseObject_Type);
+ if (is_except) {
+ type->tp_base = type_incref(reinterpret_cast<PyTypeObject*>(PyExc_Exception));
+ }
+ else {
+ type->tp_base = type_incref(&PyBaseObject_Type);
+ }
type->tp_basicsize = static_cast<ssize_t>(sizeof(instance));
type->tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HEAPTYPE;
## -404,7 +409,9 ## inline PyObject *make_object_base_type(PyTypeObject *metaclass) {
setattr((PyObject *) type, "__module__", str("pybind11_builtins"));
PYBIND11_SET_OLDPY_QUALNAME(type, name_obj);
- assert(!PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC));
+ if (!is_except) {
+ assert(!PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC));
+ }
return (PyObject *) heap_type;
}
## -565,7 +572,8 ## inline PyObject* make_new_python_type(const type_record &rec) {
auto &internals = get_internals();
auto bases = tuple(rec.bases);
- auto base = (bases.size() == 0) ? internals.instance_base
+ auto base = (bases.size() == 0) ? (rec.is_except ? internals.exception_base
+ : internals.instance_base)
And then the final change, which is the inefficiency part. In Python, everything is a PyObject, but that is really only two fields (setup with the PyObject_HEAD macro) and the actual object struct may have a lot of extra fields. And having a very precise layout is important because python uses offsetof to seek into these things some times. From the Python 2.7 source code (Include/pyerrord.h) you can see the struct that is used for base exceptions
typedef struct {
PyObject_HEAD
PyObject *dict;
PyObject *args;
PyObject *message;
} PyBaseExceptionObject;
Any pybind11 type that extends PyExc_Exception has to have a instance struct that contains the same initial layout. And in pybind11 currently, the instance struct just has PyObject_HEAD. That means if you don't change the instance struct, this will all compile, but when python seeks into this object, it will do with the assumption that hose extra fields exist and then it will seek right off the end of viable memory and you'll get all sorts of fun segfaults. So this change adds those extra fields to every class_ in pybind11. It does not seem to break normal classes to have these extra fields and it definitely seems to make exceptions work correctly. If we broke the warranty before, we just tore it up and lit it on fire.
diff --git a/src/pybind11/include/pybind11/detail/common.h b/src/pybind11/include/pybind11/detail/common.h
index dd626793..b32e0c70 100644
--- a/src/pybind11/include/pybind11/detail/common.h
+++ b/src/pybind11/include/pybind11/detail/common.h
## -392,6 +392,10 ## struct nonsimple_values_and_holders {
/// The 'instance' type which needs to be standard layout (need to be able to use 'offsetof')
struct instance {
PyObject_HEAD
+ // Necessary to support exceptions.
+ PyObject *dict;
+ PyObject *args;
+ PyObject *message;
/// Storage for pointers and holder; see simple_layout, below, for a description
However, once these changes are all made, here is what you can do.
Bind in the class
auto PyBadData = py::class_< ::example::data::BadData>(module, "BadData", py::is_except(), py::dynamic_attr())
.def(py::init<>())
.def(py::init< std::string, std::string >())
.def("__str__", &::example::data::BadData::toString)
.def("getStack", &::example::data::BadData::getStack)
.def_property("message", &::example::data::BadData::getMsg, &::example::data::BadData::setMsg)
.def("getMsg", &::example::data::BadData::getMsg);
And take a function in c++ that throws the exception
void raiseMe()
{
throw ::example::data::BadData("this is an error", "");
}
and bind that in too
module.def("raiseMe", &raiseMe, "A function throws");
Add an exception translator to put the entire python type into the exception
py::register_exception_translator([](std::exception_ptr p) {
try {
if (p) {
std::rethrow_exception(p);
}
} catch (const ::example::data::BadData &e) {
auto err = py::cast(e);
auto errType = err.get_type().ptr();
PyErr_SetObject(errType, err.ptr());
}
});
And then you get all the things you could want!
>>> import example
>>> example.raiseMe()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
example.BadData: BadData(msg=this is an error, stack=)
You can, of course, also instantiate and raise the exception from python as well
>>> import example
>>> raise example.BadData("this is my error", "no stack")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
example.BadData: BadData(msg=this is my error, stack=no stack)
In some places in exposing C++ code to python, I need to use a PyObject*. If I have an instance of a boost::python::class_ object, I can invoke ptr() on it. But what if I just have the type?
Basically, given the type list boost::python::bases<A, B, C>, I want to convert this to a boost::python::tuple of instances that I can pass into something like PyErr_NewExceptionWithDoc(). Is this possible?
Given a C++ type T, one can create a boost::python::type_id object, and then query into the Boost.Python registry for registration information. If an entry is found in the registry, then one can use it to obtain a handle to the Python class created for type T:
/// #brief Get the class object for a wrapped type that has been exposed
/// through Boost.Python.
template <typename T>
boost::python::object get_instance_class()
{
// Query into the registry for type T.
namespace python = boost::python;
python::type_info type = python::type_id<T>();
const python::converter::registration* registration =
python::converter::registry::query(type);
// If the class is not registered, return None.
if (!registration) return python::object();
python::handle<PyTypeObject> handle(python::borrowed(
registration->get_class_object()));
return python::object(handle);
}
Here is a complete example demonstrating locating a Python class object in the Boost.Python registry:
#include <boost/python.hpp>
#include <iostream>
/// #brief Get the class object for a wrapped type that has been exposed
/// through Boost.Python.
template <typename T>
boost::python::object get_instance_class()
{
// Query into the registry for type T.
namespace python = boost::python;
python::type_info type = python::type_id<T>();
const python::converter::registration* registration =
python::converter::registry::query(type);
// If the class is not registered, return None.
if (!registration) return python::object();
python::handle<PyTypeObject> handle(python::borrowed(
registration->get_class_object()));
return python::object(handle);
}
struct spam {};
int main()
{
Py_Initialize();
namespace python = boost::python;
try
{
// Create the __main__ module.
python::object main_module = python::import("__main__");
python::object main_namespace = main_module.attr("__dict__");
// Create `Spam` class.
// >>> class Spam: pass
auto spam_class_object = python::class_<spam>("Spam", python::no_init);
// >>> print Spam
main_module.attr("__builtins__").attr("print")(get_instance_class<spam>());
// >>> assert(spam is spam)
assert(spam_class_object.ptr() == get_instance_class<spam>().ptr());
}
catch (python::error_already_set&)
{
PyErr_Print();
return 1;
}
}
Output:
<class 'Spam'>
For more type related functionality, such as accepting type objects, is, and issubclass, see this answer.
I have the following C++ classes that I would like expose to python.
class PlainOldData
{
...
};
class FancyClass
{
public:
const PlainOldData& GetMyPOD() {return myPOD;}
private:
PlainOldData myPOD;
};
Because I want my python classes to be pythonic, I would like expose myPOD as a property. However, when I try to do the following:
// expose with boost::python
BOOST_PYTHON_MODULE(mymod)
{
class_<PlainOldData>("PlainOldData", init<>());
// this fails
class_<FancyClass>("FancyClass", init<>())
.add_property("myPOD", &FancyClass::GetMyPOD);
}
I get the following error: error C2027: use of undefined type 'boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<T>'
But, if I try to specify a call policy, such as:
class_<FancyClass>("FancyClass", init<>())
.add_property("myPOD", &FancyClass::GetMyPOD, return_value_policy<copy_const_reference>());
I get an incredibly long error message.
Is it possible to expose this function as a property; am I doing something wrong?
Similar to how Python's property() is passed Python callable objects, the boost::python::class_::add_property() function can accept Python callable objects that can be created with CallPolicies, such as those returned from boost::python::make_function().
For example, the property in the original code could be exposed like:
class_<FancyClass>("FancyClass", init<>())
.add_property("myPOD", make_function(&FancyClass::GetMyPOD,
return_value_policy<copy_const_reference>()));
Here is a complete minimal example:
#include <boost/python.hpp>
class egg {};
class spam
{
public:
const egg& get_egg() { return egg_; }
private:
egg egg_;
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<egg>("Egg");
python::class_<spam>("Spam")
.add_property("egg", python::make_function(&spam::get_egg,
python::return_value_policy<python::copy_const_reference>()))
;
}
Interactive usage:
>>> import example
>>> spam = example.Spam()
>>> assert(spam.egg is not spam.egg) # unique identities as spam.egg
# returns a copy
>>> egg1 = spam.egg
>>> assert(egg1 is not spam.egg)
>>> egg2 = spam.egg
>>> assert(egg1 is not egg2)
I have a c++ class with a pure virtual function which I wrap in the following way using Boost Python:
class Model {
virtual double Test() = 0;
};
class ModelWrapper : public Model, public boost::python::wrapper<Model> {
double Test() {
return this->get_override("Test")();
}
};
BOOST_PYTHON_MODULE(mymodule)
{
class_<ModelWrapper, boost::noncopyable>("Model")
.def("Test, pure_virtual(&Model::Test))
;
}
I also have a c++ function that's expecting a Model object as an argument:
double MyFunction(Model& m) {
return m.Test()
}
Now what I would like to do is to make a derived class in Python that overrides the Model::Test function but can still be passed to MyFunction. Something like this:
from mymodule import *
class NewModel(Model):
def Test(self):
return 0.5
m = NewModel()
print(MyFunction(m))
which I would want to print out 0.5 in this case. Instead I get this error:
> ArgumentError: Python argument types in
> mymodule.MyFunction(NewModel) did not match C++ signature:
> MyFunction(Model {lvalue})
I'm not sure what step I'm missing in order for NewModel to be usable where Model is expected. Any insight would be greatly appreciated.
I needed to add 'Model.__init__(self)' to the NewModel __init__ function explicitly. After that it worked fine.