I have a list of base classes in C++, I want to access them in Python as a list of their derived most classes.
Is there a build in means to cater for this in Boost.Python?
I've made an example the problem I ma facing:
// ------------------------------- Code ----------------------------------//
#include<memory>
#include<iostream>
#include<vector>
namespace boost { template<class T> T* get_pointer(std::shared_ptr<T>& p){ return p.get(); }}
struct Vehicle{ virtual ~Vehicle(){} friend bool operator==(const Vehicle& lhs, const Vehicle& rhs) { return true; }};
struct Boat: public Vehicle{
virtual ~Boat(){}
friend bool operator==(const Boat& lhs, const Boat& rhs) { return true; }
char const* testBoatSpecificMethod() { return "Floating."; }
};
struct Truck: public Vehicle{
virtual ~Truck(){}
friend bool operator==(const Truck& lhs, const Truck& rhs) { return true; }
char const* testTruckSpecificMethod() { return "Trucking."; }
};
class Garage
{
public:
Garage() {};
~Garage() {};
char const* test() { std::string temp = "Vehicle List Size: " + std::to_string(m_VehicleList.size()); return temp.c_str(); }
friend bool operator==(const Garage& lhs, const Garage& rhs) { return true; }
std::vector<std::shared_ptr<Vehicle>>& vehicleList() { return m_VehicleList; }
private:
std::vector<std::shared_ptr<Vehicle>> m_VehicleList;
};
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
BOOST_PYTHON_MODULE(garage_ext)
{
using namespace boost::python;
class_<Garage>("Garage")
.def("test", &Garage::test)
.def("vehicleList", &Garage::vehicleList, return_internal_reference<1>());
class_<Vehicle,std::shared_ptr<Vehicle>>("Vehicle");
class_<Boat,std::shared_ptr<Boat>,bases<Vehicle>>("Boat")
.def("testBoatSpecificMethod", &Boat::testBoatSpecificMethod);
class_<Truck,std::shared_ptr<Truck>,bases<Vehicle>>("Truck")
.def("testTruckSpecificMethod", &Truck::testTruckSpecificMethod);
implicitly_convertible<std::shared_ptr<Boat>,std::shared_ptr<Vehicle>>();
implicitly_convertible<std::shared_ptr<Truck>,std::shared_ptr<Vehicle>>();
class_<std::vector<std::shared_ptr<Vehicle>> >("stl_vector_Vehicle")
.def(vector_indexing_suite<std::vector<std::shared_ptr<Vehicle>> >());
}
// --------------------------- Test Script -------------------------------//
import garage_ext
g = garage_ext.Garage()
l = g.vehicleList()
l.append(garage_ext.Boat())
print "Expecting a Boat object:"
print str(l[0])
print g.vehicleList()[0].testBoatSpecificMethod()
garage_ext.f2("Done.")
// ------------------------------ Output ---------------------------------//
Expecting a Boat object
Traceback (most recent call last):
File "test_garage.py", line 7, in
print g.vehicleList()[0].testBoatSpecificMethod()
AttributeError: 'Vehicle' object has no attribute 'testBoatSpecificMethod'
'Vehicle' object has no attribute 'testBoatSpecificMethod'
Here I want Vehicle to be a Boat object.
If there is not a build-in or recommended/known Boost.Python means to handle this problem,
I'll try wrapping the list (Lots of wrapping to be done then in my library.) with a get accessor returning a boost::python::list, storing the derived most types in the python list object. Getting the derived most type possibly by calling overriden 'getAsDerivedClass' method.
I would like to avoid this. I dislike having to add python usage specific methods to the library, for our design and vision values / reasons. Another concern is that this way will introduce a lot of extramaintenance work.
EDIT:
What I want works when I am using raw pointers instead of smart pointers.
For what I feel are obvious reasons,I do not want to use raw pointers in place of smart pointers.
This does give me a relieve in that knowing what I want this concept isn't so far-fetched as I started to fear. (I am struggling still to make it work with smart pointers. The python object asks for a converter, too much work to write one by hand.)
Related
I'm trying to get a simple C++ 20 based generator pattern work with PyBind11. This is the code:
#include <pybind11/pybind11.h>
#include <coroutine>
#include <iostream>
struct Generator2 {
Generator2(){}
struct Promise;
using promise_type=Promise;
std::coroutine_handle<Promise> coro;
Generator2(std::coroutine_handle<Promise> h): coro(h) {}
~Generator2() {
if(coro)
coro.destroy();
}
int value() {
return coro.promise().val;
}
bool next() {
std::cout<<"calling coro.resume()";
coro.resume();
std::cout<<"coro.resume() called";
return !coro.done();
}
struct Promise {
void unhandled_exception() {std::rethrow_exception(std::move(std::current_exception()));}
int val;
Generator2 get_return_object() {
return Generator2{std::coroutine_handle<Promise>::from_promise(*this)};
}
std::suspend_always initial_suspend() {
return {};
}
std::suspend_always yield_value(int x) {
val=x;
return {};
}
std::suspend_never return_void() {
return {};
}
std::suspend_always final_suspend() noexcept {
return {};
}
};
};
Generator2 myCoroutineFunction() {
for(int i = 0; i < 100; ++i) {
co_yield i;
}
}
class Gen{
private:
Generator2 myCoroutineResult;
public:
Gen(){
myCoroutineResult = myCoroutineFunction();
}
int next(){
return (myCoroutineResult.next());
}
};
PYBIND11_MODULE(cmake_example, m) {
pybind11::class_<Gen>(m, "Gen")
.def(pybind11::init())
.def("next", &Gen::next);
}
However I'm getting an error:
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
Could c++ coroutines, coroutine_handles, co_yield etc. be a low-level thing that is not supported by PyBind11 yet?
Even though PyBind11 does not support coroutines directly, your problem does not mix coroutine and pybind code since you are hiding the coroutine behind Gen anyway.
The problem is that your Generator2 type uses the compiler provided copy and move constructors.
This line:
myCoroutineResult = myCoroutineFunction();
Creates a coroutine handle when you call myCoroutineFunction, and puts it in the temporary Generator2 in the right hand side. Then, you initialize myCoroutineResult from the right hand side generator. All is well, but then the temporary gets destroyed. Your destructor checks whether the handle is valid or not:
~Generator2() {
if(coro)
coro.destroy();
}
But in your implementation, the coro member of the member generator gets copied from the temporary without resetting the temporary's coro member. So the coroutine itself gets destroyed once you initialize myCoroutineResult, and you are holding onto a dangling coroutine handle. Remember that std::coroutine_handles behave like a raw pointer.
Essentially, you have a violation of the rule of 5. You have a custom destructor, but no copy/move constructors or assignment operators. Since you cannot copy construct a coroutine, you can ignore the copy constructors but you need to provide move constructors/assigment operators:
Generator2(Generator2&& rhs) : coro{std::exchange(rhs.coro, nullptr)} {
// rhs will not delete our coroutine,
// since we put nullptr to its coro member
}
Generator2& operator=(Generator2&& rhs) {
if (&rhs == this) {
return *this;
}
if (coro) {
coro.destroy();
}
coro = std::exchange(rhs.coro, nullptr);
return *this;
}
Also, use member initialization list to initialize members instead of assigning them within the constructor body. So instead of this:
Gen(){
myCoroutineResult = myCoroutineFunction();
}
Use this:
Gen() : myCoroutineResult{myCoroutineFunction()} {}
The reasoning can be seen even in this answer. The first one calls the assignment operator, which performs a bunch of additional work, whereas the second one calls the move constructor, which is as lean as it gets.
I want to create custom conversions from std::vector to Python list using boost python. For that I follow the to_python_converter approach. I used a typical converter structure, i.e.
template <class T, bool NoProxy = true>
struct vector_to_list {
static PyObject* convert(const std::vector<T>& vec) {
typedef typename std::vector<T>::const_iterator const_iter;
bp::list* l = new boost::python::list();
for (const_iter it = vec.begin(); it != vec.end(); ++it) {
if (NoProxy) {
l->append(boost::ref(*it));
} else {
l->append(*it);
}
}
return l->ptr();
}
static PyTypeObject const* get_pytype() { return &PyList_Type; }
};
which I can use successfully in plenty of cases but it doesn't work with std::vector<double>. This is the way how I declare this conversion in my boost python module as:
BOOST_PYTHON_MODULE(libmymodule_pywrap) {
.
.
.
bp::to_python_converter<std::vector<double, std::allocator<double> >,
vector_to_list<double, false>, true>(); // this doesn't work
bp::to_python_converter<std::vector<Eigen::VectorXd,
std::allocator<Eigen::VectorXd> >,
vector_to_list<Eigen::VectorXd, false>, true>(); // this works well
}
And I get the following compilation error:
/usr/include/boost/python/object/make_instance.hpp:27:9: error: no matching function for call to ‘assertion_failed(mpl_::failed************ boost::mpl::or_<boost::is_class<double>, boost::is_union<double>, mpl_::bool_<false>, mpl_::bool_<false>, mpl_::bool_<false> >::************)’
BOOST_MPL_ASSERT((mpl::or_<is_class<T>, is_union<T> >));
^
/usr/include/boost/mpl/assert.hpp:83:5: note: candidate: template<bool C> int mpl_::assertion_failed(typename mpl_::assert<C>::type)
int assertion_failed( typename assert<C>::type );
^
/usr/include/boost/mpl/assert.hpp:83:5: note: template argument deduction/substitution failed:
/usr/include/boost/python/object/make_instance.hpp:27:9: note: cannot convert ‘mpl_::assert_arg<boost::mpl::or_<boost::is_class<double>, boost::is_union<double>, mpl_::bool_<false>, mpl_::bool_<false>, mpl_::bool_<false> > >(0u, 1)’ (type ‘mpl_::failed************ boost::mpl::or_<boost::is_class<double>, boost::is_union<double>, mpl_::bool_<false>, mpl_::bool_<false>, mpl_::bool_<false> >::************’) to type ‘mpl_::assert<false>::type {aka mpl_::assert<false>}’
BOOST_MPL_ASSERT((mpl::or_<is_class<T>, is_union<T> >));
Do somebody understand what it's going on?
I am learning Boost::Python as well and unfortunately don't understand how to solve that error, but this example seems to avoid the error message, which you may be able to modify to your own needs.
template<typename T>
struct vector_to_list
{
static PyObject* convert(const std::vector<T>& src)
{
boost::python::list result;
for (int i = 0; i < src.size(); i++)
{
result.append(src[i]);
}
return boost::python::incref(result.ptr());
}
};
...
boost::python::to_python_converter<std::vector<double>, vector_to_list<double> >();
...
However, if this is to provide functionality like, for example:
getData() is declared in C++:
vector<double> getData() { return m_Data; }
where, for example, vector<double> m_Data = {1.0, 2.0, 3.0};
and you wanted in Python:
data = example.getData()
print (data)
[1.0, 2.0, 3.0]
You could implement it by creating a generic container and register each like this (courtesy of this answer):
/// #brief Type that allows for registration of conversions from
/// python iterable types.
struct iterable_converter
{
/// #note Registers converter from a python interable type to the
/// provided type.
template <typename Container>
iterable_converter&
from_python()
{
boost::python::converter::registry::push_back(
&iterable_converter::convertible,
&iterable_converter::construct<Container>,
boost::python::type_id<Container>());
// Support chaining.
return *this;
}
/// #brief Check if PyObject is iterable.
static void* convertible(PyObject* object)
{
return PyObject_GetIter(object) ? object : NULL;
}
/// #brief Convert iterable PyObject to C++ container type.
///
/// Container Concept requirements:
///
/// * Container::value_type is CopyConstructable.
/// * Container can be constructed and populated with two iterators.
/// I.e. Container(begin, end)
template <typename Container>
static void construct(
PyObject* object,
boost::python::converter::rvalue_from_python_stage1_data* data)
{
namespace python = boost::python;
// Object is a borrowed reference, so create a handle indicting it is
// borrowed for proper reference counting.
python::handle<> handle(python::borrowed(object));
// Obtain a handle to the memory block that the converter has allocated
// for the C++ type.
typedef python::converter::rvalue_from_python_storage<Container>
storage_type;
void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;
typedef python::stl_input_iterator<typename Container::value_type>
iterator;
// Allocate the C++ type into the converter's memory block, and assign
// its handle to the converter's convertible variable. The C++
// container is populated by passing the begin and end iterators of
// the python object to the container's constructor.
new (storage) Container(
iterator(python::object(handle)), // begin
iterator()); // end
data->convertible = storage;
}
};
BOOST_PYTHON_MODULE(example)
{
// Register interable conversions.
iterable_converter()
.from_python<std::vector<double> > ()
.from_python<std::vector<Eigen::VectorXd> >()
;
}
Which allows for chaining, nested vectors and an API that is more Pythonic than with indexed_vector_suite cases like:
data = example.doubleVector()
data[:] = example.getData()
you can simply use:
data = example.getData()
I want to check if an object is an instance of a certain class. In Python I can do this with instanceof. In C/C++, I found a function named PyObject_IsInstance. But it seems not to work like isinstance.
In detail (also described as sample codes below):
In C++ I defined my custom type My. The type definition is MyType and the object definition is MyObject.
Add MyType to the exported module with name My.
In Python, create a new instance my = My(), and isinstance(my, My) returns True.
While in C++ we use PyObject_IsInstance(my, (PyObject*)&MyType) to check my, and this returns 0, which means my is not an instance of the class defined by MyType.
Full C++ code:
#define PY_SSIZE_T_CLEAN
#include <python3.6/Python.h>
#include <python3.6/structmember.h>
#include <stddef.h>
typedef struct {
PyObject_HEAD
int num;
} MyObject;
static PyTypeObject MyType = []{
PyTypeObject ret = {
PyVarObject_HEAD_INIT(NULL, 0)
};
ret.tp_name = "cpp.My";
ret.tp_doc = NULL;
ret.tp_basicsize = sizeof(MyObject);
ret.tp_itemsize = 0;
ret.tp_flags = Py_TPFLAGS_DEFAULT;
ret.tp_new = PyType_GenericNew;
return ret;
}();
// check if obj is an instance of MyType
static PyObject *Py_fn_checkMy(PyObject *obj) {
if (PyObject_IsInstance(obj, (PyObject *)&MyType)) Py_RETURN_TRUE;
else Py_RETURN_FALSE;
}
static PyMethodDef modmethodsdef[] = {
{ "checkMy", (PyCFunction)Py_fn_checkMy, METH_VARARGS, NULL },
{ NULL }
};
static PyModuleDef moddef = []{
PyModuleDef ret = {
PyModuleDef_HEAD_INIT
};
ret.m_name = "cpp";
ret.m_doc = NULL;
ret.m_size = -1;
return ret;
}();
PyMODINIT_FUNC
PyInit_cpp(void)
{
PyObject *mod;
if (PyType_Ready(&MyType) < 0)
return NULL;
mod = PyModule_Create(&moddef);
if (mod == NULL)
return NULL;
Py_INCREF(&MyType);
PyModule_AddObject(mod, "My", (PyObject *)&MyType);
PyModule_AddFunctions(mod, modmethodsdef);
return mod;
}
Compile this into cpp.so, and test it in Python:
>>> import cpp
>>> isinstance(cpp.My(), cpp.My)
True
>>> cpp.checkMy(cpp.My())
False
METH_VARARGS
This is the typical calling convention, where the methods have the type PyCFunction. The function expects two PyObject* values. The first one is the self object for methods; for module functions, it is the module object. The second parameter (often called args) is a tuple object representing all arguments. This parameter is typically processed using PyArg_ParseTuple() or PyArg_UnpackTuple().
The function signature of Py_fn_checkMy does not match this. It should take two arguments. The first is the module, and this is what you are checking against MyType. The second argument (which you don't actually accept) is a tuple containing the object you passed. You should extract the argument from the tuple and check the type of this.
You'd probably be better using METH_O to specify a single argument instead of extracting arguments from a tuple:
static PyObject *Py_fn_checkMy(PyObject *self, PyObject *obj) {
if (PyObject_IsInstance(obj, (PyObject *)&MyType)) Py_RETURN_TRUE;
else Py_RETURN_FALSE;
}
static PyMethodDef modmethodsdef[] = {
{ "checkMy", (PyCFunction)Py_fn_checkMy, METH_O, NULL },
{ NULL }
};
I'm re-writing a string class (called XXXString) that implements many Pythonic operations like ==.
When overloading the [] operator I want it to return a XXXString& to its ith character, rather than a char& like in std::string.
In this way, I can do both
a_string[0] = "a"
and
a_string[0] == "a"
(assuming I have defined the == operator to compare a XXXString with a char*).
I know many C++ enthusiastic people will say no your "a" is not a 'a'. There is a '\0' in there balabala. I just want to ignore this and think Pythonic.
If you really want to do this, the solution should be to use a proxy and not return the actual XXXString itself.
#include <iostream>
#include <string>
#include <cassert>
class XXXString {
public:
class StringRefProxy {
public:
StringRefProxy(char& ch_in) : ch{ch_in} {}
bool operator==(const std::string& other) {
assert(other.length() == 1);
return other[0] == this->ch;
}
StringRefProxy& operator=(const std::string& other) {
assert(other.length() == 1);
this->ch = other[0];
return *this;
}
private:
char& ch;
};
XXXString(const std::string& str_in) : str{str_in} {}
StringRefProxy operator[](std::size_t index) {
return StringRefProxy{this->str[index]};
}
operator std::string() {
return this->str;
}
private:
std::string str;
};
int main() {
auto str = XXXString{"something"};
assert(str[0] == "s");
str[0] = "a";
assert(static_cast<std::string>(str) == "aomething");
}
Note that when C++17 is freely available, you can substitute all the const std::string& with std::string_view in the above code for better performance
Also note that as most approaches go that extract references that point to elements inside container. This too is subject to invalidation issues.
This question is a slight spin on a previous question: Accessing the underlying struct of a PyObject
Except in my version I want to know how to expose the fields of the Point struct as members of my new type.
I have looked everywhere I could think, read numerous example Python extensions and read many tutorials, docs, etc. and have not been able to find a clean way to do this.
Below is an example of what I want to do. I have some internal struct that I would like to expose via the Python extension but I don't want to have to redefine the struct in my code. It seems like the main area that is the problem is the PyMemeberDef definition...what would the offset be of x or y inside the Point struct from the context of the PointObject struct?
Any help is much appreciated, thanks.
#include <Python.h>
#include <structmember.h>
// This is actually defined elsewhere in someone else's code.
struct Point {
int x;
int y;
};
struct PointObject {
PyObject_HEAD
struct Point* my_point;
int z;
};
static PyMemberDef point_members[] = {
{"z", T_INT, offsetof(struct PointObject, z), 0, "z field"},
{"x", T_INT, offsetof(???), 0, "point x field"},
{"y", T_INT, offsetof(???), 0, "point y field"},
{NULL}
};
static PyTypeObject PointType = {
PyObject_HEAD_INIT(NULL)
.tp_name = "Point",
.tp_basicsize = sizeof(PointObject),
.tp_flags = Py_TPFLAGS_DEFAULT,
.tp_doc = "Point objects",
.tp_members = point_members,
.tp_init = (initproc)point_init,
};
...
PyMODINIT_FUNC
initMainModule(void)
{
PyObject *m = Py_InitModule(MODULE, NULL);
// Register PointType
PointType.tp_new = PyType_GenericNew;
if (PyType_Ready(&PointType) < 0)
return;
Py_INCREF(&PointType);
PyModule_AddObject(m, "Point", (PyObject *)&PointType);
...
}
I would highly recommend reading http://docs.python.org/release/2.6/_sources/extending/newtypes.txt, specifically the section on "Providing finer control over data attributes".
After reading this, it appears it is possible to do this by defining a collection of setter/getters for the specific members in the Point struct. It does seem a bit overkill for something as simple as this, but it appears to work just fine.
Below is an example on how to do it for the 'x' field of Point, based on adding the following code to my previous example:
static PyGetSetDef point_getset[] = {
{"x", (getter)point_get_x, (setter)point_set_x, "", NULL},
{NULL}
};
static PyObject*
point_get_x(struct PointObject *self, void *closure)
{
return Py_BuildValue("i", self->my_point->x);
}
static int
point_set_x(struct PointObject *self, PyObject *value, void *closure)
{
PyErr_SetString(PyExc_TypeError, "Attribute is read-only!");
return -1;
}
Lastly, add the point_getset struct to the tp_getset field in the previously defined PyTypeObject struct. It should be noted that you can still define members in the traditional PyMemberDef struct for the simple cases.