I'm trying to get a simple C++ 20 based generator pattern work with PyBind11. This is the code:
#include <pybind11/pybind11.h>
#include <coroutine>
#include <iostream>
struct Generator2 {
Generator2(){}
struct Promise;
using promise_type=Promise;
std::coroutine_handle<Promise> coro;
Generator2(std::coroutine_handle<Promise> h): coro(h) {}
~Generator2() {
if(coro)
coro.destroy();
}
int value() {
return coro.promise().val;
}
bool next() {
std::cout<<"calling coro.resume()";
coro.resume();
std::cout<<"coro.resume() called";
return !coro.done();
}
struct Promise {
void unhandled_exception() {std::rethrow_exception(std::move(std::current_exception()));}
int val;
Generator2 get_return_object() {
return Generator2{std::coroutine_handle<Promise>::from_promise(*this)};
}
std::suspend_always initial_suspend() {
return {};
}
std::suspend_always yield_value(int x) {
val=x;
return {};
}
std::suspend_never return_void() {
return {};
}
std::suspend_always final_suspend() noexcept {
return {};
}
};
};
Generator2 myCoroutineFunction() {
for(int i = 0; i < 100; ++i) {
co_yield i;
}
}
class Gen{
private:
Generator2 myCoroutineResult;
public:
Gen(){
myCoroutineResult = myCoroutineFunction();
}
int next(){
return (myCoroutineResult.next());
}
};
PYBIND11_MODULE(cmake_example, m) {
pybind11::class_<Gen>(m, "Gen")
.def(pybind11::init())
.def("next", &Gen::next);
}
However I'm getting an error:
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
Could c++ coroutines, coroutine_handles, co_yield etc. be a low-level thing that is not supported by PyBind11 yet?
Even though PyBind11 does not support coroutines directly, your problem does not mix coroutine and pybind code since you are hiding the coroutine behind Gen anyway.
The problem is that your Generator2 type uses the compiler provided copy and move constructors.
This line:
myCoroutineResult = myCoroutineFunction();
Creates a coroutine handle when you call myCoroutineFunction, and puts it in the temporary Generator2 in the right hand side. Then, you initialize myCoroutineResult from the right hand side generator. All is well, but then the temporary gets destroyed. Your destructor checks whether the handle is valid or not:
~Generator2() {
if(coro)
coro.destroy();
}
But in your implementation, the coro member of the member generator gets copied from the temporary without resetting the temporary's coro member. So the coroutine itself gets destroyed once you initialize myCoroutineResult, and you are holding onto a dangling coroutine handle. Remember that std::coroutine_handles behave like a raw pointer.
Essentially, you have a violation of the rule of 5. You have a custom destructor, but no copy/move constructors or assignment operators. Since you cannot copy construct a coroutine, you can ignore the copy constructors but you need to provide move constructors/assigment operators:
Generator2(Generator2&& rhs) : coro{std::exchange(rhs.coro, nullptr)} {
// rhs will not delete our coroutine,
// since we put nullptr to its coro member
}
Generator2& operator=(Generator2&& rhs) {
if (&rhs == this) {
return *this;
}
if (coro) {
coro.destroy();
}
coro = std::exchange(rhs.coro, nullptr);
return *this;
}
Also, use member initialization list to initialize members instead of assigning them within the constructor body. So instead of this:
Gen(){
myCoroutineResult = myCoroutineFunction();
}
Use this:
Gen() : myCoroutineResult{myCoroutineFunction()} {}
The reasoning can be seen even in this answer. The first one calls the assignment operator, which performs a bunch of additional work, whereas the second one calls the move constructor, which is as lean as it gets.
Related
I am trying to overload a python extension function that would take either a object or a string.
typedef struct
{
PyObject_HEAD
} CustomObject;
PyObject* customFunction(CustomObject* self, PyObject* args);
PyMethodDef methods[] =
{
{"customFunction", (PyCFunction) customFunction, METH_VARAGS, "A custom function"},
{NULL}
}
PyTypeObject TypeObj =
{
PyVarObject_HEAD_INIT(NULL, 0)
.tp_name = "customModule.CustomObject",
.tp_doc = "Custom Object",
.tp_basicsize = sizeof(CustomObject),
.tp_itemsize = 0,
.tp_flags = Py_TPFLAGS_DEFAULT,
.tp_methods = methods,
}
// Area of problem
PyObject* customFunction(CustomObject* self, PyObject* args)
{
const char* string;
PyObject* object;
if (PyArg_ParseTuple(args, "O!", &TypeObj, &object)) // TypeObj is the PyTypeObject fpr CustomObject
{
std::cout << "Object function\n"
// Do whatever and return PyObject*
}
else if (PyArg_ParseTuple(args, "s", &string))
{
std::cout << "String function\n"
// Do whatever and return PyObject*
}
return PyLong_FromLong(0); // In case nothing above works
}
In python I have a try except for the function and I get this error Error: <built-in method customFunction of CustomModule.CustomObject object at 0xmemoryadress> returned a result with an error set
Here are the Python docs for this PyArg_ParseTuple:
int PyArg_ParseTuple(PyObject *args, const char *format, ...)
Parse the parameters of a function that takes only positional parameters into local variables. Returns true on success; on failure, it returns false and raises the appropriate exception
I am guessing that PyArg_ParseTuple is setting an error, which is causing the entire function not to work (I do have customFunction in my method table for the module, I am just omitting that code). If I have the following Python:
import CustomModule
try:
CustomModule.customFunction("foo")
except Exception as e:
print("Error:", e)
String function does get outputted, so the code in the string if statement does work, but I assume the error occurs because PyArg_ParseTuple for the object failed, so it returns an error (not 100% sure if this is correct).
Is there a way I can prevent PyArg_ParseTuple() from raising an error, is there another function, or is there a better way to 'overload' my custom functions?
I'd probably just use PyArg_ParseTuple to get a generic unspecified object, and then handle the object types later with Py*_Check:
if (!PyArg_ParseTuple(args, "O", &object)) {
return NULL;
}
if (PyObject_IsInstance(object, (PyObject*)&PyType)) { // or a more specific function if one exists
std::cout << "Object function\n";
} else if (PyUnicode_Check(object)) {
std::cout << "String function\n";
} else {
// set an error, return NULL
}
The reason for this is that the Python "ask forgiveness, not permission" pattern of
try:
something()
except SomeException:
somethingElse()
doesn't translate very well into C, and involves quite a bit of code to handle the exceptions. If you really want to do it that way then you need to call PyErr_Clear before the second PyArg_ParseTuple, and ideally you should check it's the exception you think, and not something else entirely.
I want to create custom conversions from std::vector to Python list using boost python. For that I follow the to_python_converter approach. I used a typical converter structure, i.e.
template <class T, bool NoProxy = true>
struct vector_to_list {
static PyObject* convert(const std::vector<T>& vec) {
typedef typename std::vector<T>::const_iterator const_iter;
bp::list* l = new boost::python::list();
for (const_iter it = vec.begin(); it != vec.end(); ++it) {
if (NoProxy) {
l->append(boost::ref(*it));
} else {
l->append(*it);
}
}
return l->ptr();
}
static PyTypeObject const* get_pytype() { return &PyList_Type; }
};
which I can use successfully in plenty of cases but it doesn't work with std::vector<double>. This is the way how I declare this conversion in my boost python module as:
BOOST_PYTHON_MODULE(libmymodule_pywrap) {
.
.
.
bp::to_python_converter<std::vector<double, std::allocator<double> >,
vector_to_list<double, false>, true>(); // this doesn't work
bp::to_python_converter<std::vector<Eigen::VectorXd,
std::allocator<Eigen::VectorXd> >,
vector_to_list<Eigen::VectorXd, false>, true>(); // this works well
}
And I get the following compilation error:
/usr/include/boost/python/object/make_instance.hpp:27:9: error: no matching function for call to ‘assertion_failed(mpl_::failed************ boost::mpl::or_<boost::is_class<double>, boost::is_union<double>, mpl_::bool_<false>, mpl_::bool_<false>, mpl_::bool_<false> >::************)’
BOOST_MPL_ASSERT((mpl::or_<is_class<T>, is_union<T> >));
^
/usr/include/boost/mpl/assert.hpp:83:5: note: candidate: template<bool C> int mpl_::assertion_failed(typename mpl_::assert<C>::type)
int assertion_failed( typename assert<C>::type );
^
/usr/include/boost/mpl/assert.hpp:83:5: note: template argument deduction/substitution failed:
/usr/include/boost/python/object/make_instance.hpp:27:9: note: cannot convert ‘mpl_::assert_arg<boost::mpl::or_<boost::is_class<double>, boost::is_union<double>, mpl_::bool_<false>, mpl_::bool_<false>, mpl_::bool_<false> > >(0u, 1)’ (type ‘mpl_::failed************ boost::mpl::or_<boost::is_class<double>, boost::is_union<double>, mpl_::bool_<false>, mpl_::bool_<false>, mpl_::bool_<false> >::************’) to type ‘mpl_::assert<false>::type {aka mpl_::assert<false>}’
BOOST_MPL_ASSERT((mpl::or_<is_class<T>, is_union<T> >));
Do somebody understand what it's going on?
I am learning Boost::Python as well and unfortunately don't understand how to solve that error, but this example seems to avoid the error message, which you may be able to modify to your own needs.
template<typename T>
struct vector_to_list
{
static PyObject* convert(const std::vector<T>& src)
{
boost::python::list result;
for (int i = 0; i < src.size(); i++)
{
result.append(src[i]);
}
return boost::python::incref(result.ptr());
}
};
...
boost::python::to_python_converter<std::vector<double>, vector_to_list<double> >();
...
However, if this is to provide functionality like, for example:
getData() is declared in C++:
vector<double> getData() { return m_Data; }
where, for example, vector<double> m_Data = {1.0, 2.0, 3.0};
and you wanted in Python:
data = example.getData()
print (data)
[1.0, 2.0, 3.0]
You could implement it by creating a generic container and register each like this (courtesy of this answer):
/// #brief Type that allows for registration of conversions from
/// python iterable types.
struct iterable_converter
{
/// #note Registers converter from a python interable type to the
/// provided type.
template <typename Container>
iterable_converter&
from_python()
{
boost::python::converter::registry::push_back(
&iterable_converter::convertible,
&iterable_converter::construct<Container>,
boost::python::type_id<Container>());
// Support chaining.
return *this;
}
/// #brief Check if PyObject is iterable.
static void* convertible(PyObject* object)
{
return PyObject_GetIter(object) ? object : NULL;
}
/// #brief Convert iterable PyObject to C++ container type.
///
/// Container Concept requirements:
///
/// * Container::value_type is CopyConstructable.
/// * Container can be constructed and populated with two iterators.
/// I.e. Container(begin, end)
template <typename Container>
static void construct(
PyObject* object,
boost::python::converter::rvalue_from_python_stage1_data* data)
{
namespace python = boost::python;
// Object is a borrowed reference, so create a handle indicting it is
// borrowed for proper reference counting.
python::handle<> handle(python::borrowed(object));
// Obtain a handle to the memory block that the converter has allocated
// for the C++ type.
typedef python::converter::rvalue_from_python_storage<Container>
storage_type;
void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;
typedef python::stl_input_iterator<typename Container::value_type>
iterator;
// Allocate the C++ type into the converter's memory block, and assign
// its handle to the converter's convertible variable. The C++
// container is populated by passing the begin and end iterators of
// the python object to the container's constructor.
new (storage) Container(
iterator(python::object(handle)), // begin
iterator()); // end
data->convertible = storage;
}
};
BOOST_PYTHON_MODULE(example)
{
// Register interable conversions.
iterable_converter()
.from_python<std::vector<double> > ()
.from_python<std::vector<Eigen::VectorXd> >()
;
}
Which allows for chaining, nested vectors and an API that is more Pythonic than with indexed_vector_suite cases like:
data = example.doubleVector()
data[:] = example.getData()
you can simply use:
data = example.getData()
I want to check if an object is an instance of a certain class. In Python I can do this with instanceof. In C/C++, I found a function named PyObject_IsInstance. But it seems not to work like isinstance.
In detail (also described as sample codes below):
In C++ I defined my custom type My. The type definition is MyType and the object definition is MyObject.
Add MyType to the exported module with name My.
In Python, create a new instance my = My(), and isinstance(my, My) returns True.
While in C++ we use PyObject_IsInstance(my, (PyObject*)&MyType) to check my, and this returns 0, which means my is not an instance of the class defined by MyType.
Full C++ code:
#define PY_SSIZE_T_CLEAN
#include <python3.6/Python.h>
#include <python3.6/structmember.h>
#include <stddef.h>
typedef struct {
PyObject_HEAD
int num;
} MyObject;
static PyTypeObject MyType = []{
PyTypeObject ret = {
PyVarObject_HEAD_INIT(NULL, 0)
};
ret.tp_name = "cpp.My";
ret.tp_doc = NULL;
ret.tp_basicsize = sizeof(MyObject);
ret.tp_itemsize = 0;
ret.tp_flags = Py_TPFLAGS_DEFAULT;
ret.tp_new = PyType_GenericNew;
return ret;
}();
// check if obj is an instance of MyType
static PyObject *Py_fn_checkMy(PyObject *obj) {
if (PyObject_IsInstance(obj, (PyObject *)&MyType)) Py_RETURN_TRUE;
else Py_RETURN_FALSE;
}
static PyMethodDef modmethodsdef[] = {
{ "checkMy", (PyCFunction)Py_fn_checkMy, METH_VARARGS, NULL },
{ NULL }
};
static PyModuleDef moddef = []{
PyModuleDef ret = {
PyModuleDef_HEAD_INIT
};
ret.m_name = "cpp";
ret.m_doc = NULL;
ret.m_size = -1;
return ret;
}();
PyMODINIT_FUNC
PyInit_cpp(void)
{
PyObject *mod;
if (PyType_Ready(&MyType) < 0)
return NULL;
mod = PyModule_Create(&moddef);
if (mod == NULL)
return NULL;
Py_INCREF(&MyType);
PyModule_AddObject(mod, "My", (PyObject *)&MyType);
PyModule_AddFunctions(mod, modmethodsdef);
return mod;
}
Compile this into cpp.so, and test it in Python:
>>> import cpp
>>> isinstance(cpp.My(), cpp.My)
True
>>> cpp.checkMy(cpp.My())
False
METH_VARARGS
This is the typical calling convention, where the methods have the type PyCFunction. The function expects two PyObject* values. The first one is the self object for methods; for module functions, it is the module object. The second parameter (often called args) is a tuple object representing all arguments. This parameter is typically processed using PyArg_ParseTuple() or PyArg_UnpackTuple().
The function signature of Py_fn_checkMy does not match this. It should take two arguments. The first is the module, and this is what you are checking against MyType. The second argument (which you don't actually accept) is a tuple containing the object you passed. You should extract the argument from the tuple and check the type of this.
You'd probably be better using METH_O to specify a single argument instead of extracting arguments from a tuple:
static PyObject *Py_fn_checkMy(PyObject *self, PyObject *obj) {
if (PyObject_IsInstance(obj, (PyObject *)&MyType)) Py_RETURN_TRUE;
else Py_RETURN_FALSE;
}
static PyMethodDef modmethodsdef[] = {
{ "checkMy", (PyCFunction)Py_fn_checkMy, METH_O, NULL },
{ NULL }
};
if I call a SWIG-wrapped C/C++ function from Python, is it possible to obtain the current call stack? I would like something similar to the result of ''.join(traceback.format_stack()), but I don't want to pass this from Python to my C/C++ functions because I don't always need it. So I would like to obtain it on the fly and print it if something wrong happens on my C/C++ side.
I figured out a solution following this post, although I still prefer more natural ways of getting the same thing if there is any.
// This is similar to the python code:
// def GetScriptingLanguageCallStack():
// import traceback
// return ''.join(traceback.format_stack())
string GetScriptingLanguageCallStack() {
string result;
PyObject* module_name = PyString_FromString("traceback");
PyObject* pyth_module = PyImport_Import(module_name);
Py_DECREF(module_name);
if (pyth_module != nullptr) {
PyObject* pyth_func = PyObject_GetAttrString(pyth_module, "format_stack");
if (pyth_func != nullptr) {
if (PyCallable_Check(pyth_func)) {
PyObject* pyth_val = PyObject_CallFunctionObjArgs(pyth_func, 0);
if (pyth_val != nullptr) {
if (PyList_Check(pyth_val)) {
const int size = PyList_GET_SIZE(pyth_val);
for (int i = 0; i < size; ++i) {
PyObject* pyth_line = PyList_GET_ITEM(pyth_val, i);
result += PyString_AsString(pyth_line);
}
}
Py_DECREF(pyth_val);
}
}
Py_DECREF(pyth_func);
}
Py_DECREF(pyth_module);
}
return result;
}
By the way, I do not prefer the approach in the linked post which uses frame object, because the line number given is not pointing to the exact line which makes the further function calls, but only on the line containing the function name.
I have a list of base classes in C++, I want to access them in Python as a list of their derived most classes.
Is there a build in means to cater for this in Boost.Python?
I've made an example the problem I ma facing:
// ------------------------------- Code ----------------------------------//
#include<memory>
#include<iostream>
#include<vector>
namespace boost { template<class T> T* get_pointer(std::shared_ptr<T>& p){ return p.get(); }}
struct Vehicle{ virtual ~Vehicle(){} friend bool operator==(const Vehicle& lhs, const Vehicle& rhs) { return true; }};
struct Boat: public Vehicle{
virtual ~Boat(){}
friend bool operator==(const Boat& lhs, const Boat& rhs) { return true; }
char const* testBoatSpecificMethod() { return "Floating."; }
};
struct Truck: public Vehicle{
virtual ~Truck(){}
friend bool operator==(const Truck& lhs, const Truck& rhs) { return true; }
char const* testTruckSpecificMethod() { return "Trucking."; }
};
class Garage
{
public:
Garage() {};
~Garage() {};
char const* test() { std::string temp = "Vehicle List Size: " + std::to_string(m_VehicleList.size()); return temp.c_str(); }
friend bool operator==(const Garage& lhs, const Garage& rhs) { return true; }
std::vector<std::shared_ptr<Vehicle>>& vehicleList() { return m_VehicleList; }
private:
std::vector<std::shared_ptr<Vehicle>> m_VehicleList;
};
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
BOOST_PYTHON_MODULE(garage_ext)
{
using namespace boost::python;
class_<Garage>("Garage")
.def("test", &Garage::test)
.def("vehicleList", &Garage::vehicleList, return_internal_reference<1>());
class_<Vehicle,std::shared_ptr<Vehicle>>("Vehicle");
class_<Boat,std::shared_ptr<Boat>,bases<Vehicle>>("Boat")
.def("testBoatSpecificMethod", &Boat::testBoatSpecificMethod);
class_<Truck,std::shared_ptr<Truck>,bases<Vehicle>>("Truck")
.def("testTruckSpecificMethod", &Truck::testTruckSpecificMethod);
implicitly_convertible<std::shared_ptr<Boat>,std::shared_ptr<Vehicle>>();
implicitly_convertible<std::shared_ptr<Truck>,std::shared_ptr<Vehicle>>();
class_<std::vector<std::shared_ptr<Vehicle>> >("stl_vector_Vehicle")
.def(vector_indexing_suite<std::vector<std::shared_ptr<Vehicle>> >());
}
// --------------------------- Test Script -------------------------------//
import garage_ext
g = garage_ext.Garage()
l = g.vehicleList()
l.append(garage_ext.Boat())
print "Expecting a Boat object:"
print str(l[0])
print g.vehicleList()[0].testBoatSpecificMethod()
garage_ext.f2("Done.")
// ------------------------------ Output ---------------------------------//
Expecting a Boat object
Traceback (most recent call last):
File "test_garage.py", line 7, in
print g.vehicleList()[0].testBoatSpecificMethod()
AttributeError: 'Vehicle' object has no attribute 'testBoatSpecificMethod'
'Vehicle' object has no attribute 'testBoatSpecificMethod'
Here I want Vehicle to be a Boat object.
If there is not a build-in or recommended/known Boost.Python means to handle this problem,
I'll try wrapping the list (Lots of wrapping to be done then in my library.) with a get accessor returning a boost::python::list, storing the derived most types in the python list object. Getting the derived most type possibly by calling overriden 'getAsDerivedClass' method.
I would like to avoid this. I dislike having to add python usage specific methods to the library, for our design and vision values / reasons. Another concern is that this way will introduce a lot of extramaintenance work.
EDIT:
What I want works when I am using raw pointers instead of smart pointers.
For what I feel are obvious reasons,I do not want to use raw pointers in place of smart pointers.
This does give me a relieve in that knowing what I want this concept isn't so far-fetched as I started to fear. (I am struggling still to make it work with smart pointers. The python object asks for a converter, too much work to write one by hand.)