A c++ function has a signature:
void f*(unsigned char* buffer, int buffer_size);
It can be wrapped using pybind11:
m.def("f",&f);
I am now stuck on how I may call f in python.
For example, using ctypes, something like this would work:
buffer = bytearray(size)
cbuf_type = ctypes.c_char*len(buffer)
cbuf = cbuf_type.from_buffer(buffer)
lib.f(cbuf,size)
What would work for pybind11 ?
One simply has to use the interoperability with numpy, here an example:
https://people.duke.edu/~ccc14/cspy/18G_C++_Python_pybind11.html#Using-numpy-arrays-as-function-arguments-and-return-values
I'm making a C++ extension for Python, and I'm trying to do something like:
// this function assigns a C++ pointer to as attribute of a python object
void function1(PyObject* p){
// equivalent of p.attr = cpp_attr;
MyClass* cpp_attr = new MyClass();
PyObject* args = PyTuple_Pack(cpp_attr);
PyObject_SetAttrString(p, (char*)"attr", args);
}
I would like to retrieve this pointer and set it as attribute of another C++ object. I know how to get the PyObject* but after that I'm not sure what to do anymore
MySecondClass::MySecondClass(PyObject* p){
// get the attribute from p; equivalent of cpp_attr = p.attr
PyObject* cpp_attr = PyObject_getAttrString(p, (char*)"attr"));
// somehow get back the pointer to MyClass object created in function1
}
I looked at the documentation but I couldn't find anything that returns the original type. Is there anyway to do this?
Thanks
It's difficult to be absolutely certain, but I doubt that MyClass a Python object. This means that your attempt to store it as a Python object (e.g. using PyTuple_Pack) is completely wrong and will cause Python to malfunction in unexpected ways.
What will happen is that Python will attempt to interpret the pointer as a Python object, will try to use its normal reference counting mechanisms on that object (will change it in unpredictable ways), and ultimately try to deallocate that object (using Python mechanisms, not delete...) if some part of the object happens to equal 0.
There's a number of options, all basically centred around creating a wrapper object - a Python object defined in C++ that holds either a pointer or value of your C++ object.
Do it manually using the Python C API - This answer gives a very thorough example.
Look up the PyCapsule interface to create a quick wrapper around your object. You'd create your capsule with:
PyObject* cap = PyCapsule_New(cpp_attr, "MyClass",
[](PyObject* c) {
auto deleteme = reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass));
delete deleteme;
});
And you retrieve your C++ class from the capsule with:
reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass))
Use some tool like PyBind11, Cython, SWIG, etc to create the wrapper object for you.
Note also that PyObject_SetAttrString does not require the third argument to be a tuple (unless you specifically want to store a tuple...). You're likely getting it confused with PyObject_Call, where the args are passed as a tuple.
Assuming your call to PyTuple_Pack is correct, then you've created a PyTupleObject which has a structure:
typedef struct {
PyObject_VAR_HEAD
PyObject *ob_item[1];
} PyTupleObject;
The PyTupleObject inherits from the generic PyObject struct which has the following members:
struct _object *_ob_next;
struct _object *_ob_prev;
Py_ssize_t ob_refcnt;
struct _typeobject *ob_type;
You can access the latter two with the macrosPy_REFCNT and Py_TYPE
The ob_item[1] member should be a pointer to the memory initially allocated. Based on how Macros are written in the documentation, you should be able to access it by
((PyTupleObject *)cpp_attr)->ob_item
And if you know the data type of the C++ pointer, then you should be able to cast it back. Maybe you can try
MyClass* cpp_att_again = reinterpret_cast<MyClass*>((PyTupleObject *)cpp_attr)->ob_item
Hopefully this points you in the right direction. You might be able to glean more insight from a similar question.
I'm in the middle of trying to wrap a c++ project into a python api using SWIG and I'm running into an issue with code that has the following format.
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
typedef boost::variant<A,B,C> VariantType;
typedef std::vector<boost::variant<A,B,C>> VariantTypeList;
Classes A,B & C all come out in the python wrapper without a problem and seem to be usable. However when I try to add the following lines to the interface file
%template(VariantType) boost::variant<A,B,C>;
%template(VariantTypeList) std::vector<boost::variant<A,B,C>>;
I get an error that says
Boost\x64\include\boost\variant\variant.hpp(148): error : Syntax error in input(3).
So I go and look at the error and its a line that has a macro that is defined inside another header file specifically "boost/mpl/aux_/value_wknd.hpp" so I add that to the interface file with %include and now it appears that SWIG.exe crashes with an error helpfully stating
Access Violation
So long story short is there a way to wrap a boost::variant template type? Unfortunately this template definition is baked into the core of our library and I can't change it now. Also if it matters I'm on the MSVC 2013 compiler.
If it isn't possible to wrap the template type directly is it possible to work around this? I'm reading through the SWIG documentation to see if there is some typemap magic that can be applied but I'm fairly new to SWIG in general.
You can do this. I thought for quite a while about what the neatest Python interface to boost::variant actually is. My conclusion was that 99% of the time a Python user shouldn't even realise there's a variant type being use - unions and variants are basically just somewhat constrained duck-typing for C++.
So my goals were this:
wherever possible benefit from existing typemaps - we don't want to have to write our own std::string, int, typemaps from scratch.
anywhere a C++ function takes a boost::variant we should transparently accept any of the types the variant can hold for that function argument.
anywhere a C++ function returns a boost::variant we should transparently return it as the type the variant was holding when we got it back into Python.
allow Python users to explicitly create a variant object, e.g. an empty one, but don't expect that to ever actually happen. (Maybe that would be useful for reference output arguments, but I've not gone that far in this currently).
I didn't do this, but it would be fairly simple to add visitors from where this interface currently stands using the directors feature of SWIG.
It's pretty fiddly to do all that without adding some machinery into things. I wrapped everything up in a reusable file, this is the final working version of my boost_variant.i:
%{
#include <boost/variant.hpp>
static PyObject *this_module = NULL;
%}
%init %{
// We need to "borrow" a reference to this for our typemaps to be able to look up the right functions
this_module = m; // borrow should be fine since we can only get called when our module is loaded right?
// Wouldn't it be nice if $module worked *anywhere*
%}
#define FE_0(...)
#define FE_1(action,a1) action(0,a1)
#define FE_2(action,a1,a2) action(0,a1); action(1,a2)
#define FE_3(action,a1,a2,a3) action(0,a1); action(1,a2); action(2,a3)
#define FE_4(action,a1,a2,a3,a4) action(0,a1); action(1,a2); action(2,a3); action(3,a4)
#define FE_5(action,a1,a2,a3,a4,a5) action(0,a1); action(1,a2); action(2,a3); action(3,a4); action(4,a5)
#define GET_MACRO(_1,_2,_3,_4,_5,NAME,...) NAME
%define FOR_EACH(action,...)
GET_MACRO(__VA_ARGS__, FE_5, FE_4, FE_3, FE_2, FE_1, FE_0)(action,__VA_ARGS__)
%enddef
#define in_helper(num,type) const type & convert_type ## num () { return boost::get<type>(*$self); }
#define constructor_helper(num,type) variant(const type&)
%define %boost_variant(Name, ...)
%rename(Name) boost::variant<__VA_ARGS__>;
namespace boost {
struct variant<__VA_ARGS__> {
variant();
variant(const boost::variant<__VA_ARGS__>&);
FOR_EACH(constructor_helper, __VA_ARGS__);
int which();
bool empty();
%extend {
FOR_EACH(in_helper, __VA_ARGS__);
}
};
}
%typemap(out) boost::variant<__VA_ARGS__> {
// Make our function output into a PyObject
PyObject *tmp = SWIG_NewPointerObj(&$1, $&1_descriptor, 0); // Python does not own this object...
// Pass that temporary PyObject into the helper function and get another PyObject back in exchange
const std::string func_name = "convert_type" + std::to_string($1.which());
$result = PyObject_CallMethod(tmp, func_name.c_str(), "");
Py_DECREF(tmp);
}
%typemap(in) const boost::variant<__VA_ARGS__>& (PyObject *tmp=NULL) {
// I don't much like having to "guess" the name of the make_variant we want to use here like this...
// But it's hard to support both -builtin and regular modes and generically find the right code.
PyObject *helper_func = PyObject_GetAttrString(this_module, "new_" #Name );
assert(helper_func);
// TODO: is O right, or should it be N?
tmp = PyObject_CallFunction(helper_func, "O", $input);
Py_DECREF(helper_func);
if (!tmp) SWIG_fail; // An exception is already pending
// TODO: if we cared, we chould short-circuit things a lot for the case where our input really was a variant object
const int res = SWIG_ConvertPtr(tmp, (void**)&$1, $1_descriptor, 0);
if (!SWIG_IsOK(res)) {
SWIG_exception_fail(SWIG_ArgError(res), "Variant typemap failed, not sure if this can actually happen");
}
}
%typemap(freearg) const boost::variant<__VA_ARGS__>& %{
Py_DECREF(tmp$argnum);
%}
%enddef
This gives us a macro we can use in SWIG, %boost_variant. You can then use this in your interface file something like this:
%module test
%include "boost_variant.i"
%inline %{
struct A {};
struct B {};
%}
%include <std_string.i>
%boost_variant(TestVariant, A, B, std::string);
%inline %{
void idea(const boost::variant<A, B, std::string>&) {
}
boost::variant<A,B,std::string> make_me_a_thing() {
struct A a;
return a;
}
boost::variant<A,B,std::string> make_me_a_string() {
return "HELLO";
}
%}
Where the %boost_variant macro takes the first argument as a name for the type (much like %template would) and the remaining arguments as a list of all the types in the variant.
This is sufficient to allow us to run the following Python:
import test
a = test.A();
b = test.B();
test.idea(a)
test.idea(b)
print(test.make_me_a_thing())
print(test.make_me_a_string())
So how does that actually work?
We essentially duplicate SWIG's %template support here. (It's documented here as an option)
Most of the heavy lifting in my file is done using a FOR_EACH variadic macro. Largely that's the same as my previous answer on std::function, which was itself derived from several older Stack Overflow answers and adapted to work with SWIG's preprocessor.
Using the FOR_EACH macro we tell SWIG to wrap one constructor per type the variant can hold. This lets us explicitly construct variants from Python code, with two extra constructors added
By using constructors like this we can lean heavily on SWIG's overload resolution support. So given a Python object we can simply rely on SWIG to determine how to construct a variant from it. Which saves us a bunch of extra work, and uses the existing typemaps for each type within the variant.
The in typemap basically just delegates to the constructor, via a slightly convoluted route because it's surprisingly hard to find other functions in the same module programatically. Once that delegation has happened we use the normal conversion of a function argument to just pass the tempoary variant into the function as though it were what we were given.
We also synthesise a set of extra member functions, convert_typeN which internally just call boost::get<TYPE>(*this), where N and TYPE are the position of each type in the list of variant types.
Within the out typemap this then allows us to lookup a Python function, using which() to determine what the variant currently holds. We've then got largely SWIG generated code, using existing typemaps to make a given variant into a Python object of the underlying type. Again that saves us a lot of effort and makes everything plug and play.
If you're decided on SWIG (which wasn't clear to me from your post as you said to be fairly new to SWIG, so I'm under the assumption that this is a new project), then stop reading and ignore this answer.
But in case the bindings technology to use isn't fixed yet and you only need to bind Python, no other languages, an alternative is to use cppyy (http://cppyy.org, and full disclaimer: I'm main author). With that, the boost::variant type is directly available in Python and then you can make it look/behave more Pythonistic by writing Python code rather than SWIG .i code.
Example (note that cppyy has wheels for Windows on PyPI but built with MSVC2017, not MSVC2013, so I'll keep that caveat as to whether MSVC2013 is modern enough to build the code as I haven't tried):
import cppyy
cppyy.include("boost/variant/variant.hpp")
cppyy.include("boost/variant/get.hpp")
cpp = cppyy.gbl
std = cpp.std
boost = cpp.boost
cppyy.cppdef("""
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
""")
VariantType = boost.variant['A, B, C']
VariantTypeList = std.vector[VariantType]
v = VariantTypeList()
v.push_back(VariantType(cpp.A()))
print(v.back().which())
v.push_back(VariantType(cpp.B()))
print(v.back().which())
v.push_back(VariantType(cpp.C()))
print(v.back().which())
print(boost.get['A'](v[0]))
try:
print(boost.get['B'](v[0]))
except Exception as e:
print(e) # b/c of type-index mismatch above
print(boost.get['B'](v[1])) # now corrected
print(boost.get['C'](v[2]))
which produces the expect output of:
$ python variant.py
0
1
2
<cppyy.gbl.A object at 0x5053704>
Could not instantiate get<B>:
B& boost::get(boost::variant<A,B,C>& operand) =>
Exception: boost::bad_get: failed value get using boost::get (C++ exception)
<cppyy.gbl.B object at 0x505370c>
<cppyy.gbl.C object at 0x5053714>
I have successfully wrapped a C++ class, MyObject to Python using Swig.
class MyObject()
{
public:
double i;
};
A MyObject can be created in Python like this:
import MyModule
m = MyModule.MyObject()
The important thing about the above lines is that behind the scenes, the MyObject object is created in the pyd module that is created from C++, and "somewhere", there is a new MyObject() call, returning a pointer to a MyObject object.
Further, the following Python script has a function that returns a Python MyObject, i.e:
import MyModule
def getMyObject():
m = MyModule.MyObject()
#... do something with m, e.g.
m.i = 42
return m
My actual use case is that I initialize the Python interpreter from a C++ application.
I am executing the Python function above,from within my C++ application, i.e.
PyObject *pValue = PyObject_CallObject(pFunc, NULL);
where pFunc is a PyObject* that points to the getMyObject() python function. Code for setting up pFunc omitted for clarity. The returned value should hold a pointer to a MyObject "somewhere".
So question is, how, on the C++ side, do I go from the PyObject* pValue to a MyObject* pointer?
I do have a similar question open, see How to convert a returned Python dictionary to a C++ std::map<string, string>, but that question is more complex, as it involve the conversion from a Python dictionary, to a std class (std::map).
Having a user defined type, like MyObject, would perhaps be the first a user would want to learn to exchange. Curiously, I can't find a simple example showing how todo this.
I'm new to Swig but after doing some research it seem likely that one need to create a typemap?
Also, I'm using Embarcaderos C++ Builder, and can't use any C++11 and/or boost python libraries.
Any hints?
Update
The following code seem to properly unwrap the C++ object pointer:
PyObject* p = python.callFunction(getPluginMetaDataF);
PyObject* pThis = PyObject_GetAttrString(p, "this");
unsigned long addr = PyLong_AsLong(pThis);
MyObject* ptr = (MyObject*)(addr);
cout << ptr->i; //Prints 42!
The above code seem to properly access the object. However, the life time of the object seem a little shaky, i.e. when is this object destroyed? After Py_Finalize() ??
I'm a newbie in Python and embedding it too. And I have one problem:
There is function in my python module that recieves buffer created with ctypes.create_string_buffer(size) and fills it by content from some memory address:
def get_mem(self, address, size, data):
self.mem.read_ram_block(address, size, data)
How should I call this method with using of (char *) buffer? I want fill my C++ buffer with recieved from python.
If you only want to call the Python function ctypes.create_string_buffer(size), you could easily mirror the Python coding on the C++ side:
static PyObject* create_string_buffer(unsigned long size) {
PyObject *ctypes = PyImport_ImportModule("ctypes");
if (!ctypes) return 0;
PyObject *buf = PyObject_CallMethod(ctypes, "create_string_buffer", "k", size);
Py_DECREF(ctypes);
return buf;
}
If you'd like to use another type than unsigned long for the size, you'd need to change the format in PyObject_CallMethod as well. For example O is used for PyObject*. For a complete list of formats see the documentation for Building values.