Related
I'm in the middle of trying to wrap a c++ project into a python api using SWIG and I'm running into an issue with code that has the following format.
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
typedef boost::variant<A,B,C> VariantType;
typedef std::vector<boost::variant<A,B,C>> VariantTypeList;
Classes A,B & C all come out in the python wrapper without a problem and seem to be usable. However when I try to add the following lines to the interface file
%template(VariantType) boost::variant<A,B,C>;
%template(VariantTypeList) std::vector<boost::variant<A,B,C>>;
I get an error that says
Boost\x64\include\boost\variant\variant.hpp(148): error : Syntax error in input(3).
So I go and look at the error and its a line that has a macro that is defined inside another header file specifically "boost/mpl/aux_/value_wknd.hpp" so I add that to the interface file with %include and now it appears that SWIG.exe crashes with an error helpfully stating
Access Violation
So long story short is there a way to wrap a boost::variant template type? Unfortunately this template definition is baked into the core of our library and I can't change it now. Also if it matters I'm on the MSVC 2013 compiler.
If it isn't possible to wrap the template type directly is it possible to work around this? I'm reading through the SWIG documentation to see if there is some typemap magic that can be applied but I'm fairly new to SWIG in general.
You can do this. I thought for quite a while about what the neatest Python interface to boost::variant actually is. My conclusion was that 99% of the time a Python user shouldn't even realise there's a variant type being use - unions and variants are basically just somewhat constrained duck-typing for C++.
So my goals were this:
wherever possible benefit from existing typemaps - we don't want to have to write our own std::string, int, typemaps from scratch.
anywhere a C++ function takes a boost::variant we should transparently accept any of the types the variant can hold for that function argument.
anywhere a C++ function returns a boost::variant we should transparently return it as the type the variant was holding when we got it back into Python.
allow Python users to explicitly create a variant object, e.g. an empty one, but don't expect that to ever actually happen. (Maybe that would be useful for reference output arguments, but I've not gone that far in this currently).
I didn't do this, but it would be fairly simple to add visitors from where this interface currently stands using the directors feature of SWIG.
It's pretty fiddly to do all that without adding some machinery into things. I wrapped everything up in a reusable file, this is the final working version of my boost_variant.i:
%{
#include <boost/variant.hpp>
static PyObject *this_module = NULL;
%}
%init %{
// We need to "borrow" a reference to this for our typemaps to be able to look up the right functions
this_module = m; // borrow should be fine since we can only get called when our module is loaded right?
// Wouldn't it be nice if $module worked *anywhere*
%}
#define FE_0(...)
#define FE_1(action,a1) action(0,a1)
#define FE_2(action,a1,a2) action(0,a1); action(1,a2)
#define FE_3(action,a1,a2,a3) action(0,a1); action(1,a2); action(2,a3)
#define FE_4(action,a1,a2,a3,a4) action(0,a1); action(1,a2); action(2,a3); action(3,a4)
#define FE_5(action,a1,a2,a3,a4,a5) action(0,a1); action(1,a2); action(2,a3); action(3,a4); action(4,a5)
#define GET_MACRO(_1,_2,_3,_4,_5,NAME,...) NAME
%define FOR_EACH(action,...)
GET_MACRO(__VA_ARGS__, FE_5, FE_4, FE_3, FE_2, FE_1, FE_0)(action,__VA_ARGS__)
%enddef
#define in_helper(num,type) const type & convert_type ## num () { return boost::get<type>(*$self); }
#define constructor_helper(num,type) variant(const type&)
%define %boost_variant(Name, ...)
%rename(Name) boost::variant<__VA_ARGS__>;
namespace boost {
struct variant<__VA_ARGS__> {
variant();
variant(const boost::variant<__VA_ARGS__>&);
FOR_EACH(constructor_helper, __VA_ARGS__);
int which();
bool empty();
%extend {
FOR_EACH(in_helper, __VA_ARGS__);
}
};
}
%typemap(out) boost::variant<__VA_ARGS__> {
// Make our function output into a PyObject
PyObject *tmp = SWIG_NewPointerObj(&$1, $&1_descriptor, 0); // Python does not own this object...
// Pass that temporary PyObject into the helper function and get another PyObject back in exchange
const std::string func_name = "convert_type" + std::to_string($1.which());
$result = PyObject_CallMethod(tmp, func_name.c_str(), "");
Py_DECREF(tmp);
}
%typemap(in) const boost::variant<__VA_ARGS__>& (PyObject *tmp=NULL) {
// I don't much like having to "guess" the name of the make_variant we want to use here like this...
// But it's hard to support both -builtin and regular modes and generically find the right code.
PyObject *helper_func = PyObject_GetAttrString(this_module, "new_" #Name );
assert(helper_func);
// TODO: is O right, or should it be N?
tmp = PyObject_CallFunction(helper_func, "O", $input);
Py_DECREF(helper_func);
if (!tmp) SWIG_fail; // An exception is already pending
// TODO: if we cared, we chould short-circuit things a lot for the case where our input really was a variant object
const int res = SWIG_ConvertPtr(tmp, (void**)&$1, $1_descriptor, 0);
if (!SWIG_IsOK(res)) {
SWIG_exception_fail(SWIG_ArgError(res), "Variant typemap failed, not sure if this can actually happen");
}
}
%typemap(freearg) const boost::variant<__VA_ARGS__>& %{
Py_DECREF(tmp$argnum);
%}
%enddef
This gives us a macro we can use in SWIG, %boost_variant. You can then use this in your interface file something like this:
%module test
%include "boost_variant.i"
%inline %{
struct A {};
struct B {};
%}
%include <std_string.i>
%boost_variant(TestVariant, A, B, std::string);
%inline %{
void idea(const boost::variant<A, B, std::string>&) {
}
boost::variant<A,B,std::string> make_me_a_thing() {
struct A a;
return a;
}
boost::variant<A,B,std::string> make_me_a_string() {
return "HELLO";
}
%}
Where the %boost_variant macro takes the first argument as a name for the type (much like %template would) and the remaining arguments as a list of all the types in the variant.
This is sufficient to allow us to run the following Python:
import test
a = test.A();
b = test.B();
test.idea(a)
test.idea(b)
print(test.make_me_a_thing())
print(test.make_me_a_string())
So how does that actually work?
We essentially duplicate SWIG's %template support here. (It's documented here as an option)
Most of the heavy lifting in my file is done using a FOR_EACH variadic macro. Largely that's the same as my previous answer on std::function, which was itself derived from several older Stack Overflow answers and adapted to work with SWIG's preprocessor.
Using the FOR_EACH macro we tell SWIG to wrap one constructor per type the variant can hold. This lets us explicitly construct variants from Python code, with two extra constructors added
By using constructors like this we can lean heavily on SWIG's overload resolution support. So given a Python object we can simply rely on SWIG to determine how to construct a variant from it. Which saves us a bunch of extra work, and uses the existing typemaps for each type within the variant.
The in typemap basically just delegates to the constructor, via a slightly convoluted route because it's surprisingly hard to find other functions in the same module programatically. Once that delegation has happened we use the normal conversion of a function argument to just pass the tempoary variant into the function as though it were what we were given.
We also synthesise a set of extra member functions, convert_typeN which internally just call boost::get<TYPE>(*this), where N and TYPE are the position of each type in the list of variant types.
Within the out typemap this then allows us to lookup a Python function, using which() to determine what the variant currently holds. We've then got largely SWIG generated code, using existing typemaps to make a given variant into a Python object of the underlying type. Again that saves us a lot of effort and makes everything plug and play.
If you're decided on SWIG (which wasn't clear to me from your post as you said to be fairly new to SWIG, so I'm under the assumption that this is a new project), then stop reading and ignore this answer.
But in case the bindings technology to use isn't fixed yet and you only need to bind Python, no other languages, an alternative is to use cppyy (http://cppyy.org, and full disclaimer: I'm main author). With that, the boost::variant type is directly available in Python and then you can make it look/behave more Pythonistic by writing Python code rather than SWIG .i code.
Example (note that cppyy has wheels for Windows on PyPI but built with MSVC2017, not MSVC2013, so I'll keep that caveat as to whether MSVC2013 is modern enough to build the code as I haven't tried):
import cppyy
cppyy.include("boost/variant/variant.hpp")
cppyy.include("boost/variant/get.hpp")
cpp = cppyy.gbl
std = cpp.std
boost = cpp.boost
cppyy.cppdef("""
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
""")
VariantType = boost.variant['A, B, C']
VariantTypeList = std.vector[VariantType]
v = VariantTypeList()
v.push_back(VariantType(cpp.A()))
print(v.back().which())
v.push_back(VariantType(cpp.B()))
print(v.back().which())
v.push_back(VariantType(cpp.C()))
print(v.back().which())
print(boost.get['A'](v[0]))
try:
print(boost.get['B'](v[0]))
except Exception as e:
print(e) # b/c of type-index mismatch above
print(boost.get['B'](v[1])) # now corrected
print(boost.get['C'](v[2]))
which produces the expect output of:
$ python variant.py
0
1
2
<cppyy.gbl.A object at 0x5053704>
Could not instantiate get<B>:
B& boost::get(boost::variant<A,B,C>& operand) =>
Exception: boost::bad_get: failed value get using boost::get (C++ exception)
<cppyy.gbl.B object at 0x505370c>
<cppyy.gbl.C object at 0x5053714>
Swig generates wrapper code for a object member that has not default constructor.
Code to wrap:
class Foo {
public:
Foo (int i);
};
Class Bar {
public:
Bar(int i):foo(i)
{
...
}
Foo foo;
};
Swig Setter generated:
SWIGINTERN PyObject *_wrap_Bar_foo_set(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
Bar *arg1 = (Bar *) 0 ;
Foo arg2 ; // -> swig generates a call to a non existing default constructor
...
Then, if a try to compile the wrapper, I get an error since the default contructor does not exist:
error: no matching function for call to ‘Foo::Foo()’
Please note, the same approach is done for the getter generation.
How can I tell swig to generate a setter that accepts Foo* or Foo&?
Thanks,
Pablo
SWIG fundamentally supports this just fine, in fact I can't actually reproduce what you've seen with the code you showed. For example, this all works:
%module test
%inline %{
class Foo {
public:
Foo (int i) {}
};
class Bar {
public:
Bar(int i):foo(i)
{
}
Foo foo;
};
%}
When compiled and run with SWIG 3.0.2 (which is pretty old these days!) lets me run this Python code:
import test
f=test.Foo(0)
b=test.Bar(0)
b.foo=f
print('Well that all worked ok')
The reason this can work, even in the more general cases, is because of a feature known as the "Fulton Transform". In essence this is intended to work around a lack of copy constructor by wrapping it inside another object instead. (Although in the specific instance you've shown it's not actually needed even).
Anyway, although this should apply automatically there are a few cases were it can't won't. Fortunately though you can force this on even when it doesn't work automatically, using %feature
All you need to do is include, within your .i file, somewhere before first declaration/definition of a type without a copy ctor the following:
%feature("valuewrapper") Foo;
And that's it.
I am trying to wrap a C++ library for python, using SWIG. The library uses callback functions frequently, by passing callback functions of certain type to class methods.
Now, after wrapping the code, I would like to create the callback logic from python. Is this possible? Here is an experiment I was doing to find it out .. does not work at the moment.
The header and swig files are as follows:
paska.h :
typedef void (handleri)(int code, char* codename);
// handleri is now an alias to a function that eats int, string and returns void
void wannabe_handleri(int i, char* blah);
void handleri_eater(handleri* h);
paska.i :
%module paska
%{ // this section is copied in the front of the wrapper file
#define SWIG_FILE_WITH_INIT
#include "paska.h"
%}
// from now on, what are we going to wrap ..
%inline %{
// helper functions here
void wannabe_handleri(int i, char* blah) {
};
void handleri_eater(handleri* h) {
};
%}
%include "paska.h"
// in this case, we just put the actual .cpp code into the inline block ..
Finally, I test in python ..
import paska
def testfunc(i, st):
print i
print st
paska.handleri_eater(paska.wannabe_handleri(1,"eee")) # THIS WORKS!
paska.handleri_eater(testfunc) # THIS DOES NOT WORK!
The last line throws me "TypeError: in method 'handleri_eater', argument 1 of type 'handleri *'"
Is there any way to "cast" the python function to a type accepted by the SWIG wrapper?
Seems to me that a combination of ctypes and a SWIG typemap would be the easiest way to solve the problem. ctypes makes it easy to generate a C function that calls a Python callable. The Python code should be along the lines of:
import example
# python callback
def py_callback(i, s):
print( 'py_callback(%d, %s)'%(i, s) )
example.use_callback(py_callback)
On the SWIG side we have: (1) a Python function use_callback that wraps the Python callback with a ctypes wrapper, and passes the address the wrapper as an integer to _example.use_callback(), and (2) a SWIG typemap that extracts the address and casts itto the appropriate function pointer.
%module example
// a typemap for the callback, it expects the argument to be an integer
// whose value is the address of an appropriate callback function
%typemap(in) void (*f)(int, const char*) {
$1 = (void (*)(int i, const char*))PyLong_AsVoidPtr($input);;
}
%{
void use_callback(void (*f)(int i, const char* str));
%}
%inline
%{
// a C function that accepts a callback
void use_callback(void (*f)(int i, const char* str))
{
f(100, "callback arg");
}
%}
%pythoncode
%{
import ctypes
# a ctypes callback prototype
py_callback_type = ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_char_p)
def use_callback(py_callback):
# wrap the python callback with a ctypes function pointer
f = py_callback_type(py_callback)
# get the function pointer of the ctypes wrapper by casting it to void* and taking its value
f_ptr = ctypes.cast(f, ctypes.c_void_p).value
_example.use_callback(f_ptr)
%}
You can find this complete example with a CMakeLists.txt file here.
edit: incorporated #Flexo suggestion to move the Python part into the %pythoncode block of the SWIG file.
edit: incorporated #user87746 suggestion for Python 3.6+ compatibility.
You can implement the callback logic in Python by using "directors".
Basically, instead of passing callback functions, you pass callback objects instead. The base object can be defined in C++ and provide a virtual callback member function. This object can then be inherited from and the callback function overwritten in Python. The inherited object can then be passed to a C++ function instead of a callback function. For this to work, you need to enable the director feature for such a callback class.
This does require changing the underlying C++ library, though.
I'm using SWIG to generate Python Bindings for my qt app. I have several places where I use QLists and I would like to integrate those QLists like std::vector from the SWIG Library (see http://www.swig.org/Doc1.3/Library.html#Library_nn15).
This means:
The QList objects should be iterable from python (= they must be an iterable python object)
It should be possible to pass a python list to a function which takes a qlist
... and all the other features listed in the SWIG Library for std::vector
To achieve that I use the following Code:
https://github.com/osmandapp/OsmAnd-core/blob/master/swig/java/QList.i
Later in my classes using QLists, I add code like:
%import "qlist.i"
%template(listfilter) QList<Interface_Filter*>;
class A {
public:
//.....
QList<Interface_Filter*> get_filters();
};
This works so far, but it doesn't give me the kind of integration I get with std::vector.
I'm having trouble finding out which parts of std_vector.i, std_container.i,... make an object iterable.
How do I need to extend the QList interface file to make my QList's iterable?
What you are asking for -- a qlist.i swig file that achieves the same level of integration for QList in python as std_vector.i does for std::vector -- is a non-trivial task.
I provide a very basic extended qlist.i file (and qlisttest.i to show you how to use it) and will try to explain what steps are required.
qlist.i:
%{
#include <QList>
%}
%pythoncode %{
class QListIterator:
def __init__(self, qlist):
self.index = 0
self.qlist = qlist
def __iter__(self):
return self
def next(self):
if self.index >= self.qlist.size():
raise StopIteration;
ret = self.qlist.get(self.index)
self.index += 1
return ret
__next__ = next
%}
template<class T> class QList {
public:
class iterator;
typedef size_t size_type;
typedef T value_type;
typedef const value_type& const_reference;
QList();
size_type size() const;
void reserve(size_type n);
%rename(isEmpty) empty;
bool empty() const;
void clear();
%rename(add) push_back;
void push_back(const value_type& x);
%extend {
const_reference get(int i) throw (std::out_of_range) {
int size = int(self->size());
if (i>=0 && i<size)
return (*self)[i];
else
throw std::out_of_range("QList index out of range");
}
void set(int i, const value_type& val) throw (std::out_of_range) {
int size = int(self->size());
if (i>=0 && i<size)
(*self)[i] = val;
else
throw std::out_of_range("QList index out of range");
}
int __len__() {
return self->size();
}
const_reference __getitem__(int i) throw (std::out_of_range) {
int size = int(self->size());
if (i>=0 && i<size)
return (*self)[i];
else
throw std::out_of_range("QList index out of range");
}
%pythoncode %{
def __iter__(self):
return QListIterator(self)
%}
}
};
%define %qlist_conversions(Type...)
%typemap(in) const QList< Type > & (bool free_qlist)
{
free_qlist = false;
if ((SWIG_ConvertPtr($input, (void **) &$1, $1_descriptor, 0)) == -1) {
if (!PyList_Check($input)) {
PyErr_Format(PyExc_TypeError, "QList or python list required.");
SWIG_fail;
}
Py_ssize_t len = PyList_Size($input);
QList< Type > * qlist = new QList< Type >();
free_qlist = true;
qlist->reserve(len);
for (Py_ssize_t index = 0; index < len; ++index) {
PyObject *item = PyList_GetItem($input,index);
Type* c_item;
if ((SWIG_ConvertPtr(item, (void **) &c_item, $descriptor(Type *),0)) == -1) {
delete qlist;
free_qlist = false;
PyErr_Format(PyExc_TypeError, "List element of wrong type encountered.");
SWIG_fail;
}
qlist->append(*c_item);
}
$1 = qlist;
}
}
%typemap(freearg) const QList< Type > &
{ if (free_qlist$argnum and $1) delete $1; }
%enddef
qlisttest.i:
%module qlist;
%include "qlist.i"
%inline %{
class Foo {
public:
int foo;
};
%}
%template(QList_Foo) QList<Foo>;
%qlist_conversions(Foo);
%inline %{
int sumList(const QList<Foo> & list) {
int sum = 0;
for (int i = 0; i < list.size(); ++i) {
sum += list[i].foo;
}
return sum;
}
%}
Wrapping of QList to make it and its methods accessible from python
This is achieved by making the (partial) class definition available to swig. That is what your current qlist.i does.
Note: You might need to add a "template specialization" for the case QList<T*> that typedefs const_reference as const T* since you are using a QList of pointers. Otherwise, QList<T*>::const_reference will be const T*&, which apparently might confuse swig. (see swig/Lib/std/std_vector.i)
Automatic conversion between python list and QList
This is generally achieved by using swig typemaps. For instance, if you want a function f(const QList<int>& list) to be able to accept a python list, you need to specify an input typemap that performs the conversion from a python list to a QList<int>:
%typemap(in) const QList<int> &
{
PyObject * py_list = $input;
[check if py_list is really a python list of integers]
QList<int>* qlist = new QList<int>();
[copy the data from the py_list to the qlist]
$1 = qlist;
}
%typemap(freearg) const QList<int> &
{ if ($1) delete $1; }
Here, the situation is more difficult in several ways:
You want to be able to pass a python lists or a wrapped QList: For this to work, you need to handle both cases in the typemap.
You want to convert a python list of wrapped type T to a QList<T>:
This also involves a conversion for every element of the list from the wrapped type T to the plain T. This is achieved by the swig function SWIG_ConvertPtr.
I am not sure if you can specify typemaps with template arguments. Therefore, I wrote a swig macro %qlist_conversions(Type) that you can use to attach the typemap to the QList<Type> for a specific Type.
For the other conversion direction (QList -> python list) you should first consider what you want. Consider a C++ function that returns a QList<int>. Calling this from python, should this return a wrapped QList object, or should it automatically convert the QList to a python list?
Accessing the wrapped QList as a python sequence, i.e., make len and [] work from python
For this, you need to extend the QList class in the qlist.i file using %extend { ... } and implement __len__ and __getitem__ methods.
If slicing should also work, you need to provide a __getitem__(PySliceObject *slice)__ member method and input and "typecheck" typemaps for PySliceObjects.
If you want to be able to modify values in the wrapped QList using [] from python, you need to implement __setitem__.
For a list of all the useful methods you can implement to achieve better integration, see the python documentation on "builtin types" and "abstract base classes for containers".
Note: If you use the swig -builtin feature, then you need to additionally register the above functions to the appropriate "slots" using e.g.
%feature("python:slot", "sq_length", functype="lenfunc") __len__;
Making the wrapped QList iterable from python
For this you need to extend the QList class and implement an __iter__() method that returns a python iterator object.
A python iterator object is an object that provides the methods __iter__() and __next__() (next() for older python), where __next__() returns the next value and raises the python exception StopIteration to signal the end.
As mentioned before, you can implement the iterator object in python or C++. I show an example of doing this in python.
I hope this helps as a basis for you to tweak the functionality that you require.
You provided an answer to the question "How to make a python Object iterable", but I asked for "How do I need to extend the QList interface file to make my QList's iterable?" which is more a SWIG, than a python related question.
I tested the example from http://www.swig.org/Doc1.3/Library.html#Library_nn15 with Java, C# and Python. Only Python and C# provide iterators. The generated interface of Java doesn't implement Iterable or something like that. As far as I can see your question is related to the target language.
Maybe extending MutableSequence is an option for you. The only methods you have to implement are __getitem__, __setitem__, __delitem__, __len__ and insert by delegating them to the corresponding methods of QList. Afterwards your generated class is iterable.
As described in the docs, you need the following:
QList should have a method in python __iter__() that returns an iterator object (tp_iter if you implement it in C).
The iterator object should implement __iter__() and return itself
The iterator object should implement next() that returns the next item or raises StopIteration when it's done.
It's probably easiest to do in python, but you can implement it in C as well.
Another option is to use python generators to avoid implementing an iterator type. To do this you QList needs to implement __iter__() but instead of returning an iterator you simply yield the values.
The methods mentioned only need to be visible to python. You don't have to make them available in C/Java.
See also SWIG interfacing C library to Python (Creating 'iterable' Python data type from C 'sequence' struct)
I am working on a C++ library with Python bindings (using boost::python) representing data stored in a file. Majority of my semi-technical users will be using Python to interact with it, so I need to make it as Pythonic as possible. However, I will also have C++ programmers using the API, so I do not want to compromise on the C++ side to accommodate Python bindings.
A large part of the library will be made out of containers. To make things intuitive for the python users, I would like them to behave like python lists, i.e.:
# an example compound class
class Foo:
def __init__( self, _val ):
self.val = _val
# add it to a list
foo = Foo(0.0)
vect = []
vect.append(foo)
# change the value of the *original* instance
foo.val = 666.0
# which also changes the instance inside the container
print vect[0].val # outputs 666.0
The test setup
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
#include <boost/python/register_ptr_to_python.hpp>
#include <boost/shared_ptr.hpp>
struct Foo {
double val;
Foo(double a) : val(a) {}
bool operator == (const Foo& f) const { return val == f.val; }
};
/* insert the test module wrapping code here */
int main() {
Py_Initialize();
inittest();
boost::python::object globals = boost::python::import("__main__").attr("__dict__");
boost::python::exec(
"import test\n"
"foo = test.Foo(0.0)\n" // make a new Foo instance
"vect = test.FooVector()\n" // make a new vector of Foos
"vect.append(foo)\n" // add the instance to the vector
"foo.val = 666.0\n" // assign a new value to the instance
// which should change the value in vector
"print 'Foo =', foo.val\n" // and print the results
"print 'vector[0] =', vect[0].val\n",
globals, globals
);
return 0;
}
The way of the shared_ptr
Using the shared_ptr, I can get the same behaviour as above, but it also means that I have to represent all data in C++ using shared pointers, which is not nice from many points of view.
BOOST_PYTHON_MODULE( test ) {
// wrap Foo
boost::python::class_< Foo, boost::shared_ptr<Foo> >("Foo", boost::python::init<double>())
.def_readwrite("val", &Foo::val);
// wrap vector of shared_ptr Foos
boost::python::class_< std::vector < boost::shared_ptr<Foo> > >("FooVector")
.def(boost::python::vector_indexing_suite<std::vector< boost::shared_ptr<Foo> >, true >());
}
In my test setup, this produces the same output as pure Python:
Foo = 666.0
vector[0] = 666.0
The way of the vector<Foo>
Using a vector directly gives a nice clean setup on the C++ side. However, the result does not behave in the same way as pure Python.
BOOST_PYTHON_MODULE( test ) {
// wrap Foo
boost::python::class_< Foo >("Foo", boost::python::init<double>())
.def_readwrite("val", &Foo::val);
// wrap vector of Foos
boost::python::class_< std::vector < Foo > >("FooVector")
.def(boost::python::vector_indexing_suite<std::vector< Foo > >());
}
This produces:
Foo = 666.0
vector[0] = 0.0
Which is "wrong" - changing the original instance did not change the value inside the container.
I hope I don't want too much
Interestingly enough, this code works no matter which of the two encapsulations I use:
footwo = vect[0]
footwo.val = 555.0
print vect[0].val
Which means that boost::python is able to deal with "fake shared ownership" (via its by_proxy return mechanism). Is there any way to achieve the same while inserting new elements?
However, if the answer is no, I'd love to hear other suggestions - is there an example in the Python toolkit where a similar collection encapsulation is implemented, but which does not behave as a python list?
Thanks a lot for reading this far :)
Due to the semantic differences between the languages, it is often very difficult to apply a single reusable solution to all scenarios when collections are involved. The largest issue is that the while Python collections directly support references, C++ collections require a level of indirection, such as by having shared_ptr element types. Without this indirection, C++ collections will not be able to support the same functionality as Python collections. For instance, consider two indexes that refer to the same object:
s = Spam()
spams = []
spams.append(s)
spams.append(s)
Without pointer-like element types, a C++ collection could not have two indexes referring to the same object. Nevertheless, depending on usage and needs, there may be options that allow for a Pythonic-ish interface for the Python users while still maintaining a single implementation for C++.
The most Pythonic solution would be to use a custom converter that would convert a Python iterable object to a C++ collection. See this answer for implementation details. Consider this option if:
The collection's elements are cheap to copy.
The C++ functions operate only on rvalue types (i.e., std::vector<> or const std::vector<>&). This limitation prevents C++ from making changes to the Python collection or its elements.
Enhance vector_indexing_suite capabilities, reusing as many capabilities as possible, such as its proxies for safely handling index deletion and reallocation of the underlying collection:
Expose the model with a custom HeldType that functions as a smart pointer and delegate to either the instance or the element proxy objects returned from vector_indexing_suite.
Monkey patch the collection's methods that insert elements into the collection so that the custom HeldType will be set to delegate to a element proxy.
When exposing a class to Boost.Python, the HeldType is the type of object that gets embedded within a Boost.Python object. When accessing the wrapped types object, Boost.Python invokes get_pointer() for the HeldType. The object_holder class below provides the ability to return a handle to either an instance it owns or to an element proxy:
/// #brief smart pointer type that will delegate to a python
/// object if one is set.
template <typename T>
class object_holder
{
public:
typedef T element_type;
object_holder(element_type* ptr)
: ptr_(ptr),
object_()
{}
element_type* get() const
{
if (!object_.is_none())
{
return boost::python::extract<element_type*>(object_)();
}
return ptr_ ? ptr_.get() : NULL;
}
void reset(boost::python::object object)
{
// Verify the object holds the expected element.
boost::python::extract<element_type*> extractor(object_);
if (!extractor.check()) return;
object_ = object;
ptr_.reset();
}
private:
boost::shared_ptr<element_type> ptr_;
boost::python::object object_;
};
/// #brief Helper function used to extract the pointed to object from
/// an object_holder. Boost.Python will use this through ADL.
template <typename T>
T* get_pointer(const object_holder<T>& holder)
{
return holder.get();
}
With the indirection supported, the only thing remaining is patching the collection to set the object_holder. One clean and reusable way to support this is to use def_visitor. This is a generic interface that allows for class_ objects to be extended non-intrusively. For instance, the vector_indexing_suite uses this capability.
The custom_vector_indexing_suite class below monkey patches the append() method to delegate to the original method, and then invokes object_holder.reset() with a proxy to the newly set element. This results in the object_holder referring to the element contained within the collection.
/// #brief Indexing suite that will resets the element's HeldType to
/// that of the proxy during element insertion.
template <typename Container,
typename HeldType>
class custom_vector_indexing_suite
: public boost::python::def_visitor<
custom_vector_indexing_suite<Container, HeldType>>
{
private:
friend class boost::python::def_visitor_access;
template <typename ClassT>
void visit(ClassT& cls) const
{
// Define vector indexing support.
cls.def(boost::python::vector_indexing_suite<Container>());
// Monkey patch element setters with custom functions that
// delegate to the original implementation then obtain a
// handle to the proxy.
cls
.def("append", make_append_wrapper(cls.attr("append")))
// repeat for __setitem__ (slice and non-slice) and extend
;
}
/// #brief Returned a patched 'append' function.
static boost::python::object make_append_wrapper(
boost::python::object original_fn)
{
namespace python = boost::python;
return python::make_function([original_fn](
python::object self,
HeldType& value)
{
// Copy into the collection.
original_fn(self, value.get());
// Reset handle to delegate to a proxy for the newly copied element.
value.reset(self[-1]);
},
// Call policies.
python::default_call_policies(),
// Describe the signature.
boost::mpl::vector<
void, // return
python::object, // self (collection)
HeldType>() // value
);
}
};
Wrapping needs to occur at runtime and custom functor objects cannot be directly defined on the class via def(), so the make_function() function must be used. For functors, it requires both CallPolicies and a MPL front-extensible sequence representing the signature.
Here is a complete example that demonstrates using the object_holder to delegate to proxies and custom_vector_indexing_suite to patch the collection.
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
/// #brief Mockup type.
struct spam
{
int val;
spam(int val) : val(val) {}
bool operator==(const spam& rhs) { return val == rhs.val; }
};
/// #brief Mockup function that operations on a collection of spam instances.
void modify_spams(std::vector<spam>& spams)
{
for (auto& spam : spams)
spam.val *= 2;
}
/// #brief smart pointer type that will delegate to a python
/// object if one is set.
template <typename T>
class object_holder
{
public:
typedef T element_type;
object_holder(element_type* ptr)
: ptr_(ptr),
object_()
{}
element_type* get() const
{
if (!object_.is_none())
{
return boost::python::extract<element_type*>(object_)();
}
return ptr_ ? ptr_.get() : NULL;
}
void reset(boost::python::object object)
{
// Verify the object holds the expected element.
boost::python::extract<element_type*> extractor(object_);
if (!extractor.check()) return;
object_ = object;
ptr_.reset();
}
private:
boost::shared_ptr<element_type> ptr_;
boost::python::object object_;
};
/// #brief Helper function used to extract the pointed to object from
/// an object_holder. Boost.Python will use this through ADL.
template <typename T>
T* get_pointer(const object_holder<T>& holder)
{
return holder.get();
}
/// #brief Indexing suite that will resets the element's HeldType to
/// that of the proxy during element insertion.
template <typename Container,
typename HeldType>
class custom_vector_indexing_suite
: public boost::python::def_visitor<
custom_vector_indexing_suite<Container, HeldType>>
{
private:
friend class boost::python::def_visitor_access;
template <typename ClassT>
void visit(ClassT& cls) const
{
// Define vector indexing support.
cls.def(boost::python::vector_indexing_suite<Container>());
// Monkey patch element setters with custom functions that
// delegate to the original implementation then obtain a
// handle to the proxy.
cls
.def("append", make_append_wrapper(cls.attr("append")))
// repeat for __setitem__ (slice and non-slice) and extend
;
}
/// #brief Returned a patched 'append' function.
static boost::python::object make_append_wrapper(
boost::python::object original_fn)
{
namespace python = boost::python;
return python::make_function([original_fn](
python::object self,
HeldType& value)
{
// Copy into the collection.
original_fn(self, value.get());
// Reset handle to delegate to a proxy for the newly copied element.
value.reset(self[-1]);
},
// Call policies.
python::default_call_policies(),
// Describe the signature.
boost::mpl::vector<
void, // return
python::object, // self (collection)
HeldType>() // value
);
}
// .. make_setitem_wrapper
// .. make_extend_wrapper
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Expose spam. Use a custom holder to allow for transparent delegation
// to different instances.
python::class_<spam, object_holder<spam>>("Spam", python::init<int>())
.def_readwrite("val", &spam::val)
;
// Expose a vector of spam.
python::class_<std::vector<spam>>("SpamVector")
.def(custom_vector_indexing_suite<
std::vector<spam>, object_holder<spam>>())
;
python::def("modify_spams", &modify_spams);
}
Interactive usage:
>>> import example
>>> spam = example.Spam(5)
>>> spams = example.SpamVector()
>>> spams.append(spam)
>>> assert(spams[0].val == 5)
>>> spam.val = 21
>>> assert(spams[0].val == 21)
>>> example.modify_spams(spams)
>>> assert(spam.val == 42)
>>> spams.append(spam)
>>> spam.val = 100
>>> assert(spams[1].val == 100)
>>> assert(spams[0].val == 42) # The container does not provide indirection.
As the vector_indexing_suite is still being used, the underlying C++ container should only be modified using the Python object's API. For instance, invoking push_back on the container may cause a reallocation of the underlying memory and cause problems with existing Boost.Python proxies. On the other hand, one can safely modify the elements themselves, such as was done via the modify_spams() function above.
Unfortunately, the answer is no, you can't do what you want. In python, everything is a pointer, and lists are a container of pointers. The C++ vector of shared pointers work because the underlying data structure is more or less equivalent to a python list. What you are requesting is to have the C++ vector of allocated memory act like a vector of pointers, which can't be done.
Let's see what's happening in python lists, with C++ equivalent pseudocode:
foo = Foo(0.0) # Foo* foo = new Foo(0.0)
vect = [] # std::vector<Foo*> vect
vect.append(foo) # vect.push_back(foo)
At this point, foo and vect[0] both point to the same allocated memory, so changing *foo changes *vect[0].
Now with the vector<Foo> version:
foo = Foo(0.0) # Foo* foo = new Foo(0.0)
vect = FooVector() # std::vector<Foo> vect
vect.append(foo) # vect.push_back(*foo)
Here, vect[0] has it's own allocated memory, and is a copy of *foo. Fundamentally, you can't make vect[0] be the same memory as *foo.
On a side note, be careful with lifetime management of footwo when using std::vector<Foo>:
footwo = vect[0] # Foo* footwo = &vect[0]
A subsequent append may require moving the allocated storage for the vector, and may invalidate footwo (&vect[0] may change).