We Keep Coding

Down Casting and Template Solution Vs Python

this is basic C++ but I am getting to a point where python actually seems way simpler. Suppose:
class Base
{
public:
virtual ~Base() = default;
virtual std::string type() const { return "base"; };
};
class Derived1 : public Base
{
public:
virtual std::string type() const { return "dervied1"; };
};
class Derived2 : public Base
{
public:
virtual std::string type() const { return "dervied2"; };
};
I find my self having other functions of type:
void process(Base& derived_from_base)
{
};
Q1: How do I know what I am taking as input, should I call type() and later down cast, problem is I think type() will give always "base"?
Q2: This is so annoying. I later have to down case the input to the correct derived class. I am just wondering if python is doing this in the background, am I sure that with all of this I am faster than python?
Q3: Is it true I can replace virtual function/inheritance and all casting using templates? (heard this somewhere and not sure).
Thank you very much.

Q1: How do I know what I am taking as input?
A1: Keep the full type instead of erasing it to the base class. E.g. instead of
void process(Base& base) {
if (base.type() == "derived1") process_derived1(static_cast<Derived1&>(base));
else process_anything_else(base);
}
int main() {
std::unique_ptr<Base> base = std::make_unique<Derived1>();
process(*base);
}
use
void process(Derived1& derived1) { process_derived1(derived1); }
void process(auto& t) { process_anything_else(t); }
int main() {
Derived derived1;
process(derived1);
}
Q2: I am just wondering if python is doing this in the background, am I sure that with all of this I am faster than python?
A2: Python has something like this:
int main() {
Object derived1; // a hash map
derived1["__type__"] = "Derived1";
}
With the approach from A1, you are faster than anything (assuming everything else in the program isn't worse) because of static dispatch: thanks to templates, overload resolution happens at compile time and therefore costs nothing.
Q3: Is it true I can replace virtual function/inheritance and all casting using templates? (heard this somewhere and not sure)
A3: With proper design, you can do that most of the times, e.g. see A1. However, some things force linking dynamically: OS API, game plugins etc. In such cases, consider localizing the clumsy borderline part, so most of the code can be written as usual.
virtual function/inheritance
Note: inheritance without virtual functions is perfectly fine and zero-cost (on its own), e.g. see CRTP.

Garbage collection and custom getter in SWIG

I'm not the author, but there's a public software package I use that seems to be leaking memory (Github issue). I'm trying to figure out how to patch it to make it work correctly.
To narrow the problem down, there's a struct, call it xxx_t. First %extend is used to make a member of the struct available in Python:
%extend xxx_t {
char *surface;
}
Then there's a custom getter. What exactly it does here isn't important except that it uses new to create a char*.
%{
char* xxx_t_surface_get(xxx *n) {
char *s = new char [n->length + 1];
memcpy (s, n->surface, n->length);
s[n->length] = '\0';
return s;
}
%}
Currently the code has this line to handle garbage collection:
%newobject surface;
This does not seem to work as expected. %newobject xxx_t::surface; also doesn't work. If I replace it with %newobject xxx_t_surface_get; that doesn't work because the getter function is escaped (inside %{ ... %}).
What is the right way to tell SWIG about the char* so it gets freed?

Before getting start it's worth pointing out one thing: Because you return char* it ends up using SWIG's normal string typemaps to produce a Python string.
Having said that let's understand what the code that currently gets generated looks like. We can start our investigation with the following SWIG interface definition to experiment with:
%module test
%inline %{
struct foobar {
};
%}
%extend foobar {
char *surface;
}
If we run something like this through SWIG we'll see a generated function which wraps your _surface_get code, something like this:
SWIGINTERN PyObject *_wrap_foobar_surface_get(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
foobar *arg1 = (foobar *) 0 ;
void *argp1 = 0 ;
int res1 = 0 ;
PyObject * obj0 = 0 ;
char *result = 0 ;
if (!PyArg_ParseTuple(args,(char *)"O:foobar_surface_get",&obj0)) SWIG_fail;
res1 = SWIG_ConvertPtr(obj0, &argp1,SWIGTYPE_p_foobar, 0 | 0 );
if (!SWIG_IsOK(res1)) {
SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "foobar_surface_get" "', argument " "1"" of type '" "foobar *""'");
}
arg1 = reinterpret_cast< foobar * >(argp1);
result = (char *)foobar_surface_get(arg1);
resultobj = SWIG_FromCharPtr((const char *)result);
/* result is never used again from here onwards */
return resultobj;
fail:
return NULL;
}
The thing to note here however is that the result of calling your getter is lost when this wrapper returns. That is to say that it isn't even tied to the lifespan of the Python string object that gets returned.
So there are several ways we could fix this:
One option would be to ensure that the generated wrapper calls delete[] on the result of calling your getter, after the SWIG_FromCharPtr has happened. This is exactly what %newobject does in this instance. (See below).
Another alternative would be to keep the allocated buffer between calls, probably in some thread local storage and track the size to minimise allocations
Alternatively we could use some kind of RAII based object to own the temporary buffer and make sure it gets removed. (We could do something neat with operator void* if we wanted even).
If we change our interface to add %newobject like so:
%module test
%inline %{
struct foobar {
};
%}
%newobject surface;
%extend foobar {
char *surface;
}
Then we see that our generated code now looks like this:
// ....
result = (char *)foobar_surface_get(arg1);
resultobj = SWIG_FromCharPtr((const char *)result);
delete[] result;
We can see this in the real code from github too, so this isn't the bug that you're looking for.
Typically for C++ I'd lean towards the RAII option. And as it happens there's a neat way to do this from both a SWIG perspective and a C++ one: std::string. So we can fix your leak in a simple and clean way just by doing something like this:
%include <std_string.i> /* If you don't already have this... */
%extend xxx_t {
std::string surface;
}
%{
std::string xxx_t_surface_get(xxx *n) {
return std::string(n->surface, n->length);
}
%}
(You'll need to change the setter to match too though, unless you made it const so there is no setter)
The thing about this though is that it's still making two sets of allocations for the same output. Firstly the std::string object makes one allocation and then secondly an allocation occurs for the Python string object. That's all for something where the buffer already exists in C++ anyway. So whilst this change would be sufficient and correct to solve the leak you can also go further and write a version that does less duplicitous copying:
%extend xxx_t {
PyObject *surface;
}
%{
PyObject *xxx_t_surface_get(xxx *n) {
return SWIG_FromCharPtrAndSize(n->surface, n->length);
}
%}

swig: How to make a QList<T> iterable, like std::vector

I'm using SWIG to generate Python Bindings for my qt app. I have several places where I use QLists and I would like to integrate those QLists like std::vector from the SWIG Library (see http://www.swig.org/Doc1.3/Library.html#Library_nn15).
This means:
The QList objects should be iterable from python (= they must be an iterable python object)
It should be possible to pass a python list to a function which takes a qlist
... and all the other features listed in the SWIG Library for std::vector
To achieve that I use the following Code:
https://github.com/osmandapp/OsmAnd-core/blob/master/swig/java/QList.i
Later in my classes using QLists, I add code like:
%import "qlist.i"
%template(listfilter) QList<Interface_Filter*>;
class A {
public:
//.....
QList<Interface_Filter*> get_filters();
};
This works so far, but it doesn't give me the kind of integration I get with std::vector.
I'm having trouble finding out which parts of std_vector.i, std_container.i,... make an object iterable.
How do I need to extend the QList interface file to make my QList's iterable?

What you are asking for -- a qlist.i swig file that achieves the same level of integration for QList in python as std_vector.i does for std::vector -- is a non-trivial task.
I provide a very basic extended qlist.i file (and qlisttest.i to show you how to use it) and will try to explain what steps are required.
qlist.i:
%{
#include <QList>
%}
%pythoncode %{
class QListIterator:
def __init__(self, qlist):
self.index = 0
self.qlist = qlist
def __iter__(self):
return self
def next(self):
if self.index >= self.qlist.size():
raise StopIteration;
ret = self.qlist.get(self.index)
self.index += 1
return ret
__next__ = next
%}
template<class T> class QList {
public:
class iterator;
typedef size_t size_type;
typedef T value_type;
typedef const value_type& const_reference;
QList();
size_type size() const;
void reserve(size_type n);
%rename(isEmpty) empty;
bool empty() const;
void clear();
%rename(add) push_back;
void push_back(const value_type& x);
%extend {
const_reference get(int i) throw (std::out_of_range) {
int size = int(self->size());
if (i>=0 && i<size)
return (*self)[i];
else
throw std::out_of_range("QList index out of range");
}
void set(int i, const value_type& val) throw (std::out_of_range) {
int size = int(self->size());
if (i>=0 && i<size)
(*self)[i] = val;
else
throw std::out_of_range("QList index out of range");
}
int __len__() {
return self->size();
}
const_reference __getitem__(int i) throw (std::out_of_range) {
int size = int(self->size());
if (i>=0 && i<size)
return (*self)[i];
else
throw std::out_of_range("QList index out of range");
}
%pythoncode %{
def __iter__(self):
return QListIterator(self)
%}
}
};
%define %qlist_conversions(Type...)
%typemap(in) const QList< Type > & (bool free_qlist)
{
free_qlist = false;
if ((SWIG_ConvertPtr($input, (void **) &$1, $1_descriptor, 0)) == -1) {
if (!PyList_Check($input)) {
PyErr_Format(PyExc_TypeError, "QList or python list required.");
SWIG_fail;
}
Py_ssize_t len = PyList_Size($input);
QList< Type > * qlist = new QList< Type >();
free_qlist = true;
qlist->reserve(len);
for (Py_ssize_t index = 0; index < len; ++index) {
PyObject *item = PyList_GetItem($input,index);
Type* c_item;
if ((SWIG_ConvertPtr(item, (void **) &c_item, $descriptor(Type *),0)) == -1) {
delete qlist;
free_qlist = false;
PyErr_Format(PyExc_TypeError, "List element of wrong type encountered.");
SWIG_fail;
}
qlist->append(*c_item);
}
$1 = qlist;
}
}
%typemap(freearg) const QList< Type > &
{ if (free_qlist$argnum and $1) delete $1; }
%enddef
qlisttest.i:
%module qlist;
%include "qlist.i"
%inline %{
class Foo {
public:
int foo;
};
%}
%template(QList_Foo) QList<Foo>;
%qlist_conversions(Foo);
%inline %{
int sumList(const QList<Foo> & list) {
int sum = 0;
for (int i = 0; i < list.size(); ++i) {
sum += list[i].foo;
}
return sum;
}
%}
Wrapping of QList to make it and its methods accessible from python
This is achieved by making the (partial) class definition available to swig. That is what your current qlist.i does.
Note: You might need to add a "template specialization" for the case QList<T*> that typedefs const_reference as const T* since you are using a QList of pointers. Otherwise, QList<T*>::const_reference will be const T*&, which apparently might confuse swig. (see swig/Lib/std/std_vector.i)
Automatic conversion between python list and QList
This is generally achieved by using swig typemaps. For instance, if you want a function f(const QList<int>& list) to be able to accept a python list, you need to specify an input typemap that performs the conversion from a python list to a QList<int>:
%typemap(in) const QList<int> &
{
PyObject * py_list = $input;
[check if py_list is really a python list of integers]
QList<int>* qlist = new QList<int>();
[copy the data from the py_list to the qlist]
$1 = qlist;
}
%typemap(freearg) const QList<int> &
{ if ($1) delete $1; }
Here, the situation is more difficult in several ways:
You want to be able to pass a python lists or a wrapped QList: For this to work, you need to handle both cases in the typemap.
You want to convert a python list of wrapped type T to a QList<T>:
This also involves a conversion for every element of the list from the wrapped type T to the plain T. This is achieved by the swig function SWIG_ConvertPtr.
I am not sure if you can specify typemaps with template arguments. Therefore, I wrote a swig macro %qlist_conversions(Type) that you can use to attach the typemap to the QList<Type> for a specific Type.
For the other conversion direction (QList -> python list) you should first consider what you want. Consider a C++ function that returns a QList<int>. Calling this from python, should this return a wrapped QList object, or should it automatically convert the QList to a python list?
Accessing the wrapped QList as a python sequence, i.e., make len and [] work from python
For this, you need to extend the QList class in the qlist.i file using %extend { ... } and implement __len__ and __getitem__ methods.
If slicing should also work, you need to provide a __getitem__(PySliceObject *slice)__ member method and input and "typecheck" typemaps for PySliceObjects.
If you want to be able to modify values in the wrapped QList using [] from python, you need to implement __setitem__.
For a list of all the useful methods you can implement to achieve better integration, see the python documentation on "builtin types" and "abstract base classes for containers".
Note: If you use the swig -builtin feature, then you need to additionally register the above functions to the appropriate "slots" using e.g.
%feature("python:slot", "sq_length", functype="lenfunc") __len__;
Making the wrapped QList iterable from python
For this you need to extend the QList class and implement an __iter__() method that returns a python iterator object.
A python iterator object is an object that provides the methods __iter__() and __next__() (next() for older python), where __next__() returns the next value and raises the python exception StopIteration to signal the end.
As mentioned before, you can implement the iterator object in python or C++. I show an example of doing this in python.
I hope this helps as a basis for you to tweak the functionality that you require.

You provided an answer to the question "How to make a python Object iterable", but I asked for "How do I need to extend the QList interface file to make my QList's iterable?" which is more a SWIG, than a python related question.
I tested the example from http://www.swig.org/Doc1.3/Library.html#Library_nn15 with Java, C# and Python. Only Python and C# provide iterators. The generated interface of Java doesn't implement Iterable or something like that. As far as I can see your question is related to the target language.
Maybe extending MutableSequence is an option for you. The only methods you have to implement are __getitem__, __setitem__, __delitem__, __len__ and insert by delegating them to the corresponding methods of QList. Afterwards your generated class is iterable.

As described in the docs, you need the following:
QList should have a method in python __iter__() that returns an iterator object (tp_iter if you implement it in C).
The iterator object should implement __iter__() and return itself
The iterator object should implement next() that returns the next item or raises StopIteration when it's done.
It's probably easiest to do in python, but you can implement it in C as well.
Another option is to use python generators to avoid implementing an iterator type. To do this you QList needs to implement __iter__() but instead of returning an iterator you simply yield the values.
The methods mentioned only need to be visible to python. You don't have to make them available in C/Java.
See also SWIG interfacing C library to Python (Creating 'iterable' Python data type from C 'sequence' struct)

Wrapping an std::vector using boost::python vector_indexing_suite

I am working on a C++ library with Python bindings (using boost::python) representing data stored in a file. Majority of my semi-technical users will be using Python to interact with it, so I need to make it as Pythonic as possible. However, I will also have C++ programmers using the API, so I do not want to compromise on the C++ side to accommodate Python bindings.
A large part of the library will be made out of containers. To make things intuitive for the python users, I would like them to behave like python lists, i.e.:
# an example compound class
class Foo:
def __init__( self, _val ):
self.val = _val
# add it to a list
foo = Foo(0.0)
vect = []
vect.append(foo)
# change the value of the *original* instance
foo.val = 666.0
# which also changes the instance inside the container
print vect[0].val # outputs 666.0
The test setup
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
#include <boost/python/register_ptr_to_python.hpp>
#include <boost/shared_ptr.hpp>
struct Foo {
double val;
Foo(double a) : val(a) {}
bool operator == (const Foo& f) const { return val == f.val; }
};
/* insert the test module wrapping code here */
int main() {
Py_Initialize();
inittest();
boost::python::object globals = boost::python::import("__main__").attr("__dict__");
boost::python::exec(
"import test\n"
"foo = test.Foo(0.0)\n" // make a new Foo instance
"vect = test.FooVector()\n" // make a new vector of Foos
"vect.append(foo)\n" // add the instance to the vector
"foo.val = 666.0\n" // assign a new value to the instance
// which should change the value in vector
"print 'Foo =', foo.val\n" // and print the results
"print 'vector[0] =', vect[0].val\n",
globals, globals
);
return 0;
}
The way of the shared_ptr
Using the shared_ptr, I can get the same behaviour as above, but it also means that I have to represent all data in C++ using shared pointers, which is not nice from many points of view.
BOOST_PYTHON_MODULE( test ) {
// wrap Foo
boost::python::class_< Foo, boost::shared_ptr<Foo> >("Foo", boost::python::init<double>())
.def_readwrite("val", &Foo::val);
// wrap vector of shared_ptr Foos
boost::python::class_< std::vector < boost::shared_ptr<Foo> > >("FooVector")
.def(boost::python::vector_indexing_suite<std::vector< boost::shared_ptr<Foo> >, true >());
}
In my test setup, this produces the same output as pure Python:
Foo = 666.0
vector[0] = 666.0
The way of the vector<Foo>
Using a vector directly gives a nice clean setup on the C++ side. However, the result does not behave in the same way as pure Python.
BOOST_PYTHON_MODULE( test ) {
// wrap Foo
boost::python::class_< Foo >("Foo", boost::python::init<double>())
.def_readwrite("val", &Foo::val);
// wrap vector of Foos
boost::python::class_< std::vector < Foo > >("FooVector")
.def(boost::python::vector_indexing_suite<std::vector< Foo > >());
}
This produces:
Foo = 666.0
vector[0] = 0.0
Which is "wrong" - changing the original instance did not change the value inside the container.
I hope I don't want too much
Interestingly enough, this code works no matter which of the two encapsulations I use:
footwo = vect[0]
footwo.val = 555.0
print vect[0].val
Which means that boost::python is able to deal with "fake shared ownership" (via its by_proxy return mechanism). Is there any way to achieve the same while inserting new elements?
However, if the answer is no, I'd love to hear other suggestions - is there an example in the Python toolkit where a similar collection encapsulation is implemented, but which does not behave as a python list?
Thanks a lot for reading this far :)

Due to the semantic differences between the languages, it is often very difficult to apply a single reusable solution to all scenarios when collections are involved. The largest issue is that the while Python collections directly support references, C++ collections require a level of indirection, such as by having shared_ptr element types. Without this indirection, C++ collections will not be able to support the same functionality as Python collections. For instance, consider two indexes that refer to the same object:
s = Spam()
spams = []
spams.append(s)
spams.append(s)
Without pointer-like element types, a C++ collection could not have two indexes referring to the same object. Nevertheless, depending on usage and needs, there may be options that allow for a Pythonic-ish interface for the Python users while still maintaining a single implementation for C++.
The most Pythonic solution would be to use a custom converter that would convert a Python iterable object to a C++ collection. See this answer for implementation details. Consider this option if:
The collection's elements are cheap to copy.
The C++ functions operate only on rvalue types (i.e., std::vector<> or const std::vector<>&). This limitation prevents C++ from making changes to the Python collection or its elements.
Enhance vector_indexing_suite capabilities, reusing as many capabilities as possible, such as its proxies for safely handling index deletion and reallocation of the underlying collection:
Expose the model with a custom HeldType that functions as a smart pointer and delegate to either the instance or the element proxy objects returned from vector_indexing_suite.
Monkey patch the collection's methods that insert elements into the collection so that the custom HeldType will be set to delegate to a element proxy.
When exposing a class to Boost.Python, the HeldType is the type of object that gets embedded within a Boost.Python object. When accessing the wrapped types object, Boost.Python invokes get_pointer() for the HeldType. The object_holder class below provides the ability to return a handle to either an instance it owns or to an element proxy:
/// #brief smart pointer type that will delegate to a python
/// object if one is set.
template <typename T>
class object_holder
{
public:
typedef T element_type;
object_holder(element_type* ptr)
: ptr_(ptr),
object_()
{}
element_type* get() const
{
if (!object_.is_none())
{
return boost::python::extract<element_type*>(object_)();
}
return ptr_ ? ptr_.get() : NULL;
}
void reset(boost::python::object object)
{
// Verify the object holds the expected element.
boost::python::extract<element_type*> extractor(object_);
if (!extractor.check()) return;
object_ = object;
ptr_.reset();
}
private:
boost::shared_ptr<element_type> ptr_;
boost::python::object object_;
};
/// #brief Helper function used to extract the pointed to object from
/// an object_holder. Boost.Python will use this through ADL.
template <typename T>
T* get_pointer(const object_holder<T>& holder)
{
return holder.get();
}
With the indirection supported, the only thing remaining is patching the collection to set the object_holder. One clean and reusable way to support this is to use def_visitor. This is a generic interface that allows for class_ objects to be extended non-intrusively. For instance, the vector_indexing_suite uses this capability.
The custom_vector_indexing_suite class below monkey patches the append() method to delegate to the original method, and then invokes object_holder.reset() with a proxy to the newly set element. This results in the object_holder referring to the element contained within the collection.
/// #brief Indexing suite that will resets the element's HeldType to
/// that of the proxy during element insertion.
template <typename Container,
typename HeldType>
class custom_vector_indexing_suite
: public boost::python::def_visitor<
custom_vector_indexing_suite<Container, HeldType>>
{
private:
friend class boost::python::def_visitor_access;
template <typename ClassT>
void visit(ClassT& cls) const
{
// Define vector indexing support.
cls.def(boost::python::vector_indexing_suite<Container>());
// Monkey patch element setters with custom functions that
// delegate to the original implementation then obtain a
// handle to the proxy.
cls
.def("append", make_append_wrapper(cls.attr("append")))
// repeat for __setitem__ (slice and non-slice) and extend
;
}
/// #brief Returned a patched 'append' function.
static boost::python::object make_append_wrapper(
boost::python::object original_fn)
{
namespace python = boost::python;
return python::make_function([original_fn](
python::object self,
HeldType& value)
{
// Copy into the collection.
original_fn(self, value.get());
// Reset handle to delegate to a proxy for the newly copied element.
value.reset(self[-1]);
},
// Call policies.
python::default_call_policies(),
// Describe the signature.
boost::mpl::vector<
void, // return
python::object, // self (collection)
HeldType>() // value
);
}
};
Wrapping needs to occur at runtime and custom functor objects cannot be directly defined on the class via def(), so the make_function() function must be used. For functors, it requires both CallPolicies and a MPL front-extensible sequence representing the signature.
Here is a complete example that demonstrates using the object_holder to delegate to proxies and custom_vector_indexing_suite to patch the collection.
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
/// #brief Mockup type.
struct spam
{
int val;
spam(int val) : val(val) {}
bool operator==(const spam& rhs) { return val == rhs.val; }
};
/// #brief Mockup function that operations on a collection of spam instances.
void modify_spams(std::vector<spam>& spams)
{
for (auto& spam : spams)
spam.val *= 2;
}
/// #brief smart pointer type that will delegate to a python
/// object if one is set.
template <typename T>
class object_holder
{
public:
typedef T element_type;
object_holder(element_type* ptr)
: ptr_(ptr),
object_()
{}
element_type* get() const
{
if (!object_.is_none())
{
return boost::python::extract<element_type*>(object_)();
}
return ptr_ ? ptr_.get() : NULL;
}
void reset(boost::python::object object)
{
// Verify the object holds the expected element.
boost::python::extract<element_type*> extractor(object_);
if (!extractor.check()) return;
object_ = object;
ptr_.reset();
}
private:
boost::shared_ptr<element_type> ptr_;
boost::python::object object_;
};
/// #brief Helper function used to extract the pointed to object from
/// an object_holder. Boost.Python will use this through ADL.
template <typename T>
T* get_pointer(const object_holder<T>& holder)
{
return holder.get();
}
/// #brief Indexing suite that will resets the element's HeldType to
/// that of the proxy during element insertion.
template <typename Container,
typename HeldType>
class custom_vector_indexing_suite
: public boost::python::def_visitor<
custom_vector_indexing_suite<Container, HeldType>>
{
private:
friend class boost::python::def_visitor_access;
template <typename ClassT>
void visit(ClassT& cls) const
{
// Define vector indexing support.
cls.def(boost::python::vector_indexing_suite<Container>());
// Monkey patch element setters with custom functions that
// delegate to the original implementation then obtain a
// handle to the proxy.
cls
.def("append", make_append_wrapper(cls.attr("append")))
// repeat for __setitem__ (slice and non-slice) and extend
;
}
/// #brief Returned a patched 'append' function.
static boost::python::object make_append_wrapper(
boost::python::object original_fn)
{
namespace python = boost::python;
return python::make_function([original_fn](
python::object self,
HeldType& value)
{
// Copy into the collection.
original_fn(self, value.get());
// Reset handle to delegate to a proxy for the newly copied element.
value.reset(self[-1]);
},
// Call policies.
python::default_call_policies(),
// Describe the signature.
boost::mpl::vector<
void, // return
python::object, // self (collection)
HeldType>() // value
);
}
// .. make_setitem_wrapper
// .. make_extend_wrapper
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Expose spam. Use a custom holder to allow for transparent delegation
// to different instances.
python::class_<spam, object_holder<spam>>("Spam", python::init<int>())
.def_readwrite("val", &spam::val)
;
// Expose a vector of spam.
python::class_<std::vector<spam>>("SpamVector")
.def(custom_vector_indexing_suite<
std::vector<spam>, object_holder<spam>>())
;
python::def("modify_spams", &modify_spams);
}
Interactive usage:
>>> import example
>>> spam = example.Spam(5)
>>> spams = example.SpamVector()
>>> spams.append(spam)
>>> assert(spams[0].val == 5)
>>> spam.val = 21
>>> assert(spams[0].val == 21)
>>> example.modify_spams(spams)
>>> assert(spam.val == 42)
>>> spams.append(spam)
>>> spam.val = 100
>>> assert(spams[1].val == 100)
>>> assert(spams[0].val == 42) # The container does not provide indirection.
As the vector_indexing_suite is still being used, the underlying C++ container should only be modified using the Python object's API. For instance, invoking push_back on the container may cause a reallocation of the underlying memory and cause problems with existing Boost.Python proxies. On the other hand, one can safely modify the elements themselves, such as was done via the modify_spams() function above.

Unfortunately, the answer is no, you can't do what you want. In python, everything is a pointer, and lists are a container of pointers. The C++ vector of shared pointers work because the underlying data structure is more or less equivalent to a python list. What you are requesting is to have the C++ vector of allocated memory act like a vector of pointers, which can't be done.
Let's see what's happening in python lists, with C++ equivalent pseudocode:
foo = Foo(0.0) # Foo* foo = new Foo(0.0)
vect = [] # std::vector<Foo*> vect
vect.append(foo) # vect.push_back(foo)
At this point, foo and vect[0] both point to the same allocated memory, so changing *foo changes *vect[0].
Now with the vector<Foo> version:
foo = Foo(0.0) # Foo* foo = new Foo(0.0)
vect = FooVector() # std::vector<Foo> vect
vect.append(foo) # vect.push_back(*foo)
Here, vect[0] has it's own allocated memory, and is a copy of *foo. Fundamentally, you can't make vect[0] be the same memory as *foo.
On a side note, be careful with lifetime management of footwo when using std::vector<Foo>:
footwo = vect[0] # Foo* footwo = &vect[0]
A subsequent append may require moving the allocated storage for the vector, and may invalidate footwo (&vect[0] may change).

Boost.Python - How to return by reference?

I'm using Boost.Python to create Python modules from C++ classes. And I ran into a problem with references.
Condider the following case where I have a class Foo with overloaded get methods that can either return by value or reference.
Specifying that the return by value should be used was easy once I typedefed a signature. But I think it should be possible return a reference as well by using a return_value_policy. However, using what seemed appropriate (doc); return_value_policy<reference_existing_object> did not seem to work.
Have I misunderstood what it does?
struct Foo {
Foo(float x) { _x = x; }
float& get() { return _x; }
float get() const { return _x; }
private:
float _x;
};
// Wrapper code
BOOST_PYTHON_MODULE(my_module)
{
using namespace boost::python;
typedef float (Foo::*get_by_value)() const;
typedef float& (Foo::*get_by_ref)();
class_<Foo>("Foo", init<float>())
.def("get", get_by_value(&Foo::get))
.def("get_ref", get_by_ref(&Foo::get),
return_value_policy<reference_existing_object>())//Doesn't work
;
}
Note: I know it could be dangerous to reference existing object without life-time managing.
Update:
It looks like it works for objects but not basic data types.
Take this revised example:
struct Foo {
Foo(float x) { _x = x; }
float& get() { return _x; }
float get() const { return _x; }
void set( float f ){ _x = f;}
Foo& self(){return *this;}
private:
float _x;
};
// Wrapper code
using namespace boost::python;
BOOST_PYTHON_MODULE(my_module)
{
typedef float (Foo::*get_by_value)() const;
class_<Foo>("Foo", init<float>())
.def("get", get_by_value(&Foo::get))
.def("get_self", &Foo::self,
return_value_policy<reference_existing_object>())
.def("set", &Foo::set);
;
}
Which in a test gave the expected result:
>>> foo1 = Foo(123)
>>> foo1.get()
123.0
>>> foo2 = foo1.get_self()
>>> foo2.set(1)
>>> foo1.get()
1.0
>>> id(foo1) == id(foo2)
False

In Python, there's the concept of immutable types. An immutable type can't have its value changed. Examples of built-in immutable types are int, float and str.
Having said that, you can't do what you want with boost::python, because Python itself does not allow you to change the value of the float returned by the function.
Your second example shows one solution, another would be to create thin-wrappers and exposing that:
void Foo_set_x(Foo& self, float value) {
self.get() = value;
}
class_<Foo>("Foo", init<float>())
...
.def("set", &Foo_set_x);
;
Which is a better solution than having to change the original C++ class.

I think you want return internal reference instead. I have used it before to do something similar.
Edit: Latest doc

I don't know much about Boost.Python, so I may misunderstand the question, in which case this is completely unhelpful. But here goes:
In Python you can't choose between returning by reference or by value, the distinction doesn't make sense in Python. I find it's easiest to think of it as everything being handled by reference.
You just have objects, and you have names for those objects. So
foo = "ryiuy"
Creates the string object "ryiuy" and then lets you refer to that string object with the name "foo". So in Python, when you get passed something, you get passed that object. There is no "values" as such, so you can't pass the value. But then again, it's also a valid viewpoint that there aren't references either, just objects and their names.
So the answer is, I guess, is that when you get a reference in C, you need to pass a reference to the object that reference references into Python. And when you get a value in C, you need to pass a reference to the object that you create from that value into Python.

Are you sure that the c++ object is being copied? You will get a new python object each time but which references the same c++ object. How are you determining that the object has been copied?