We Keep Coding

How do I stop pybind11 from deallocating an object constructed from Python?

So, I know that pybind lets you set a return value policy for methods that you wrap up. However, that doesn't seem to be working for me when I try to use this policy on a constructor. I have a class to wrap my C++ type that looks like this:
class PyComponent{
public:
static Component* Create(ComponentType type) {
Component* c = new Component(type);
// Irrelevant stuff removed here
return c;
}
/// #brief Wrap a behavior for Python
static void PyInitialize(py::module_& m);
};
void PyComponent::PyInitialize(py::module_ & m)
{
py::class_<Component>(m, "Component")
.def(py::init<>(&PyComponent::Create), py::return_value_policy::reference)
;
}
However, this does NOT stop my Component type from getting deallocated from the Python side if I call Component() and the created object goes out of scope. Any suggestions?

I did figure out the solution to this. It's to pass py::nodelete to the wrapper for my class
void PyComponent::PyInitialize(py::module_ & m)
{
py::class_<Component, std::unique_ptr<Component, py::nodelete>>(m, "Component")
.def(py::init<>(&PyComponent::Create), py::return_value_policy::reference)
;
}

Calling a function of an object instance using embedded Python

I want to be able to run Python scripts in my app to allow automating stuff and modifying existing objects/calling methods of existing objects.
In my application there is a BasicWindow class and MainWindow class that derives from the former. For now at application start I initialize one instance of MainWindow. This object has many functions, among them there is one that loads files (LoadFile()), and I will use it as example here.
Lets say that I want to call that particular function (but not limited to that function, it is just an example of the functionality that I want to achieve from Python) of that particular object instance.
This method is not a static one. For this I am using Boost.Python and I am creating a module this way:
BOOST_PYTHON_MODULE(MyModule)
{
MainWindow::PythonExpose(); //not really sure how to operate here
//more stuff
}
The idea is that I could call from Python something like:
MainWindow.LoadFile()
or even better, just:
LoadFile()
One solution could be to create static, application scoped functions and then just expose those functions. In C++ I could find the particular instance of MainWindow: (both methods are static)
void AppHelper::LoadFile()
{
GetMainWindow()->LoadFile();
}
void AppHelper::PythonExposeGlobal()
{
using namespace boost::python;
def("LoadFile", &AppHelper::LoadFile);
}
Is it possible to achieve this? The general question would be: is it possible to call methods of existing objects (in C++) from Python? If so, how to do it? If not, what can I do to mimic this behavior?
For example, I could easily enable scripting capabilities in my C# application and sharing instances of existing objects. (But of course C# has reflection).

If you can guarantee that the object will live as long as any scripts using it run, then there's a fairly simple approach that I use.
I'll use a primitive counter class for demonstration:
class counter
{
public:
counter() : count(0) {}
void increment() { ++count; }
int count;
};
Now, I expose this class to python, such that it considers it non-copyable, and doesn't allow construction of new instances. I also expose any members that I want to use from the scripts.
BOOST_PYTHON_MODULE(example)
{
bp::class_<counter, boost::noncopyable>("Counter", bp::no_init)
.def("increment", &counter::increment)
;
}
Next step is to create a Python object that uses an existing instance, and allow the script to use it (e.g. add it as an attribute of some module, such as the main one).
counter c;
bp::object main_module(bp::import("__main__"));
main_module.attr("c") = bp::object(bp::ptr(&c));
Now your scripts can use this instance:
c.increment()
Sample program:
#include <boost/python.hpp>
#include <iostream>
namespace bp = boost::python;
// Simple counter that can be incremented
class counter
{
public:
counter() : count(0) {}
void increment() { ++count; }
int count;
};
// Expose the counter class to Python
// We don't need constructor, since we only intend to use instance
// already existing on the C++ side
BOOST_PYTHON_MODULE(example)
{
bp::class_<counter, boost::noncopyable>("Counter", bp::no_init)
.def("increment", &counter::increment)
;
}
int main()
{
Py_InitializeEx(0);
// Bind our class
initexample();
counter c;
bp::object main_module(bp::import("__main__"));
bp::object main_namespace(main_module.attr("__dict__"));
// Add the current instance of counter to Python as attribute c of the main module
main_module.attr("c") = bp::object(bp::ptr(&c));
std::cout << "Before: " << c.count << '\n';
// Increment the counter from Python side
bp::exec("c.increment()", main_namespace);
std::cout << "After: " << c.count << '\n';
Py_Finalize();
return 0;
}
Console Output:
Before: 0
After: 1

swig: How to make a QList<T> iterable, like std::vector

I'm using SWIG to generate Python Bindings for my qt app. I have several places where I use QLists and I would like to integrate those QLists like std::vector from the SWIG Library (see http://www.swig.org/Doc1.3/Library.html#Library_nn15).
This means:
The QList objects should be iterable from python (= they must be an iterable python object)
It should be possible to pass a python list to a function which takes a qlist
... and all the other features listed in the SWIG Library for std::vector
To achieve that I use the following Code:
https://github.com/osmandapp/OsmAnd-core/blob/master/swig/java/QList.i
Later in my classes using QLists, I add code like:
%import "qlist.i"
%template(listfilter) QList<Interface_Filter*>;
class A {
public:
//.....
QList<Interface_Filter*> get_filters();
};
This works so far, but it doesn't give me the kind of integration I get with std::vector.
I'm having trouble finding out which parts of std_vector.i, std_container.i,... make an object iterable.
How do I need to extend the QList interface file to make my QList's iterable?

What you are asking for -- a qlist.i swig file that achieves the same level of integration for QList in python as std_vector.i does for std::vector -- is a non-trivial task.
I provide a very basic extended qlist.i file (and qlisttest.i to show you how to use it) and will try to explain what steps are required.
qlist.i:
%{
#include <QList>
%}
%pythoncode %{
class QListIterator:
def __init__(self, qlist):
self.index = 0
self.qlist = qlist
def __iter__(self):
return self
def next(self):
if self.index >= self.qlist.size():
raise StopIteration;
ret = self.qlist.get(self.index)
self.index += 1
return ret
__next__ = next
%}
template<class T> class QList {
public:
class iterator;
typedef size_t size_type;
typedef T value_type;
typedef const value_type& const_reference;
QList();
size_type size() const;
void reserve(size_type n);
%rename(isEmpty) empty;
bool empty() const;
void clear();
%rename(add) push_back;
void push_back(const value_type& x);
%extend {
const_reference get(int i) throw (std::out_of_range) {
int size = int(self->size());
if (i>=0 && i<size)
return (*self)[i];
else
throw std::out_of_range("QList index out of range");
}
void set(int i, const value_type& val) throw (std::out_of_range) {
int size = int(self->size());
if (i>=0 && i<size)
(*self)[i] = val;
else
throw std::out_of_range("QList index out of range");
}
int __len__() {
return self->size();
}
const_reference __getitem__(int i) throw (std::out_of_range) {
int size = int(self->size());
if (i>=0 && i<size)
return (*self)[i];
else
throw std::out_of_range("QList index out of range");
}
%pythoncode %{
def __iter__(self):
return QListIterator(self)
%}
}
};
%define %qlist_conversions(Type...)
%typemap(in) const QList< Type > & (bool free_qlist)
{
free_qlist = false;
if ((SWIG_ConvertPtr($input, (void **) &$1, $1_descriptor, 0)) == -1) {
if (!PyList_Check($input)) {
PyErr_Format(PyExc_TypeError, "QList or python list required.");
SWIG_fail;
}
Py_ssize_t len = PyList_Size($input);
QList< Type > * qlist = new QList< Type >();
free_qlist = true;
qlist->reserve(len);
for (Py_ssize_t index = 0; index < len; ++index) {
PyObject *item = PyList_GetItem($input,index);
Type* c_item;
if ((SWIG_ConvertPtr(item, (void **) &c_item, $descriptor(Type *),0)) == -1) {
delete qlist;
free_qlist = false;
PyErr_Format(PyExc_TypeError, "List element of wrong type encountered.");
SWIG_fail;
}
qlist->append(*c_item);
}
$1 = qlist;
}
}
%typemap(freearg) const QList< Type > &
{ if (free_qlist$argnum and $1) delete $1; }
%enddef
qlisttest.i:
%module qlist;
%include "qlist.i"
%inline %{
class Foo {
public:
int foo;
};
%}
%template(QList_Foo) QList<Foo>;
%qlist_conversions(Foo);
%inline %{
int sumList(const QList<Foo> & list) {
int sum = 0;
for (int i = 0; i < list.size(); ++i) {
sum += list[i].foo;
}
return sum;
}
%}
Wrapping of QList to make it and its methods accessible from python
This is achieved by making the (partial) class definition available to swig. That is what your current qlist.i does.
Note: You might need to add a "template specialization" for the case QList<T*> that typedefs const_reference as const T* since you are using a QList of pointers. Otherwise, QList<T*>::const_reference will be const T*&, which apparently might confuse swig. (see swig/Lib/std/std_vector.i)
Automatic conversion between python list and QList
This is generally achieved by using swig typemaps. For instance, if you want a function f(const QList<int>& list) to be able to accept a python list, you need to specify an input typemap that performs the conversion from a python list to a QList<int>:
%typemap(in) const QList<int> &
{
PyObject * py_list = $input;
[check if py_list is really a python list of integers]
QList<int>* qlist = new QList<int>();
[copy the data from the py_list to the qlist]
$1 = qlist;
}
%typemap(freearg) const QList<int> &
{ if ($1) delete $1; }
Here, the situation is more difficult in several ways:
You want to be able to pass a python lists or a wrapped QList: For this to work, you need to handle both cases in the typemap.
You want to convert a python list of wrapped type T to a QList<T>:
This also involves a conversion for every element of the list from the wrapped type T to the plain T. This is achieved by the swig function SWIG_ConvertPtr.
I am not sure if you can specify typemaps with template arguments. Therefore, I wrote a swig macro %qlist_conversions(Type) that you can use to attach the typemap to the QList<Type> for a specific Type.
For the other conversion direction (QList -> python list) you should first consider what you want. Consider a C++ function that returns a QList<int>. Calling this from python, should this return a wrapped QList object, or should it automatically convert the QList to a python list?
Accessing the wrapped QList as a python sequence, i.e., make len and [] work from python
For this, you need to extend the QList class in the qlist.i file using %extend { ... } and implement __len__ and __getitem__ methods.
If slicing should also work, you need to provide a __getitem__(PySliceObject *slice)__ member method and input and "typecheck" typemaps for PySliceObjects.
If you want to be able to modify values in the wrapped QList using [] from python, you need to implement __setitem__.
For a list of all the useful methods you can implement to achieve better integration, see the python documentation on "builtin types" and "abstract base classes for containers".
Note: If you use the swig -builtin feature, then you need to additionally register the above functions to the appropriate "slots" using e.g.
%feature("python:slot", "sq_length", functype="lenfunc") __len__;
Making the wrapped QList iterable from python
For this you need to extend the QList class and implement an __iter__() method that returns a python iterator object.
A python iterator object is an object that provides the methods __iter__() and __next__() (next() for older python), where __next__() returns the next value and raises the python exception StopIteration to signal the end.
As mentioned before, you can implement the iterator object in python or C++. I show an example of doing this in python.
I hope this helps as a basis for you to tweak the functionality that you require.

You provided an answer to the question "How to make a python Object iterable", but I asked for "How do I need to extend the QList interface file to make my QList's iterable?" which is more a SWIG, than a python related question.
I tested the example from http://www.swig.org/Doc1.3/Library.html#Library_nn15 with Java, C# and Python. Only Python and C# provide iterators. The generated interface of Java doesn't implement Iterable or something like that. As far as I can see your question is related to the target language.
Maybe extending MutableSequence is an option for you. The only methods you have to implement are __getitem__, __setitem__, __delitem__, __len__ and insert by delegating them to the corresponding methods of QList. Afterwards your generated class is iterable.

As described in the docs, you need the following:
QList should have a method in python __iter__() that returns an iterator object (tp_iter if you implement it in C).
The iterator object should implement __iter__() and return itself
The iterator object should implement next() that returns the next item or raises StopIteration when it's done.
It's probably easiest to do in python, but you can implement it in C as well.
Another option is to use python generators to avoid implementing an iterator type. To do this you QList needs to implement __iter__() but instead of returning an iterator you simply yield the values.
The methods mentioned only need to be visible to python. You don't have to make them available in C/Java.
See also SWIG interfacing C library to Python (Creating 'iterable' Python data type from C 'sequence' struct)

Good way to dereference boost::shared_ptr in swig interface

Is there a good, automated way to allow me to pass shared_ptr's into functions that expect references in a SWIG interface?
I have a library that provides functions like:
// Module message
typedef boost::shared_ptr< Item > ItemPtr;
class Message{
public:
ItemPtr getItem( int index) ...
...
};
And then another library that uses them:
// Module client
class Client{
public:
void processItem( const Item& item );
}
These are in separate modules.
Obviously, I want to do something like:
>>> item=message.getItem(4)
>>> client.processItem( item )
in python,
but the types do not line up, the item is a shared_ptr while the function expects a reference.
So far my solution is to override the functions in the interface file
%extend client {
void processItem( ItemPtr ptr){
$self->processItem( *ptr);
}
}
Is there an approach where where I can avoid having to duplicate all of the interface functions that expect a reference, but where I'll have shared_ptr's?

Just return an Item instance from your getItem(int index) function. Ohterwise you'll have to overload the functions in order to receive both arguments types (shared_ptr and Item references)

Boost.Python: How to expose std::unique_ptr

I am fairly new to boost.python and trying to expose the return value of a function to python.
The function signature looks like this:
std::unique_ptr<Message> someFunc(const std::string &str) const;
When calling the function in python, I get the following error:
TypeError: No to_python (by-value) converter found for C++ type: std::unique_ptr<Message, std::default_delete<Message> >
My function call in python looks like this:
a = mymodule.MyClass()
a.someFunc("some string here") # error here
I tried to expose the std::unique_ptr but just cant get it to work..
Does someone know how to properly expose the pointer class?
Thanks!
Edit:
I tried the following:
class_<std::unique_ptr<Message, std::default_delete<Message>>, bost::noncopyable ("Message", init<>())
;
This example compiles, but I still get the error mentioned above.
Also, I tried to expose the class Message itself
class_<Message>("Message", init<unsigned>())
.def(init<unsigned, unsigned>())
.def("f", &Message::f)
;

In short, Boost.Python does not support move-semantics, and therefore does not support std::unique_ptr. Boost.Python's news/change log has no indication that it has been updated for C++11 move-semantics. Additionally, this feature request for unique_ptr support has not been touched for over a year.
Nevertheless, Boost.Python supports transferring exclusive ownership of an object to and from Python via std::auto_ptr. As unique_ptr is essentially a safer version of auto_ptr, it should be fairly straight forward to adapt an API using unique_ptr to an API that uses auto_ptr:
When C++ transfers ownership to Python, the C++ function must:
be exposed with CallPolicy of boost::python::return_value_policy with a boost::python::manage_new_object result converter.
have unique_ptr release control via release() and return a raw pointer
When Python transfers ownership to C++, the C++ function must:
accept the instance via auto_ptr. The FAQ mentions that pointers returned from C++ with a manage_new_object policy will be managed via std::auto_ptr.
have auto_ptr release control to a unique_ptr via release()
Given an API/library that cannot be changed:
/// #brief Mockup Spam class.
struct Spam;
/// #brief Mockup factory for Spam.
struct SpamFactory
{
/// #brief Create Spam instances.
std::unique_ptr<Spam> make(const std::string&);
/// #brief Delete Spam instances.
void consume(std::unique_ptr<Spam>);
};
The SpamFactory::make() and SpamFactory::consume() need to be wrapped via auxiliary functions.
Functions transferring ownership from C++ to Python can be generically wrapped by a function that will create Python function objects:
/// #brief Adapter a member function that returns a unique_ptr to
/// a python function object that returns a raw pointer but
/// explicitly passes ownership to Python.
template <typename T,
typename C,
typename ...Args>
boost::python::object adapt_unique(std::unique_ptr<T> (C::*fn)(Args...))
{
return boost::python::make_function(
[fn](C& self, Args... args) { return (self.*fn)(args...).release(); },
boost::python::return_value_policy<boost::python::manage_new_object>(),
boost::mpl::vector<T*, C&, Args...>()
);
}
The lambda delegates to the original function, and releases() ownership of the instance to Python, and the call policy indicates that Python will take ownership of the value returned from the lambda. The mpl::vector describes the call signature to Boost.Python, allowing it to properly manage function dispatching between the languages.
The result of adapt_unique is exposed as SpamFactory.make():
boost::python::class_<SpamFactory>(...)
.def("make", adapt_unique(&SpamFactory::make))
// ...
;
Generically adapting SpamFactory::consume() is a more difficult, but it is easy enough to write a simple auxiliary function:
/// #brief Wrapper function for SpamFactory::consume_spam(). This
/// is required because Boost.Python will pass a handle to the
/// Spam instance as an auto_ptr that needs to be converted to
/// convert to a unique_ptr.
void SpamFactory_consume(
SpamFactory& self,
std::auto_ptr<Spam> ptr) // Note auto_ptr provided by Boost.Python.
{
return self.consume(std::unique_ptr<Spam>{ptr.release()});
}
The auxiliary function delegates to the original function, and converts the auto_ptr provided by Boost.Python to the unique_ptr required by the API. The SpamFactory_consume auxiliary function is exposed as SpamFactory.consume():
boost::python::class_<SpamFactory>(...)
// ...
.def("consume", &SpamFactory_consume)
;
Here is a complete code example:
#include <iostream>
#include <memory>
#include <boost/python.hpp>
/// #brief Mockup Spam class.
struct Spam
{
Spam(std::size_t x) : x(x) { std::cout << "Spam()" << std::endl; }
~Spam() { std::cout << "~Spam()" << std::endl; }
Spam(const Spam&) = delete;
Spam& operator=(const Spam&) = delete;
std::size_t x;
};
/// #brief Mockup factor for Spam.
struct SpamFactory
{
/// #brief Create Spam instances.
std::unique_ptr<Spam> make(const std::string& str)
{
return std::unique_ptr<Spam>{new Spam{str.size()}};
}
/// #brief Delete Spam instances.
void consume(std::unique_ptr<Spam>) {}
};
/// #brief Adapter a non-member function that returns a unique_ptr to
/// a python function object that returns a raw pointer but
/// explicitly passes ownership to Python.
template <typename T,
typename ...Args>
boost::python::object adapt_unique(std::unique_ptr<T> (*fn)(Args...))
{
return boost::python::make_function(
[fn](Args... args) { return fn(args...).release(); },
boost::python::return_value_policy<boost::python::manage_new_object>(),
boost::mpl::vector<T*, Args...>()
);
}
/// #brief Adapter a member function that returns a unique_ptr to
/// a python function object that returns a raw pointer but
/// explicitly passes ownership to Python.
template <typename T,
typename C,
typename ...Args>
boost::python::object adapt_unique(std::unique_ptr<T> (C::*fn)(Args...))
{
return boost::python::make_function(
[fn](C& self, Args... args) { return (self.*fn)(args...).release(); },
boost::python::return_value_policy<boost::python::manage_new_object>(),
boost::mpl::vector<T*, C&, Args...>()
);
}
/// #brief Wrapper function for SpamFactory::consume(). This
/// is required because Boost.Python will pass a handle to the
/// Spam instance as an auto_ptr that needs to be converted to
/// convert to a unique_ptr.
void SpamFactory_consume(
SpamFactory& self,
std::auto_ptr<Spam> ptr) // Note auto_ptr provided by Boost.Python.
{
return self.consume(std::unique_ptr<Spam>{ptr.release()});
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<Spam, boost::noncopyable>(
"Spam", python::init<std::size_t>())
.def_readwrite("x", &Spam::x)
;
python::class_<SpamFactory>("SpamFactory", python::init<>())
.def("make", adapt_unique(&SpamFactory::make))
.def("consume", &SpamFactory_consume)
;
}
Interactive Python:
>>> import example
>>> factory = example.SpamFactory()
>>> spam = factory.make("a" * 21)
Spam()
>>> spam.x
21
>>> spam.x *= 2
>>> spam.x
42
>>> factory.consume(spam)
~Spam()
>>> spam.x = 100
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
None.None(Spam, int)
did not match C++ signature:
None(Spam {lvalue}, unsigned int)

My suggestion is to get the raw pointer from the std::unique_ptr container with get(). You will have to careful to keep the unique_ptr in scope for for whole time that you wish to use the raw pointer value, otherwise the object will be deleted and you'll have a pointer to an invalid area of memory.

Boost supports movable semantics and unique_ptr since v.1.55.
But in my project I used previous version and made such simple wrapper:
class_<unique_ptr<HierarchyT>, noncopyable>(typpedName<LinksT>("hierarchy", false)
, "hierarchy holder")
.def("__call__", &unique_ptr<HierarchyT>::get,
return_internal_reference<>(),
"get holding hierarchy")
.def("reset", &unique_ptr<HierarchyT>::reset,
"reset holding hierarhy")
;
to create unique_ptr<HierarchyT> as Python shierarchy and pass it to the function that accepts it by reference.
Python code:
hier = mc.shierarchy()
mc.clusterize(hier, nds)
where C++ function is float clusterize(unique_ptr<HierarchyT>& hier,...).
Then to access results in Python make a call hier() to get the wrapped object from the unique_ptr:
output(hier(), nds)

I think nowadays there is no way to do what you are looking for... The reason is because std::unique_ptr<Message> someFunc(const std::string &str) is returning by value, which means one of two things:
The return value is going to be copied (but unique_ptr is not copyable);
The return value is going to be moved (now the problem is that boost::python doesn't provide support to move semantics). (heyy, I'm using boost 1,53, not sure in the newest versions);
Is someFunc() creating the object? In case YES, I think the solution is to create a wrapper, in case NO, you can return by reference:
std::unique_ptr<Message>& someFunc(const std::string &str)
expose the class:
class_<std::unique_ptr<Message, std::default_delete<Message>>, boost::noncopyable>("unique_ptr_message")
.def("get", &std::unique_ptr<Message>::get, return_value_policy<reference_existing_object>())
;
and also the functions:
def("someFunc", someFunc, return_value_policy<reference_existing_object>());