We Keep Coding

Increase ref_count of pybind11 handle in C++

I have a virtual object that I subclass and implement in python using pybind11, and that object then is used in C++ code but the python object goes out of scope when the python script finished and python garbage collector destroys my object before I can use it in C++ side.
To avoid the destruction I have added a "keep" method to the binding code that increases the ref_count, so that python garbage collector doesn't destroy it.
C++ code for the bindings
VNCS::Real VNCS::py::BlendingField::blending(const VNCS::Point_3 &p) const
{
std::array<double, 3> pyP{p[0], p[1], p[2]};
PYBIND11_OVERLOAD_PURE(VNCS::Real, /* Return type */
VNCS::BlendingField, /* Parent class */
blending, /* Name of function in C++ (must match Python name) */
pyP /* Argument(s) */
);
}
void VNCS::py::module::blendingField(pybind11::module &m)
{
pybind11::class_<VNCS::BlendingField, VNCS::py::BlendingField, std::shared_ptr<VNCS::BlendingField>>(
m, "BlendingField")
.def(pybind11::init(
[](pybind11::args &args, pybind11::kwargs &kwargs) { return std::make_shared<VNCS::py::BlendingField>(); }))
.def("keep", [](pybind11::handle handle) { handle.inc_ref(); });
}
Python code
class BlendingField(PyVNCS.BlendingField):
def __init__(self, *args, **kwargs):
PyVNCS.BlendingField.__init__(self, *args, **kwargs)
def __del__(self):
print("dtor from python")
def blending(self, point):
return point[0]
blendingField = BlendingField()
blendingField.keep()
simCreator = initObject.addObject("SimCreator",
name="simCreator",
coarseMesh="#lowResTopology",
detailMesh="#highResTopology")
simCreator.setBlendingField(blendingField)
This doesn't look very nice, as the keep method there feels like a terrible hack. What would be the correct way to implement this?

I don't think messing with the ref count is the best solution. If you want to use an object in C++-land, use C++ lifetime management tools.
Pybind integrates lifetime management into the language bindings, you just need to change it's behavior. To quote the docs:
The binding generator for classes, class_, can be passed a template type that denotes a special holder type that is used to manage references to the object. If no such holder type template argument is given, the default for a type named Type is std::unique_ptr<Type>, which means that the object is deallocated when Python’s reference count goes to zero.
It is possible to switch to other types of reference counting wrappers or smart pointers, which is useful in codebases that rely on them. For instance, the following snippet causes std::shared_ptr to be used instead.
Here's an example lifted from the tests:
// Object managed by a std::shared_ptr<>
class MyObject2 {
public:
MyObject2(const MyObject2 &) = default;
MyObject2(int value) : value(value) { print_created(this, toString()); }
std::string toString() const { return "MyObject2[" + std::to_string(value) + "]"; }
virtual ~MyObject2() { print_destroyed(this); }
private:
int value;
};
py::class_<MyObject2, std::shared_ptr<MyObject2>>(m, "MyObject2")
.def(py::init<int>());
m.def("make_myobject2_1", []() { return new MyObject2(6); });
m.def("make_myobject2_2", []() { return std::make_shared<MyObject2>(7); });
m.def("print_myobject2_1", [](const MyObject2 *obj) { py::print(obj->toString()); });
m.def("print_myobject2_2", [](std::shared_ptr<MyObject2> obj) { py::print(obj->toString()); });
m.def("print_myobject2_3", [](const std::shared_ptr<MyObject2> &obj) { py::print(obj->toString()); });
m.def("print_myobject2_4", [](const std::shared_ptr<MyObject2> *obj) { py::print((*obj)->toString()); });
If you do go that route, which I would recommend, this question would be a duplicate of this one.
Note that there are some minor pitfalls here especially if youre codebase uses raw pointers around python code. The docs point to a possible double-free if a C++ method returning a raw pointer to an shared object is called from python. Again stealing from the docs:
class Child { };
class Parent {
public:
Parent() : child(std::make_shared<Child>()) { }
Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */
private:
std::shared_ptr<Child> child;
};
PYBIND11_MODULE(example, m) {
py::class_<Child, std::shared_ptr<Child>>(m, "Child");
py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent")
.def(py::init<>())
.def("get_child", &Parent::get_child);
}
from example import Parent
print(Parent().get_child()) // UB or Segfault
Though these issue don't seem to me to be unique to shared_ptr over unique_ptr.

How to return a list of dicts from C++ to python using boost.python without leaking memory?

I have some code in C++ and I want to call it from Python. The C++ code returns lists of dicts to the python code. I use boost.python to achieve this, in the following manner:
struct MyStruct {
int foo;
};
class MyClass
{
public:
std::vector<MyStruct> get_stuff();
};
struct MyStructConverter
{
static PyObject* convert(const std::vector<MyStruct>& v)
{
boost::python::list ret;
for (auto c : v) {
boost::python::dict *r = new boost::python::dict();
(*r)["foo"] = c.foo;
ret.append(boost::python::object(*r));
}
return boost::python::incref(ret.ptr());
}
};
/** Interface between C++ and Python
*/
BOOST_PYTHON_MODULE(glue)
{
boost::python::to_python_converter<std::vector<MyStruct>, MyStructConverter>();
class_<MyClass, boost::noncopyable>("MyClass")
.def("get_stuff", &MyClass::get_stuff);
}
This works to the extent that I can pass data from C++ to python. However, when used as follows:
# test.py
x = MyClass()
while True:
u = x.get_stuff()
# do something with u
it leaks memory (i.e. the VSZ of the python process keeps growing until the whole system grinds to a halt).
What am I doing wrong? How can I avoid this?

Memory Core Dump C++

Now, I have called python to C++. Using ctype to connect between both of them. And I have a problem about core dump when in running time.
I have a library which is called "libfst.so"
This is my code.
NGramFST.h
#include <iostream>
class NGramFST{
private:
static NGramFST* m_Instace;
public:
NGramFST(){
}
static NGramFST* getInstance() {
if (m_Instace == NULL){
m_Instace = new NGramFST();
}
return m_Instace;
}
double getProbabilityOfWord(std::string word, std::string context) {
std::cout << "reloading..." << std::endl;
return 1;
}
};
NGramFST.cpp
#include "NGramFST.h"
NGramFST* NGramFST::m_Instace = NULL;
extern "C" {
double FST_getProbability(std::string word, std::string context){
return NGramFST::getInstance()->getProbabilityOfWord(word, context);
}
}
And this is my python code.
from ctypes import cdll
lib = cdll.LoadLibrary('./libfst.so')
#-------------------------main code------------------------
class FST(object):
def __init__(self):
print 'Initializing'
def getProbabilityOfWord(self, word, context):
lib.FST_getProbability(word, context)
fst = FST()
print fst.getProbabilityOfWord(c_wchar_p('jack london'), c_wchar_p('my name is'))
This is error
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)
I reviewed again but I can not detect where is my problem.

ctypes does not understand C++ types (it's not called c++types). It cannot handle std::string. It wouldn't know that your function expects std::string arguments anyway.
In order to work with ctypes, your library needs a C-compatible interface. extern "C" is necessary but not sufficient. The functions need to be actually callable from C.
Better yet, use a modern C++/Python binding library such as pybind11.

It work when I change python code below
string1 = "my string 1"
string2 = "my string 2"
# create byte objects from the strings
b_string1 = string1.encode('utf-8')
b_string2 = string2.encode('utf-8')
print fst.getProbabilityOfWord(b_string1, b_string2)
and c++ code change type of param bellow
FST_getProbability(const char* word, const char* context)

Converting Strings in Linux using SWIG for Python

I have a C++ class that is able to output strings in normal ASCII or wide format. I want to get the output in Python as a string. I am using SWIG (version 3.0.4) and have read the SWIG documentation. I'm using the following typemap to convert from a standard c string to my C++ class:
%typemap(out) myNamespace::MyString &
{
$result = PyString_FromString(const char *v);
}
This works fine in Windows with the VS2010 compiler, but it is not working completely in Linux. When I compile the wrap file under Linux, I get the following error:
error: cannot convert ‘std::string*’ to ‘myNamespace::MyString*’ in assignment
So I tried adding an extra typemap to the Linux interface file as so:
%typemap(in) myNamespace::MyString*
{
$result = PyString_FromString(std::string*);
}
But I still get the same error. If I manually go into the wrap code and fix the assignment like so:
arg2 = (myNamespace::MyString*) ptr;
then the code compiles just fine. I don't see why my additional typemap isn't working. Any ideas or solutions would be greatly appreciated. Thanks in advance.

It doesn't look like your typemap is using the arguments quite correctly. You should have something like this instead:
%typemap(out) myNamespace::MyString &
{
$result = PyString_FromString($1);
}
Where the '$1' is the first argument. See the SWIG special variables for more information [http://www.swig.org/Doc3.0/Typemaps.html#Typemaps_special_variables]
EDIT:
To handle the input typemap, you will need something like this:
%typemap(in) myNamespace::MyString*
{
const char* pChars = "";
if(PyString_Check($input))
{
pChars = PyString_AsString($input);
}
$1 = new myNamespace::MyString(pChars);
}
You can do more error checking and handle Unicode with the following code:
%typemap(in) myNamespace::MyString*
{
const char* pChars = "";
PyObject* pyobj = $input;
if(PyString_Check(pyobj))
{
pChars = PyString_AsString(pyobj);
$1 = new myNamespace::MyString(pChars);
}
else if(PyUnicode_Check(pyobj))
{
PyObject* tmp = PyUnicode_AsUTF8String(pyobj);
pChars = PyString_AsString(tmp);
$1 = new myNamespace::MyString(pChars);
}
else
{
std::string strTemp;
int rrr = SWIG_ConvertPtr(pyobj, (void **) &strTemp, $descriptor(String), 0);
if(!SWIG_IsOK(rrr))
SWIG_exception_fail(SWIG_ArgError(rrr), "Expected a String "
"in method '$symname', argument $argnum of type '$type'");
$1 = new myNamespace::MyString(strTemp);
}
}

Create a property with a call policy - boost::python

I have the following C++ classes that I would like expose to python.
class PlainOldData
{
...
};
class FancyClass
{
public:
const PlainOldData& GetMyPOD() {return myPOD;}
private:
PlainOldData myPOD;
};
Because I want my python classes to be pythonic, I would like expose myPOD as a property. However, when I try to do the following:
// expose with boost::python
BOOST_PYTHON_MODULE(mymod)
{
class_<PlainOldData>("PlainOldData", init<>());
// this fails
class_<FancyClass>("FancyClass", init<>())
.add_property("myPOD", &FancyClass::GetMyPOD);
}
I get the following error: error C2027: use of undefined type 'boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<T>'
But, if I try to specify a call policy, such as:
class_<FancyClass>("FancyClass", init<>())
.add_property("myPOD", &FancyClass::GetMyPOD, return_value_policy<copy_const_reference>());
I get an incredibly long error message.
Is it possible to expose this function as a property; am I doing something wrong?

Similar to how Python's property() is passed Python callable objects, the boost::python::class_::add_property() function can accept Python callable objects that can be created with CallPolicies, such as those returned from boost::python::make_function().
For example, the property in the original code could be exposed like:
class_<FancyClass>("FancyClass", init<>())
.add_property("myPOD", make_function(&FancyClass::GetMyPOD,
return_value_policy<copy_const_reference>()));
Here is a complete minimal example:
#include <boost/python.hpp>
class egg {};
class spam
{
public:
const egg& get_egg() { return egg_; }
private:
egg egg_;
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<egg>("Egg");
python::class_<spam>("Spam")
.add_property("egg", python::make_function(&spam::get_egg,
python::return_value_policy<python::copy_const_reference>()))
;
}
Interactive usage:
>>> import example
>>> spam = example.Spam()
>>> assert(spam.egg is not spam.egg) # unique identities as spam.egg
# returns a copy
>>> egg1 = spam.egg
>>> assert(egg1 is not spam.egg)
>>> egg2 = spam.egg
>>> assert(egg1 is not egg2)