We Keep Coding

Boost Python issue with converters - static linking

I have a question regarding the below code.
It's an example how to pass a custom class via shared_ptr to embedded python code and it works when boost is dynamically linked.
Unfortunately the same code with statically linked boost doesn't work with the following error message:
"No to_python (by-value) converter found for C++ type: class boost::shared_ptr".
I don't understand why a different linking can affect type recognition of a registered converter. What am I missing?
Can anybody help me out?
Thanks,
Dominik
Example from here.
#include <boost/shared_ptr.hpp>
#include <boost/make_shared.hpp>
#include <boost/python.hpp>
#include <string>
#include <iostream>
namespace bp = boost::python;
struct Foo{
Foo(){}
Foo(std::string const& s) : m_string(s){}
void doSomething() {
std::cout << "Foo:" << m_string << std::endl;
}
std::string m_string;
};
typedef boost::shared_ptr<Foo> foo_ptr;
BOOST_PYTHON_MODULE(hello)
{
bp::class_<Foo, foo_ptr>("Foo")
.def("doSomething", &Foo::doSomething)
;
};
int main(int argc, char **argv)
{
Py_Initialize();
try {
PyRun_SimpleString(
"a_foo = None\n"
"\n"
"def setup(a_foo_from_cxx):\n"
" print 'setup called with', a_foo_from_cxx\n"
" global a_foo\n"
" a_foo = a_foo_from_cxx\n"
"\n"
"def run():\n"
" a_foo.doSomething()\n"
"\n"
"print 'main module loaded'\n"
);
foo_ptr a_cxx_foo = boost::make_shared<Foo>("c++");
inithello();
bp::object main = bp::object(bp::handle<>(bp::borrowed(
PyImport_AddModule("__main__")
)));
// pass the reference to a_cxx_foo into python:
bp::object setup_func = main.attr("setup");
setup_func(a_cxx_foo);
// now run the python 'main' function
bp::object run_func = main.attr("run");
run_func();
}
catch (bp::error_already_set) {
PyErr_Print();
}
Py_Finalize();
return 0;
}

I far as I understand the documentation about Boost Python linkage, it seems that the conversion registry used for automatic conversion of Python object into C++ object is not available when statically linked. I'm facing the same issue and that's a pity it is actually the case. I would have imagined at least the required converter to be bundle but I'm afraid it is not the case for some reason.

Swigging a class member function that is a template function

This question builds on the question: How to instantiate a template method of a template class with swig? .
However, compared to that question, the code I'm trying to wrap is a little different:
class MyClass {
public:
template <class T>
void f1(const string& firstArg, const T& value);
};
The MyClass is a regular C++ class, with one template function f1.
Attempt to wrap MyClass::f1:, i.e. the Swig .i file
%template(f1String) MyClass::f1<std::string>;
With the above, a Python client can do
o = MyClass
str1 = "A String"
o.f1String("", str1)
This interface require the Python client to learn about all different f1 function names, each one different depending on the type. Not so clean.
A cleaner interface can be obtained by overloading, extending in the interface file, e.g.
%extend MyClass {
void f1(const string& s, const string& s1){
$self->f1(s, s1);
}
void f1(const string& s, const int& anInt){
$self->f1(s, anInt);
}
}
This allow client code like this:
o = MyClass
str1 = "A String"
anInt = 34
o.f1("", str1)
o.f1("", anInt)
Question is, is there any way to obtain the interface above (by extending), without extending, using Swig?

Luckily the Python wrapper supports overloading, so you can simply instantiate the two methods with the same name and SWIG will do its magic to resolve the overloads at runtime. See 6.18 Templates in the chapter “SWIG and C++” of the documentation for more details.
test.i
%module example
%{
#include<iostream>
class MyClass {
public:
template <class T>
void f1(const std::string& firstArg, const T& value) {
std::cout << firstArg << ',' << value << '\n';
}
};
%}
%include <std_string.i>
class MyClass {
public:
template <class T>
void f1(const std::string& firstArg, const T& value);
};
%extend MyClass {
%template(f1) f1<std::string>;
%template(f1) f1<int>;
}
test.py
from example import *
o = MyClass()
str1 = "A String"
anInt = 34
o.f1("X", str1)
o.f1("Y", anInt)
Example workflow to compile and run:
$ swig -python -c++ test.i
$ g++ -Wall -Wextra -Wpedantic -I /usr/include/python2.7/ -fPIC -shared test_wrap.cxx -o _example.so -lpython2.7
$ python2.7 test.py
X,A String
Y,34

Calling python object's method from c++

I am trying to achieve the following: passing a python object to a c++ callback chain (which are typical in many popular c++ libraries). In the c++ code, callbacks pass on objects that have necessary information for consecutive callbacks in the cascade/chain.
Here is a small test code I wrote: we pass a python object to a c routine (case 1) and call it's method. That works ok. But when I pass the python object to a c++ object and try to call it "inside" the c++ object, I get segfault.. :(
Here it goes:
c++ module ("some.cpp"):
#include <stdint.h>
#include <iostream>
#include <Python.h>
/* objective:
* create c++ objects / routines that accept python objects
* then call methods of the python objects inside c++
*
* python objects (including its variables and methods) could be passed along, for example in c++ callback chains ..
* .. and in the end we could call a python callback
*
* Compile and test as follows:
* python setup.py build_ext
* [copy/link some.so where test.py is]
* python test.py
*
*/
class testclass {
public:
testclass(int* i, PyObject* po) {
std::cerr << "testclass constructor! \n";
i=i; po=po;
}
~testclass() {}
void runpo() {
PyObject* name;
const char* mname="testmethod";
name=PyString_FromString(mname);
std::cerr << "about to run the python method .. \n";
PyObject_CallMethodObjArgs(po, name, NULL);
std::cerr << ".. you did it - i will buy you a beer!\n";
}
public:
int* i;
PyObject* po;
};
/* Docstrings */
static char module_docstring[] = "hand-made python module";
/* Available functions */
static PyObject* regi_wrapper(PyObject * self, PyObject * args);
void regi(int* i, PyObject* po);
/* Module specification */
static PyMethodDef module_methods[] = {
{"regi_wrapper",regi_wrapper, METH_VARARGS, "lets see if we can wrap this sucker"},
{NULL, NULL, 0, NULL}
};
/* Initialize the module */
PyMODINIT_FUNC initsome(void)
{
PyObject *m = Py_InitModule3("some", module_methods, module_docstring);
if (m == NULL)
return;
// import_array(); // numpy not required here ..
}
static PyObject* regi_wrapper(PyObject * self, PyObject * args)
{
int* input_i; // whatever input variable
PyObject* input_po; // python object
PyObject* ret; // return variable
// parse arguments
if (!PyArg_ParseTuple(args, "iO", &input_i, &input_po)) {
return NULL;
}
// https://stackoverflow.com/questions/16606872/calling-python-method-from-c-or-c-callback
// Py_INCREF(input_po); // need this, right? .. makes no difference
/* // seems not to make any difference ..
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
*/
regi(input_i, input_po);
// PyGILState_Release(gstate); // .. makes no difference
// Py_DECREF(input_po); // .. makes no difference
Py_RETURN_TRUE;
}
void regi(int* i, PyObject* po) {
// search variable and methods from PyObject "po" and call its methods?
PyObject* name;
const char* mname="testmethod";
testclass* testobj;
testobj=new testclass(i,po);
/* [insert // in front of this line to test case 1]
// ***** this one works! *********
name=PyString_FromString(mname);
PyObject_CallMethodObjArgs(po, name, NULL);
*/ // [insert // in front of this line to test case 1]
// *** I WOULD LIKE THIS TO WORK *** but it gives segfault.. :(
testobj->runpo(); // [uncomment to test case 2]
}
setup.py:
from distutils.core import setup, Extension
# the c++ extension module
extension_mod = Extension("some", ["some.cpp"])
setup(name = "some", ext_modules=[extension_mod])
test.py:
import some
class sentinel:
def __init__(self):
pass
def testmethod(self):
print "hello from sentinel.testmethod"
pass
se=sentinel()
some.regi_wrapper(1,se)
This question seems relevant:
Calling python method from C++ (or C) callback
.. however the answer did not help me.
What am I missing/misunderstanding here (my c++ sucks big time, so I might have missed something obvious) .. ?
Also, some bonus questions:
a) I am familiar with swig and swig "directors".. however, I would like to use swig for general wrapping of the code, but my custom wrapping for the sort of things described in this question (i.e. without directors). Is there any way to achieve this?
b) Any other suggestions to achieve what I am trying to achieve here, are highly appreciated.. is this possible or just pure insanity?

Using in the constructor
po=this->po
solves the "issue". Sorry for the spam! I will leave here this thing as an example.. maybe someone finds it useful.

python pointer to C structure in SWIG -- accessing the struct members

I'm trying to get a very basic Python-to-C interface working with SWIG where I can pass a pointer to a structure to a C function and the members get populated, and I can then access the members in python.
In the below example, everything works except when I try to print the structure members:
print swig_test.diags_mem0_get(var)
Results in:
$ ./runpython.py
Traceback (most recent call last):
File "./runpython.py", line 11, in <module>
print swig_test.diags_mem0_get(var)
AttributeError: 'module' object has no attribute 'diags_mem0_get'
Whereas this:
print var.mem0
Results in:
$ ./runpython.py
<Swig Object of type 'uint16_t *' at 0x7f8261e15b40>swig/python detected a memory leak of type 'uint16_t *', no destructor found.
I am following the SWIG 3.0 Documentation, specifically section "5.5 Structures and unions" here: http://swig.org/Doc3.0/SWIGDocumentation.html#SWIG_nn31
What am I doing wrong?
I have distilled the example down to bare bones:
swig_test.h
typedef struct diags_t {
uint16_t mem0;
uint16_t mem1;
} diags;
diags *malloc_diags(void);
void free_diags(diags *pdiag);
int get_diags(diags *pdiags);
swig_test.c
#include <stdlib.h> // malloc()
#include <stdint.h> // uint16_t
#include "swig_test.h"
int main(int argc, char **argv) {
return 0;
}
int get_diags(diags *pdiags) {
pdiags->mem0 = 0xdead;
pdiags->mem1 = 0xbeef;
return 0;
}
diags *malloc_diags(void) {
diags *dptr = malloc(sizeof(diags));
return dptr;
}
void free_diags(diags *pdiag) {
if (pdiag != NULL)
free(pdiag);
}
swig_test.i
%module swig_test
%{
#include "swig_test.h"
%}
%include "swig_test.h"
Makefile
CXX = gcc
INCLUDES = -I./
COMPFLAGS = -c -Wall -fPIC
PYINC = /usr/include/python2.7
SWIG = /usr/bin/swig
all: swig_test _swig_test.so
swig_test: swig_test.o
$(CXX) -Wall $^ -o $#
swig_test.o: swig_test.c
$(CXX) $(COMPFLAGS) $(INCLUDES) $^
_swig_test.so: swig_test_wrap.o swig_test.o
$(CXX) -shared $^ -L$(PYLIB) -lpython2.7 -o $#
swig_test_wrap.o: swig_test_wrap.c
$(CXX) $(COMPFLAGS) $(INCLUDES) -I$(PYINC) $^
swig_test_wrap.c: swig_test.i
$(SWIG) -python $(INCLUDES) $^
And finally the simple python example:
runpython.py
#!/usr/bin/python2
import swig_test
var = swig_test.malloc_diags()
if var == 'NULL':
print "Error, no memory left"
else:
ret = swig_test.get_diags(var)
if ret == 0:
print swig_test.diags_mem0_get(var)
print var.mem0
swig_test.free_diags(var)

The functionality you're looking for comes in typemaps. The documentation freely admits that "At first glance, this code will look a little confusing." Here's what worked for me.
In essence, a typemap is a few lines of code that SWIG swaps in when it needs to convert between Python and C. They're separately defined for Python to C (%typemap(in)) and C to Python (%typemap(out)). SWIG's documentation also defines a few magic variables:
$input refers to an input object that needs to be converted to C/C++.
$result refers to an object that is going to be returned by a wrapper function.
$1 refers to a C/C++ variable that has the same type as specified in the typemap declaration (an int in this example).
For unsigned integer support, you just need in and out maps for uint8_t, uint16_t, and uint32_t
The lines below provide that functionality. They can go into SWIG's .i file, or the main header (with an ifdef SWIG guard around them).
/* uintXX_t mapping: Python -> C */
%typemap(in) uint8_t {
$1 = (uint8_t) PyInt_AsLong($input);
}
%typemap(in) uint16_t {
$1 = (uint16_t) PyInt_AsLong($input);
}
%typemap(in) uint32_t {
$1 = (uint32_t) PyInt_AsLong($input);
}
/* uintXX_t mapping: C -> Python */
%typemap(out) uint8_t {
$result = PyInt_FromLong((long) $1);
}
%typemap(out) uint16_t {
$result = PyInt_FromLong((long) $1);
}
%typemap(out) uint32_t {
$result = PyInt_FromLong((long) $1);
}
Resources I found useful:
http://www.swig.org/Doc3.0/Typemaps.html#Typemaps_nn3
https://docs.python.org/2/c-api/int.html?highlight=pyint_aslong

One "fix" is to change the uint16_t members to int types. Then this syntax works:
print var.mem0
Clearly SWIG has some problems with non-integer types.
I'd love it if somebody proposed an alternative solution that lets me keep my uint16_t types.

how do i get a structure in c++ from python?

My C++ program:
#include <iostream>
using namespace std;
struct FirstStructure
{
public:
int first_int;
int second_int;
};
struct SecondStructure
{
public:
int third_int;
FirstStructure ft;
};
int test_structure(SecondStructure ss)
{
int sum = ss.ft.first_int + ss.ft.second_int + ss.third_int;
return sum;
}
extern "C"
{
int test(SecondStructure ss)
{
return test_structure(ss);
}
}
And I compile the cpp file use this command "g++ -fPIC -shared -o array.so array.cpp".
Then I call the file array.so use python,my python program as these:
#coding=utf-8
import ctypes
from ctypes import *
class FirstStructure(Structure):
_fields_ = [
("first_int", c_int),
("second_int", c_int)
]
class SecondStructure(Structure):
_fields_ = [
("third_int", c_int),
("ft", FirstStructure)
]
if __name__ == '__main__':
fs = FirstStructure(1, 2)
ss = SecondStructure(3, fs)
print ss.ft.first_int
lib = ctypes.CDLL("./array.so")
print lib.test(ss)
When I run the python program,the console show an error, the error is "segmentation fault".I read the documentation from the url "https://docs.python.org/2/library/ctypes.html",how to fix the bug.

You have to declare a function's argument and return types in python, in order to be able to call it properly.
So, insert the following before calling the test function:
lib.test.argtypes = [SecondStructure]
lib.test.restype = ctypes.c_int
Things should work then, as far as I can see...
As long as the amount of C-to-python interfaces remains "small" (whatever that is), I think ctypes is just fine.

ok,I got it,modified code as these:
#include <iostream>
using namespace std;
extern "C"
{
struct FirstStructure
{
public:
int first_int;
int second_int;
};
struct SecondStructure
{
public:
int third_int;
FirstStructure ft;
};
int test_structure(SecondStructure *ss)
{
int sum = ss->ft.first_int + ss->ft.second_int + ss->third_int;
return sum;
}
int test(SecondStructure *ss)
{
return test_structure(ss);
}
}
and then,I fixed the bug.

Well if you are intending to design communication medium between C++ and python then I would suggest go for combination zmq and google protocol buffers.
where proto buf would serve for serialization/deserialization and zmq for communication medium.

You might want to have a look at Boost.python
https://wiki.python.org/moin/boost.python/SimpleExample
It will allow you to compile python modules from C++ and define how the python is allowed to access the c++ code in an easy to understand fashion