We Keep Coding

How to wrap a C++ object using pure Python Extension API (python3)?

I want to know how to wrap a C++ object with Python Extension API (and distutils) without external tools (like Cython, Boost, SWIG, ...). Just in pure Python way without creating a dll.
Note that my C++ object has memory allocations so destructor has to be called to avoid memory leaks.
#include "Voice.h"
namespace transformation
{
Voice::Voice(int fftSize) { mem=new double[fftSize]; }
Voice::~Voice() { delete [] mem; }
int Voice::method1() { /*do stuff*/ return (1); }
}
I just want to do somethings like that in Python :
import voice
v=voice.Voice(512)
result=v.method1()

Seems that the answer was in fact here : https://docs.python.org/3.6/extending/newtypes.html
With examples, but not really easy.
EDIT 1 :
In fact, it is not really for wrapping a C++ object in a Python object, but rather to create a Python object with C code. (edit2 : and so you can wrap C++ object!)
EDIT 2 :
Here is a solution using the Python newtypes
.
Original C++ file : Voice.cpp
#include <cstdio>
#include "Voice.h"
namespace transformation
{
Voice::Voice(int fftSize) {
printf("c++ constructor of voice\n");
this->fftSize=fftSize;
mem=new double[fftSize];
}
Voice::~Voice() { delete [] mem; }
int Voice::filter(int freq) {
printf("c++ voice filter method\n");
return (doubleIt(3));
}
int Voice::doubleIt(int i) { return 2*i; }
}
.
Original h file : Voice.h
namespace transformation {
class Voice {
public:
double *mem;
int fftSize;
Voice(int fftSize);
~Voice();
int filter(int freq);
int doubleIt(int i);
};
}
.
C++ Python wrapper file : voiceWrapper.cpp
#include <Python.h>
#include <cstdio>
//~ #include "structmember.h"
#include "Voice.h"
using transformation::Voice;
typedef struct {
PyObject_HEAD
Voice * ptrObj;
} PyVoice;
static PyModuleDef voicemodule = {
PyModuleDef_HEAD_INIT,
"voice",
"Example module that wrapped a C++ object",
-1,
NULL, NULL, NULL, NULL, NULL
};
static int PyVoice_init(PyVoice *self, PyObject *args, PyObject *kwds)
// initialize PyVoice Object
{
int fftSize;
if (! PyArg_ParseTuple(args, "i", &fftSize))
return -1;
self->ptrObj=new Voice(fftSize);
return 0;
}
static void PyVoice_dealloc(PyVoice * self)
// destruct the object
{
delete self->ptrObj;
Py_TYPE(self)->tp_free(self);
}
static PyObject * PyVoice_filter(PyVoice* self, PyObject* args)
{
int freq;
int retval;
if (! PyArg_ParseTuple(args, "i", &freq))
return Py_False;
retval = (self->ptrObj)->filter(freq);
return Py_BuildValue("i",retval);
}
static PyMethodDef PyVoice_methods[] = {
{ "filter", (PyCFunction)PyVoice_filter, METH_VARARGS, "filter the mem voice" },
{NULL} /* Sentinel */
};
static PyTypeObject PyVoiceType = { PyVarObject_HEAD_INIT(NULL, 0)
"voice.Voice" /* tp_name */
};
PyMODINIT_FUNC PyInit_voice(void)
// create the module
{
PyObject* m;
PyVoiceType.tp_new = PyType_GenericNew;
PyVoiceType.tp_basicsize=sizeof(PyVoice);
PyVoiceType.tp_dealloc=(destructor) PyVoice_dealloc;
PyVoiceType.tp_flags=Py_TPFLAGS_DEFAULT;
PyVoiceType.tp_doc="Voice objects";
PyVoiceType.tp_methods=PyVoice_methods;
//~ PyVoiceType.tp_members=Noddy_members;
PyVoiceType.tp_init=(initproc)PyVoice_init;
if (PyType_Ready(&PyVoiceType) < 0)
return NULL;
m = PyModule_Create(&voicemodule);
if (m == NULL)
return NULL;
Py_INCREF(&PyVoiceType);
PyModule_AddObject(m, "Voice", (PyObject *)&PyVoiceType); // Add Voice object to the module
return m;
}
.
distutils file : setup.py
from distutils.core import setup, Extension
setup(name='voicePkg', version='1.0', \
ext_modules=[Extension('voice', ['voiceWrapper.cpp','Voice.cpp'])])
.
python test file : test.py
import voice
v=voice.Voice(512)
result=v.filter(5)
print('result='+str(result))
.
and magic :
sudo python3 setup.py install
python3 test.py
Output is :
c++ constructor of voice
c++ voice filter method
result=6
Enjoy !
Doom

SWIG: How to return a struct using %apply? 'No typemaps defined' warning

I currently have a function that uses a struct as a buffer to return some information, like so:
int example_reader(int code, void* return_struct);
My goal is to make it so that when I wrap this function using SWIG so that it can be used in Python, I will return the struct along with the function's regular return value. Thus far, I have been doing so using the %apply command like so:
%apply struct ret_struct *OUTPUT {void* return_struct};
However, when I add the above line to my .i file and try to run SWIG, I get the following warning:
"warning 453: Can't apply (struct ret_struct *OUTPUT. No typemaps are defined"
I believe I'm including the .h file that defines the struct I'm trying to return, so I've had trouble pinpointing the issue. Please correct me if the issue seems to involve the improper inclusion of the struct. I've tried reading through the SWIG documentation as well as other Stack Overflow posts to get some inkling of what the problem might be, but I haven't been able to figure it out thus far. The problem is made slightly trickier because I am trying to return a void pointer to a struct, and the code I'm trying to wrap could have multiple kinds of structs for me to return. What would be a wise way of handling the return of this struct? Thank you!

I have given here a full C example, where an interface is used for returning a struct to the target language together with a return value. In this way you can make a proper interface, where no implementation is given in the header. That is no default implementation of a virtual destructor. If you don't want to use an interface, you can let SWIG and Python know how data are represented.
Interface header: foo.h
typedef struct _Foo Foo;
int foo_new(Foo **obj);
int foo_free(Foo *obj);
int foo_get_value_a(Foo *obj, int *result);
int foo_set_value_a(Foo *obj, int value);
int foo_get_value_b(Foo *obj, char **result);
int foo_set_value_b(Foo *obj, char *value);
SWIG interface: foo.i
%module foo
%{
#include "foo.h"
%}
%include "typemaps.i"
%typemap(in, numinputs=0) Foo ** (Foo *temp) {
$1 = &temp;
}
%typemap(argout) Foo ** {
PyObject* temp = NULL;
if (!PyList_Check($result)) {
temp = $result;
$result = PyList_New(1);
PyList_SetItem($result, 0, temp);
}
temp = SWIG_NewPointerObj(*$1, SWIGTYPE_p__Foo, SWIG_POINTER_NEW);
PyList_Append($result, temp);
Py_DECREF(temp);
}
%delobject foo_free; // Protect for double deletion
struct _Foo {};
%extend _Foo {
~_Foo() {
foo_free($self);
}
};
%ignore _Foo;
Some implementation of the interface: foo.c
%include "foo.h"
#include "foo.h"
#include "stdlib.h"
#include "string.h"
struct FooImpl {
char* c;
int i;
};
int foo_new(Foo **obj)
{
struct FooImpl* f = (struct FooImpl*) malloc(sizeof(struct FooImpl));
f->c = NULL;
*obj = (Foo*) f;
return 0;
}
int foo_free(Foo *obj)
{
struct FooImpl* impl = (struct FooImpl*) obj;
if (impl) {
if (impl->c) {
free(impl->c);
impl->c = NULL;
}
}
return 0;
}
int foo_get_value_a(Foo *obj, int *result)
{
struct FooImpl* impl = (struct FooImpl*) obj;
*result = impl->i;
return 0;
}
int foo_set_value_a(Foo *obj, int value)
{
struct FooImpl* impl = (struct FooImpl*) obj;
impl->i = value;
return 0;
}
int foo_get_value_b(Foo *obj, char **result)
{
struct FooImpl* impl = (struct FooImpl*) obj;
*result = impl->c;
return 0;
}
int foo_set_value_b(Foo *obj, char *value)
{
struct FooImpl* impl = (struct FooImpl*) obj;
int len = strlen(value);
if (impl->c) {
free(impl->c);
}
impl->c = (char*)malloc(len+1);
strcpy(impl->c,value);
return 0;
}
Script for building
#!/usr/bin/env python
from distutils.core import setup, Extension
import os
os.environ['CC'] = 'gcc';
setup(name='foo',
version='1.0',
ext_modules =[Extension('_foo',
['foo.i','foo.c'])])
Usage:
import foo
OK, f = foo.foo_new()
OK = foo.foo_set_value_b(f, 'Hello world!')
OK = foo.foo_free(f)
OK, f = foo.foo_new()
# Test safe to double delete
del f

Expose an underlying struct as a member of a custom type within a Python extension

This question is a slight spin on a previous question: Accessing the underlying struct of a PyObject
Except in my version I want to know how to expose the fields of the Point struct as members of my new type.
I have looked everywhere I could think, read numerous example Python extensions and read many tutorials, docs, etc. and have not been able to find a clean way to do this.
Below is an example of what I want to do. I have some internal struct that I would like to expose via the Python extension but I don't want to have to redefine the struct in my code. It seems like the main area that is the problem is the PyMemeberDef definition...what would the offset be of x or y inside the Point struct from the context of the PointObject struct?
Any help is much appreciated, thanks.
#include <Python.h>
#include <structmember.h>
// This is actually defined elsewhere in someone else's code.
struct Point {
int x;
int y;
};
struct PointObject {
PyObject_HEAD
struct Point* my_point;
int z;
};
static PyMemberDef point_members[] = {
{"z", T_INT, offsetof(struct PointObject, z), 0, "z field"},
{"x", T_INT, offsetof(???), 0, "point x field"},
{"y", T_INT, offsetof(???), 0, "point y field"},
{NULL}
};
static PyTypeObject PointType = {
PyObject_HEAD_INIT(NULL)
.tp_name = "Point",
.tp_basicsize = sizeof(PointObject),
.tp_flags = Py_TPFLAGS_DEFAULT,
.tp_doc = "Point objects",
.tp_members = point_members,
.tp_init = (initproc)point_init,
};
...
PyMODINIT_FUNC
initMainModule(void)
{
PyObject *m = Py_InitModule(MODULE, NULL);
// Register PointType
PointType.tp_new = PyType_GenericNew;
if (PyType_Ready(&PointType) < 0)
return;
Py_INCREF(&PointType);
PyModule_AddObject(m, "Point", (PyObject *)&PointType);
...
}

I would highly recommend reading http://docs.python.org/release/2.6/_sources/extending/newtypes.txt, specifically the section on "Providing finer control over data attributes".
After reading this, it appears it is possible to do this by defining a collection of setter/getters for the specific members in the Point struct. It does seem a bit overkill for something as simple as this, but it appears to work just fine.
Below is an example on how to do it for the 'x' field of Point, based on adding the following code to my previous example:
static PyGetSetDef point_getset[] = {
{"x", (getter)point_get_x, (setter)point_set_x, "", NULL},
{NULL}
};
static PyObject*
point_get_x(struct PointObject *self, void *closure)
{
return Py_BuildValue("i", self->my_point->x);
}
static int
point_set_x(struct PointObject *self, PyObject *value, void *closure)
{
PyErr_SetString(PyExc_TypeError, "Attribute is read-only!");
return -1;
}
Lastly, add the point_getset struct to the tp_getset field in the previously defined PyTypeObject struct. It should be noted that you can still define members in the traditional PyMemberDef struct for the simple cases.

Return CTypes pointer from C

I'm writing a Python C Extension that needs to return a CTypes pointer to a char array in memory (I need to interface with another Python library that expects a CTypes pointer).
I cannot find any documentation on any kind of CTypes interface for C. Are there any workarounds, like calling the Python pointer constructor?
static PyObject *get_pointer(myObject *self)
{
char *my_pointer = self->internal_pointer;
return PyCTypes_Pointer(my_pointer); /* how do I turn 'my_pointer' into
a Python object representing a CTypes pointer? */
}
Thanks in advance.

Even though you're programming in C, try using Python itself. Sadly I don't know of any kind of CTypes interface for C and couldn't find any hint in the sources. Maybe you'll be able to write your own PyCTypes_Pointer method with the following.
I didn't do any cleanup, error checking or handling here, but it's working with Python 2.7 and 3.4 for me.
#include <Python.h>
static const char *test = "Testing Python ctypes char pointer ...";
static PyObject *c_char_p = NULL;
static PyObject *c_void_p = NULL;
static PyObject *cast = NULL;
static PyObject * char_p ( void ) {
PyObject *p = PyLong_FromVoidPtr((void *) test);
p = PyObject_CallFunction(c_void_p, "O", p);
return PyObject_CallFunction(cast, "OO", p, c_char_p);
}
static PyObject *len ( void ) {
return PyLong_FromSize_t(strlen(test));
}
static PyMethodDef methods[] = {
{"char_p", (PyCFunction) char_p, METH_NOARGS, ""},
{"len", (PyCFunction) len, METH_NOARGS, ""},
{ NULL }
};
#if PY_MAJOR_VERSION >= 3
static PyModuleDef module = {PyModuleDef_HEAD_INIT, "pyt", "", -1, methods,
NULL, NULL, NULL, NULL};
#define INIT_FUNC PyInit_pyp
#else
#define INIT_FUNC initpyp
#endif
PyMODINIT_FUNC INIT_FUNC ( void ) {
PyObject* m;
PyObject *ctypes = PyImport_ImportModule("ctypes");
PyObject *c_char = PyObject_GetAttrString(ctypes, "c_char");
c_char_p = PyObject_CallMethod(ctypes, "POINTER", "O", c_char);
c_void_p = PyObject_GetAttrString(ctypes, "c_void_p");
cast = PyObject_GetAttrString(ctypes, "cast");
#if PY_MAJOR_VERSION >= 3
m = PyModule_Create(&module);
#else
m = Py_InitModule("pyp", methods);
#endif
return (PyMODINIT_FUNC) m;
}
Just build with
from __future__ import print_function
from distutils.core import Extension, setup
import sys
sys.argv.extend(["build_ext", "-i"])
setup(ext_modules = [Extension('pyp', ['pyp.c'])])
import pyp
c = pyp.char_p()
print(c[:pyp.len()])
and get an output like
<class 'ctypes.LP_c_char'> Testing Python ctypes char pointer ...
If you're handling zero-terminated strings only, you might be able to use ctypes.c_char_p directly, maybe ...

OpenCV: memory leak with Python interface but not in the C version

I am asking here because I haven't gotten any help from the OpenCV developers so far. I reduced the problem to a very simple test case so probably anyone with some background with CPython could help here.
This C code does not leak:
int main() {
while(true) {
int hist_size[] = {40};
float range[] = {0.0f,255.0f};
float* ranges[] = {range};
CvHistogram* hist = cvCreateHist(1, hist_size, CV_HIST_ARRAY, ranges, 1);
cvReleaseHist(&hist);
}
}
This Python code does leak:
while True: cv.CreateHist([40], cv.CV_HIST_ARRAY, [[0,255]], 1)
I searched through the CPython code (of OpenCVs current SVN trunk code) and found this:
struct cvhistogram_t {
PyObject_HEAD
CvHistogram h;
PyObject *bins;
};
...
/* cvhistogram */
static void cvhistogram_dealloc(PyObject *self)
{
cvhistogram_t *cvh = (cvhistogram_t*)self;
Py_DECREF(cvh->bins);
PyObject_Del(self);
}
static PyTypeObject cvhistogram_Type = {
PyObject_HEAD_INIT(&PyType_Type)
0, /*size*/
MODULESTR".cvhistogram", /*name*/
sizeof(cvhistogram_t), /*basicsize*/
};
static PyObject *cvhistogram_getbins(cvhistogram_t *cvh)
{
Py_INCREF(cvh->bins);
return cvh->bins;
}
static PyGetSetDef cvhistogram_getseters[] = {
{(char*)"bins", (getter)cvhistogram_getbins, (setter)NULL, (char*)"bins", NULL},
{NULL} /* Sentinel */
};
static void cvhistogram_specials(void)
{
cvhistogram_Type.tp_dealloc = cvhistogram_dealloc;
cvhistogram_Type.tp_getset = cvhistogram_getseters;
}
...
static PyObject *pycvCreateHist(PyObject *self, PyObject *args, PyObject *kw)
{
const char *keywords[] = { "dims", "type", "ranges", "uniform", NULL };
PyObject *dims;
int type;
float **ranges = NULL;
int uniform = 1;
if (!PyArg_ParseTupleAndKeywords(args, kw, "Oi|O&i", (char**)keywords, &dims, &type, convert_to_floatPTRPTR, (void*)&ranges, &uniform)) {
return NULL;
}
cvhistogram_t *h = PyObject_NEW(cvhistogram_t, &cvhistogram_Type);
args = Py_BuildValue("Oi", dims, CV_32FC1);
h->bins = pycvCreateMatND(self, args);
Py_DECREF(args);
if (h->bins == NULL) {
return NULL;
}
h->h.type = CV_HIST_MAGIC_VAL;
if (!convert_to_CvArr(h->bins, &(h->h.bins), "bins"))
return NULL;
ERRWRAP(cvSetHistBinRanges(&(h->h), ranges, uniform));
return (PyObject*)h;
}
And from the OpenCV C headers:
typedef struct CvHistogram
{
int type;
CvArr* bins;
float thresh[CV_MAX_DIM][2]; /* For uniform histograms. */
float** thresh2; /* For non-uniform histograms. */
CvMatND mat; /* Embedded matrix header for array histograms. */
}
CvHistogram;
I don't exactly understand everything because I never worked with the C-interface to Python before. But probably the bug I am searching for is somewhere in this code.
Am I right? Or where should I search for the bug? How would I fix it?
(Note for people who have seen an earlier version of this question: I looked at the wrong code. Their SWIG interface was deprecated and not used anymore (but the code was still there in SVN, this is why I confused it. So don't look into interfaces/swig, this code is old and not used. The current code lives in modules/python.)
Upstream bug report: memleak in OpenCV Python CreateHist

It has been fixed.
Changed 3 weeks ago by jamesb
status changed from accepted to closed
resolution set to fixed
Fixed in r4526
The ranges parameters were not being freed, and the iterator over ranges was not being DECREF'ed. Regressions now pass, and original loop does not leak.

I think you have garbage collection issue, in that you never leave the loop.
Does this work more as expected?
while True:
cv.CreateHist([40], cv.CV_HIST_ARRAY, [[0,255]], 1)
cv = None