We Keep Coding

Cython: return double * heap array from cython function to python as a 1D np.ndarray [duplicate]

Following this answer to "Can I force a numpy ndarray to take ownership of its memory?" I attempted to use the Python C API function PyArray_ENABLEFLAGS through Cython's NumPy wrapper and found it is not exposed.
The following attempt to expose it manually (this is just a minimum example reproducing the failure)
from libc.stdlib cimport malloc
import numpy as np
cimport numpy as np
np.import_array()
ctypedef np.int32_t DTYPE_t
cdef extern from "numpy/ndarraytypes.h":
void PyArray_ENABLEFLAGS(np.PyArrayObject *arr, int flags)
def test():
cdef int N = 1000
cdef DTYPE_t *data = <DTYPE_t *>malloc(N * sizeof(DTYPE_t))
cdef np.ndarray[DTYPE_t, ndim=1] arr = np.PyArray_SimpleNewFromData(1, &N, np.NPY_INT32, data)
PyArray_ENABLEFLAGS(arr, np.NPY_ARRAY_OWNDATA)
fails with a compile error:
Error compiling Cython file:
------------------------------------------------------------
...
def test():
cdef int N = 1000
cdef DTYPE_t *data = <DTYPE_t *>malloc(N * sizeof(DTYPE_t))
cdef np.ndarray[DTYPE_t, ndim=1] arr = np.PyArray_SimpleNewFromData(1, &N, np.NPY_INT32, data)
PyArray_ENABLEFLAGS(arr, np.NPY_ARRAY_OWNDATA)
^
------------------------------------------------------------
/tmp/test.pyx:19:27: Cannot convert Python object to 'PyArrayObject *'
My question: Is this the right approach to take in this case? If so, what am I doing wrong? If not, how do I force NumPy to take ownership in Cython, without going down to a C extension module?

You just have some minor errors in the interface definition. The following worked for me:
from libc.stdlib cimport malloc
import numpy as np
cimport numpy as np
np.import_array()
ctypedef np.int32_t DTYPE_t
cdef extern from "numpy/arrayobject.h":
void PyArray_ENABLEFLAGS(np.ndarray arr, int flags)
cdef data_to_numpy_array_with_spec(void * ptr, np.npy_intp N, int t):
cdef np.ndarray[DTYPE_t, ndim=1] arr = np.PyArray_SimpleNewFromData(1, &N, t, ptr)
PyArray_ENABLEFLAGS(arr, np.NPY_OWNDATA)
return arr
def test():
N = 1000
cdef DTYPE_t *data = <DTYPE_t *>malloc(N * sizeof(DTYPE_t))
arr = data_to_numpy_array_with_spec(data, N, np.NPY_INT32)
return arr
This is my setup.py file:
from distutils.core import setup, Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("_owndata", ["owndata.pyx"])]
setup(cmdclass={'build_ext': build_ext}, ext_modules=ext_modules)
Build with python setup.py build_ext --inplace. Then verify that the data is actually owned:
import _owndata
arr = _owndata.test()
print arr.flags
Among others, you should see OWNDATA : True.
And yes, this is definitely the right way to deal with this, since numpy.pxd does exactly the same thing to export all the other functions to Cython.

#Stefan's solution works for most scenarios, but is somewhat fragile. Numpy uses PyDataMem_NEW/PyDataMem_FREE for memory-management and it is an implementation detail, that these calls are mapped to the usual malloc/free + some memory tracing (I don't know which effect Stefan's solution has on the memory tracing, at least it seems not to crash).
There are also more esoteric cases possible, in which free from numpy-library doesn't use the same memory-allocator as malloc in the cython code (linked against different run-times for example as in this github-issue or this SO-post).
The right tool to pass/manage the ownership of the data is PyArray_SetBaseObject.
First we need a python-object, which is responsible for freeing the memory. I'm using a self-made cdef-class here (mostly because of logging/demostration), but there are obviously other possiblities as well:
%%cython
from libc.stdlib cimport free
cdef class MemoryNanny:
cdef void* ptr # set to NULL by "constructor"
def __dealloc__(self):
print("freeing ptr=", <unsigned long long>(self.ptr)) #just for debugging
free(self.ptr)
#staticmethod
cdef create(void* ptr):
cdef MemoryNanny result = MemoryNanny()
result.ptr = ptr
print("nanny for ptr=", <unsigned long long>(result.ptr)) #just for debugging
return result
...
Now, we use a MemoryNanny-object as sentinel for the memory, which gets freed as soon as the parent-numpy-array gets destroyed. The code is a little bit awkward, because PyArray_SetBaseObject steals the reference, which is not handled by Cython automatically:
%%cython
...
from cpython.object cimport PyObject
from cpython.ref cimport Py_INCREF
cimport numpy as np
#needed to initialize PyArray_API in order to be able to use it
np.import_array()
cdef extern from "numpy/arrayobject.h":
# a little bit awkward: the reference to obj will be stolen
# using PyObject* to signal that Cython cannot handle it automatically
int PyArray_SetBaseObject(np.ndarray arr, PyObject *obj) except -1 # -1 means there was an error
cdef array_from_ptr(void * ptr, np.npy_intp N, int np_type):
cdef np.ndarray arr = np.PyArray_SimpleNewFromData(1, &N, np_type, ptr)
nanny = MemoryNanny.create(ptr)
Py_INCREF(nanny) # a reference will get stolen, so prepare nanny
PyArray_SetBaseObject(arr, <PyObject*>nanny)
return arr
...
And here is an example, how this functionality can be called:
%%cython
...
from libc.stdlib cimport malloc
def create():
cdef double *ptr=<double*>malloc(sizeof(double)*8);
ptr[0]=42.0
return array_from_ptr(ptr, 8, np.NPY_FLOAT64)
which can be used as follows:
>>> m = create()
nanny for ptr= 94339864945184
>>> m.flags
...
OWNDATA : False
...
>>> m[0]
42.0
>>> del m
freeing ptr= 94339864945184
with results/output as expected.
Note: the resulting arrays doesn't really own the data (i.e. flags return OWNDATA : False), because the memory is owned be the memory-nanny, but the result is the same: the memory gets freed as soon as array is deleted (because nobody holds a reference to the nanny anymore).
MemoryNanny doesn't have to guard a raw C-pointer. It can be anything else, for example also a std::vector:
%%cython -+
from libcpp.vector cimport vector
cdef class VectorNanny:
#automatically default initialized/destructed by Cython:
cdef vector[double] vec
#staticmethod
cdef create(vector[double]& vec):
cdef VectorNanny result = VectorNanny()
result.vec.swap(vec) # swap and not copy
return result
# for testing:
def create_vector(int N):
cdef vector[double] vec;
vec.resize(N, 2.0)
return VectorNanny.create(vec)
The following test shows, that the nanny works:
nanny=create_vector(10**8) # top shows additional 800MB memory are used
del nanny # top shows, this additional memory is no longer used.

The latest Cython version allows you to do with with minimal syntax, albeit slightly more overhead than the lower-level solutions suggested.
numpy_array = np.asarray(<np.int32_t[:10, :10]> my_pointer)
https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html#coercion-to-numpy
This alone does not pass ownership.
Notably, a Cython array is generated with this call, via array_cwrapper.
This generates a cython.array, without allocating memory. The cython.array uses the stdlib.h malloc and free by default, so it would be expected that you use the default malloc, as well, instead of any special CPython/Numpy allocators.
free is only called if ownership is set for this cython.array, which it is by default only if it allocates data. For our case, we can manually set it via:
my_cyarr.free_data = True
So to return a 1D array, it would be as simple as:
from cython.view cimport array as cvarray
# ...
cdef cvarray cvarr = <np.int32_t[:N]> data
cvarr.free_data = True
return np.asarray(cvarr)

Cython throws an error when pass function with void * argument to dict

I have a function with void * parameter and I want to store it in the dict.
What I do is:
%%cython
cdef void foo(void * bar):
pass
cdef dict foobar = {'foo': foo}
But this code raises an Error: Cannot convert 'void (void *)' to Python object Any ways to overcome this problem?

The easiest solution is to create a cdef class that can wrap this function. Since the cdef class is a Python object it can be stored in a dictionary like any other Python object.
ctypedef void (*void_func_ptr)(void*)
cdef class VoidFuncWrapper:
cdef void_func_ptr func
def __cinit__(self):
self.func = NULL
#staticmethod
cdef VoidFuncWrapper make_from_ptr(void_func_ptr f):
cdef VoidFuncWrapper out = VoidFuncWrapper()
out.func = f
return out
Then you can simply do:
cdef dict foobar = {'foo': VoidFuncWrapper.make_from_ptr(foo)}

Your problem here is that void* has no defined converter from a Python object, so a Python wrapper function (that accepts Python objects, converts them, and passes them to the underlying C function) can't be defined, either explicitly with cpdef or implicitly by putting the cdef function in a Python object (the dict in this case).
You could make this work by defining the argument to be something Cython knows how to convert to, e.g.:
cpdef void foo(const char * bar):
pass
cdef dict foobar = {'foo': foo}
Try it online!
But that may not work for your scenario if the function needs to accept an arbitrary pointer. If that's the case, you may want to switch to using C++ containers that can hold your function pointer type directly:
from libcpp.unordered_map cimport unordered_map
from libcpp.string cimport string
cdef void foo(void *bar):
pass
ctypedef void (*f_type)(void *)
cdef unordered_map[string, f_type] foobar
foobar['foo'.encode('ascii')] = foo

Cython - 'Abstract declarator' compiler warning

The situation is as follows: I want to wrap the method Unit.getDistance
//Unit.h
class Unit{
....
int getDistance(PositionOrUnit target) const;
};
//PositionUnit.h
class PositionOrUnit{
PositionOrUnit(Unit unit = nullptr);
PositionOrUnit(Position pos);
};
The library uses converting constructors to allow Unit or Position to be automatically constructed into a PositionOrUnit object by the compiler, so that it is possible to pass a Unit or Position object directly into this method.
#cUnit.pxd
ctypedef UnitInterface *Unit
cdef cppclass UnitInterface:
int getDistance(PositionOrUnit target) const
#cPositionUnit.pxd
cdef cppclass PositionOrUnit:
PositionOrUnit() #fake for nullptr unit
PositionOrUnit(Unit unit)
PositionOrUnit(Position pos)
Now I don't know how to make a converting constructor in Python, so I use subclass polymorphism to get this to work. I declare a base class PositionUnitConverter and have Unit and Position subclass from it.
#PositionUnit.pyx
cdef class PositionUnitConverter:
cdef cPositionUnit.PositionOrUnit getPositionOrUnit(self):
return cPositionUnit.PositionOrUnit()
cdef class PositionOrUnit(PositionUnitConverter):
cdef cPositionUnit.PositionOrUnit thisobj
cdef cPositionUnit.PositionOrUnit getPositionOrUnit(self):
return self.thisobj
#Unit.pyx
cdef class Unit(PositionUnitConverter):
cdef cUnit.Unit thisptr
cdef cPositionUnit.PositionOrUnit getPositionOrUnit(self):
return cPositionUnit.PositionOrUnit(self.thisptr)
def getDistance(self, PositionUnitConverter target):
return self.thisptr.getDistance(target.getPositionOrUnit())
The end result is that in my Python code I can call Unit.getDistance with either type of object still.
# this works
unit.getDistance(unit2)
But now I get several of these warnings:
CyBW\CyBW.cpp(15698) : warning C4190: 'abstract declarator' has C-linkage specified, but
returns UDT 'BWAPI::PositionOrUnit' which is incompatible with C
../include/BWAPI/PositionUnit.h(13) : see declaration of 'BWAPI::PositionOrUnit'
The .cpp code generated at that line is: (split to be easier to read)
__pyx_vtable_4CyBW_5BWAPI_PositionUnitConverter.getPositionOrUnit = (BWAPI::PositionOrUnit
(*)(struct __pyx_obj_4CyBW_5BWAPI_PositionUnitConverter *))
__pyx_f_4CyBW_5BWAPI_21PositionUnitConverter_getPositionOrUnit;
My Questions are:
Am I doing something wrong to get this warning
How can I avoid this warning?
or should I ignore this warning, and how?
If there is anything else I can provide to help, please comment.

Cython: overloaded constructor initialization using raw pointer [duplicate]

This question already has answers here:
Wrapping a pre-initialized pointer in a cython class
(3 answers)
Closed 1 year ago.
I'm trying to wrap two C++ classes: Cluster and ClusterTree. ClusterTree has a method get_current_cluster() that instantiates a Cluster object, and returns a reference to it. ClusterTree owns the Cluster object, and manages its creation and deletion in C++.
I've wrapped Cluster with cython, resulting in PyCluster.
PyCluster should have two ways of creation:
1) By passing in two arrays, which then implies that Python should then automatically handle deletion (via __dealloc__)
2) By directly passing in a raw C++ pointer (created by ClusterTree's get_current_cluster()). In this case, ClusterTree then assumes responsibility of deleting the underlying pointer.
from libcpp cimport bool
from libcpp.vector cimport vector
cdef extern from "../include/Cluster.h" namespace "Terran":
cdef cppclass Cluster:
Cluster(vector[vector[double]],vector[int]) except +
cdef class PyCluster:
cdef Cluster* __thisptr
__autoDelete = True
def __cinit__(self, vector[vector[double]] data, vector[int] period):
self.__thisptr = new Cluster(data, period)
#classmethod
def __constructFromRawPointer(self, raw_ptr):
self.__thisptr = raw_ptr
self.__autoDelete = False
def __dealloc__(self):
if self.__autoDelete:
del self.__thisptr
cdef extern from "../include/ClusterTree.h" namespace "Terran":
cdef cppclass ClusterTree:
ClusterTree(vector[vector[double]],vector[int]) except +
Cluster& getCurrentCluster()
cdef class PyClusterTree:
cdef ClusterTree *__thisptr
def __cinit__(self, vector[vector[double]] data, vector[int] period):
self.__thisptr = new ClusterTree(data,period)
def __dealloc__(self):
del self.__thisptr
def get_current_cluster(self):
cdef Cluster* ptr = &(self.__thisptr.getCurrentCluster())
return PyCluster.__constructFromRawPointer(ptr)
This results in:
Error compiling Cython file:
------------------------------------------------------------
...
def get_current_cluster(self):
cdef Cluster* ptr = &(self.__thisptr.getCurrentCluster())
return PyCluster.__constructFromRawPointer(ptr)
^
------------------------------------------------------------
terran.pyx:111:54: Cannot convert 'Cluster *' to Python object
Note I cannot cdef __init__ or #classmethods.

Pointers can only be passed to cdef'd functions as arguments, and cinit has to be def'd. But providing a classmethod is almost the way to go!
cdef Cluster* __thisptr
cdef bool __wrapped ## defaults to False
#staticmethod
cdef PyCluster wrap(Cluster* ptr):
cdef PyCluster pc = PyCluster([], []) ## Initialize as cheaply as possible
del pc.__thisptr ## delete the old pointer to avoid memory leaks!
pc.__thisptr = ptr
pc.__wrapped = True
return pc

I know this is an old question, but after my own recent struggles with Cython I thought I'd post an answer for the sake of posterity.
It seems to me you could use a copy constructor to create a new PyCluster object from an existing Cluster object.
Define the copy constructor in your C code, then call the copy constructor in the Python class definition (in this case, when a pointer is passed) using new. This will work, although it may not be the best or most performant solution.

Passing C++ pointer as argument into Cython function

cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
def __cinit__(self, Bar *b)
bar = b
This will always give me something like:
Cannot convert Python object argument to type 'Bar *'
Is there a way to accomplish this, or do I need to extract everything from a Bar object, create a Python equivalent, pass it in, then reconstruct it in PyClass?

I came across this problem trying to wrap C code with structs as python classes. The issue seems to be that "special" function including __init__ and __cinit__ must be declared as def rather than cdef. This means that they can be called from normal python, so the type parameters are effectively ignored and everything is treated as object.
In J.F. Sebastian's answer the fix is not the wrapping - a double is a basic numeric type and so there is a default conversion between the C/C++ type and the Python object. Czarek's answer is basically correct - you need to use a fake constructor idiom, using a global function. It is not possible to use a #staticmethod decorator as they cannot be applied to cdef functions. The answer looks simpler on the original example provided.
cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
cdef PyClass_Init(Bar *b):
result = PyClass()
result.bar = b
return result

As of Cython 0.21 it has been possible to declare cdef methods with the #staticmethod decorator. This allows static creator methods that take non-Python arguments:
cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
#staticmethod
cdef create(Bar *bar):
cdef PyClass pc = PyClass()
pc.bar = bar
return pc

For each cdef class create a global cdef function that acts as a constructor, CefResponse is a C++ object, PyResponse a python equivalent of a c++ object:
cdef object CreatePyResponse(CefRefPtr[CefResponse] cefResponse):
pyResponse = PyResponse()
pyResponse.cefResponse = cefResponse
return pyResponse
cdef class PyResponse:
cdef CefRefPtr[CefResponse] cefResponse
def GetStatus(self):
return (<CefResponse*>(self.cefResponse.get())).GetStatus()
So instead of resp = PyResponse(cppObject) call resp = CreatePyResponse(cppObject).
Example taken from CEF Python:
https://code.google.com/p/cefpython/source/browse/cefpython/response.pyx?r=0250b65e046a

Python class accepts Python arguments. To pass a C++ argument you need to wrap it:
# distutils: language = c++
cdef extern from "Foo.h" namespace "baz":
cdef cppclass Bar:
Bar(double d)
double get()
cdef class PyBar: # wrap Bar class
cdef Bar *thisptr
def __cinit__(self, double d):
self.thisptr = new Bar(d)
def __dealloc__(self):
del self.thisptr
property d:
def __get__(self):
return self.thisptr.get()
PyBar instances can be used as any other Python objects both from Cython and pure Python:
class PyClass:
def __init__(self, PyBar bar):
self.bar = bar
print(PyClass(PyBar(1)).bar.d)