We Keep Coding

Cython: efficient custom numpy 1D array for cdef class

Say we have a class in cython that wraps (via a pointer) a C++ class with unknown/variable size in memory:
//poly.h
class Poly{
std::vector[int] v
// [...] Methods to initialize/add/multiply/... coefficients [...] e.g.,
Poly(int len, int val){for (int i=0; i<len; i++){this->v.push_back(val)};};
void add(Poly& p) {for (int i=0; i<this->v.size();i++){this->v[i] += p->v[i];};};
};
We can conveniently expose operations like add in PyPoly using operator overloads (e.g., __add__/__iadd__):
cdef extern from "poly.h":
cdef cppclass Poly:
Poly(int len, int val)
void add(Poly& p)
#pywrapper.pyx
cdef class PyPoly
cdef Poly* c_poly
cdef __cinit__(self, int l, int val):
self.c_poly = new Poly(l, val)
cdef __dealloc__(self):
del self.c_poly
def __add__(self, PyPoly other):
new_poly = PyPoly(self.c_poly.size(), 0)
new_poly.c_poly.add(self.c_poly)
new_poly.c_poly.add(other.c_poly)
return new_poly
How to create an efficient 1D numpy array with this cdef class?
The naive way I'm using so far involves a np.ndarray of type object, which benefits from the existing operator overloads:
pypoly_arr = np.array([PyPoly(l=10, val) for val in range(10)])
pypoly_sum = np.sum(pypoly_arr) # Works thanks to implemented PyPoly.__add__
However, the above solution has to go through python code to understand the data type and the proper way to deal with __add__, which becomes quite cumbersome for big array sizes.
Inspired by https://stackoverflow.com/a/45150611/9670056, I tried with an array wrapper of my own, but I'm not sure how to create a vector[PyPoly], whether I should do it or instead just hold a vector of borrowed references vector[Poly*], so that the call to np.sum could be treated (and paralellized) at C++ level.
Any help/suggestions will be highly appreciated! (specially to rework the question/examples to make it as generic as possible & runnable)

This is not possible to do that in Cython. Indeed, Numpy does not support native Cython classes as a data type. The reason is that the Numpy code is written in C and it already compiled when your Cython code is compiled. This means Numpy cannot directly use your native type. It has to do an indirection and this indirection is made possible through the object CPython type which has the downside of being slow (mainly because of the actual indirection but also a bit because of CPython compiler overheads). Cython do not reimplement Numpy primitives as it would be a huge work. Numpy only supports a restricted predefined set of data types. It supports custom user types such types are not as powerful as CPython classes (eg. you cannot reimplement custom operators on items like you did).
Just-in-time (JIT) compiler modules like Numba can theoretically supports this because they reimplement Numpy and generate a code at runtime. However, the support of JIT classes in Numba is experimental and AFAIK array of JIT classes are not yet supported.
Note that you do not need to build an array in this case. A basic loop is faster and use less memory. Something (untested) like:
cdef int val
cdef PyPoly pypoly_sum
pypoly_sum = PyPoly(l=10, 0)
for val in range(1, 10):
pypoly_sum += PyPoly(l=10, val)

Wrapping a function returning a Vector of Complex types in Cython

I'm currently trying to wrap a C++ class I have created. One of the functions returns a vector of vectors which contain complex floats:
std::vector<std::vector<std::complex<float>>>
My DataBridge.pxd file is shown below:
# distutils: language = c++
from libcpp cimport bool
from libcpp cimport float
from libcpp.vector cimport vector
from libcpp.complex cimport complex
from libc.float cimport float
cimport numpy as np
import numpy as np
cdef extern from "projectxcpp/bridge.cpp":
pass
# Declare the class with cdef
cdef extern from "projectxcpp/bridge.hpp":
cdef cppclass DataBridge:
DataBridge() except +
void start()
void stop()
bool isDataReady()
vector[vector[complex[float]]] getData()
My bridge.hpp file is shown below:
#ifndef BRIDGE_HPP
#define BRIDGE_HPP
#include <vector>
#include <complex>
...
class DataBridge {
public:
int start(void);
int stop(void);
bool isDataReady(void);
std::vector<std::vector<std::complex<float>>> getData(void);
private:
...
};
#endif
Then, trying to run the setup.py, the following message is given:
[1/1] Cythonizing snowconecpp.pyx
Error compiling Cython file:
------------------------------------------------------------
...
DataBridge() except +
void start()
void stop()
bool isDataReady()
vector[vector[complex[float]]] getData()
^
------------------------------------------------------------
DataBridge.pxd:24:29: Array size must be a compile time constant
Error compiling Cython file:
------------------------------------------------------------
...
DataBridge() except +
void start()
void stop()
bool isDataReady()
vector[vector[complex[float]]] getData()
^
------------------------------------------------------------
DataBridge.pxd:24:29: unknown type in template argument
Traceback (most recent call last):
File "setup.py", line 15, in <module>
setup(ext_modules=cythonize("projectxcpp.pyx", language_level="3"))
File ".../lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1096, in cythonize
cythonize_one(*args)
File ".../lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1219, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: projectxcpp.pyx
My setup.py file contains simply:
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules=cythonize("projectxcpp.pyx", language_level="3"))
When declaring types in my wrapper as either complex[float] e or vector[vector[float]] e, these will parse and move to the next step of compilation.
Just wondering if there's a special requirement for putting a complex type within vectors for Cython - cannot seem to find answers for this or similar situations where C++ wrapping is involved.
My end goal is to be able to read this data out into a numpy array. If there is a better way of going about this, please make me aware of it too - Thank you!

I think the issue is that complex is already a type in Python/Cython (and one that's slightly badly handled in places too, due to the overlap with C or C++ complex...). It works if you either rename it, or address it by it's full path
from libcpp.complex cimport complex as cpp_complex
# later
vector[vector[cpp_complex[float]]]
or
cimport libcpp.complex
# later
vector[vector[libcpp.complex.complex[float]]]
For the more general part of the question: my preference would be not to provide the data for a Numpy array like that. Part of this is that I really dislike 2D arrays being defined as pointer-to-pointer or vector-of-vector. You're better off using a 1D array/vector stored alongside shape information that lets you index it.
There's a few options for interfacing with Numpy:
If you know the size in advance then allocate the memory with Numpy and pass that memory into C++ (you can get a pointer to the first element of a Numpy array). This way the C++ code modifies Numpy's memory and you don't have to copy the data.
If you want to allocate the memory with C++ then you can wrap it with a Cython class that implements the buffer protocol. The memory is managed by C++, but Numpy can directly access the memory without copying. (You may need to implement a move constructor to avoid copying, but that is fairly straightforward).
You can use Numpy C API functions PyArray_SimpleNewFromData to get Numpy to wrap the memory. This requires care to ensure that it's freed at the right time.
You can do what you're doing currently and copy the data over. This is very simple to implement, but does add a copy operation every time you want to move data from C++ to Python.

How to use #define as size of char array in Cython

c++ header (some.h) contains:
#define NAME_SIZE 42
struct s_X{
char name[NAME_SIZE + 1]
} X;
I want to use X structure in Python. How could I make it?
I write:
cdef extern from "some.h":
cdef int NAME_SIZE # 42
ctypedef struct X:
char name[NAME_SIZE + 1]
And got an error: Not allowed in a constant expression

It often doesn't really matter what you tell Cython when declaring types - it uses the information for checking you aren't doing anything obviously wrong with type casting and that's it. The cdef extern "some.h" statement ensures that some.h is included into to c-file Cython creates and ultimately that determines what is complied.
Therefore, in this particular case, you can just insert an arbitary number and it will work fine
cdef extern "some.h":
cdef int NAME_SIZE # 42
ctypedef struct X:
char name[2] # you can pick a number at random here
In situations it won't work though, especially where Cython has to actually use the number in the C code it generates. For example:
def some_function():
cdef char_array[NAME_SIZE+1] # won't work! Cython needs to know NAME_SIZE to generate the C code...
# other code follows
(I don't currently have a suggestion as to what to do in this case)

NAME_SIZE doesn't actually exist in your program so you'll probably have to hardcode it in the Python.
Despite how it looks in your C source code, you hardcoded it in the C array declaration, too.

Cython's `cdef` raises a NameError where a `def` works just fine

Maybe I am understanding the cdef for function definition incorrectly. E.g., assume I want to write a function to convert a Python list to a C array:
%%cython
cimport cython
from libc.stdlib cimport malloc, free
cdef int* list_to_array(a_list):
""" Converts a Python list into a C array """
cdef int *c_array
cdef int count, n
c_array = <int *>malloc(len(a_list)*cython.sizeof(int))
count = len(a_list)
for i in range(count):
c_array[i] = a_list[i]
return c_array
when I call the function now via
list_to_array([1,2,3])
I get a
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-32-8f3f777d7883> in <module>()
----> 1 list_to_array([1,2,3])
NameError: name 'list_to_array' is not defined
However, when I just use the def, the function can be called (although it doesn't return what I want, it is just for illustrating my problem...)
%%cython
cimport cython
from libc.stdlib cimport malloc, free
def list_to_array1(a_list):
""" Converts a Python list into a C array """
cdef int *c_array
cdef int count, n
c_array = <int *>malloc(len(a_list)*cython.sizeof(int))
count = len(a_list)
for i in range(count):
c_array[i] = a_list[i]
return 1
list_to_array1([1,2,3])
1
When I tried to use cpdef instead of cdef, I encounter a different issue:
Error compiling Cython file:
------------------------------------------------------------
...
cimport cython
from libc.stdlib cimport malloc, free
cpdef int* list_to_carray(a_list):
^
------------------------------------------------------------
/Users/sebastian/.ipython/cython/_cython_magic_c979dc7a52cdfb492e901a4b337ed2d2.pyx:3:6: Cannot convert 'int *' to Python object

Citing the docs, "The cdef statement is used to make C level declarations"
Then if you scroll a bit down here, you see that cdef functions can not be called from python, hence your NameError. Try using cpdef.
Note that if you plan to use that function in python code it will leak memory. You may also want to have a look at this answer on how/why you should return pass a list to/from cython (disclaimer: the answer is mine) to avoid the leakage.
EDIT, in reply to updated question:
The error once you introduce cpdef happens because a pointer cannot be converted to a python object in a trivial way. Cython does the hard work for you in the simplest cases, see here. The question you should ask here is why you want to return a C pointer to the python environment, that does not provide pointers.

Creating a PyCObject pointer in Cython

A few SciPy functions (like scipy.ndimage.interpolation.geometric_transform) can take pointers to C functions as arguments to avoid having to call a Python callable on each point of the input array.
In a nutshell :
Define a function called my_function somewhere in the C module
Return a PyCObject with the &my_function pointer and (optionally) a void* pointer to pass some global data around
The related API method is PyCObject_FromVoidPtrAndDesc, and you can read Extending ndimage in C to see it in action.
I am very interested in using Cython to keep my code more manageable, but I'm not sure how exactly I should create such an object. Any, well,... pointers?

Just do in Cython the same thing you would do in C, call PyCObject_FromVoidPtrAndDesc directly. Here is an example from your link ported to Cython:
###### example.pyx ######
from libc.stdlib cimport malloc, free
from cpython.cobject cimport PyCObject_FromVoidPtrAndDesc
cdef int _shift_function(int *output_coordinates, double* input_coordinates,
int output_rank, int input_rank, double *shift_data):
cdef double shift = shift_data[0]
cdef int ii
for ii in range(input_rank):
input_coordinates[ii] = output_coordinates[ii] - shift
return 1
cdef void _shift_destructor(void* cobject, void *shift_data):
free(shift_data)
def shift_function(double shift):
"""This is the function callable from python."""
cdef double* shift_data = <double*>malloc(sizeof(shift))
shift_data[0] = shift
return PyCObject_FromVoidPtrAndDesc(&_shift_function,
shift_data,
&_shift_destructor)
Performance should be identical to pure C version.
Note that Cyhton requires operator & to get function address. Also, Cython lacks pointer dereference operator *, indexing equivalent is used instead (*ptr -> ptr[0]).

I think that is a bad idea. Cython was created to avoid writing PyObjects also! Moreover, in this case, writing the code through Cython probably doesn't improve code maintenance...
Anyway, you can import the PyObject with
from cpython.ref cimport PyObject
in your Cython code.
UPDATE
from cpython cimport *
is safer.
Cheers,
Davide