A few SciPy functions (like scipy.ndimage.interpolation.geometric_transform) can take pointers to C functions as arguments to avoid having to call a Python callable on each point of the input array.
In a nutshell :
Define a function called my_function somewhere in the C module
Return a PyCObject with the &my_function pointer and (optionally) a void* pointer to pass some global data around
The related API method is PyCObject_FromVoidPtrAndDesc, and you can read Extending ndimage in C to see it in action.
I am very interested in using Cython to keep my code more manageable, but I'm not sure how exactly I should create such an object. Any, well,... pointers?
Just do in Cython the same thing you would do in C, call PyCObject_FromVoidPtrAndDesc directly. Here is an example from your link ported to Cython:
###### example.pyx ######
from libc.stdlib cimport malloc, free
from cpython.cobject cimport PyCObject_FromVoidPtrAndDesc
cdef int _shift_function(int *output_coordinates, double* input_coordinates,
int output_rank, int input_rank, double *shift_data):
cdef double shift = shift_data[0]
cdef int ii
for ii in range(input_rank):
input_coordinates[ii] = output_coordinates[ii] - shift
return 1
cdef void _shift_destructor(void* cobject, void *shift_data):
free(shift_data)
def shift_function(double shift):
"""This is the function callable from python."""
cdef double* shift_data = <double*>malloc(sizeof(shift))
shift_data[0] = shift
return PyCObject_FromVoidPtrAndDesc(&_shift_function,
shift_data,
&_shift_destructor)
Performance should be identical to pure C version.
Note that Cyhton requires operator & to get function address. Also, Cython lacks pointer dereference operator *, indexing equivalent is used instead (*ptr -> ptr[0]).
I think that is a bad idea. Cython was created to avoid writing PyObjects also! Moreover, in this case, writing the code through Cython probably doesn't improve code maintenance...
Anyway, you can import the PyObject with
from cpython.ref cimport PyObject
in your Cython code.
UPDATE
from cpython cimport *
is safer.
Cheers,
Davide
Related
Say we have a class in cython that wraps (via a pointer) a C++ class with unknown/variable size in memory:
//poly.h
class Poly{
std::vector[int] v
// [...] Methods to initialize/add/multiply/... coefficients [...] e.g.,
Poly(int len, int val){for (int i=0; i<len; i++){this->v.push_back(val)};};
void add(Poly& p) {for (int i=0; i<this->v.size();i++){this->v[i] += p->v[i];};};
};
We can conveniently expose operations like add in PyPoly using operator overloads (e.g., __add__/__iadd__):
cdef extern from "poly.h":
cdef cppclass Poly:
Poly(int len, int val)
void add(Poly& p)
#pywrapper.pyx
cdef class PyPoly
cdef Poly* c_poly
cdef __cinit__(self, int l, int val):
self.c_poly = new Poly(l, val)
cdef __dealloc__(self):
del self.c_poly
def __add__(self, PyPoly other):
new_poly = PyPoly(self.c_poly.size(), 0)
new_poly.c_poly.add(self.c_poly)
new_poly.c_poly.add(other.c_poly)
return new_poly
How to create an efficient 1D numpy array with this cdef class?
The naive way I'm using so far involves a np.ndarray of type object, which benefits from the existing operator overloads:
pypoly_arr = np.array([PyPoly(l=10, val) for val in range(10)])
pypoly_sum = np.sum(pypoly_arr) # Works thanks to implemented PyPoly.__add__
However, the above solution has to go through python code to understand the data type and the proper way to deal with __add__, which becomes quite cumbersome for big array sizes.
Inspired by https://stackoverflow.com/a/45150611/9670056, I tried with an array wrapper of my own, but I'm not sure how to create a vector[PyPoly], whether I should do it or instead just hold a vector of borrowed references vector[Poly*], so that the call to np.sum could be treated (and paralellized) at C++ level.
Any help/suggestions will be highly appreciated! (specially to rework the question/examples to make it as generic as possible & runnable)
This is not possible to do that in Cython. Indeed, Numpy does not support native Cython classes as a data type. The reason is that the Numpy code is written in C and it already compiled when your Cython code is compiled. This means Numpy cannot directly use your native type. It has to do an indirection and this indirection is made possible through the object CPython type which has the downside of being slow (mainly because of the actual indirection but also a bit because of CPython compiler overheads). Cython do not reimplement Numpy primitives as it would be a huge work. Numpy only supports a restricted predefined set of data types. It supports custom user types such types are not as powerful as CPython classes (eg. you cannot reimplement custom operators on items like you did).
Just-in-time (JIT) compiler modules like Numba can theoretically supports this because they reimplement Numpy and generate a code at runtime. However, the support of JIT classes in Numba is experimental and AFAIK array of JIT classes are not yet supported.
Note that you do not need to build an array in this case. A basic loop is faster and use less memory. Something (untested) like:
cdef int val
cdef PyPoly pypoly_sum
pypoly_sum = PyPoly(l=10, 0)
for val in range(1, 10):
pypoly_sum += PyPoly(l=10, val)
I'm trying to implement a simple code in cython using Jupyter notebook (I use python 2) and using gmp arithmetic in order to handle very large integers. I'm not a gmp/cython expert. My question is : how do I print the value a in the function fib().
The following code returns {}.
As fas as I can understand it has to do with stdout. For instance I tried gmp_printf and it didn't work.
%%cython --link-args=-lgmp
cdef extern from "gmp.h":
ctypedef struct mpz_t:
pass
cdef void mpz_init(mpz_t)
cdef void mpz_init_set_ui(mpz_t, unsigned int)
cdef void mpz_add(mpz_t, mpz_t, mpz_t)
cdef void mpz_sub(mpz_t, mpz_t, mpz_t)
cdef void mpz_add_ui(mpz_t, const mpz_t, unsigned long int)
cdef void mpz_set(mpz_t, mpz_t)
cdef void mpz_clear(mpz_t)
cdef unsigned long int mpz_get_ui(mpz_t)
cdef void mpz_set_ui(mpz_t, unsigned long int)
cdef int gmp_printf (const char*, ...)
cdef size_t mpz_out_str (FILE , int , const mpz_t)
def fib(unsigned long int n):
cdef mpz_t a,b
mpz_init(a)
mpz_init(b)
mpz_init_set_ui(a,1)
mpz_init_set_ui(b,1)
cdef int i
for i in range(n):
mpz_add(a,a,b)
mpz_sub(b,a,b)
return a
And the result
fib(10)
{}
If I use return mpz_get_ui(a) instead of return a
the code is working fine, but this is not the thing I really want (to get a long integer).
EDIT.
I compared the previous code with another one again in cython but not using mpz.
%%cython
def pyfib(unsigned long int n):
a,b=1,1
for i in range(n):
a=a+b
b=a-b
return a
and finally the same code but using mpz from gmpy2
%%cython
import gmpy2
from gmpy2 import mpz
def pyfib_with_gmpy2(unsigned long int n):
cdef int i
a,b=mpz(1),mpz(1)
for i in range(n):
a=a+b
b=a-b
return a
Then
timeit fib(700000)
1 loops, best of 3: 3.19 s per loop
and
timeit pyfib(700000)
1 loops, best of 3: 11 s per loop
and
timeit pyfib_with_gmpy2(700000)
1 loops, best of 3: 3.28 s per loop
(Answer mostly summarises a bunch of comments)
The immediate issue you were having was that Python has no real way to handle C structs. To get around this, Cython tries to convert structs to dictionaries when they are passed to Python (if possible). In this particular case, mpz_t is treated as "opaque" by C (and thus Cython) so you aren't supposed to know about its members. Therefore Cython "helpfully" converts it to an empty dictionary (a correct representation of all the members it knows about).
For an immediate fix I suggested using the gmpy library, which is an existing Python/Cython wrapping of GMP. This is probably a better choice than repeating the effort to wrap it.
As a general solution to this sort of problem there are two obvious options.
You could create a cdef wrapper class. The documentation I have linked is for C++, but the idea could be applied to C as well (with new/'del' replaced with 'malloc'/'free'). This is ultimately a Python class (so can be returned from Cython to Python) but contains a C struct, which you can manipulate directly in Cython. The approach is pretty well documented and doesn't need repeating here.
You could convert the mpz_t back to a Python integer at the end of the function. I feel this makes most sense, since ultimately they represent the same thing. The code shown below is a rough outline and hasn't been tested (I don't have gmp installed):
cdef mpz_to_py_int(mpz_t x):
# get bytes that describe the integer
cdef const mp_limb_t* x_data = mpz_limbs_read(x)
# view as a unsigned char* (i.e. as bytes)
cdef unsigned char* x_data_bytes = <unsigned char*>x_data
# cast to a memoryview then pass that to the int classmethod "from_bytes"
# assuming big endian (python 3.2+ required)
out = int.from_bytes(<unsigned char[:mpz_size(x):1]>x_data_bytes,'big')
# correct using sign
if mpz_sign(x) < 0:
return -out
else
return out
I'm trying to wrap a parallel sort written in c++ as a template, to use it with numpy arrays of any numeric type. I'm trying to use Cython to do this.
My problem is that I don't know how to pass a pointer to the numpy array data (of a correct type) to a c++ template. I believe I should use fused dtypes for this, but I don't quite understand how.
The code in .pyx file is below
# importing c++ template
cdef extern from "test.cpp":
void inPlaceParallelSort[T](T* arrayPointer,int arrayLength)
def sortNumpyArray(np.ndarray a):
# This obviously will not work, but I don't know how to make it work.
inPlaceParallelSort(a.data, len(a))
In the past I did similar tasks with ugly for-loops over all possible dtypes, but I believe there should be a better way to do this.
Yes, you want to use a fused type to have Cython call the sorting template for the appropriate specialization of the template.
Here's a working example for all non-complex data types that does this with std::sort.
# cython: wraparound = False
# cython: boundscheck = False
cimport cython
cdef extern from "<algorithm>" namespace "std":
cdef void sort[T](T first, T last) nogil
ctypedef fused real:
cython.char
cython.uchar
cython.short
cython.ushort
cython.int
cython.uint
cython.long
cython.ulong
cython.longlong
cython.ulonglong
cython.float
cython.double
cpdef void npy_sort(real[:] a) nogil:
sort(&a[0], &a[a.shape[0]-1])
Maybe I am understanding the cdef for function definition incorrectly. E.g., assume I want to write a function to convert a Python list to a C array:
%%cython
cimport cython
from libc.stdlib cimport malloc, free
cdef int* list_to_array(a_list):
""" Converts a Python list into a C array """
cdef int *c_array
cdef int count, n
c_array = <int *>malloc(len(a_list)*cython.sizeof(int))
count = len(a_list)
for i in range(count):
c_array[i] = a_list[i]
return c_array
when I call the function now via
list_to_array([1,2,3])
I get a
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-32-8f3f777d7883> in <module>()
----> 1 list_to_array([1,2,3])
NameError: name 'list_to_array' is not defined
However, when I just use the def, the function can be called (although it doesn't return what I want, it is just for illustrating my problem...)
%%cython
cimport cython
from libc.stdlib cimport malloc, free
def list_to_array1(a_list):
""" Converts a Python list into a C array """
cdef int *c_array
cdef int count, n
c_array = <int *>malloc(len(a_list)*cython.sizeof(int))
count = len(a_list)
for i in range(count):
c_array[i] = a_list[i]
return 1
list_to_array1([1,2,3])
1
When I tried to use cpdef instead of cdef, I encounter a different issue:
Error compiling Cython file:
------------------------------------------------------------
...
cimport cython
from libc.stdlib cimport malloc, free
cpdef int* list_to_carray(a_list):
^
------------------------------------------------------------
/Users/sebastian/.ipython/cython/_cython_magic_c979dc7a52cdfb492e901a4b337ed2d2.pyx:3:6: Cannot convert 'int *' to Python object
Citing the docs, "The cdef statement is used to make C level declarations"
Then if you scroll a bit down here, you see that cdef functions can not be called from python, hence your NameError. Try using cpdef.
Note that if you plan to use that function in python code it will leak memory. You may also want to have a look at this answer on how/why you should return pass a list to/from cython (disclaimer: the answer is mine) to avoid the leakage.
EDIT, in reply to updated question:
The error once you introduce cpdef happens because a pointer cannot be converted to a python object in a trivial way. Cython does the hard work for you in the simplest cases, see here. The question you should ask here is why you want to return a C pointer to the python environment, that does not provide pointers.
I'm working in with a C library that repeatedly calls a user supplied function pointer to get more data. I'd like to write a Cython wrapper in such a way that the Python implementation of that callback can return any reasonable data type like str, bytearray, memory mapped files, and so on (specifically, supports the Buffer interface). what I have so far is:
from cpython.buffer cimport PyBUF_SIMPLE
from cpython.buffer cimport Py_buffer
from cpython.buffer cimport PyObject_GetBuffer
from cpython.buffer cimport PyBuffer_Release
from libc.string cimport memmove
cdef class _callback:
cdef public object callback
cdef public object data
cdef uint16_t GetDataCallback(void * userdata,
uint32_t wantlen, unsigned char * data,
uint32_t * gotlen):
cdef Py_buffer gotdata
box = <_callback> userdata
gotdata_object = box.callback(box.data, wantlen)
if not PyObject_CheckBuffer(gotdata_object):
# sulk
return 1
try:
PyObject_GetBuffer(gotdata_object, &gotdata, PyBUF_SIMPLE)
if not (0 < gotdata.len <= wantlen):
# sulk
return 1
memmove(data, gotdata.buf, gotdata.len)
return 0
finally:
PyBuffer_Release(&gotdata)
The code I want to write would produce equivalent C code, but look like this:
from somewhere cimport something
from libc.string cimport memmove
cdef class _callback:
cdef public object callback
cdef public object data
cdef uint16_t GetDataCallback(void * userdata,
uint32_t wantlen, unsigned char * data,
uint32_t * gotlen):
cdef something gotdata
box = <_callback> userdata
gotdata = box.callback(box.data, wantlen)
if not (0 < gotdata.len <= wantlen):
# sulk
return 1
memmove(data, gotdata.buf, gotdata.len)
return 0
The generated C code looks like what I think it should be doing; but this seems like digging around in the Python API unnecessarily. Does Cython provide a nicer syntax to achieve this effect?
If you want to support everything that implements every variation of the new-style or old-style buffer interface, then you have to use the C API.
But if you don't care about old-style buffers, you can almost always use a memoryview:
Cython memoryviews support nearly all objects exporting the interface of Python new style buffers. This is the buffer interface described in PEP 3118. NumPy arrays support this interface, as do Cython arrays. The “nearly all” is because the Python buffer interface allows the elements in the data array to themselves be pointers; Cython memoryviews do not yet support this.
This of course includes str (or, in 3.x, bytes), bytearray, etc—if you followed the link, you may notice that it links to the same page to explain what it supports that you linked to explain what you want to support.
For 1D arrays of characters (like str), it's:
cdef char [:] gotdata