I am working on speeding up some Python/Numpy code with Cython, and am a bit unclear on the effects of "locally" setting (as defined here in the docs) compiler directives. In my case I'd like to use:
#cython.wraparound (False) #turn off negative indexing
#cython.boundscheck(False) #turn off bounds-checking
I understand that I can globally define this in my setup.py file, but I'm developing for non-Cython users and would like the directives to be obvious from the .pyx file.
If I am writing a .pyx file with several functions defined in it, need I only set these once, or will they only apply to the next function defined? The reason I ask is that the documentation often says things like "turn off boundscheck for this function," making me wonder whether it only applies to the next function defined.
In other words, do I need to do this:
import numpy as np
cimport numpy as np
cimport cython
ctypedef np.float64_t DTYPE_FLOAT_t
#cython.wraparound (False) #turn off negative indexing
#cython.boundscheck(False) # turn off bounds-checking
def myfunc1(np.ndarray[DTYPE_FLOAT_t] a):
do things here
def myfunc2(np.ndarray[DTYPE_FLOAT_t] b):
do things here
Or do I need to do this:
import numpy as np
cimport numpy as np
cimport cython
ctypedef np.float64_t DTYPE_FLOAT_t
#cython.wraparound (False) #turn off negative indexing
#cython.boundscheck(False) # turn off bounds-checking
def myfunc1(np.ndarray[DTYPE_FLOAT_t] a):
do things here
#cython.wraparound (False) #turn off negative indexing
#cython.boundscheck(False) # turn off bounds-checking
def myfunc2(np.ndarray[DTYPE_FLOAT_t] b):
do things here
Thanks!
The documentation states that if you want to set a compiler directive globally, you need to do so with a comment at the top of the file. eg.
#!python
#cython: language_level=3, boundscheck=False
The manual does not say explicitly, but these directives are using decorator notation. In plain Python,
#decorator2
#decorator1
def fn(args):
body
is syntactic sugar for
def fn(args):
body
fn = decorator2(decorator1(fn))
So my expectation is that the directives work like that, which would mean that they apply only to the next function.
The Cython manual also says that the compiler directives can be used in with statements. What it doesn't say, unfortunately, is whether those can appear at top level:
with cython.wraparound(False), cython.boundscheck(False):
def myfunc1(np.ndarray[DTYPE_FLOAT_t] a):
do things here
def myfunc2(np.ndarray[DTYPE_FLOAT_t] b):
do things here
might be what you're looking for, or then again it might be a syntax error or a no-op. You're going to have to try it and see.
Related
In Cython when using numpy, what is the point of writing:
cimport numpy as np
import numpy as np
ctypedef np.int_t DTYPE_t
and then using DTYPE_t everywhere instead of just using np.int_t? Does the ctypedef actually do anything differently in the resulting code here?
You can read the notes from the docs for cython, reading the notes they explain the reason for the use of this notation and imports.
from __future__ import division
import numpy as np
# "cimport" is used to import special compile-time information
# about the numpy module (this is stored in a file numpy.pxd which is
# currently part of the Cython distribution).
cimport numpy as np
# We now need to fix a datatype for our arrays. I've used the variable
# DTYPE for this, which is assigned to the usual NumPy runtime
# type info object.
DTYPE = np.int
# "ctypedef" assigns a corresponding compile-time type to DTYPE_t. For
# every type in the numpy module there's a corresponding compile-time
# type with a _t-suffix.
ctypedef np.int_t DTYPE_t
# "def" can type its arguments but not have a return type. The type of the
# arguments for a "def" function is checked at run-time when entering the
# function.
#
# The arrays f, g and h is typed as "np.ndarray" instances. The only effect
# this has is to a) insert checks that the function arguments really are
# NumPy arrays, and b) make some attribute access like f.shape[0] much
# more efficient. (In this example this doesn't matter though.)
Let us have script foo.pyx with function in it:
def hello():
cdef int* i = <int *> malloc(sizeof(int))
i[0] = 1
trol(i)
print i
and script with function noo.pyx:
cdef trol(int * i):
i[0] = 42
the question is, how do I now import the trol function from file noo.pyx to foo.pyx, so I can use it in hello function.
This is only model example, but I think, that it illustrate the problem fair enough.
I tried simple
from noo import trol
but that throws "Cannot convert 'int *' to Python object"
Edit: I would better add, that this example will work just fine if I put both functions to same file.
This seems like something obvious to try, but did you try:
from noo cimport trol
If you use import instead of cimport, I think it will try to cast trol as a python function and generate the error you're getting.
The solution eventually was to create additional .pxd file, which something very similar to classic header .h file in C. It stores functions declarations and when cimport is called, it is in this file where it looks for functions and structures.
So to be specific, all I needed to do was to create file noo.pxd containing:
cdef trol(int * i)
and than we can simply cimport this function from foo.pyx by calling
from noo cimport trol
A few SciPy functions (like scipy.ndimage.interpolation.geometric_transform) can take pointers to C functions as arguments to avoid having to call a Python callable on each point of the input array.
In a nutshell :
Define a function called my_function somewhere in the C module
Return a PyCObject with the &my_function pointer and (optionally) a void* pointer to pass some global data around
The related API method is PyCObject_FromVoidPtrAndDesc, and you can read Extending ndimage in C to see it in action.
I am very interested in using Cython to keep my code more manageable, but I'm not sure how exactly I should create such an object. Any, well,... pointers?
Just do in Cython the same thing you would do in C, call PyCObject_FromVoidPtrAndDesc directly. Here is an example from your link ported to Cython:
###### example.pyx ######
from libc.stdlib cimport malloc, free
from cpython.cobject cimport PyCObject_FromVoidPtrAndDesc
cdef int _shift_function(int *output_coordinates, double* input_coordinates,
int output_rank, int input_rank, double *shift_data):
cdef double shift = shift_data[0]
cdef int ii
for ii in range(input_rank):
input_coordinates[ii] = output_coordinates[ii] - shift
return 1
cdef void _shift_destructor(void* cobject, void *shift_data):
free(shift_data)
def shift_function(double shift):
"""This is the function callable from python."""
cdef double* shift_data = <double*>malloc(sizeof(shift))
shift_data[0] = shift
return PyCObject_FromVoidPtrAndDesc(&_shift_function,
shift_data,
&_shift_destructor)
Performance should be identical to pure C version.
Note that Cyhton requires operator & to get function address. Also, Cython lacks pointer dereference operator *, indexing equivalent is used instead (*ptr -> ptr[0]).
I think that is a bad idea. Cython was created to avoid writing PyObjects also! Moreover, in this case, writing the code through Cython probably doesn't improve code maintenance...
Anyway, you can import the PyObject with
from cpython.ref cimport PyObject
in your Cython code.
UPDATE
from cpython cimport *
is safer.
Cheers,
Davide
I'm trying to port some python code to cython and I'm encountering some minor problems.
Below you see a code snippet (simplified example) of the code.
cimport numpy as np
cimport cython
#cython.boundscheck(False) # turn of bounds-checking for entire function
#cython.wraparound(False)
#cython.nonecheck(False)
def Interpolation(cells, int nmbcellsx):
cdef np.ndarray[float,ndim=1] celle
cdef int cellnonzero
cdef int i,l
for i in range(nmbcellsx):
celle = cells[i].e
cellnonzero = cells[i].nonzero
for l in range(cellnonzero):
celle[l] = celle[l] * celle[l]
I don't understand why the inner-most loop does not fully translate to C code (i.e. the last line, celle[l] = ...), see output from cython -a feedback:
What am I missing here?
Thanks a lot.
I finally realized that a simple "return 0" at the very end of the function solves this problem. However, this behaviour seems quite strange to me. Is this actually a bug?
I want to create a signal processing algorithm that needs to hold some internal state in a numpy array.
For speed, I coded that in cython and declared the state a global variable like this:
import numpy as np
cimport numpy as np
cdef np.ndarray delay_buffer
However, what I really would like to do is this:
import numpy as np
cimport numpy as np
DTYPE = np.float32
ctypedef np.float32_t DTYPE_t
cdef np.ndarray[DTYPE_t] delay_buffer
This I can do anyhwere else, but not in the global scope. Is there any way to accomplish this?
Is there any way to accomplish this?
No. As the error says, Buffer types only allowed as function local variables.
One alternative is to use a monolithic main function. This really only takes indenting everything but it means that you can only share so much.
My favourite alternative would be to upgrade to the modern method of using memoryviews:
cdef DTYPE_t[:] delay_buffer
The should be faster, cleaner and no less powerful.