I have a list of numpy.ndarrays (with different length) in python and need to have very fast access to those in python. I think an array of pointers would do the trick. I tried:
float_type_t* list_of_arrays[no_of_arrays]
for data_array in python_list_of_arrays:
list_of_arrays[0] = data_array
But cython complains:
no_of_arrays < Not allowed in a constant expression
I have tried several ways to constify this variable:
cdef extern from *:
ctypedef int const_int "const int"
(there have been more creative attempts) - however it unfortunatley doesn't work.
Please help.
Why don't you use a numpy object array instead of a list of arrays?
I think the problem you're having is because you are declaring list_of_arrays in the stack, and its size must be known at compile-time. You can try some dynamic allocation, like this:
from libc.stdlib cimport malloc, free
cdef float_type_t *list_of_arrays = \
<float_type_t *>malloc(no_of_arrays * sizeof(float_type_t*))
for i in range(no_of_arrays):
list_of_arrays[i] = &(data_array[i].data)
# don't forget to free list_of_arrays!
(This assumes data_array is a numpy array.)
But this is still guessing a bit what you want to do.
Related
Is it possible to write a Cython function where a numpy array is passed by reference (perhaps a memory view?) and can I use such function in Python?
I tried:
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2]& sums):
...
...
Here counter and sums would be passed by reference. I am interested in the latter (but will happily accept critics and suggestions regarding both).
The compiler throws:
Reference base type cannot be a Python object
I get the same message with cdef either and I don't understand the full etiology of the problem.
Yes - sort of.
A Numpy array is a Python object. All Python objects are passed "by reference" anyway (i.e. if you change the contents of a Numpy array inside the function you change it outside).
Therefore you just do
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2] sums):
Specifying it as a C++ reference doesn't really make much sense. For a Python object there's quite a bit of detail that Cython hides from you.
However, cpdef functions are designed to be callable from Python and Cython. Python ints (and other numeric types) are immutable. Therefore changing counter will not work as you think when called from Python, and if it did what you hoped it'd potentially break the interpreter.
I'd therefore avoid using references in cpdef functions (and probably mostly avoid cpdef functions completely - it's usually better just to decide on def or cdef)
I'm trying to send float16 data to an Nvidia P100 card from some Cython code. When I was using float32, I could define my types in Cython like so:
DTYPE = np.float32
ctypedef np.float32_t DTYPE_t
cdef np.ndarray[DTYPE_t, ndim=2] mat = np.empty((100, 100), dtype=DTYPE)
But Cython can't find a defined type for np.float16_t, so I can't just replace the 32 with 16. If I try to provide another type that takes the same amount of space, like np.uint16_t, I get errors like:
Does not understand character buffer dtype format string ('e')
When I google, all I can find is a thread from 2011 about people trying to figure out how to support it...surely there must be a solution by now?
I think the answer is "sort of, but it's a reasonable amount of work if you want to do any real calculations".
The basic problem is that C doesn't support a 16-bit float type on most PCs (because the processor instructions don't exist). Therefore, what numpy has done is typedefed a 16-bit unsigned int to store the 16-bit float, and then written a set of functions to convert that to/from the supported float types. Any calculations using np.float16 are actually done on 32-bit or 64-bit floats but the data is stored in the 16-bit format between calculations.
The consequence of this is that Cython doesn't have an easy way of generating valid C code for any calculations it needs to do. The consequence is that you may need to write out this C code yourself.
There are a number of levels of complexity, depending on what you actually want to do:
1) Don't type anything
Cython doesn't actually need you to specify any types - it compiles Python code happy. Therefore, don't assign types to the half-float arrays and just let them by Python objects. This may not be terribly fast but it's worth remembering that it will work.
2) To move data you can view it as uint16
If you're just shuffling data around then can define uint16 arrays and use those to copy it from one place to another. Use the numpy view function to get the data in a format that Cython recognises and to get it back. However, you can't do maths in this mode (the answers will be meaningless).
from libc.stdint cimport uint16_t
import numpy as np
def just_move_stuff(x):
assert x.dtype == np.float16
# I've used memoryviews by cdef np.ndarray should be fine too
cdef uint16_t[:] x_as_uint = x.view(np.uint16)
cdef uint16_t[:] y_as_uint = np.empty(x.shape,dtype=np.uint16)
for n in range(x_as_uint.shape[0]):
y_as_uint[n] = x_as_uint[n]
return np.asarray(y_as_uint).view(dtype=np.float16)
The view function doesn't make a copy so is pretty cheap to use.
3) Do maths with manual conversions
If you want to do any calculations you'll need to use numpy's conversion functions to change your "half-float" data to full floats and back. If you forget to do this the answers you get will be meaningless. Start by including them from numpy/halffloat.h:
cdef extern from "numpy/halffloat.h":
ctypedef uint16_t npy_half
# conversion functions
float npy_half_to_float(npy_half h);
npy_half npy_float_to_half(float f);
def do_some_maths(x):
assert x.dtype == np.float16
cdef uint16_t[:] x_as_uint = x.view(np.uint16)
cdef uint16_t[:] y_as_uint = np.empty(x.shape,dtype=np.uint16)
for n in range(x_as_uint.shape[0]):
y_as_uint[n] = npy_float_to_half(2*npy_half_to_float(x_as_uint[n]))
return np.asarray(y_as_uint).view(dtype=np.float16)
This code requires you to link against the numpy core math library:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from numpy.distutils.misc_util import get_info
info = get_info('npymath')
ext_modules = [Extension("module_name", ["module_name.pyx"],**info)]
setup(
ext_modules = cythonize(ext_modules)
)
I am trying to speed up a Python script. I have profiled the code and re-factored quite a lot already in pure Python. It seems that I am still spending a lot of time in accessing some numpy arrays in a way that looks like:
KeyArray[BoolArray[index]]
where KeyArray is ndim=2 and contains strings, BoolArray is ndim=1 and contains bool and index is an int.
I am trying to learn Cython to see how faster it could be. I wrote the following script that does not work:
import numpy as np
cimport numpy as np
def fastindexer(np.ndarray[np.str_t,ndim=1] KeyArray, np.ndarray [np.bool_t,ndim=2] BoolArray, np.int_t DateIndex):
cdef np.ndarray[np.str_t,ndim=1] FArray = KeyArray[BoolArray[DateIndex]]
return FArray
I understand that types str/bool are not available 'as is' in np arrays. I tried to cast as well but I don't understand how this should be written.
All help welcome
As #Joe said, moving a single indexing statement to Cython won't give you speed. If you decide to move more of your program to Cython, you need to fix a number of problems.
1) You use def instead of cdef, limiting you to Python-only functionality.
2) You use the old buffer syntax. Read about memoryviews
3) Slicing a 2-D array is slow because a new memoryview is created each time. That said, it still is a lot faster than Python, but for peak performance you would have to use a different approach.
Heres something to get you started.
cpdef func():
cdef int i
cdef bool[:] my_bool_array = np.zeros(10, dtype=bool)
# I'm not if this next line is correct
cdef char[:,:] my_string_array = np.chararray((10, 10))
cdef char answer
for i in range(10):
answer = my_string_array[ my_bool_array[i] ]
I have a function in c++
myfun(double* array, int n1, int n2);
that I interface with numpy in python. In my interface file I have
%apply (double* INPLACE_FARRAY2, int DIM1, int DIM2) {(double* inarray, int n1, int n2)}
Now, I want to pass an array b = array([[3,27.0],[2,9],[10,1]],order='F') into myfun within python, but I get the following error
TypeError: Array must be contiguous. A non-contiguous array was given.
What am I doing wrong? Is the double data type in my %apply statement incorrect?
Fortran order (order='F') may be the issue, since this is opposite the C order, I'm not sure because I'm unable to find a clear definition of continguous vs non for numpy. So it may be worth trying:
b = array([[3,27.0],[2,9],[10,1]],order='C')
Also, it may be worth trying
myfun( numpy.ascontiguousarray(b) )
I found the numpy swig reference doc useful to understand the difference between INPLACE_ and IN_ typemaps. Basically the latter is used when c function will only read and not write, as long input is a sequence of some sort the SWIG code will be able to handle it. The former indicates that the c function will write to the array so the ordering in original container is must match.
I want to create a signal processing algorithm that needs to hold some internal state in a numpy array.
For speed, I coded that in cython and declared the state a global variable like this:
import numpy as np
cimport numpy as np
cdef np.ndarray delay_buffer
However, what I really would like to do is this:
import numpy as np
cimport numpy as np
DTYPE = np.float32
ctypedef np.float32_t DTYPE_t
cdef np.ndarray[DTYPE_t] delay_buffer
This I can do anyhwere else, but not in the global scope. Is there any way to accomplish this?
Is there any way to accomplish this?
No. As the error says, Buffer types only allowed as function local variables.
One alternative is to use a monolithic main function. This really only takes indenting everything but it means that you can only share so much.
My favourite alternative would be to upgrade to the modern method of using memoryviews:
cdef DTYPE_t[:] delay_buffer
The should be faster, cleaner and no less powerful.