For given ctype array or a python list, how does one cast python object to cython void ptr?
The way I'm doing it now is something like this(_arr is python list):
int *int_t = <int*> malloc(cython.sizeof(int) * len(_arr))
if int_t is NULL:
raise MemoryError()
for i in xrange(len(_arr)):
int_t[i] = _arr[i]
After this I have int_t, in which I have entire array. But I wan't the thing to be more general and support other types, not just int. Do I have to do the same thing for each type or is there any generic way in which this can be done?
First of all, you should be aware that numpy arrays support quite a range of data types. Here is a general introduction on how to work with numpy arrays in cython. If you want to do numerical stuff that should do the trick. Also, numpy does support arrays of Python objects. But then again, the array lookups are optimized but interaction with the objects is not, since they are still Python objects.
If you are thinking about doing this with arbitrary types, i.e. converting arbitrary Python objects into some kind of C type or C++ object, you have to do it manually for every type. And this makes sense, too, since automatic casting between a potentially dynamic data structure and a static one is not always obvious, especially not to a compiler.
Related
I am using ctypes to call function in C. Function expects pointer to the first element of the contiguous data and number of data.
One thing that works is something like that
a=15 # a could be any number
temp = numpy.array([a]*14, dtype=numpy.int8)
c_function(temp.ctypes.data_as(ctypes.c_void_p), 14)
This is really cumbersome, requires both numpy and ctypes. Is there any other more simple way that works both in Python2 and Python3 (AFAIK bytes([a]*14) works but only for Python3)
EDIT: More interestingly this also works (!)
a=15 # a could be any number
temp = chr(a)*14
c_function(temp, 14)
There were suggestions in other threads that one could pass something pointer to the first element of the contiguous data, like here Passing memoryview to C function, but I was just unable to make this work.
Preliminaries
Python does not have pointers. You cannot create a pointer in Python, though Python variables act in some ways like pointers. You can create Python objects that represent lower-level pointers, but what you actually seem to want is to feed your C function a pointer to Python-managed data, which is an altogether different thing.
What ctypes does for you
You seem to have settled on using ctypes for the actual function call, so the general question expressed in the question title is a little overbroad for what you actually want to know. The real question seems to be more like "How do I get ctypes to pass a C pointer to Python-managed data to a C function?"
According to the ctypes Python 2 docs, in Python 2,
None, integers, longs, byte strings and unicode strings are the only
native Python objects that can directly be used as parameters in these
function calls. None is passed as a C NULL pointer, byte strings and
unicode strings are passed as pointer to the memory block that
contains their data (char * or wchar_t *). [...]
(emphasis added).
It's more or less the same list in Python 3 ...
None, integers, bytes objects and (unicode) strings
... with the same semantics.
Note well that ctypes takes care of the conversion from Python object to corresponding C representation -- nowhere does Python code handle C pointers per se, nor even direct representations of them.
Relevant C details
In many C implementations, all object pointer types have the same representation and can be used semi-interchangeably, but pointers of type char * are guaranteed by the standard to have the same size and representation as pointers of type void *. These two pointer types are guranteed to be interchangeable as function parameters and return values, among other things.
Synthesis
How convenient! It is acceptable to call your C function with a first argument of type char * when the function declares that parameter to be of type void *, and that is exactly what ctypes will arrange for you when the Python argument is a byte string (Python 2) or a bytes object (Python 3). The C function will receive a pointer to the object's data, not to the object itself. This provides a simpler and better way forward than going through numpy or a similar package, and it is basically the approach that you appended to your question. Thus, supposing that c_function identifies a ctypes-wrapped C function, you could do this (python3):
len = 15
c_function(b'0' * len, len)
Of course, you can also create a variable for the object and pass that, instead, which would allow you to afterward see whatever the C function has done with the contents of the object.
Do note, however, that
Byte strings and bytes objects are immutable as far as Python is concerned. You can get yourself in trouble if you use a C function to change the contents of a bytes object that other Python code assumes will never change.
The C side cannot determine the size of the data from a pointer to it. That is presumably the purpose of the second parameter. If you tell the function that the object is larger than it really is, and the function relies on that to try to modify bytes past the end of the actual data, then you will have difficult to debug trouble, from corruption of other data to memory leaks. If you're lucky, your program will crash.
It depends on what Python implementation you use, but typically the elements of a Unicode string are larger than one byte each. Save yourself some trouble and use byte strings / bytes instead.
I would like to have a cython array of a cdef class:
cdef class Child:
cdef int i
def do(self):
self.i += 1
cdef class Mother:
cdef Child[:] array_of_child
def __init__(self):
for i in range(100):
self.array_of_child[i] = Child()
The answer is no - it is not really possible in a useful way: newsgroup post of essentially the same question
It wouldn't be possible to have a direct array (allocated in a single chunk) of Childs. Partly because, if somewhere else ever gets a reference to a Child in the array, that Child has to be kept alive (but not the whole array) which wouldn't be possible to ensure if they were all allocated in the same chunk of memory. Additionally, were the array to be resized (if this is a requirement) then it would invalidate any other references to the objects within the array.
Therefore you're left with having an array of pointers to Child. Such a structure would be fine, but internally would look almost exactly like a Python list (so there's really no benefit to doing something more complicated in Cython...).
There are a few sensible workarounds:
The workaround suggested in the newsgroup post is just to use a python list. You could also use a numpy array with dtype=object. If you need to to access a cdef function in the class you can do a cast first:
cdef Child c = <Child?>a[0] # omit the ? if you don't want
# the overhead of checking the type.
c.some_cdef_function()
Internally both these options are stored as an C array of PyObject pointers to your Child objects and so are not as inefficient as you probably assume.
A further possibility might be to store your data as a C struct (cdef struct ChildStruct: ....) which can be readily stored as an array. When you need a Python interface to that struct you can either define Child so it contains a copy of ChildStruct (but modifications won't propagate back to your original array), or a pointer to ChildStruct (but you need to be careful with ensuring that the memory is not freed which the Child pointing to it is alive).
You could use a Numpy structured array - this is pretty similar to using an array of C structs except Numpy handles the memory, and provides a Python interface.
The memoryview syntax in your question is valid: cdef Child[:] array_of_child. This can be initialized from a numpy array of dtype object:
array_of_child = np.array([(Child() for i in range(100)])
In terms of data-structure, this is an array of pointers (i.e. the same as a Python list, but can be multi-dimensional). It avoids the need for <Child> casting. The important thing it doesn't do is any kind of type-checking - if you feed an object that isn't Child into the array then it won't notice (because the underlying dtype is object), but will give nonsense answers or segmentation faults.
In my view this approach gives you a false sense of security about two things: first that you have made a more efficient data structure (you haven't, it's basically the same as a list); second that you have any kind of type safety. However, it does exist. (If you want to use memoryviews, e.g. for multi-dimensional arrays, it would probably be better to use a memoryview of type object - this is honest about the underlying dtype)
I am aware of this question, but I was looking for a simpler way to generate 2d memoryviews from C arrays. Since I am a C and Cython noobie, could someone please explain why something like
cdef int[:, :] get_zeros(int d):
# get 2-row array of zeros with d as second dimension
cdef int i
cdef int *arr = <int *> malloc(sizeof(int) * d)
for i in range(d):
arr[i] = 0
cdef int[:, :] arr_view
arr_view[0, :] = <int[:d]>arr
arr_view[1, :] = <int[:d]>arr
return arr_view
won't work?
When compiling it I get Cannot assign type 'int[::1]' to 'int' as error. Does this mean, that the 2d memview is collapsed by the first assign statement to 1d or is it because memoryviews need contiguous blocks etc.?
It's obviously quite hard to "explain why something [...] won't work", because ultimately it's just a design decision that could have been taken differently. But:
Cython memoryviews are designed to be pretty dumb. All they do is provide some nice syntax to access the memory of something that implements the Python buffer protocol, and then have a tiny bit of additional syntax to let you do things like get a 1D memoryview of a pointer.
Additionally, the memoryview as a whole wraps something. When you create cdef int[:, :] arr_view it's invalid until you do arr_view = something. Attempts to assign to part of it are nonsense, since (a) it'd delegate the assignment to the thing it wraps using the buffer protocol and (b) exactly how the assignment would work would depend on what format of buffer protocol you were wrapping. What you've done might be valid if wrapping an "indirect" buffer protocol object but would make no sense if wrapping a contiguous array. Since arr_view could be wrapping either the Cython compiler has to treat it as an error.
The question you link to implements the buffer protocol and so is the correct way to implement this kind of array. What you're attempting to do is to take the extra syntax that gives a 1D memoryview from a pointer and force that into part of a 2D memoryview in the vague hope that this might work. This requires a lot of logic that goes well beyond the scope of what a Cython memoryview was designed to do.
There's probably a couple of additional points worth making:
Memoryviews of pointers don't handle freeing of pointers (since it'd be pretty much impossible for them to second-guess what you want). You have to handle this logic. Your current design would leak memory, if it worked. In the design you linked to the wrapping class could implement this in __dealloc__ (although it isn't shown in that answer) and thus much better.
My personal view is that "ragged arrays" (2D arrays of pointers to pointers) are awful. They require a lot of allocation and deallocation. There's lots of opportunity to half-initialize them. Access to them requires a couple of levels of indirection and so is slow. The only thing going for them is that they provide a arr[idx1][idx2] syntax in C. In general I much prefer Numpy's approach of allocating a 1D array and using shape/strides to work out where to index. (Obviously if you're wrapping an existing library then you may not be your choice...)
In addition to the wonderful answer #DavidW has provided, I would like to add some more info. In your included code, I see that you are malloc-ing an array of ints and then zeroing out the contents in a for-loop. A more convenient way of accomplishing this is to use C's calloc function instead, which guarantees a pointer to zeroed memory and would not require a for loop afterwards.
Additionally, you could create a single int * that points to an "array" of data that is calloced to a total size of 2 * d * sizeof(int). This would ensure that both of the "rows" of data are contiguous with each other instead of separate and ragged. This could then be cast directly to a 2d memoryview.
As promised in the comments, here is what that conversion code could look like (with calloc use included):
cdef int[:, :] get_zeros(int d):
cdef int *arr = <int *>calloc(2 * d, sizeof(int))
cdef int[:, :] arr_view = <int[:2, :d]>arr
return arr_view
There also appears to be a calloc equivalent in the python c-api per the docs if you want to try it out. However, it does not appear to be wrapped in cython's mem.pxd module, which is why you were likely not able to find it. You could declare a similar extern block in your code to wrap it like the other functions included in that link.
And here is a bonus link if you want to know more about writing an allocator to dole out memory from a large block if you go the pre-allocation route (i.e. what PyMem_* functions likely do behind the scenes, but more tunable and under your control for your specific use case).
I was told the following extension type may not be very efficient due to the need for a Python object type-declaration for the use of DefaultDict. Can someone explain why this is, and whether DefaultDict could still be worth using (as opposed to dict)?
cdef class CythonClass(object):
cdef int var1, var2
cdef object defaultdict
def __init__(self, a, b):
self.var1 = a
self.var2 = b
self.defaultdict = DefaultDict(DefaultDict([]))
I may have overstated the efficiency part in my other answer. What I meant was: don't expect huge speedups (more than 1.5-2x) when you have to use Python objects.
You can use them and it won't be slower (except for very rare cases) than using them in python code. However the great power with Cython is that you can utilize native c types and homogeneous data structures like c-arrays (which can be much faster than python lists or even dictionaries) or if you go c++ then also vector, unordered_map and such.
One point to remember when dealing with Python objects is that all python objects are pointers to some structs, so each one adds one layer of indirection - that's even true for Python ints. However the Cython int is a C-integer without that indirection. That's one of the main reasons why for-loops in Cython are much faster. (However they are limit to the range of 64bits not unlimited precision like in Python, that's the trade-off).
Another point is that operating on python objects means that you need to go through pythons lookups, pythons operations, etc. However for built-in objects cython can use the Python C API to gain additional speed by avoiding the python based lookups (DefaultDict is not among them, I guess), for example with declared dicts the following code compiles differently:
def dumb_func(dict dct, str key):
return dct[key] # translates to: __Pyx_PyDict_GetItem(__pyx_v_dct, __pyx_v_key)
def dumb_func(object dct, object key): # different signature
return dct[key] # translates to: PyObject_GetItem(__pyx_v_dct, __pyx_v_key)
You probably could guess which one is faster, the one that adresses the dict directly (__Pyx_PyDict_GetItem is probably a sophisticated wrapper for PyDict_GetItem and PyDict_GetItemString) or the one that just adresses a python object with PyObject_GetItem (going through the python lookups). This won't be a huge speedup neither but it's noticable.
In the end I would say that normal (and declared) dicts would definetly be faster than DefaultDict (except that's some C or C++ class) in Cython code.
I'm trying to subclass numpy.complex64 in order to make use of the way numpy stores the data, (contiguous, alternating real and imaginary part) but use my own __add__, __sub__, ... routines.
My problem is that when I make a numpy.ndarray, setting dtype=mysubclass, I get a numpy.ndarray with dtype='numpy.complex64' in stead, which results in numpy not using my own functions for additions, subtractions and so on.
Example:
import numpy as np
class mysubclass(np.complex64):
pass
a = mysubclass(1+1j)
A = np.empty(2, dtype=mysubclass)
print type(a)
print repr(A)
Output:
<class '__main__.mysubclass'>
array([ -2.07782988e-20 +4.58546896e-41j, -2.07782988e-20 +4.58546896e-41j], dtype=complex64)'
Does anyone know how to do this?
Thanks in advance - Soren
The NumPy type system is only designed to be extended from C, via the PyArray_RegisterDataType function. It may be possible to access this functionality from Python using ctypes but I wouldn't recommend it; better to write an extension in C or Cython, or subclass ndarray as #seberg describes.
There's a simple example dtype in the NumPy source tree: newdtype_example/floatint.c. If you're into Pyrex, reference.pyx in the pytables source may be worth a look.
Note that scalars and arrays are quite different in numpy. np.complex64 (this is 32-bit float, just to note, not double precision). You will not be able to change the array like that, you will need to subclass the array instead and then override its __add__ and __sub__.
If that is all you want to do, it should just work otherwise look at http://docs.scipy.org/doc/numpy/user/basics.subclassing.html since subclassing an array is not that simple.
However if you want to use this type also as a scalar. For example you want to index scalars out, it gets more difficult at least currently. You can get a little further by defining __array_wrap__ to convert to scalars to your own scalar type for some reduce functions, for indexing to work in all cases it appears to me that you may have define your own __getitem__ currently.
In all cases with this approach, you still use the complex datatype, and all functions that are not explicitly overridden will still behave the same. #ecatmur mentioned that you can create new datatypes from the C side, if that is really what you want.