I'm writing a python wrapper to a C class and I'm allocating memory using PyMem_Malloc as explained here
cdef class SomeMemory:
cdef double* data
def __cinit__(self, size_t number):
# allocate some memory (uninitialised, may contain arbitrary data)
self.data = <my_data_t*> PyMem_Malloc(number * sizeof(my_data_t))
if not self.data:
raise MemoryError()
Then I import and use the class in another script using:
import SomeMemory
sm = SomeMemory(10)
I now would like to access the elements of self.data but I encounter 2 problems
if I type self.data and hit enter, the ipython kernel crashes
if I try to loop on self data
like:
for p in self.data:
print p
I get an error that self.data is not iterable.
How can I access self.data? Do I need to cast the elements to my_data_t first?
(Heavily edited in light of the updated question)
So - my understanding is that self.data needs to be declared as public to access it from Python:
cdef class SomeMemory:
cdef public double* data
# also make sure you define __dealloc__ to remove the data
However, that doesn't seem to be enforced here.
However, your real problems are that Python doesn't know what to do with an object of type double *. It's certainly never going to be iterable because the information about when to stop is simply not stored (so it will always go off the end.
There are a range of better alternatives:
you store your data as a Python array (see http://docs.cython.org/src/tutorial/array.html for a guide). You have quick access to the raw c pointer from within Cython if you want, but also Python knows what to do with it
Code follows
from cpython cimport array as c_array
from array import array
cdef class SomeMemory:
cdef c_array.array data
def __cinit__(self, size_t number):
self.data = array('d', some initial value goes here)
You store your data as a numpy array. This is possibly slightly more complicated, but has much the same advantages. I won't give an example, but it's easier to find.
You use Cython's typed memory view: http://docs.cython.org/src/userguide/memoryviews.html. The advantage of this is that it lets you control the memory management yourself (if that's absolutely important to you). However, I'm not 100% sure that you can access them cleanly in Python (too lazy to test!)
You wrap the data in your own class implementing __getitem__ and __len__ so it can be used as an iterator. This is likely more trouble that it's worth.
I recommend the first option, unless you have good reason for one of the others.
Related
Is it possible to write a Cython function where a numpy array is passed by reference (perhaps a memory view?) and can I use such function in Python?
I tried:
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2]& sums):
...
...
Here counter and sums would be passed by reference. I am interested in the latter (but will happily accept critics and suggestions regarding both).
The compiler throws:
Reference base type cannot be a Python object
I get the same message with cdef either and I don't understand the full etiology of the problem.
Yes - sort of.
A Numpy array is a Python object. All Python objects are passed "by reference" anyway (i.e. if you change the contents of a Numpy array inside the function you change it outside).
Therefore you just do
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2] sums):
Specifying it as a C++ reference doesn't really make much sense. For a Python object there's quite a bit of detail that Cython hides from you.
However, cpdef functions are designed to be callable from Python and Cython. Python ints (and other numeric types) are immutable. Therefore changing counter will not work as you think when called from Python, and if it did what you hoped it'd potentially break the interpreter.
I'd therefore avoid using references in cpdef functions (and probably mostly avoid cpdef functions completely - it's usually better just to decide on def or cdef)
I am using C-contiguous memoryviews in my Python code and I would like to use dgemm which needs Fortran-contiguous memoryviews.
I would like to use the function PyMemoryView_GetContiguous found here but I don't know how to access it.
Does someone know which import I have to do ?
I don't want to use the function copy_fortran() as it really slows down my code.
PyMemoryView_GetContiguous doesn't look to be exposed as part of the Cython standard includes unfortunately. It should be reasonably easy to wrap though:
from cpython.buffer cimport PyBUF_READ # we'll need this later
cdef extern from "Python.h":
# copy the signature replacing PyObject* with object to let Cython
# handle the reference counting
object PyMemoryView_GetContiguous(object, int, char)
def test(double[:,::1] c_contig):
f_contig = PyMemoryView_GetContiguous(c_contig, PyBuf_READ,'F')
# .... do something useful
Be aware that this will still involve copying all the memory (this is absolutely unavoidable!) so is unlikely to be significantly faster than copy_fortran.
There's a problem though - PyMemoryView_GetContiguous won't return a writeable memoryview unless it does not have to make a copy, and Cython requires things assigned to a typed memoryview be writeable, so you can only use it as a Python object.
You can get a pointer to the first element though - the underlying object that created is a bytes/str object, so you can get a char* then cast that to whatever pointer you need. This should be enough to call your Fortran functions:
cdef char* as_bytes = f_contig.obj
some_function(<double*>as_bytes)
I have written a small C++ library to simulate a system of ODEs, and have used Cython to wrap a function that initializes and solves the system. The C++ function takes in a handful of parameters, and returns a struct containing two matrices: a set of patterns used to construct the system, and the solved state of the system at every time step.
These matrices are implemented using the Eigen library, so I am using the Eigency interface to transfer the underlying contents of these data structures to NumPy arrays.
All is good if, in Python, I return a view to one of these matrices. However, I would like to return a tuple containing both of these matrices, which requires temporarily storing this struct, and then returning each of its matrix members:
## kernel.pyx
from eigency.core cimport *
cdef extern from "kernel_cpp.h":
cdef cppclass network:
MatrixXf state
MatrixXf patterns
cdef network _run "run" (int N, int K, int P, double g, double T, double tau, double dt)
def run(N=10000, K=100, P=30, g=1.5, T=0.5, tau=1e-3, dt=1e-3):
return ndarray_view(_run(N,K,P,g,T,tau,dt).state) # <--- This works
# This does not:
# cdef network net = _run(N,K,P,g,T,tau,dt):
# return (ndarray_view(net.state), ndarray_view(net.patterns))
The error that I get when building the commented code with python setup.py build_ext --inplace is
kernel.cpp:1552:11: error: no matching constructor for initialization of 'network'
network __pyx_v_net;
^`
and this makes sense to me -- Cython is trying to call a constructor for network. But I want the construction to take place within "run." What is the correct type declaration here?
In case it helps, a stripped-down version of the code is here.
Thanks!
It's worth knowing how Cython translates code which stack allocates C++ variables:
def run(N=10000, K=100, P=30, g=1.5, T=0.5, tau=1e-3, dt=1e-3):
cdef network net = _run(N,K,P,g,T,tau,dt)
the inside of the function gets translated to:
{
// start of function
network net; // actually some mangled variable name (see your error message)
// body of function
net = _run(N,K,P,g,T,tau,dt);
}
i.e. it requires both a default constructor and a copy/move assignment operator. This doesn't match how C++ code tends to be written so occasionally causes problems. Cython usually assumes the existence of these (unless you tell it about other constructors) so you don't need to explicitly write these out in your cdef cppclass block.
I think your error message looks to occur when compiling the C++ code and it's not defining a default constructor. (The copy assignment later shouldn't be a problem since C++ will automatically generate this). You have two options:
If you're happy modifying the C++ code then define a default constructor. This will allow the Cython code to work as written. If you're using >=C++11 then that could be as easy as:
network() = default;
It's possible that this won't work depending on whether the internal Eigen types can be default constructed though.
If you don't want modify the C++ code, or if it turns out you can't easily define a default constructor, then you'll have to modify the Cython code to allocate the network with new. This also means you'll have to deal with deallocation yourself:
# earlier
cdef cppclass network:
# need to tell Cython about copy constructor
network(const network&)
# everything else as before
def run(N=10000, K=100, P=30, g=1.5, T=0.5, tau=1e-3, dt=1e-3):
cdef network* net
try:
net = new network(_run(N,K,P,g,T,tau,dt))
return (ndarray_view(net.state), ndarray_view(net.patterns))
finally:
del net
In this case no default constructor is needed, only a copy/move constructor (which you have to have anyway to be able to return a network from a function). Note use of finally to ensure than we free the memory.
I have a C++ class with a variable that is a dynamic array. It is extremely simple, currently only for testing purposes:
class testClass {
public:
int *x;
testClass();
~testClass();
};
The variable x is initialised with some values, currently through the constructor. I am trying to write a python wrapper code for the C++ class through Cython that can access x. How can I do this?
Best thing would be to be able to access the variable without copying a lot of data, as x might be large. Can I access it as a numpy array?
Example, that it behave as numpy array, that could be read only for instance? I want to be able to use the data in x in other calculations with python, so a numpy array is preferred.
I suppose I could make a GET method that initialize a numpy array, passes it it to the GET method and populate it with the data from x with a loop, but this would copy the data, and does not seem very elegant. Hoping for a better solution.
I have tried with a static array, and found a solution that sort of worked. If x is static, I could do the following in a .pyx file:
cdef extern from "testClass.h":
cdef cppclass testClass:
testClass()
int x[5]
cdef class pyTestClass:
cdef testClass *thisptr
def __cinit__(self):
self.thisptr = new testClass()
def __dealloc__(self):
del self.thisptr
property x:
def __get__(self):
return self.thisptr.x
If I access x in Python, I will get a Python list with the values back.
How to access dynamic C++ array variable, belonging to an object, with Python?
Have a look at this example that explains how to expose arrays allocated in C to numpy without data copies. The relevant line to create, for instance, a 1D numpy array of integers in Cython with already allocated data is,
ndarray = np.PyArray_SimpleNewFromData(1, shape, np.NPY_INT, data_pointer)
See the corresponding Numpy documentation.
We have some code written in python that uses a few classes that are really just "structs" -- instances of these classes just have a bunch of fields in them and no methods. Example:
class ResProperties:
def __init__(self):
self.endDayUtilities = 0
self.marginalUtilities = []
self.held = 0
self.idleResource = True
self.experience = 0.0
self.resSetAside = 0
self.unitsGatheredToday = 0
Our main code uses a bunch of instances of this class.
To speed up the code, I thought I'd cython-ized this class:
cdef class ResProperties:
cdef public float endDayUtilities
cdef public list marginalUtilities
cdef public int held
cdef public int idleResource
cdef public float experience
cdef public int resSetAside
cdef public int unitsGatheredToday
def __init__(self):
self.endDayUtilities = 0
# etc: code just like above.
However, the result is that the code runs something like 25% slower now!
How do I figure out what is causing the code to run slower now?
Thanks.
You converted these classes to Cython but are still using them from Python code?
Conversion of data from C to Python and back is going to incur overhead. For example, your endDayUtilities member is a C-style float. When you access it from Python, a float() object must be constructed before your Python code can do anything with it. The same thing has to happen in reverse when you assign to that attribute from Python.
Off the top of my head, I'd estimate the performance overhead of these data conversions at... oh, about 25%. :-)
You're not going to see a performance boost until you move some of your code that uses that data to Cython. Basically, the more you can stay in C-land, the better you'll do. Going back and forth will kill you.
As another, simpler approach, you might want to try Psyco or PyPy instead of Cython.