I am using C-contiguous memoryviews in my Python code and I would like to use dgemm which needs Fortran-contiguous memoryviews.
I would like to use the function PyMemoryView_GetContiguous found here but I don't know how to access it.
Does someone know which import I have to do ?
I don't want to use the function copy_fortran() as it really slows down my code.
PyMemoryView_GetContiguous doesn't look to be exposed as part of the Cython standard includes unfortunately. It should be reasonably easy to wrap though:
from cpython.buffer cimport PyBUF_READ # we'll need this later
cdef extern from "Python.h":
# copy the signature replacing PyObject* with object to let Cython
# handle the reference counting
object PyMemoryView_GetContiguous(object, int, char)
def test(double[:,::1] c_contig):
f_contig = PyMemoryView_GetContiguous(c_contig, PyBuf_READ,'F')
# .... do something useful
Be aware that this will still involve copying all the memory (this is absolutely unavoidable!) so is unlikely to be significantly faster than copy_fortran.
There's a problem though - PyMemoryView_GetContiguous won't return a writeable memoryview unless it does not have to make a copy, and Cython requires things assigned to a typed memoryview be writeable, so you can only use it as a Python object.
You can get a pointer to the first element though - the underlying object that created is a bytes/str object, so you can get a char* then cast that to whatever pointer you need. This should be enough to call your Fortran functions:
cdef char* as_bytes = f_contig.obj
some_function(<double*>as_bytes)
Related
Although I few similar questions already being asked, but I couldn't get head around on how to fix.
Basically I have this function:
Module one.pyx:
cdef char *isInData(data, dataLength):
cdef char *ret = <char *>malloc(200)
memset(ret, 0x00, 200)
if (b"Hello" in data and b"World" in data):
strcat(ret, b"Hello World!")
return ret
Module two.pyx:
import one
from libc.stdlib cimport malloc,realloc, free
cdef char *tempo():
cdef char *str
str = one.isInData(b"Hello what is up World", 23)
# do some stuff
free(str)
Issue occurs on line str = one.isInData("Hello what is up World", 23), I am assuming that as soon as isInData->ret is assigned to str. isInData->ret is deleted which causes this issue. But annyone help me on how to fix this?
import one
This line does a Python import of one. It doesn't know about any cdef functions defined in one (or even that one is a Cython module...). Therefore it assumes that isInData is a Python object that it can look up and that it'll be a callable returning another Python object.
cdf char* str = some_python_function() is unsafe because str points into the Python object. However the Python object is only a temporary and is most likely freed almost immediately.
What you mean is:
cimport one
which tells Cython that it's a Cython module that you're importing, and it should expect to find the declarations at compile-time. You'll probably need to write a pxd file to give the declarations.
I haven't looked in detail at the rest of your code so don't know if you are handling C strings right. But in general you may find it easier just to use Python strings rather than messing around with C string handling.
Is it possible to write a Cython function where a numpy array is passed by reference (perhaps a memory view?) and can I use such function in Python?
I tried:
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2]& sums):
...
...
Here counter and sums would be passed by reference. I am interested in the latter (but will happily accept critics and suggestions regarding both).
The compiler throws:
Reference base type cannot be a Python object
I get the same message with cdef either and I don't understand the full etiology of the problem.
Yes - sort of.
A Numpy array is a Python object. All Python objects are passed "by reference" anyway (i.e. if you change the contents of a Numpy array inside the function you change it outside).
Therefore you just do
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2] sums):
Specifying it as a C++ reference doesn't really make much sense. For a Python object there's quite a bit of detail that Cython hides from you.
However, cpdef functions are designed to be callable from Python and Cython. Python ints (and other numeric types) are immutable. Therefore changing counter will not work as you think when called from Python, and if it did what you hoped it'd potentially break the interpreter.
I'd therefore avoid using references in cpdef functions (and probably mostly avoid cpdef functions completely - it's usually better just to decide on def or cdef)
My machine is Win7 with anaconda.
i am recently converting C++ dll functions to a python project.
There are numbers of difficulty I had overcame yet I have no idea how to deal with the following conversion:
typedef int (__stdcall *p_API_GetOrder)(vector<ApiOrder>& apiOrderList);
where,
class ApiOrder(Structure):
_fields_ = [
('Timestamp', c_long),
('Item', c_char * 16),
('Qty', c_long),
]
In python, I tried,
mydll.API_GetOrder(POINTER(ApiOrder()))
The error is:
TypeError: must be a ctypes type
i am not an export in C++ or programming. So not quite sure what byref is. It would be nice if someone can clear my concept.
POINTER(…) constructs a new pointer type, not a value of that type. So, when you do this:
mydll.API_GetOrder(POINTER(ApiOrder()))
… you’re passing a Python type object, not a ctypes wrapper around a C pointer object.
To get a pointer to a ctypes wrapper object, you want to call either pointer or byref. The former constructs a POINTER(…) instance, sets it to point to your object, and passes the wrapped pointer; the latter just directly passes a pointer to your object without constructing a pointer wrapper object, and usually that’s all you need. See Passing pointers in the docs for further details.
However, I don’t think this is going to do much good, for two reasons.
First, most functions that take a pointer to some struct and return an int are doing it so they can fill in that struct with useful values. Constructing a new empty struct and passing a pointer to it and not holding onto a reference to it means you have no way to look at whatever values got filled in.
Also, you probably want to check the return value.
In general, you need to do something like this:
order = ApiOrder()
ret = mydll.API_GetOrder(byref(order))
if ret:
do some error handling with either ret or errno
else:
so something with order
While we’re at it, you almost certainly want to set the argtypes and restype of the function, so ctypes knows how to convert things properly, and can give you an exception if you do something that makes no sense, instead of making it guess how to convert and pass things and segfault if it guesses wrong.
Also, for the case of functions that return a success-or-error int, it's usually better to assign a function to the restype, which looks up the error and raises an appropriate exception. (Use an errcheck if you need anything more flexible than just checking that an int return is nonzero or a pointer return is zero.)
But even this isn’t going to help here, because the function you’re trying to call doesn’t take a pointer to an ApiOrder in the first place, it takes a reference to a std::vector of them. So you need to call into the C++ stdlib to construct an object of that type, then you can byref that as the argument.
But usually, it’s easier to write some C++ code that provides a C API to the library, then use ctypes to call that C API, instead of trying to build and use C++ objects from Python.
Your C++ code would look something like this:
int call_getorder(p_API_GetOrder func, ApiOrder *apiOrderArray, size_t apiOrderCount) {
std::vector<ApiOrder> vec(apiOrderArray, apiOrderCount);
ret = func(vec);
if (ret) return ret;
std::copy(std::begin(vec), std::end(vec), apiOrderArray);
return 0;
}
Now, you can call this from Python by creating an array of 1 ApiOrder (or creating a POINTER to an ApiOrder and passing it directly, if you prefer):
orders = (ApiOrder*1)()
ret = mywrapperdll.call_order(mydll.API_GetOrder, byref(order), 1)
if ret:
do some error handling with either ret or errno
else:
do something with order[0]
Of course you're still going to want argtypes and restype.
Please read this post before answering: Pass a closure from Cython to C++
In the accepted answers, it is neatly shown how a python function is converted into a std::function using Boost Python.
Following this example I'm able to wrap functions taking an std::function as an argument and call them using a python function as input. However, this only works when the std::function parameters are primitives like int, double, string etc.
Any guidance on how to make this work for custom types as well will be highly appreciated.
This won't be a complete answer - it assumes you can fill in the gaps from my previous answer that the question was based on. Unfortunately it is a little bit more complicated than that case.
Just to define the problem - assume you have a parameter of a custom C++ class, like:
class cpp_class {
// some non-trivial contents
};
and thus your C++ interface looks like this:
void call_some_std_func(std::function<void(cpp_class&)> callback) {
callback(5,std::string("hello"));
}
The first thing to do is to write a Cython wrapper for your C++ class (in principle you could make a Boost Python wrapper instead). Here you need to make a choice of about "ownership" of the C++ object. The first choice is to make a copy:
cdef extern from "cpp_file.hpp":
cppclass cpp_class:
pass # details
cdef class CyWrapper:
cdef cpp_class* ptr
def __dealloc__(self):
del self.ptr
# other details following standard wrapper pattern
cdef public make_CyWrapper(cpp_class& x):
obj = CyWrapper()
obj.ptr = new cpp_class(x)
return obj
I've created a wrapper class with a destructor that handles the memory and a publicly accessible constructor function that can be called from external code. This version is safe because your wrapper owns the object it holds and so there can be no writes to invalid memory. However, because it makes a copy, you can't make changes to the original C++ object.
A second option is to hold a pointer to an object you don't own. The code is basically identical except you remove the __dealloc__ and avoid making a copy in make_CyWrapper:
obj.ptr = &x // instead of new cpp_class(x)
This is unsafe - you need to ensure your C++ object outlives the Cython wrapper - but allows you to modify the object.
You could also imagine a few other options: you could take ownership of an existing object with your Cython wrapper (Such a scheme would have to pass by pointer rather than reference, or it could use move constructors); you could deconstruct your C++ class into a representation expressed in basic types and pass those to Python; you could use shared pointers to split the ownership; or you have a more elaborate way of marking your Cython wrapper as "invalid" once your held C++ instance is destructed.
What you do next depends on whether you're using Boost Python (for it's convenient, callable wrapping of Python objects) or if you're making your own version. (I showed both possibilities in the previous answer).
Assuming Boost Python, you need to do two things - tell it about the conversion and make sure that it imports the module that your wrapper is defined in (if you don't do this you get exciting segmentation faults)
struct convert_to_PyWrapper {
static PyObject* convert(const cpp_class& rhs) {
// the const_cast here is a bit dodgy, but was needed to make it work
return make_CyWrapper(const_cast<cpp_class&>(rhs));
}
};
inline void setup_boost_python() {
PyInit_your_module_name(); // named inityour_module_name in Python 2
boost::python::to_python_converter<
cpp_class,
convert_to_PyWrapper>();
}
You need to make sure that your Python/Cython code calls "setup_boost_python" before attempting to use the callback (if you put it at module level it's done on import, which is ideal).
If you're following my "manual" scheme (avoiding the dependency on Boost Python) then you need to modify the call_obj Cython function that does the C++ to Cython type conversion.
cdef public void call_obj(obj, cpp_class& c):
obj(make_CyWrapper(c))
You also need to ensure the wrapper Cython module is imported before use (otherwise you get segmentation faults). I did this in "py_object_wrapper.hpp" but providing it's done once somewhere you can place it where you like.
void operator()(cpp_class& a) {
PyInit_your_module_name();
if (held) {
call_obj(held,a);
}
}
I have written a small C++ library to simulate a system of ODEs, and have used Cython to wrap a function that initializes and solves the system. The C++ function takes in a handful of parameters, and returns a struct containing two matrices: a set of patterns used to construct the system, and the solved state of the system at every time step.
These matrices are implemented using the Eigen library, so I am using the Eigency interface to transfer the underlying contents of these data structures to NumPy arrays.
All is good if, in Python, I return a view to one of these matrices. However, I would like to return a tuple containing both of these matrices, which requires temporarily storing this struct, and then returning each of its matrix members:
## kernel.pyx
from eigency.core cimport *
cdef extern from "kernel_cpp.h":
cdef cppclass network:
MatrixXf state
MatrixXf patterns
cdef network _run "run" (int N, int K, int P, double g, double T, double tau, double dt)
def run(N=10000, K=100, P=30, g=1.5, T=0.5, tau=1e-3, dt=1e-3):
return ndarray_view(_run(N,K,P,g,T,tau,dt).state) # <--- This works
# This does not:
# cdef network net = _run(N,K,P,g,T,tau,dt):
# return (ndarray_view(net.state), ndarray_view(net.patterns))
The error that I get when building the commented code with python setup.py build_ext --inplace is
kernel.cpp:1552:11: error: no matching constructor for initialization of 'network'
network __pyx_v_net;
^`
and this makes sense to me -- Cython is trying to call a constructor for network. But I want the construction to take place within "run." What is the correct type declaration here?
In case it helps, a stripped-down version of the code is here.
Thanks!
It's worth knowing how Cython translates code which stack allocates C++ variables:
def run(N=10000, K=100, P=30, g=1.5, T=0.5, tau=1e-3, dt=1e-3):
cdef network net = _run(N,K,P,g,T,tau,dt)
the inside of the function gets translated to:
{
// start of function
network net; // actually some mangled variable name (see your error message)
// body of function
net = _run(N,K,P,g,T,tau,dt);
}
i.e. it requires both a default constructor and a copy/move assignment operator. This doesn't match how C++ code tends to be written so occasionally causes problems. Cython usually assumes the existence of these (unless you tell it about other constructors) so you don't need to explicitly write these out in your cdef cppclass block.
I think your error message looks to occur when compiling the C++ code and it's not defining a default constructor. (The copy assignment later shouldn't be a problem since C++ will automatically generate this). You have two options:
If you're happy modifying the C++ code then define a default constructor. This will allow the Cython code to work as written. If you're using >=C++11 then that could be as easy as:
network() = default;
It's possible that this won't work depending on whether the internal Eigen types can be default constructed though.
If you don't want modify the C++ code, or if it turns out you can't easily define a default constructor, then you'll have to modify the Cython code to allocate the network with new. This also means you'll have to deal with deallocation yourself:
# earlier
cdef cppclass network:
# need to tell Cython about copy constructor
network(const network&)
# everything else as before
def run(N=10000, K=100, P=30, g=1.5, T=0.5, tau=1e-3, dt=1e-3):
cdef network* net
try:
net = new network(_run(N,K,P,g,T,tau,dt))
return (ndarray_view(net.state), ndarray_view(net.patterns))
finally:
del net
In this case no default constructor is needed, only a copy/move constructor (which you have to have anyway to be able to return a network from a function). Note use of finally to ensure than we free the memory.