Q: Is there something like numpy.(C/C++)vectorize ?
Suppose I have a numpy vector and a C++ DLL (with C extern interface) containing a function accepting a float and returning a float (no array!). With ctypes I can wrap the DLL function and do numpy.vectorize on that, so I can write:
myfunc = numpy.vectorize(lambda x: ctypes_wrapped_DLL_fun(x))
x_vec = numpy.linspace(...) # a vector
print myfunc(x_vec)
Works like a charm. Of course this function call is slow, because it has to pass from the C-ish numpy context with every single value through python, through the DLL and back.
I have found a mindboggling array of ways to interface C/C++ with numpy. Essentially only the chapter about Cython 2.8.5.2. "Numpy Support" gives an indication how to deal with vectors without going through the numpy/python/C interfaces all the time with every single element of the vector. The downside of this is that I would have to change the interface of the DLL from simple floats to arrays (and it liked to crash because I probably did things wrong).
numpy.ctypeslibalso seems not to be what I need, because it also would require the DLL to have an array interface.
Any good examples for that (? I do not mind use Cython or another solution requiring a compiler. I would strongly prefer not to change the interface of the DLL.
Related
Is it possible to write a Cython function where a numpy array is passed by reference (perhaps a memory view?) and can I use such function in Python?
I tried:
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2]& sums):
...
...
Here counter and sums would be passed by reference. I am interested in the latter (but will happily accept critics and suggestions regarding both).
The compiler throws:
Reference base type cannot be a Python object
I get the same message with cdef either and I don't understand the full etiology of the problem.
Yes - sort of.
A Numpy array is a Python object. All Python objects are passed "by reference" anyway (i.e. if you change the contents of a Numpy array inside the function you change it outside).
Therefore you just do
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2] sums):
Specifying it as a C++ reference doesn't really make much sense. For a Python object there's quite a bit of detail that Cython hides from you.
However, cpdef functions are designed to be callable from Python and Cython. Python ints (and other numeric types) are immutable. Therefore changing counter will not work as you think when called from Python, and if it did what you hoped it'd potentially break the interpreter.
I'd therefore avoid using references in cpdef functions (and probably mostly avoid cpdef functions completely - it's usually better just to decide on def or cdef)
I am using C-contiguous memoryviews in my Python code and I would like to use dgemm which needs Fortran-contiguous memoryviews.
I would like to use the function PyMemoryView_GetContiguous found here but I don't know how to access it.
Does someone know which import I have to do ?
I don't want to use the function copy_fortran() as it really slows down my code.
PyMemoryView_GetContiguous doesn't look to be exposed as part of the Cython standard includes unfortunately. It should be reasonably easy to wrap though:
from cpython.buffer cimport PyBUF_READ # we'll need this later
cdef extern from "Python.h":
# copy the signature replacing PyObject* with object to let Cython
# handle the reference counting
object PyMemoryView_GetContiguous(object, int, char)
def test(double[:,::1] c_contig):
f_contig = PyMemoryView_GetContiguous(c_contig, PyBuf_READ,'F')
# .... do something useful
Be aware that this will still involve copying all the memory (this is absolutely unavoidable!) so is unlikely to be significantly faster than copy_fortran.
There's a problem though - PyMemoryView_GetContiguous won't return a writeable memoryview unless it does not have to make a copy, and Cython requires things assigned to a typed memoryview be writeable, so you can only use it as a Python object.
You can get a pointer to the first element though - the underlying object that created is a bytes/str object, so you can get a char* then cast that to whatever pointer you need. This should be enough to call your Fortran functions:
cdef char* as_bytes = f_contig.obj
some_function(<double*>as_bytes)
I'm trying to write a small, modular program in Python that will dynamically load C functions and use them to execute computationally intensive code. In this program I am creating a couple of large matrices that I will be passing back and forth between my Python code to different C functions. I would prefer to pass these matrices by reference to avoid additional computational overhead.
I've tried reading through the Python docs for ctypes, but it doesn't seem to explain how to do this. I understand, for instance, that I can use byref() or pointer() to pass a pointer from Python to a C function, but how to I pass a pointer from an external C function back to Python? Given that variables are names in Python, is this just done "automatically" (for lack of a better term) when Python receives a value from a C function?
As a concrete example, this is what I'm trying to do (in pseudo-code):
foo = ctypes.CDLL("pathToFoo")
bar = ctypes.CDLL("pathToBar")
# Generate a large matrix by calling a C function.
reallyBigMatrix = foo.generateReallyBigMatrix()
# Pass reallyBigMatrix to another function and perform some operation
# on it. Since the matrix is really big, I would prefer to pass a
# reference to this matrix to my next C function rather than passing
# the matrix by value.
modifiedReallyBigMatrix = bar.modifyReallyBigMatrix(reallBigMatrix)
Alternatively, I'm using Python and C in conjunction as I need an easy way to dynamically load C functions in my program. I may pass paths to different C files to my Python program so that the Python program will execute the same code on different functions. As an example, I may want to run my program two different ways: keep the same "generateReallyBigMatrix" function in both runs, but used a different "modifyReallyBigMatrix" program between run 1 and run 2. If there is an easy, cross-platform way to do this in C or C++ I would be happy to implement that solution rather than using ctypes and Python. However, I haven't been able to find a simple, cross-platform solution.
You've mentioned that you are writing all the code, both Python and C, from yourself. I suggest not using ctypes for this, as ctypes is best suited for using C libraries that cannot be modified.
Instead, write a module in C using the Python C API. It will expose a single function to start with, like this:
PyObject* generateReallyBigMatrix(void);
Now, instead of trying to return a raw C pointer, you can return any Python object that you like. Good choices here would be to return a NumPy array (using the NumPy C API), or to return a Python "buffer" (from which a NumPy array can be constructed in Python if desired).
Either way, once this function is written in C using the appropriate APIs, your Python code will be simple:
import foo
reallyBigMatrix = foo.generateReallyBigMatrix()
To do it using the NumPy C API, your C code will look like this:
PyObject* generateReallyBigMatrix(void)
{
npy_intp dimension = 100;
PyArray_Descr* descr;
PyArray_DescrAlignConverter2("float64", &descr); // use any dtype
PyObject* array = PyArray_Empty(1, &dimension, descr, 0/*fortran*/);
Py_DECREF(descr);
void* data = PyArray_DATA(array);
// TODO: populate data
return array;
}
static PyMethodDef methods[] = {
{"generateReallyBigMatrix", generateReallyBigMatrix, METH_VARARGS, "doc"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
PyMODINIT_FUNC initfoo(void)
{
import_array(); // enable NumPy C API
Py_InitModule("foo", methods);
}
Note that the NumPy C API requires a slightly strange initialization ritual. See also Numpy C API: Link several object files
You then compile the code as a shared library called foo.so (no lib prefix).
I have a C/C++ DLL with libraries I need to call from NumPy, where the format inside the DLL is all boost::multi_array format. While I have seen some posts involving this project: https://github.com/mdboom/numpy-boost
there really is very little documentation and examples from users out there on the steps required to wrap the boost::multi_array to NumPy interface with this library. My questions: while the examples are enough to guess the C++ part of the interface (other than strings, how is this handled? Q1), what do you have to do on the Python side to use the compiled DLL (Q2)? It seems Ctypes wouldn't work (correct me if I'm wrong) so does this have to be rewritten in Cython?
For reference, this is the C part with all the datatypes I'm trying to wrap:
extern "C"
{
DECLDIR void Cfunction(
boost::multi_array<double, 2>& p_result,
const vector<string>& p_calcType,
const string& p_optionType,
unsigned long p_nTimeStep = 50,
const vector<double>& p_premium = vector<double>());
With ctypes you can load DLL dynamically and call functions from it. From my point of view, it's easiest way for using compiled code within Python code. Here is how: How can I use a DLL file from Python?
numpy allows you to get pointer to memory location. Look at that: http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.ndarray.ctypes.html
If you don't have source code of your DLL and cannot change its interface, you create C++ wrapper (your own separate DLL) that accepts raw pointer from Python instead of boost::multi_array.
I have dll, that builded in Delphi, and I try to call function from it. Declaration of function looks like this:
function GetUid(UID:Pointer):Integer; stdcall;
This is equivalent to this C function signature:
int GetUID(void *pointer);
Library handled using ctypes:
from ctypes import *
lib = cdll.LoadLibrary("mylib.dll")
But i stuck here:
res = lib.GetUid(?)
What I need to pass in this function?
Pointer is void *, but how make this rightly in python?
Python is a high level language. You do not typically import a DLL from a C or Pascal native library and invoke it and pass variables from Python into a C or Pascal function taking a void * type raw pointer and then manipulate raw memory this way.
In short if you knew what you were doing you would know better than to try to do what you're doing here.
Let's suppose that your implementation is like this:
function GetUid(UID:Pointer):Integer; stdcall;
var
P2:^Integer;
begin
P2 := UID;
P2^ := 0;
end;
Then, what you would want to do is pass in an address to a 32 bit integer. Of course my example above is absurd, because what would have made sense above is to just declare the parameter as an "int *pointer" (in C terms) rather than as a "void *pointer".
Whatever it is you're doing, the next thing that will likely happen is that you will corrupt your python interpreter's heap, and cause lots of fun crashes and errors.
A far more sensible approach is to read the Python documentation on writing C extensions that can manipulate native Python types (PyObject), and doing the same thing but in pascal, if you like.
p4d appears to be a workable way of writing extension DLLs in delphi:
https://code.google.com/p/python4delphi/source/list