I am trying to speed up my Python by translating it into Cython. It uses the function scipy.integrate.quad, which requires a python callable as one of its arguments. Is there any way to define and pass a function from within a cdef statement in Cython so I can use this?
Thanks.
Yes, you have just to wrap the function with a python function.
def py_callback(a):
return c_callback(a)
cdef float c_callback(float a):
return a * a
...
scipy.integrate.quad(py_callback)
There is a bit of conversion overload of course.
An alternative is to look up how that function is implemented.
If is a wrapper to a C function try to call directly the C function from the pyx file
(creating a pxd that wraps the scipy C-module).
If is pure Python attempt to translate it to cython.
Otherwise you have to use some other pure C integration function and skip the scipy one.
Related
Is it possible to write a Cython function where a numpy array is passed by reference (perhaps a memory view?) and can I use such function in Python?
I tried:
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2]& sums):
...
...
Here counter and sums would be passed by reference. I am interested in the latter (but will happily accept critics and suggestions regarding both).
The compiler throws:
Reference base type cannot be a Python object
I get the same message with cdef either and I don't understand the full etiology of the problem.
Yes - sort of.
A Numpy array is a Python object. All Python objects are passed "by reference" anyway (i.e. if you change the contents of a Numpy array inside the function you change it outside).
Therefore you just do
cpdef void my_function(bytes line, int& counter, np.ndarray[np.uint32_t, ndim=2] sums):
Specifying it as a C++ reference doesn't really make much sense. For a Python object there's quite a bit of detail that Cython hides from you.
However, cpdef functions are designed to be callable from Python and Cython. Python ints (and other numeric types) are immutable. Therefore changing counter will not work as you think when called from Python, and if it did what you hoped it'd potentially break the interpreter.
I'd therefore avoid using references in cpdef functions (and probably mostly avoid cpdef functions completely - it's usually better just to decide on def or cdef)
I know that a Numba-jitted function calling another jitted function will recognize this and automatically use a fast C calling convention rather than going through the Python object layer, and therefore avoid the high Python function call overhead:
import numba
#numba.jit
def foo(x):
return x**2
#numba.jit
def bar(x):
return 4 * foo(x) # this will be a fast function call
My question is whether the same is true if I call a Cython function from Numba. So let's say I have a Cython module, foo.pyx:
cpdef double foo(double x):
return x**2
As well as a standard Python module bar.py:
import numba
import foo
#numba.jit
def bar(x):
return 4 * foo.foo(x) # will this be a fast function call?
Will Numba recognize foo.foo as a C-callable function automatically or do I need to tell it manually by, say, setting up a CFFI wrapper?
EDIT: Upon further reflection, Cython functions are just standard "builtin" functions from the view of the Python interpreter. So the question can be made more general: does Numba optimize calls to builtin functions and methods to bypass the Python calling overhead?
It is possible to use Cython's cpdef/cdef-functions (but not the def-functions) in nopython-numba:
step: cdef/cpdef function must be marked as api in the Cython code.
step: numba.extending.get_cython_function_address can be used to get the address of the cpdef-function.
step: ctypes can be used to create a CFunction from the address of the cpdef-function, which can be used in numba-nopython code.
Read on for a more detailed explanation.
Even if the builtin-functions (PyCFunction, the same as Cython's def-functions) are written in C, they don't have a signature which could be used by nopython-numba-code.
For example the acos function from the math-module, doesn't have the signature
`double acos(double)`
as one could expect, but its signature is
static PyObject * math_acos(PyObject *self, PyObject *args)
So basically in order to call this function, numba would need to build a Python-float from the C-float at hand, but this is prohibited by nopython=True.
However, Cythons cpdef-functions are a little bit different: it is a small wrapper around a real cdef-function, whose arguments are raw C-types like double, int and so on. This cdef-function could be used by numba, only if its address were known.
Cython offers a way to find out the addresses of cdef-functions in a portable way: the addresses can be found in the attribute __pyx_capi__ of the cythonized module.
However, not all cdef and cpdef functions are exposed in this way, but only ones which are explicitly marked as C-api declarations or implicitly by being shared through a pxd-file.
Once the function foo of the foomodule is marked as api:
cpdef api double foo(double x):
return x*x
the address of the cpdef-function foo can be found in foomodule.__pyx_capi__-dictionary:
import foomodule
foomodule.__pyx_capi
# {'foo': <capsule object "double (double)" at 0x7fe0a46f0360>}
It is surprisingly hard to extract the address from a PyCapsule in Python. One possibility is to use ctypes.pythonapi, another (maybe easier one) is to utilize Cython to access Python's C-API:
%%cython
from cpython.pycapsule cimport PyCapsule_GetPointer, PyCapsule_GetName
def address_from_capsule(object capsule):
name = PyCapsule_GetName(capsule)
return <unsigned long long int> PyCapsule_GetPointer(capsule, name)
which can be used as:
addr = address_from_capsule(foomodule.__pyx_capi__['foo'])
However, numba offers a similar functionality out of the box - get_cython_function_address :
from numba.extending import get_cython_function_address
addr = get_cython_function_address("foomodule", "foo")
Once we got the address of the c-function, we can construct a ctypes-function:
import ctypes
foo_functype = ctypes.CFUNCTYPE(ctypes.c_double, ctypes.c_double)
foo_for_numba = foo_functype(addr)
This function can be utilized for example as follows from nopython-numba:
from numba import njit
#njit
def use_foo(x):
return foo_for_numba(x)
and now:
use_foo(5)
# 25.0
yields the expected result.
Arguments of ctype-functions which are understood by numba out of the box can be found here and are:
_FROM_CTYPES = {
ctypes.c_bool: types.boolean,
ctypes.c_int8: types.int8,
ctypes.c_int16: types.int16,
ctypes.c_int32: types.int32,
ctypes.c_int64: types.int64,
ctypes.c_uint8: types.uint8,
ctypes.c_uint16: types.uint16,
ctypes.c_uint32: types.uint32,
ctypes.c_uint64: types.uint64,
ctypes.c_float: types.float32,
ctypes.c_double: types.float64,
ctypes.c_void_p: types.voidptr,
ctypes.py_object: types.ffi_forced_object,
}
There is a limited set of builtin functions (from both the python standard library and numpy) that numba knows how to translate into native code:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html
Anything else will not be able to be jitted by Numba in nopython mode, thus resorting to objectmode which is much slower.
There is no direct way to pass a cython function to Numba and have it be recognized in nopython mode. Numba does have hooks for cffi:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html#cffi
that can be leveraged to call outside C code, that you might be able to rig up to call cython if you could create a low level wrapper at the C-level; I'm not 100% sure if this is possible though. I wrote about doing this for calling RMath functions from Numba:
https://web.archive.org/web/20160611082327/https://www.continuum.io/blog/developer-blog/calling-c-libraries-numba-using-cffi
It might be helpful in getting you started if you go that route.
I'm currently writing a wrapper for a C++ project that use std::complex<double>, available in cython as libcpp.complex.complex[double].
However there is no implicit conversion between this and Python complex, I'm trying to find the best way to do this conversion.
The obvious is to use
cdef libcpp.complex.complex[double] x = ...
X = complex(x.real(),x.imag()
And
cdef complex Y = ...
cdef libcpp.complex.complex[double] y = libcpp.complex.complex[double](Y.real, Y.imag)
And
cdef libcpp.complex.complex[double] z
cdef complex Z = ...
z.real(Z.real)
z.imag(Z.imag)
But is there a better way, preferably to make cython do the conversion automatically.
For one this occurs in some cpdef functions, and I would like to avoid using Python complex in calls from cython for improved speed.
However as far as I can tell cython can't do this conversion implicitly, and thus I can't avoid using the Python complex for code that can be called from Python or C++.
This functionality will be released in the next version of Cython.
https://github.com/cython/cython/commit/2c3c04b25620c437ce41d7851e7581eb1f0c313c
I have dll, that builded in Delphi, and I try to call function from it. Declaration of function looks like this:
function GetUid(UID:Pointer):Integer; stdcall;
This is equivalent to this C function signature:
int GetUID(void *pointer);
Library handled using ctypes:
from ctypes import *
lib = cdll.LoadLibrary("mylib.dll")
But i stuck here:
res = lib.GetUid(?)
What I need to pass in this function?
Pointer is void *, but how make this rightly in python?
Python is a high level language. You do not typically import a DLL from a C or Pascal native library and invoke it and pass variables from Python into a C or Pascal function taking a void * type raw pointer and then manipulate raw memory this way.
In short if you knew what you were doing you would know better than to try to do what you're doing here.
Let's suppose that your implementation is like this:
function GetUid(UID:Pointer):Integer; stdcall;
var
P2:^Integer;
begin
P2 := UID;
P2^ := 0;
end;
Then, what you would want to do is pass in an address to a 32 bit integer. Of course my example above is absurd, because what would have made sense above is to just declare the parameter as an "int *pointer" (in C terms) rather than as a "void *pointer".
Whatever it is you're doing, the next thing that will likely happen is that you will corrupt your python interpreter's heap, and cause lots of fun crashes and errors.
A far more sensible approach is to read the Python documentation on writing C extensions that can manipulate native Python types (PyObject), and doing the same thing but in pascal, if you like.
p4d appears to be a workable way of writing extension DLLs in delphi:
https://code.google.com/p/python4delphi/source/list
Here's the scenario: I am using ctypes to load a C DLL into a Python program. I want to register a callback so that code in the DLL can call a function in Python.
This is all great until I want to have the function be variadic. For example, say I wanted to implement "printf" in Python (for argument's sake, I'll also have it return an integer):
def printf(format, *args):
(...)
return 0
I am creating the callback function pointer using the CFUNCTYPE factory in ctypes:
callback_type = CFUNCTYPE(c_int, c_wchar_p)
Of course, since I only specify one input argument (the format string as a Unicode char array), the callback works but *args is empty.
So, is there any way of making this work with ctypes, or do I need to rewrite my C code to not require a variadic callback function?
No, it's not supported by the underlying libffi. You need indeed to write custom C code.
(Fwiw, CFFI is a ctypes alternative. Also based on libffi, it suffers from the same limitation --- but at least it makes it easy to write custom pieces of C code to do the interfacing.)