I am trying to write a Cython wrapper around a C library. I am very new to Cython, so my apologies in advance if the problem is obvious.
In a file wrapper.pxd, I define a struct (reduced example):
cdef extern from "thiscouldbeyourlibrary.h":
cdef struct foo:
double **output
I then have a class:
cdef class Bar:
cdef wrapper.foo __stuff
cdef do_something(self):
self.__stuff.output = NULL
This fails:
Cannot convert 'void *' to Python object.
Apparently, Cython cannot be sure that self.__stuff.output is always a pointer. But I declared its type and the class is a "cdef" class, so I don't really understand why.
The problem is the incompatibility between NULL and double **. You could assign it to a char, int or void *, for example, doing:
wrapper.pyd:
cdef extern from "thiscouldbeyourlibrary.h":
cdef struct foo:
char a
int b
void *c
double **output
thiscouldbeyourlibrary.h:
struct foo
{
char a;
int b;
void *c;
double **output;
};
main.pyx:
cimport wrapper
cdef class Bar:
cdef wrapper.foo __stuff
def __init__(self):
self.__stuff.a = <char>NULL
self.__stuff.b = <int>NULL
self.__stuff.c = NULL
def main():
bar = Bar()
print bar.__stuff.a
print bar.__stuff.b
If you had allocated the memory for output before, you could have done it:
self.__stuff.output[0] = NULL
without allocating it will crash...
Related
I have a function with void * parameter and I want to store it in the dict.
What I do is:
%%cython
cdef void foo(void * bar):
pass
cdef dict foobar = {'foo': foo}
But this code raises an Error: Cannot convert 'void (void *)' to Python object Any ways to overcome this problem?
The easiest solution is to create a cdef class that can wrap this function. Since the cdef class is a Python object it can be stored in a dictionary like any other Python object.
ctypedef void (*void_func_ptr)(void*)
cdef class VoidFuncWrapper:
cdef void_func_ptr func
def __cinit__(self):
self.func = NULL
#staticmethod
cdef VoidFuncWrapper make_from_ptr(void_func_ptr f):
cdef VoidFuncWrapper out = VoidFuncWrapper()
out.func = f
return out
Then you can simply do:
cdef dict foobar = {'foo': VoidFuncWrapper.make_from_ptr(foo)}
Your problem here is that void* has no defined converter from a Python object, so a Python wrapper function (that accepts Python objects, converts them, and passes them to the underlying C function) can't be defined, either explicitly with cpdef or implicitly by putting the cdef function in a Python object (the dict in this case).
You could make this work by defining the argument to be something Cython knows how to convert to, e.g.:
cpdef void foo(const char * bar):
pass
cdef dict foobar = {'foo': foo}
Try it online!
But that may not work for your scenario if the function needs to accept an arbitrary pointer. If that's the case, you may want to switch to using C++ containers that can hold your function pointer type directly:
from libcpp.unordered_map cimport unordered_map
from libcpp.string cimport string
cdef void foo(void *bar):
pass
ctypedef void (*f_type)(void *)
cdef unordered_map[string, f_type] foobar
foobar['foo'.encode('ascii')] = foo
I have a C function which signature looks like this:
typedef double (*func_t)(double*, int)
int some_f(func_t myFunc);
I would like to pass a Python function (not necessarily explicitly) as an argument for some_f. Unfortunately, I can't afford to alter declaration of some_f, that's it: I shouldn't change C code.
One obvious thing I tried to do is to create a basic wrapping function like this:
cdef double wraping_f(double *d, int i /*?, object f */):
/*do stuff*/
return <double>f(d_t)
However, I can't come up with a way to actually "put" it inside wrapping_f's body.
There is a very bad solution to this problem: I could use a global object variable, however this forces me copy-n-paste multiple instances of essentially same wrapper function that will use different global functions (I am planning to use multiple Python functions simultaneously).
I keep my other answer for historical reasons - it shows, that there is no way to do what you want without jit-compilation and helped me to understand how great #DavidW's advise in this answer was.
For the sake of simplicity, I use a slightly simpler signature of functions and trust you to change it accordingly to your needs.
Here is a blueprint for a closure, which lets ctypes do the jit-compilation behind the scenes:
%%cython
#needs Cython > 0.28 to run because of verbatim C-code
cdef extern from *: #fill some_t with life
"""
typedef int (*func_t)(int);
static int some_f(func_t fun){
return fun(42);
}
"""
ctypedef int (*func_t)(int)
int some_f(func_t myFunc)
#works with any recent Cython version:
import ctypes
cdef class Closure:
cdef object python_fun
cdef object jitted_wrapper
def inner_fun(self, int arg):
return self.python_fun(arg)
def __cinit__(self, python_fun):
self.python_fun=python_fun
ftype = ctypes.CFUNCTYPE(ctypes.c_int,ctypes.c_int) #define signature
self.jitted_wrapper=ftype(self.inner_fun) #jit the wrapper
cdef func_t get_fun_ptr(self):
return (<func_t *><size_t>ctypes.addressof(self.jitted_wrapper))[0]
def use_closure(Closure closure):
print(some_f(closure.get_fun_ptr()))
And now using it:
>>> cl1, cl2=Closure(lambda x:2*x), Closure(lambda x:3*x)
>>> use_closure(cl1)
84
>>> use_closure(cl2)
126
This answer is more in Do-It-Yourself style and while not unintersting you should refer to my other answer for a concise recept.
This answer is a hack and a little bit over the top, it only works for Linux64 and probably should not be recommended - yet I just cannot stop myself from posting it.
There are actually four versions:
how easy the life could be, if the API would take the possibility of closures into consideration
using a global state to produce a single closure [also considered by you]
using multiple global states to produce multiple closures at the same time [also considered by you]
using jit-compiled functions to produce an arbitrary number of closures at the same time
For the sake of simplicity I chose a simpler signature of func_t - int (*func_t)(void).
I know, you cannot change the API. Yet I cannot embark on a journey full of pain, without mentioning how simple it could be... There is a quite common trick to fake closures with function pointers - just add an additional parameter to your API (normally void *), i.e:
#version 1: Life could be so easy
# needs Cython >= 0.28 because of verbatim C-code feature
%%cython
cdef extern from *: #fill some_t with life
"""
typedef int (*func_t)(void *);
static int some_f(func_t fun, void *params){
return fun(params);
}
"""
ctypedef int (*func_t)(void *)
int some_f(func_t myFunc, void *params)
cdef int fun(void *obj):
print(<object>obj)
return len(<object>obj)
def doit(s):
cdef void *params = <void*>s
print(some_f(&fun, params))
We basically use void *params to pass the inner state of the closure to fun and so the result of fun can depend on this state.
The behavior is as expected:
>>> doit('A')
A
1
But alas, the API is how it is. We could use a global pointer and a wrapper to pass the information:
#version 2: Use global variable for information exchange
# needs Cython >= 0.28 because of verbatim C-code feature
%%cython
cdef extern from *:
"""
typedef int (*func_t)();
static int some_f(func_t fun){
return fun();
}
static void *obj_a=NULL;
"""
ctypedef int (*func_t)()
int some_f(func_t myFunc)
void *obj_a
cdef int fun(void *obj):
print(<object>obj)
return len(<object>obj)
cdef int wrap_fun():
global obj_a
return fun(obj_a)
cdef func_t create_fun(obj):
global obj_a
obj_a=<void *>obj
return &wrap_fun
def doit(s):
cdef func_t fun = create_fun(s)
print(some_f(fun))
With the expected behavior:
>>> doit('A')
A
1
create_fun is just convenience, which sets the global object and return the corresponding wrapper around the original function fun.
NB: It would be safer to make obj_a a Python-object, because void * could become dangling - but to keep the code nearer to versions 1 and 4 we use void * instead of object.
But what if there are more than one closure in use at the same time, let's say 2? Obviously with the approach above we need 2 global objects and two wrapper functions to achieve our goal:
#version 3: two function pointers at the same time
%%cython
cdef extern from *:
"""
typedef int (*func_t)();
static int some_f(func_t fun){
return fun();
}
static void *obj_a=NULL;
static void *obj_b=NULL;
"""
ctypedef int (*func_t)()
int some_f(func_t myFunc)
void *obj_a
void *obj_b
cdef int fun(void *obj):
print(<object>obj)
return len(<object>obj)
cdef int wrap_fun_a():
global obj_a
return fun(obj_a)
cdef int wrap_fun_b():
global obj_b
return fun(obj_b)
cdef func_t create_fun(obj) except NULL:
global obj_a, obj_b
if obj_a == NULL:
obj_a=<void *>obj
return &wrap_fun_a
if obj_b == NULL:
obj_b=<void *>obj
return &wrap_fun_b
raise Exception("Not enough slots")
cdef void delete_fun(func_t fun):
global obj_a, obj_b
if fun == &wrap_fun_a:
obj_a=NULL
if fun == &wrap_fun_b:
obj_b=NULL
def doit(s):
ss = s+s
cdef func_t fun1 = create_fun(s)
cdef func_t fun2 = create_fun(ss)
print(some_f(fun2))
print(some_f(fun1))
delete_fun(fun1)
delete_fun(fun2)
After compiling, as expected:
>>> doit('A')
AA
2
A
1
But what if we have to provide an arbitrary number of function-pointers at the same time?
The problem is, that we need to create the wrapper-functions at the run time, because there is no way to know how many we will need while compiling, so the only thing I can think of is to jit-compile these wrapper-functions when they are needed.
The wrapper function looks quite simple, here in assembler:
wrapper_fun:
movq address_of_params, %rdi ; void *param is the parameter of fun
movq address_of_fun, %rax ; addresse of the function which should be called
jmp *%rax ;jmp instead of call because it is last operation
The addresses of params and of fun will be known at run time, so we just have to link - replace the placeholder in the resulting machine code.
In my implementation I'm following more or less this great article: https://eli.thegreenplace.net/2017/adventures-in-jit-compilation-part-4-in-python/
#4. version: jit-compiled wrapper
%%cython
from libc.string cimport memcpy
cdef extern from *:
"""
typedef int (*func_t)(void);
static int some_f(func_t fun){
return fun();
}
"""
ctypedef int (*func_t)()
int some_f(func_t myFunc)
cdef extern from "sys/mman.h":
void *mmap(void *addr, size_t length, int prot, int flags,
int fd, size_t offset);
int munmap(void *addr, size_t length);
int PROT_READ # #define PROT_READ 0x1 /* Page can be read. */
int PROT_WRITE # #define PROT_WRITE 0x2 /* Page can be written. */
int PROT_EXEC # #define PROT_EXEC 0x4 /* Page can be executed. */
int MAP_PRIVATE # #define MAP_PRIVATE 0x02 /* Changes are private. */
int MAP_ANONYMOUS # #define MAP_ANONYMOUS 0x20 /* Don't use a file. */
# |-----8-byte-placeholder ---|
blue_print = b'\x48\xbf\x00\x00\x00\x00\x00\x00\x00\x00' # movabs 8-byte-placeholder,%rdi
blue_print+= b'\x48\xb8\x00\x00\x00\x00\x00\x00\x00\x00' # movabs 8-byte-placeholder,%rax
blue_print+= b'\xff\xe0' # jmpq *%rax ; jump to address in %rax
cdef func_t link(void *obj, void *fun_ptr) except NULL:
cdef size_t N=len(blue_print)
cdef char *mem=<char *>mmap(NULL, N,
PROT_READ | PROT_WRITE | PROT_EXEC,
MAP_PRIVATE | MAP_ANONYMOUS,
-1,0)
if <long long int>mem==-1:
raise OSError("failed to allocated mmap")
#copy blueprint:
memcpy(mem, <char *>blue_print, N);
#inject object address:
memcpy(mem+2, &obj, 8);
#inject function address:
memcpy(mem+2+8+2, &fun_ptr, 8);
return <func_t>(mem)
cdef int fun(void *obj):
print(<object>obj)
return len(<object>obj)
cdef func_t create_fun(obj) except NULL:
return link(<void *>obj, <void *>&fun)
cdef void delete_fun(func_t fun):
munmap(fun, len(blue_print))
def doit(s):
ss, sss = s+s, s+s+s
cdef func_t fun1 = create_fun(s)
cdef func_t fun2 = create_fun(ss)
cdef func_t fun3 = create_fun(sss)
print(some_f(fun2))
print(some_f(fun1))
print(some_f(fun3))
delete_fun(fun1)
delete_fun(fun2)
delete_fun(fun3)
And now, the expected behavior:
>>doit('A')
AA
2
A
1
AAA
3
After looking at this, maybe there is a change the API can be changed?
Cython equivalent of c define
#define myfunc(Node x,...) SetNode(x.getattributeNode(),__VA_ARGS__)
I have a c api SetNode which takes first argument a node of struct type node and N variables (N is variable number from 0-N)
here is a c example to solve such problum
exampleAPI.c
#include<stdarg.h>
float sumN(int len,...){
va_list argp;
int i;
float s=0;
va_start(argp,len);
for(i=0;i<len;i++){
s+=va_arg(argp,int);
}
va_end(argp);
}
exampleAPI.h
#include<stdarg.h>
float sumN(int len,...)
examplecode.c
#include<stdarg.h>
#include"exampleAPI.h"
int len(float first,...){
va_list argp;
int i=1;
va_start(argp,first);
while(1){
if(va_arg(argp,float)==NULL){
return i
}
else{
i++;
}
}
va_end(argp);
}
#define sum(...) sumN(len(__VA_ARGS__),__VA_ARGS__)
Now calling
sum(1,2,3,4);
will return 10.000000
sum(1.5,6.5);
will return 8.00000
I need a cython alternative for bellow c definition and not above example
because I have a C-API which has SetNode function which takes variable number of arguments and I want to wrap it in cython and call from python
#define myfunc(Node x,...) SetNode(x.getattributeNode(),__VA_ARGS__)
here Node is a class defined in cython which holds a c stuct as attribute and getattributeNode() is a function of Node class which returns c struct that needs to be passed into C-API.
cdef extern "Network.h":
ctypedef struct node_bn:
pass
node_bn* SetNode(node_bn* node,...)
cdef class Node:
cdef node_bn *node
cdef getattributeNode(self):
return self.node
def setNode(self,*arg):
self.node=SetNode(self.node,*arg) # Error cannot convert python objects to c type
Alternative thing I tried
cdef extern from "stdarg.h":
ctypedef struct va_list:
pass
ctypedef struct fake_type:
pass
void va_start(va_list, void* arg)
void* va_arg(va_list, fake_type)
void va_end(va_list)
fake_type int_type "int"
cdef extern "Network.h":
ctypedef struct node_bn:
pass
node_bn* VSetNode(node_bn* node,va_list argp)
cdef class Node:
cdef node_bn *node
cdef getattributeNode(self):
return self.node
cpdef _setNode(self,node_bn *node,...):
cdef va_list agrp
va_start(va_list, node)
self.node=VSetNode(node,argp)
va_end(va_list)
def setNode(self,*arg):
self._setNode(self.node,*arg)
works fine when argument list is empty
n = Node()
n.setNode() #This works
n.SetNode("top",1) # error takes exactly one argument given 3 in self._setNode(self.node,*arg)
If anyone could suggest cython equivalent to it, it would be great.
I don't think it's easily done though Cython (the problem is telling Cython what type conversions to do for an arbitrary number of arguments). The best I can suggest is to use the standard library ctypes library for this specific case and wrap the rest in Cython.
For the sake of an example, I've used a very simple sum function. va_sum.h contains:
typedef struct { double val; } node_bn;
node_bn* sum_va(node_bn* node,int len, ...);
/* on windows this must be:
__declspec(dllexport) node_bn* sum_va(node_bn* node,int len, ...);
*/
and va_sum.c contains:
#include <stdarg.h>
#include "va_sum.h"
node_bn* sum_va(node_bn* node,int len, ...) {
int i;
va_list vl;
va_start(vl,len);
for (i=0; i<len; ++i) {
node->val += va_arg(vl,double);
}
va_end(vl);
return node;
}
I've written it so it adds everything to a field in a structure just to demonstrate that you can pass pointers to structures without too much trouble.
The Cython file is:
# definition of a structure
cdef extern from "va_sum.h":
ctypedef struct node_bn:
double val;
# obviously you'll want to wrap things in terms of Python accessible classes, but this atleast demonstrates how it works
def test_sum(*args):
cdef node_bn input_node;
cdef node_bn* output_node_p;
input_node.val = 5.0 # create a node, and set an initial value
from ctypes import CDLL, c_double,c_void_p
import os.path
# load the Cython library with ctypes to gain access to the "sum_va" function
# Assuming you've linked it in when you build the Cython module
full_path = os.path.realpath(__file__)
this_file_library = CDLL(full_path)
# I treat all my arguments as doubles - you may need to do
# something more sophisticated, but the idea is the same:
# convert them to the c-type the function is expecting
args = [ c_double(arg) for arg in args ]
sum_va = this_file_library.sum_va
sum_va.restype = c_void_p # it returns a pointer
# pass the pointers as a void pointer
# my c compiler warns me if I used int instead of long
# but which integer type you have to use is likely system dependent
# and somewhere you have to be careful
output_node_p_as_integer = sum_va(c_void_p(<long>&input_node),len(args),
*args)
# unfortunately getting the output needs a two stage typecast - first to a long, then to a pointer
output_node_p = <node_bn*>(<long>(output_node_p_as_integer))
return output_node_p.val
You need to compile your va_sum.c together with your Cython file (e.g. by adding sources = ['cython_file.pyx','va_sum.c'] in setup.py)
Ctypes is probably a bit slower than Cython (I think there's a reasonable overhead on each call), and it's odd to mix them, but this should at least let you write the main wrapper in Cython, and use ctypes to get round the specific limitation.
This is probably not the proper answer, since I am not sure I understand the question fully. I would have replied in a comment, but the code formatting is too poor.
In Python the functions sum and len are available:
def my_len(*args):
return len(args)
def my_sum(*args):
return sum(args)
print "len =", my_len("hello", 123, "there")
print "sum =", my_sum(6.5, 1.5, 2.0)
outputs:
len = 3
sum = 10.0
How can I convert Python object to C void type using Cython?
Currently I am getting this message when I try to cast
Casting temporary Python object to non-numeric non-Python type
This can be done like this :
1. Cast from Python to C
If you really meant void * this would be :
some_pyobj = "abc"
cdef void *ptr
ptr = <void *>some_pyobj
If you meant PyObject * this would be :
cdef PyObject *ptr
ptr = <PyObject *>some_pyobj # Cast from Python object to C pointer
Then, from C side, the PyObject struct is available by including Python.h.
Here is the reference (from object.h Python include file) :
/* Nothing is actually declared to be a PyObject, but every pointer to
* a Python object can be cast to a PyObject*. This is inheritance built
* by hand. Similarly every pointer to a variable-size Python object can,
* in addition, be cast to PyVarObject*.
*/
typedef struct _object {
PyObject_HEAD
} PyObject;
2. Cast from C to Python
It works in both ways, meaning that the following is also possible :
cdef PyObject *ptr
ptr = <PyObject *>some_pyobj
cdef object some_other_pyobj
some_other_pyobj = <object>ptr # Cast from C pointer to Python object
You can:
def run():
pyobj = 'abc'
pyobj_void_star = <void *>pyobj
cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
def __cinit__(self, Bar *b)
bar = b
This will always give me something like:
Cannot convert Python object argument to type 'Bar *'
Is there a way to accomplish this, or do I need to extract everything from a Bar object, create a Python equivalent, pass it in, then reconstruct it in PyClass?
I came across this problem trying to wrap C code with structs as python classes. The issue seems to be that "special" function including __init__ and __cinit__ must be declared as def rather than cdef. This means that they can be called from normal python, so the type parameters are effectively ignored and everything is treated as object.
In J.F. Sebastian's answer the fix is not the wrapping - a double is a basic numeric type and so there is a default conversion between the C/C++ type and the Python object. Czarek's answer is basically correct - you need to use a fake constructor idiom, using a global function. It is not possible to use a #staticmethod decorator as they cannot be applied to cdef functions. The answer looks simpler on the original example provided.
cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
cdef PyClass_Init(Bar *b):
result = PyClass()
result.bar = b
return result
As of Cython 0.21 it has been possible to declare cdef methods with the #staticmethod decorator. This allows static creator methods that take non-Python arguments:
cdef extern from "Foo.h":
cdef cppclass Bar:
pass
cdef class PyClass:
cdef Bar *bar
#staticmethod
cdef create(Bar *bar):
cdef PyClass pc = PyClass()
pc.bar = bar
return pc
For each cdef class create a global cdef function that acts as a constructor, CefResponse is a C++ object, PyResponse a python equivalent of a c++ object:
cdef object CreatePyResponse(CefRefPtr[CefResponse] cefResponse):
pyResponse = PyResponse()
pyResponse.cefResponse = cefResponse
return pyResponse
cdef class PyResponse:
cdef CefRefPtr[CefResponse] cefResponse
def GetStatus(self):
return (<CefResponse*>(self.cefResponse.get())).GetStatus()
So instead of resp = PyResponse(cppObject) call resp = CreatePyResponse(cppObject).
Example taken from CEF Python:
https://code.google.com/p/cefpython/source/browse/cefpython/response.pyx?r=0250b65e046a
Python class accepts Python arguments. To pass a C++ argument you need to wrap it:
# distutils: language = c++
cdef extern from "Foo.h" namespace "baz":
cdef cppclass Bar:
Bar(double d)
double get()
cdef class PyBar: # wrap Bar class
cdef Bar *thisptr
def __cinit__(self, double d):
self.thisptr = new Bar(d)
def __dealloc__(self):
del self.thisptr
property d:
def __get__(self):
return self.thisptr.get()
PyBar instances can be used as any other Python objects both from Cython and pure Python:
class PyClass:
def __init__(self, PyBar bar):
self.bar = bar
print(PyClass(PyBar(1)).bar.d)