We Keep Coding

How to use the `restrict` keyword in Cython?

I'm using Cython in CPython 3.6 and I'd like to mark some pointers as "not aliased", to improve performance and be explicit semantically.
In C this is done with the restrict (__restrict, __restrict__) keyword. But how do I use restrict on my cdef variables from a Cython .pyx code?
Thanks!

Cython doesn't have syntax/support for the restrict-keyword (yet?). Your best bet is to code this function in C, most convenient is probably to use verbatim-C-code, e.g. here for a dummy example:
%%cython
cdef extern from *:
"""
void c_fun(int * CYTHON_RESTRICT a, int * CYTHON_RESTRICT b){
a[0] = b[0];
}
"""
void c_fun(int *a, int *b)
# example usage
def doit(k):
cdef int i=k;
cdef int j=k+1;
c_fun(&i,&j)
print(i)
Here I use Cython's (undocumented) define CYTHON_RESTRICT, which is defined as
// restrict
#ifndef CYTHON_RESTRICT
#if defined(__GNUC__)
#define CYTHON_RESTRICT __restrict__
#elif defined(_MSC_VER) && _MSC_VER >= 1400
#define CYTHON_RESTRICT __restrict
#elif defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L
#define CYTHON_RESTRICT restrict
#else
#define CYTHON_RESTRICT
#endif
#endif
so it works not only for C99-compliant c-compiler. In the long run however, it is probably best to define something similar and not to depend on undocumented features.

Making a Cython wrapper around Python function

I have a C function which signature looks like this:
typedef double (*func_t)(double*, int)
int some_f(func_t myFunc);
I would like to pass a Python function (not necessarily explicitly) as an argument for some_f. Unfortunately, I can't afford to alter declaration of some_f, that's it: I shouldn't change C code.
One obvious thing I tried to do is to create a basic wrapping function like this:
cdef double wraping_f(double *d, int i /*?, object f */):
/*do stuff*/
return <double>f(d_t)
However, I can't come up with a way to actually "put" it inside wrapping_f's body.
There is a very bad solution to this problem: I could use a global object variable, however this forces me copy-n-paste multiple instances of essentially same wrapper function that will use different global functions (I am planning to use multiple Python functions simultaneously).

I keep my other answer for historical reasons - it shows, that there is no way to do what you want without jit-compilation and helped me to understand how great #DavidW's advise in this answer was.
For the sake of simplicity, I use a slightly simpler signature of functions and trust you to change it accordingly to your needs.
Here is a blueprint for a closure, which lets ctypes do the jit-compilation behind the scenes:
%%cython
#needs Cython > 0.28 to run because of verbatim C-code
cdef extern from *: #fill some_t with life
"""
typedef int (*func_t)(int);
static int some_f(func_t fun){
return fun(42);
}
"""
ctypedef int (*func_t)(int)
int some_f(func_t myFunc)
#works with any recent Cython version:
import ctypes
cdef class Closure:
cdef object python_fun
cdef object jitted_wrapper
def inner_fun(self, int arg):
return self.python_fun(arg)
def __cinit__(self, python_fun):
self.python_fun=python_fun
ftype = ctypes.CFUNCTYPE(ctypes.c_int,ctypes.c_int) #define signature
self.jitted_wrapper=ftype(self.inner_fun) #jit the wrapper
cdef func_t get_fun_ptr(self):
return (<func_t *><size_t>ctypes.addressof(self.jitted_wrapper))[0]
def use_closure(Closure closure):
print(some_f(closure.get_fun_ptr()))
And now using it:
>>> cl1, cl2=Closure(lambda x:2*x), Closure(lambda x:3*x)
>>> use_closure(cl1)
84
>>> use_closure(cl2)
126

This answer is more in Do-It-Yourself style and while not unintersting you should refer to my other answer for a concise recept.
This answer is a hack and a little bit over the top, it only works for Linux64 and probably should not be recommended - yet I just cannot stop myself from posting it.
There are actually four versions:
how easy the life could be, if the API would take the possibility of closures into consideration
using a global state to produce a single closure [also considered by you]
using multiple global states to produce multiple closures at the same time [also considered by you]
using jit-compiled functions to produce an arbitrary number of closures at the same time
For the sake of simplicity I chose a simpler signature of func_t - int (*func_t)(void).
I know, you cannot change the API. Yet I cannot embark on a journey full of pain, without mentioning how simple it could be... There is a quite common trick to fake closures with function pointers - just add an additional parameter to your API (normally void *), i.e:
#version 1: Life could be so easy
# needs Cython >= 0.28 because of verbatim C-code feature
%%cython
cdef extern from *: #fill some_t with life
"""
typedef int (*func_t)(void *);
static int some_f(func_t fun, void *params){
return fun(params);
}
"""
ctypedef int (*func_t)(void *)
int some_f(func_t myFunc, void *params)
cdef int fun(void *obj):
print(<object>obj)
return len(<object>obj)
def doit(s):
cdef void *params = <void*>s
print(some_f(&fun, params))
We basically use void *params to pass the inner state of the closure to fun and so the result of fun can depend on this state.
The behavior is as expected:
>>> doit('A')
A
1
But alas, the API is how it is. We could use a global pointer and a wrapper to pass the information:
#version 2: Use global variable for information exchange
# needs Cython >= 0.28 because of verbatim C-code feature
%%cython
cdef extern from *:
"""
typedef int (*func_t)();
static int some_f(func_t fun){
return fun();
}
static void *obj_a=NULL;
"""
ctypedef int (*func_t)()
int some_f(func_t myFunc)
void *obj_a
cdef int fun(void *obj):
print(<object>obj)
return len(<object>obj)
cdef int wrap_fun():
global obj_a
return fun(obj_a)
cdef func_t create_fun(obj):
global obj_a
obj_a=<void *>obj
return &wrap_fun
def doit(s):
cdef func_t fun = create_fun(s)
print(some_f(fun))
With the expected behavior:
>>> doit('A')
A
1
create_fun is just convenience, which sets the global object and return the corresponding wrapper around the original function fun.
NB: It would be safer to make obj_a a Python-object, because void * could become dangling - but to keep the code nearer to versions 1 and 4 we use void * instead of object.
But what if there are more than one closure in use at the same time, let's say 2? Obviously with the approach above we need 2 global objects and two wrapper functions to achieve our goal:
#version 3: two function pointers at the same time
%%cython
cdef extern from *:
"""
typedef int (*func_t)();
static int some_f(func_t fun){
return fun();
}
static void *obj_a=NULL;
static void *obj_b=NULL;
"""
ctypedef int (*func_t)()
int some_f(func_t myFunc)
void *obj_a
void *obj_b
cdef int fun(void *obj):
print(<object>obj)
return len(<object>obj)
cdef int wrap_fun_a():
global obj_a
return fun(obj_a)
cdef int wrap_fun_b():
global obj_b
return fun(obj_b)
cdef func_t create_fun(obj) except NULL:
global obj_a, obj_b
if obj_a == NULL:
obj_a=<void *>obj
return &wrap_fun_a
if obj_b == NULL:
obj_b=<void *>obj
return &wrap_fun_b
raise Exception("Not enough slots")
cdef void delete_fun(func_t fun):
global obj_a, obj_b
if fun == &wrap_fun_a:
obj_a=NULL
if fun == &wrap_fun_b:
obj_b=NULL
def doit(s):
ss = s+s
cdef func_t fun1 = create_fun(s)
cdef func_t fun2 = create_fun(ss)
print(some_f(fun2))
print(some_f(fun1))
delete_fun(fun1)
delete_fun(fun2)
After compiling, as expected:
>>> doit('A')
AA
2
A
1
But what if we have to provide an arbitrary number of function-pointers at the same time?
The problem is, that we need to create the wrapper-functions at the run time, because there is no way to know how many we will need while compiling, so the only thing I can think of is to jit-compile these wrapper-functions when they are needed.
The wrapper function looks quite simple, here in assembler:
wrapper_fun:
movq address_of_params, %rdi ; void *param is the parameter of fun
movq address_of_fun, %rax ; addresse of the function which should be called
jmp *%rax ;jmp instead of call because it is last operation
The addresses of params and of fun will be known at run time, so we just have to link - replace the placeholder in the resulting machine code.
In my implementation I'm following more or less this great article: https://eli.thegreenplace.net/2017/adventures-in-jit-compilation-part-4-in-python/
#4. version: jit-compiled wrapper
%%cython
from libc.string cimport memcpy
cdef extern from *:
"""
typedef int (*func_t)(void);
static int some_f(func_t fun){
return fun();
}
"""
ctypedef int (*func_t)()
int some_f(func_t myFunc)
cdef extern from "sys/mman.h":
void *mmap(void *addr, size_t length, int prot, int flags,
int fd, size_t offset);
int munmap(void *addr, size_t length);
int PROT_READ # #define PROT_READ 0x1 /* Page can be read. */
int PROT_WRITE # #define PROT_WRITE 0x2 /* Page can be written. */
int PROT_EXEC # #define PROT_EXEC 0x4 /* Page can be executed. */
int MAP_PRIVATE # #define MAP_PRIVATE 0x02 /* Changes are private. */
int MAP_ANONYMOUS # #define MAP_ANONYMOUS 0x20 /* Don't use a file. */
# |-----8-byte-placeholder ---|
blue_print = b'\x48\xbf\x00\x00\x00\x00\x00\x00\x00\x00' # movabs 8-byte-placeholder,%rdi
blue_print+= b'\x48\xb8\x00\x00\x00\x00\x00\x00\x00\x00' # movabs 8-byte-placeholder,%rax
blue_print+= b'\xff\xe0' # jmpq *%rax ; jump to address in %rax
cdef func_t link(void *obj, void *fun_ptr) except NULL:
cdef size_t N=len(blue_print)
cdef char *mem=<char *>mmap(NULL, N,
PROT_READ | PROT_WRITE | PROT_EXEC,
MAP_PRIVATE | MAP_ANONYMOUS,
-1,0)
if <long long int>mem==-1:
raise OSError("failed to allocated mmap")
#copy blueprint:
memcpy(mem, <char *>blue_print, N);
#inject object address:
memcpy(mem+2, &obj, 8);
#inject function address:
memcpy(mem+2+8+2, &fun_ptr, 8);
return <func_t>(mem)
cdef int fun(void *obj):
print(<object>obj)
return len(<object>obj)
cdef func_t create_fun(obj) except NULL:
return link(<void *>obj, <void *>&fun)
cdef void delete_fun(func_t fun):
munmap(fun, len(blue_print))
def doit(s):
ss, sss = s+s, s+s+s
cdef func_t fun1 = create_fun(s)
cdef func_t fun2 = create_fun(ss)
cdef func_t fun3 = create_fun(sss)
print(some_f(fun2))
print(some_f(fun1))
print(some_f(fun3))
delete_fun(fun1)
delete_fun(fun2)
delete_fun(fun3)
And now, the expected behavior:
>>doit('A')
AA
2
A
1
AAA
3
After looking at this, maybe there is a change the API can be changed?

Cython - Passing variable number of arguments from Python to C ellipsis [duplicate]

Cython equivalent of c define
#define myfunc(Node x,...) SetNode(x.getattributeNode(),__VA_ARGS__)
I have a c api SetNode which takes first argument a node of struct type node and N variables (N is variable number from 0-N)
here is a c example to solve such problum
exampleAPI.c
#include<stdarg.h>
float sumN(int len,...){
va_list argp;
int i;
float s=0;
va_start(argp,len);
for(i=0;i<len;i++){
s+=va_arg(argp,int);
}
va_end(argp);
}
exampleAPI.h
#include<stdarg.h>
float sumN(int len,...)
examplecode.c
#include<stdarg.h>
#include"exampleAPI.h"
int len(float first,...){
va_list argp;
int i=1;
va_start(argp,first);
while(1){
if(va_arg(argp,float)==NULL){
return i
}
else{
i++;
}
}
va_end(argp);
}
#define sum(...) sumN(len(__VA_ARGS__),__VA_ARGS__)
Now calling
sum(1,2,3,4);
will return 10.000000
sum(1.5,6.5);
will return 8.00000
I need a cython alternative for bellow c definition and not above example
because I have a C-API which has SetNode function which takes variable number of arguments and I want to wrap it in cython and call from python
#define myfunc(Node x,...) SetNode(x.getattributeNode(),__VA_ARGS__)
here Node is a class defined in cython which holds a c stuct as attribute and getattributeNode() is a function of Node class which returns c struct that needs to be passed into C-API.
cdef extern "Network.h":
ctypedef struct node_bn:
pass
node_bn* SetNode(node_bn* node,...)
cdef class Node:
cdef node_bn *node
cdef getattributeNode(self):
return self.node
def setNode(self,*arg):
self.node=SetNode(self.node,*arg) # Error cannot convert python objects to c type
Alternative thing I tried
cdef extern from "stdarg.h":
ctypedef struct va_list:
pass
ctypedef struct fake_type:
pass
void va_start(va_list, void* arg)
void* va_arg(va_list, fake_type)
void va_end(va_list)
fake_type int_type "int"
cdef extern "Network.h":
ctypedef struct node_bn:
pass
node_bn* VSetNode(node_bn* node,va_list argp)
cdef class Node:
cdef node_bn *node
cdef getattributeNode(self):
return self.node
cpdef _setNode(self,node_bn *node,...):
cdef va_list agrp
va_start(va_list, node)
self.node=VSetNode(node,argp)
va_end(va_list)
def setNode(self,*arg):
self._setNode(self.node,*arg)
works fine when argument list is empty
n = Node()
n.setNode() #This works
n.SetNode("top",1) # error takes exactly one argument given 3 in self._setNode(self.node,*arg)
If anyone could suggest cython equivalent to it, it would be great.

I don't think it's easily done though Cython (the problem is telling Cython what type conversions to do for an arbitrary number of arguments). The best I can suggest is to use the standard library ctypes library for this specific case and wrap the rest in Cython.
For the sake of an example, I've used a very simple sum function. va_sum.h contains:
typedef struct { double val; } node_bn;
node_bn* sum_va(node_bn* node,int len, ...);
/* on windows this must be:
__declspec(dllexport) node_bn* sum_va(node_bn* node,int len, ...);
*/
and va_sum.c contains:
#include <stdarg.h>
#include "va_sum.h"
node_bn* sum_va(node_bn* node,int len, ...) {
int i;
va_list vl;
va_start(vl,len);
for (i=0; i<len; ++i) {
node->val += va_arg(vl,double);
}
va_end(vl);
return node;
}
I've written it so it adds everything to a field in a structure just to demonstrate that you can pass pointers to structures without too much trouble.
The Cython file is:
# definition of a structure
cdef extern from "va_sum.h":
ctypedef struct node_bn:
double val;
# obviously you'll want to wrap things in terms of Python accessible classes, but this atleast demonstrates how it works
def test_sum(*args):
cdef node_bn input_node;
cdef node_bn* output_node_p;
input_node.val = 5.0 # create a node, and set an initial value
from ctypes import CDLL, c_double,c_void_p
import os.path
# load the Cython library with ctypes to gain access to the "sum_va" function
# Assuming you've linked it in when you build the Cython module
full_path = os.path.realpath(__file__)
this_file_library = CDLL(full_path)
# I treat all my arguments as doubles - you may need to do
# something more sophisticated, but the idea is the same:
# convert them to the c-type the function is expecting
args = [ c_double(arg) for arg in args ]
sum_va = this_file_library.sum_va
sum_va.restype = c_void_p # it returns a pointer
# pass the pointers as a void pointer
# my c compiler warns me if I used int instead of long
# but which integer type you have to use is likely system dependent
# and somewhere you have to be careful
output_node_p_as_integer = sum_va(c_void_p(<long>&input_node),len(args),
*args)
# unfortunately getting the output needs a two stage typecast - first to a long, then to a pointer
output_node_p = <node_bn*>(<long>(output_node_p_as_integer))
return output_node_p.val
You need to compile your va_sum.c together with your Cython file (e.g. by adding sources = ['cython_file.pyx','va_sum.c'] in setup.py)
Ctypes is probably a bit slower than Cython (I think there's a reasonable overhead on each call), and it's odd to mix them, but this should at least let you write the main wrapper in Cython, and use ctypes to get round the specific limitation.

This is probably not the proper answer, since I am not sure I understand the question fully. I would have replied in a comment, but the code formatting is too poor.
In Python the functions sum and len are available:
def my_len(*args):
return len(args)
def my_sum(*args):
return sum(args)
print "len =", my_len("hello", 123, "there")
print "sum =", my_sum(6.5, 1.5, 2.0)
outputs:
len = 3
sum = 10.0

How to extern from <algorithm> in Cython

I'm trying to use partial_sort from the <algorithm> library within Cython, but I just cannot find the correct way to properly extern it.
reference
Here's my failed attempt:
%%cython -f
cdef extern from "<algorithm>" namespace "std":
void partial_sort[RandomAccessIterator](RandomAccessIterator first, RandomAccessIterator middle, RandomAccessIterator last)
void partial_sort[RandomAccessIterator, Compare](RandomAccessIterator first, RandomAccessIterator middle, RandomAccessIterator last, Compare comp)
Error message when using Cython 0.19.1:
Error compiling Cython file:
------------------------------------------------------------
...
cdef extern from "<algorithm>" namespace "std":
cdef cppclass RandomAccessIterator:
cppclass Compare
void partial_sort[RandomAccessIterator]
^
------------------------------------------------------------
/Users/richizy/.ipython/cython/_cython_magic_cf4fbe14563c3de19c8c3af3253a182e.pyx:5:46: Not allowed in a constant expression
Error compiling Cython file:
------------------------------------------------------------
...
cdef extern from "<algorithm>" namespace "std":
cdef cppclass RandomAccessIterator:
cppclass Compare
void partial_sort[RandomAccessIterator]
^
------------------------------------------------------------
/Users/richizy/.ipython/cython/_cython_magic_cf4fbe14563c3de19c8c3af3253a182e.pyx:5:46: Array dimension not integer
Error compiling Cython file:
------------------------------------------------------------
...
cdef extern from "<algorithm>" namespace "std":
cdef cppclass RandomAccessIterator:
cppclass Compare
void partial_sort[RandomAccessIterator]
^
------------------------------------------------------------
/Users/richizy/.ipython/cython/_cython_magic_cf4fbe14563c3de19c8c3af3253a182e.pyx:5:25: Array element type 'void' is incomplete
Error message when using Cython 0.20.1:
CompileError: command 'gcc' failed with exit status 1
warning: .ipython/cython/_cython_magic_121a91d1fdd64d85c4b01e6540fd86d6.pyx:4:52: Function signature does not match previous declaration
Edit: As of 2/22/14, for Cython 0.20.1
https://groups.google.com/forum/#!topic/cython-users/H4UEM6IlvpM
Correct, Cython does not support default template specializations (for
functions or classes). Nor does it support non-typename template
parameters (with out some hacking). Both are missing features that
we'd like to get to someday.
Robert

It seems that Cython doesn't work well with template specialization. The following code works for me (Cython version 0.20, Python 2.7.5, g++ (SUSE Linux) 4.8.1)
from libcpp.vector cimport vector
cdef extern from "<algorithm>" namespace "std":
void partial_sort[RandomAccessIterator](RandomAccessIterator first, RandomAccessIterator middle, RandomAccessIterator last)
# void partial_sort[RandomAccessIterator, Compare](RandomAccessIterator first, RandomAccessIterator middle, RandomAccessIterator last, Compare comp)
cpdef bla():
cdef vector[int] v
cdef int i = 0
cdef list res = []
v.push_back(4)
v.push_back(6)
v.push_back(2)
v.push_back(5)
partial_sort[vector[int].iterator](v.begin(), v.end(), v.end())
for i in v:
res.append(i)
return res
Then
>>> import bla
>>> bla.bla()
[2, 4, 5, 6]
However uncommenting the line break the code with
bla.pyx:15:16: Wrong number of template arguments: expected 2, got 1
Here is a workaround: You declare the different specialization of the template function under two different names:
cdef extern from "<algorithm>" namespace "std":
void partial_sort_1 "std::partial_sort"[RandomAccessIterator](RandomAccessIterator first, RandomAccessIterator middle, RandomAccessIterator last)
void partial_sort_2 "std::partial_sort"[RandomAccessIterator, Compare](RandomAccessIterator first, RandomAccessIterator middle, RandomAccessIterator last, Compare comp)
And then you use the correct one as in:
partial_sort_1[vector[int].iterator](v.begin(), v.end(), v.end())
Note: I got from Cython-Users that this is a know problem and that having template partial specialization on Cython is on their wish list for a moment.

Python ImportError : cannot find symbol - for custom C++ module

I have created 2 modules in C++ and imported them in Python via Cython.
The first module is called Generator and it contains 2 classes : MersenneTwister, which implements algorithms for generating pseud-random numbers and a Generator class which is a singleton, creating a MersenneTwister object.
The second module is called KMeans and contains a function calls KMeans implementing the K-Means algorithm for clustering data.
For both modules I have created the corresponding .pyx files, which where compiles OK by cython and then, using g++ I have created the .so files and then imported the KMeans.so files in python, receiving : ImportError: ./KMeans.so: undefined symbol: _ZN9Generator8instanceEv
I have compiles the .cpp files using : g++ -shared -pthread -fPIC -fwrapv -fno-strict-aliasing -o file.so file.cpp
# KMeans.pyx files
include "Generator.pyx"
from libcpp.vector cimport vector
cdef import from "KMeans.h" :
cdef struct Pair:
vector[vector[double]] C
vector[unsigned int] J
Pair(vector[vector[double]] _C, vector[unsigned int] _J)
cdef double s(vector[double] X, vector[double] c)
cdef void AddToCodeword(vector[double] X, vector[double] c)
cdef Pair KMeans(unsigned int k, vector[vector[double]] X) except +
def KMean(unsigned k, X):
cdef Pair returnValue = KMeans(k, X)
return returnValue.C, returnValue.J
#Generator.pyx file
cdef extern from "mt.h":
cppclass MersenneTwister:
MersenneTwister()
#public members
double random()
void init_genrand(unsigned long s)
void init_by_array(unsigned long* init_key, int key_length)
unsigned long genrand_int32()
long genrand_int31()
double genrand_real1()
double genrand_real2()
double genrand_real3()
double genrand_res53()
#private members
unsigned long* mt_
int mti_
unsigned long* init_key_
int key_length_
unsigned long s_
int seeded_by_array_
int seeded_by_int_
#static members
cdef extern from "mt.h" namespace "MersenneTwister":
int N = 624
int M = 397
unsigned long MATRIX_A = 0x9908b0dfUL
unsigned long UPPER_MASK = 0x80000000UL
unsigned long LOWER_MASK = 0x7fffffffUL
cdef extern from "generator.h" namespace "Generator":
MersenneTwister *_ptr_
MersenneTwister *instance()
MersenneTwister *instance(unsigned int seed)