Let us have script foo.pyx with function in it:
def hello():
cdef int* i = <int *> malloc(sizeof(int))
i[0] = 1
trol(i)
print i
and script with function noo.pyx:
cdef trol(int * i):
i[0] = 42
the question is, how do I now import the trol function from file noo.pyx to foo.pyx, so I can use it in hello function.
This is only model example, but I think, that it illustrate the problem fair enough.
I tried simple
from noo import trol
but that throws "Cannot convert 'int *' to Python object"
Edit: I would better add, that this example will work just fine if I put both functions to same file.
This seems like something obvious to try, but did you try:
from noo cimport trol
If you use import instead of cimport, I think it will try to cast trol as a python function and generate the error you're getting.
The solution eventually was to create additional .pxd file, which something very similar to classic header .h file in C. It stores functions declarations and when cimport is called, it is in this file where it looks for functions and structures.
So to be specific, all I needed to do was to create file noo.pxd containing:
cdef trol(int * i)
and than we can simply cimport this function from foo.pyx by calling
from noo cimport trol
Related
As part of larger cython project I am getting an error that seems to indicate that maybe the automatic type merging is failing. It is possible I am doing something silly or that the fact that I am structuring things in this way is a bad smell. I think I can get around the issue by using void* and then casts everywhere but this seems quite messy for what should be a simple thing.
It seems similar to this issue, but that involved function pointers.
Wrong signature error using function types in multiple modules
I have put together a minimal example that demonstrates the issue. The files are as follows.
a.pxd
cdef struct A:
int a
cdef A makeA(int a)
a.pyx
cdef struct A:
pass
cdef A makeA(int a):
cdef A sA
sA.a = a
return sA
b.pyx
from cfr.a cimport A, makeA
cdef int get_A_value():
cdef A sA = makeA(5)
return sA.a
If I compile and then try to import b in a jupyter-lab notebook I get the following error:
b.pyx in init b()
TypeError: C function cfr.a.makeA has wrong signature (expected struct __pyx_t_3cfr_1a_A (int), got struct __pyx_t_1a_A (int))
EDIT: It looks like it might be something to do with my directory structure. Note that I do from cfr.a cimport A (as this happens to be the layout of my project). I think I need to do from a import A and get the import to work through the use of my pythonpath.
So it turned out the issue was in the from cfr.a cimport A line. This was somehow meaning that the A type was not being seen as the same type as A in the a.pyx file.
To fix use:
from a cimport A
(You may then get a not found error, but we can fix this with our pythonpath).
In the comments adding a __init__.pxd file was suggested, but adding a blank one did not work for me.
Although I few similar questions already being asked, but I couldn't get head around on how to fix.
Basically I have this function:
Module one.pyx:
cdef char *isInData(data, dataLength):
cdef char *ret = <char *>malloc(200)
memset(ret, 0x00, 200)
if (b"Hello" in data and b"World" in data):
strcat(ret, b"Hello World!")
return ret
Module two.pyx:
import one
from libc.stdlib cimport malloc,realloc, free
cdef char *tempo():
cdef char *str
str = one.isInData(b"Hello what is up World", 23)
# do some stuff
free(str)
Issue occurs on line str = one.isInData("Hello what is up World", 23), I am assuming that as soon as isInData->ret is assigned to str. isInData->ret is deleted which causes this issue. But annyone help me on how to fix this?
import one
This line does a Python import of one. It doesn't know about any cdef functions defined in one (or even that one is a Cython module...). Therefore it assumes that isInData is a Python object that it can look up and that it'll be a callable returning another Python object.
cdf char* str = some_python_function() is unsafe because str points into the Python object. However the Python object is only a temporary and is most likely freed almost immediately.
What you mean is:
cimport one
which tells Cython that it's a Cython module that you're importing, and it should expect to find the declarations at compile-time. You'll probably need to write a pxd file to give the declarations.
I haven't looked in detail at the rest of your code so don't know if you are handling C strings right. But in general you may find it easier just to use Python strings rather than messing around with C string handling.
There are 2 issues both relating to importing that may or may not be cython related?
I have the following simplified files to recreate the problem. All files are in the same directory. The .pyx files have successfully compiled into *.so, *.pyc and *.c files.
setup.py:
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules=cythonize("*.pyx"),
)
cy1.pyx: (cython)
cdef int timestwo(int x):
return x * 2
cy1.pxd:
cdef int timestwo(int x)
cy3.py: (normal python)
def tripleit(x):
return x*3
go.py:
from cy1 import timestwo
print str(timestwo(5))
Gives me the error: ImportError: cannot import name timestwo
if I change it to:
go.py:
import pyximport; pyximport.install()
import cy1
print str(cy1.timestwo(5))
it tells me: AttributeError: 'module' object has no attribute 'timestwo'
if I take out the cython all together and try to use normal python call from cy3.py:
go.py:
import cy3
print str(cy3.tripeleit(3))
I get: AttributeError: 'module' object has no attribute 'tripeleit'
and last if I do:
go.py:
from cy3 import tripleit
print str(tripeleit(3))
I get:
NameError: name 'tripeleit' is not defined
Sorry if this is super basic but I cannot seem to figure it out.
The problem is that in go.py:
from cy1 import timestwo
print str(timestwo(5))
you are trying to import a function defined as cdef.
To expose this function to Python you have to either use def or cpdef. Possibly you have to keep as cdef in order to cimport from other Cython files, justifying why you also have the pxd file. In that case I usually have a C-like function and a wrapper that can be called from Python.
In this case your cy1.pyx file would look like:
cdef int ctimestwo(int x):
return x * 2
def timestwo(x): # <-- small wrapper to expose ctimestwo() to Python
return ctimestwo(x)
and your cy1.pxd file:
cdef int ctimestwo(int x)
such that you can cimport only the ctimestwo function.
Regarding your second error, you have a typo:
print str(tripeleit(3))
It should be:
print str(tripleit(3))
Maybe I am understanding the cdef for function definition incorrectly. E.g., assume I want to write a function to convert a Python list to a C array:
%%cython
cimport cython
from libc.stdlib cimport malloc, free
cdef int* list_to_array(a_list):
""" Converts a Python list into a C array """
cdef int *c_array
cdef int count, n
c_array = <int *>malloc(len(a_list)*cython.sizeof(int))
count = len(a_list)
for i in range(count):
c_array[i] = a_list[i]
return c_array
when I call the function now via
list_to_array([1,2,3])
I get a
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-32-8f3f777d7883> in <module>()
----> 1 list_to_array([1,2,3])
NameError: name 'list_to_array' is not defined
However, when I just use the def, the function can be called (although it doesn't return what I want, it is just for illustrating my problem...)
%%cython
cimport cython
from libc.stdlib cimport malloc, free
def list_to_array1(a_list):
""" Converts a Python list into a C array """
cdef int *c_array
cdef int count, n
c_array = <int *>malloc(len(a_list)*cython.sizeof(int))
count = len(a_list)
for i in range(count):
c_array[i] = a_list[i]
return 1
list_to_array1([1,2,3])
1
When I tried to use cpdef instead of cdef, I encounter a different issue:
Error compiling Cython file:
------------------------------------------------------------
...
cimport cython
from libc.stdlib cimport malloc, free
cpdef int* list_to_carray(a_list):
^
------------------------------------------------------------
/Users/sebastian/.ipython/cython/_cython_magic_c979dc7a52cdfb492e901a4b337ed2d2.pyx:3:6: Cannot convert 'int *' to Python object
Citing the docs, "The cdef statement is used to make C level declarations"
Then if you scroll a bit down here, you see that cdef functions can not be called from python, hence your NameError. Try using cpdef.
Note that if you plan to use that function in python code it will leak memory. You may also want to have a look at this answer on how/why you should return pass a list to/from cython (disclaimer: the answer is mine) to avoid the leakage.
EDIT, in reply to updated question:
The error once you introduce cpdef happens because a pointer cannot be converted to a python object in a trivial way. Cython does the hard work for you in the simplest cases, see here. The question you should ask here is why you want to return a C pointer to the python environment, that does not provide pointers.
A few SciPy functions (like scipy.ndimage.interpolation.geometric_transform) can take pointers to C functions as arguments to avoid having to call a Python callable on each point of the input array.
In a nutshell :
Define a function called my_function somewhere in the C module
Return a PyCObject with the &my_function pointer and (optionally) a void* pointer to pass some global data around
The related API method is PyCObject_FromVoidPtrAndDesc, and you can read Extending ndimage in C to see it in action.
I am very interested in using Cython to keep my code more manageable, but I'm not sure how exactly I should create such an object. Any, well,... pointers?
Just do in Cython the same thing you would do in C, call PyCObject_FromVoidPtrAndDesc directly. Here is an example from your link ported to Cython:
###### example.pyx ######
from libc.stdlib cimport malloc, free
from cpython.cobject cimport PyCObject_FromVoidPtrAndDesc
cdef int _shift_function(int *output_coordinates, double* input_coordinates,
int output_rank, int input_rank, double *shift_data):
cdef double shift = shift_data[0]
cdef int ii
for ii in range(input_rank):
input_coordinates[ii] = output_coordinates[ii] - shift
return 1
cdef void _shift_destructor(void* cobject, void *shift_data):
free(shift_data)
def shift_function(double shift):
"""This is the function callable from python."""
cdef double* shift_data = <double*>malloc(sizeof(shift))
shift_data[0] = shift
return PyCObject_FromVoidPtrAndDesc(&_shift_function,
shift_data,
&_shift_destructor)
Performance should be identical to pure C version.
Note that Cyhton requires operator & to get function address. Also, Cython lacks pointer dereference operator *, indexing equivalent is used instead (*ptr -> ptr[0]).
I think that is a bad idea. Cython was created to avoid writing PyObjects also! Moreover, in this case, writing the code through Cython probably doesn't improve code maintenance...
Anyway, you can import the PyObject with
from cpython.ref cimport PyObject
in your Cython code.
UPDATE
from cpython cimport *
is safer.
Cheers,
Davide