There are 2 issues both relating to importing that may or may not be cython related?
I have the following simplified files to recreate the problem. All files are in the same directory. The .pyx files have successfully compiled into *.so, *.pyc and *.c files.
setup.py:
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules=cythonize("*.pyx"),
)
cy1.pyx: (cython)
cdef int timestwo(int x):
return x * 2
cy1.pxd:
cdef int timestwo(int x)
cy3.py: (normal python)
def tripleit(x):
return x*3
go.py:
from cy1 import timestwo
print str(timestwo(5))
Gives me the error: ImportError: cannot import name timestwo
if I change it to:
go.py:
import pyximport; pyximport.install()
import cy1
print str(cy1.timestwo(5))
it tells me: AttributeError: 'module' object has no attribute 'timestwo'
if I take out the cython all together and try to use normal python call from cy3.py:
go.py:
import cy3
print str(cy3.tripeleit(3))
I get: AttributeError: 'module' object has no attribute 'tripeleit'
and last if I do:
go.py:
from cy3 import tripleit
print str(tripeleit(3))
I get:
NameError: name 'tripeleit' is not defined
Sorry if this is super basic but I cannot seem to figure it out.
The problem is that in go.py:
from cy1 import timestwo
print str(timestwo(5))
you are trying to import a function defined as cdef.
To expose this function to Python you have to either use def or cpdef. Possibly you have to keep as cdef in order to cimport from other Cython files, justifying why you also have the pxd file. In that case I usually have a C-like function and a wrapper that can be called from Python.
In this case your cy1.pyx file would look like:
cdef int ctimestwo(int x):
return x * 2
def timestwo(x): # <-- small wrapper to expose ctimestwo() to Python
return ctimestwo(x)
and your cy1.pxd file:
cdef int ctimestwo(int x)
such that you can cimport only the ctimestwo function.
Regarding your second error, you have a typo:
print str(tripeleit(3))
It should be:
print str(tripleit(3))
Related
I am trying to compile and run a Cython extension. There are three files in the same directory:
main.pxd
cdef class Function:
cdef object f
main.pyx
cdef class Function:
def __init__(Function self, object f):
if callable(f):
self.f = f
elif type(f) in [staticmethod,classmethod]:
self.f = f
else:
raise TypeError("constructor argument must be callable")
setup.py
# import setuptools
from distutils.core import Extension, setup
from Cython.Build import cythonize
# define an extension that will be cythonized and compiled
ext = Extension(name="Do", sources=["main.pyx"])
setup(ext_modules=cythonize(ext,language_level=3))
I call python setup.py build_ext --inplace. The code compiles without errors. However, when I try to create a new Function object in another Python module, I get the following error:
File "main.pyx", line 4, in Do.Function.__init__
AttributeError: 'Do.Function' object has no attribute 'f'
This tells me that the compiler is not aware of main.pxd. Adding cimport main to the top of main.pyx causes a compiler error.
How do I get Cython to see main.pxd, and include it in the binary?
Python 3.6, Cython 0.29.23, Windows 10
Update:
The Cython docs have an example. In the Sharing Extension Types section there is code for a Shrubbery class. I copied the code exactly into main.pxd and main.pyx. I get a similar error as before: 'Do.Shrubbery' object has no attribute 'width'.
Update:
Added public to cdef in pxd file. Same error occurs.
Problem solved by making the extension name the same as the names of the pyx and pxd files.
I am attempting to extend some sklearn classes in the sklearn.neighbors.dist_metrics module. However, this entire module is written in Cython, and the things I want to do with my custom class must implement a cdef dist (as opposed to a def dist), apparently.
Thus, in my own module (I am using jupyter %%cython for now, actually) I would like to implement my custom class in Cython, implementing the cdef dist method as required. However, there are a number of other cdef things in the same module, in the sklearn.neighbors.typedefs module, and possibly in other modules that I need to import. When I try to import those things, I get errors of various sorts.
When I try a naive import:
%%cython
import numpy as np
from sklearn.neighbors.dist_metrics import DistanceMetric
from sklearn.neighbors.typedefs import DTYPE_t, ITYPE_t
cdef class NewDistance(sklearn.neighbors.DistanceMetric):
cdef inline DTYPE_t dist(self, DTYPE_t* x1, DTYPE_t* x2,
ITYPE_t size) nogil except -1:
return 5
I get errors suggesting that once the things have been "pythonized" I cannot use them in a "cython" definition:
Error compiling Cython file:
...
First base of 'KernelDistance' is not an extension type
...
'DTYPE_t' is not a type identifier
There is a cimport keyword, so:
%%cython
import numpy as np
from sklearn.neighbors.dist_metrics cimport DistanceMetric
from sklearn.neighbors.typedefs cimport DTYPE_t, ITYPE_t
cdef class NewDistance(sklearn.neighbors.DistanceMetric):
cdef inline DTYPE_t dist(self, DTYPE_t* x1, DTYPE_t* x2,
ITYPE_t size) nogil except -1:
return 5
Annnnndddd...
Error compiling Cython file:
....
'sklearn/neighbors/dist_metrics.pxd' not found
....
'sklearn/neighbors/typedefs.pxd' not found
How does one go about importing cython files from other libraries?
I am new to cython.
Now, I am trying to import standard c library and define a simple function in pyx file:
from libc.math cimport sin
cdef double f(double x):
return sin(x*x)
I compiled with this file:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules=[
Extension("demo",
["demo.pyx"],
libraries=["m"]) # Unix-like specific
]
setup(
name = "Demos",
cmdclass = {"build_ext": build_ext},
ext_modules = ext_modules
)
and generate a library called demo.so
now I am trying to call this "f" function in a python file:
import demo
print demo.f(2)
The compiler said,
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'f'
Does anyone know, how can I call the function in pyx file? thanks!
Functions defined with cdef cannot be accessed from python. You can write them and use them inside the cython code but they cannot be exposed.
If you want to expose a function either define it with def, or define it using cpdef.
Using def you'll create a normal python function, which means using that function in the cython code might require more conversions then using cdef (and hence more overhead).
Using cpdef cython will generate two functions. One is exactly the function that would be defined using cdef and it will also create a python function that acts as a wrapper for this function. The cython code will use the pure-C version of the function, thus reducing the overhead, and the library will expose the wrapper.
Let us have script foo.pyx with function in it:
def hello():
cdef int* i = <int *> malloc(sizeof(int))
i[0] = 1
trol(i)
print i
and script with function noo.pyx:
cdef trol(int * i):
i[0] = 42
the question is, how do I now import the trol function from file noo.pyx to foo.pyx, so I can use it in hello function.
This is only model example, but I think, that it illustrate the problem fair enough.
I tried simple
from noo import trol
but that throws "Cannot convert 'int *' to Python object"
Edit: I would better add, that this example will work just fine if I put both functions to same file.
This seems like something obvious to try, but did you try:
from noo cimport trol
If you use import instead of cimport, I think it will try to cast trol as a python function and generate the error you're getting.
The solution eventually was to create additional .pxd file, which something very similar to classic header .h file in C. It stores functions declarations and when cimport is called, it is in this file where it looks for functions and structures.
So to be specific, all I needed to do was to create file noo.pxd containing:
cdef trol(int * i)
and than we can simply cimport this function from foo.pyx by calling
from noo cimport trol
I am having problems compiling cdef-ed types in different packages and I couldn't find an explanation in cython docs.
I have this setup.py in the root of my python src tree:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("flink.pytk.defs.FragIdx",
sources = ["flink/pytk/defs/FragIdx.pyx"]),
Extension("flink.pytk.fragments.STK_idx",
sources = ["flink/pytk/fragments/STK_idx.pyx"])
]
)
FragIdx is a cdef-ed type, defined in flink/pytk/defs/FragIdx.pyx:
cdef class FragIdx:
cdef public FragIdx parent
cdef public FragIdx root
cdef public tuple label
...
And STK_idx is an extension of FragIdx, defined in flink/pytk/fragments/STK_idx.pyx:
from flink.pytk.defs.FragIdx import FragIdx
cdef class STK_idx(FragIdx):
...
When I try to compile using the setup.py listed at the beginning of the post, FragIdx is compiled all right, but when it comes to STK_idx I get the following error message:
flink/pytk/fragments/STK_idx.pyx:5:5: 'FragIdx' is not a type name
Please note that the root directory of my source tree is listed in $PYTHONPATH.
I would really appreciate if anyone could shed any light on this, thanks a lot!
Daniele
Oh, well, for those having a similar problem, it looks like maybe I found the answer.
I was expecting python to automatically scan the symbols compiled into the shared library FragIdx.so, instead it looks like this information must be provided explicitly as a .pxd file (which becomes a C header file after Cython is run).
There are basically two steps involved in the process:
Creation of a definition (.pxd) file for the superclass;
Importing of the the superclass definition via cimport (as opposed to import) in the subclass module.
So, to make it more general.
Suppose that you have defined your cdef-ed type A in module pkg1.mod1. Then you cdef a type B in pkg2.mod2 that subclasses A.
Your directory structure would look something like this:
pkg1/
mod1.pyx
mod1.pxd
pkg2/
mod2.pyx
mod2.pxd
In pkg1/mod1.pxd you would have, say:
cdef class A:
cdef int a
cdef int b
And in pkg1/mod1.pyx you would provide the methods of your class.
In pkg2/mod2.pxd, you would have:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
cdef ... # your attributes here
And again, in pkg2/mod2.pyx you would have to cimport the A symbol again:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
... # your methods here
Interestingly enough, if you just want to use A in your python code, as opposed to using it to define a subtype, the definitions file mod1.pxd is not needed. This is related to the fact that when creating an extension type you need the definitions to be available for the C compiler, whereas you don't have this problem when running python code, but since it is not very intuitive maybe it's important to point it out.
This information is actually available in the Cython docs, though maybe it could be a little bit more explicit.
Hope this information can save some to someone.