I am trying to compile and run a Cython extension. There are three files in the same directory:
main.pxd
cdef class Function:
cdef object f
main.pyx
cdef class Function:
def __init__(Function self, object f):
if callable(f):
self.f = f
elif type(f) in [staticmethod,classmethod]:
self.f = f
else:
raise TypeError("constructor argument must be callable")
setup.py
# import setuptools
from distutils.core import Extension, setup
from Cython.Build import cythonize
# define an extension that will be cythonized and compiled
ext = Extension(name="Do", sources=["main.pyx"])
setup(ext_modules=cythonize(ext,language_level=3))
I call python setup.py build_ext --inplace. The code compiles without errors. However, when I try to create a new Function object in another Python module, I get the following error:
File "main.pyx", line 4, in Do.Function.__init__
AttributeError: 'Do.Function' object has no attribute 'f'
This tells me that the compiler is not aware of main.pxd. Adding cimport main to the top of main.pyx causes a compiler error.
How do I get Cython to see main.pxd, and include it in the binary?
Python 3.6, Cython 0.29.23, Windows 10
Update:
The Cython docs have an example. In the Sharing Extension Types section there is code for a Shrubbery class. I copied the code exactly into main.pxd and main.pyx. I get a similar error as before: 'Do.Shrubbery' object has no attribute 'width'.
Update:
Added public to cdef in pxd file. Same error occurs.
Problem solved by making the extension name the same as the names of the pyx and pxd files.
Related
Python's inspect module doesn't seem to be able to inspect the signatures of "built-in" functions, which include functions defined in C extension modules, like those defined by Cython. Is there any way to get the signature of a Python function you have defined in such a module, and specifically in Cython? I am looking to be able to find the available keyword arguments.
MWE:
# mwe.pyx
def example(a, b=None):
pass
and
import pyximport; pyximport.install()
import mwe
import inspect
inspect.signature(mwe.example)
yields:
Traceback (most recent call last):
File "mwe_py.py", line 5, in <module>
inspect.signature(mwe.example)
File "/nix/store/134l79vxb91w8mhxxkj6kb5llf7dmwpm-python3-3.4.5/lib/python3.4/inspect.py", line 2063, in signature
return _signature_internal(obj)
File "/nix/store/134l79vxb91w8mhxxkj6kb5llf7dmwpm-python3-3.4.5/lib/python3.4/inspect.py", line 1965, in _signature_internal
skip_bound_arg=skip_bound_arg)
File "/nix/store/134l79vxb91w8mhxxkj6kb5llf7dmwpm-python3-3.4.5/lib/python3.4/inspect.py", line 1890, in _signature_from_builtin
raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in function example>
In Python 3.4.5 and Cython 0.24.1
I've retracted my duplicate suggestion (saying that it was impossible...) having investigated further. It seems to work fine with reasonably recent versions of Cython (v0.23.4) and Python 3.4.4.
import cython
import inspect
scope = cython.inline("""def f(a,*args,b=False): pass """)
print(inspect.getfullargspec(scope['f']))
gives the output
FullArgSpec(args=['a'], varargs='args', varkw=None, defaults=None, kwonlyargs=['b'], kwonlydefaults={'b': False}, annotations={})
Also mentioned in the documentation is the compilation option "binding" which apparently makes this detail more accessible (although I didn't need it).
I have a feeling that this may depend on improvements to inspect made relatively recently (possibly this fix) so if you're using Python 2 you're probably out of luck.
Edit: your example works if you using the binding compilation option:
import cython
#cython.binding(True)
def example(a, b=None):
pass
I suspect that inline adds it automatically (but the code to do inline is sufficiently convoluted that I can't find proof of that either way). You can also set it as a file-level option.
The answer above using the binding decorator works for me when running code that has been cythonized. But, when I was running that same code within a Django 2.2 app, the application would fail on start with an error that cython has no attribute 'binding'. To avoid this I have added this "special cython header" at the top of my file containing the cythonized function as documented here to achieve the same results.
# cython: binding=True
def example(a, b=None):
pass
I'm in the process of writing a Cython wrapper for a complex C++ library. I think I've mostly figured out how to write the necessary .pxd and .pyx files. My problem now is that although my C++ program has about 100 separate namespaces, the namespace of the Cython-compiled python library is totally flat.
For example, if I have this in my .pxd file:
cdef extern from "lm/io/hdf5/SimulationFile.h" namespace "lm::io::hdf5":
cdef cppclass CppHdf5File "lm::io::hdf5::Hdf5File":
...
and this in my .pyx file:
cdef class Hdf5File:
cdef CppHdf5File* thisptr
...
then the Cython-compiled Python library contains a class named Hdf5File. Ideally, I'd like the Python to contain a lm.io.hdf5.Hdf5File class (i.e. a Hdf5File class in a lm.io.hdf5 module). In other words, I'd like it if there was a way to translate the C++ :: scoping operator into the Python . dot operator.
Is there a way to get Cython to play nicely with my existing C++ namespaces?
Suppose your .pyx file is named source.pyx. I would write a setup.py as below:
from setuptools import Extension, setup
from Cython.Build import cythonize
extensions = [
Extension(
name='lm.io.hdf5',
# ^^^^^^^^^^ -- note the name here
sources=[
'path/to/source.pyx',
# other sources like c++ files ...
],
# other options ...
),
]
# Call `setup` as you wish, e.g.:
#setup(
# ext_modules=cythonize(extensions, language_level='3'),
# zip_safe=False,
#)
This will generate lm/io/hdf5.so or like if compilation is successful.
Then in Python, you may import like this:
from lm.io.hdf5 import Hdf5File
Reference: setuptools doc (doc for name field)
There are 2 issues both relating to importing that may or may not be cython related?
I have the following simplified files to recreate the problem. All files are in the same directory. The .pyx files have successfully compiled into *.so, *.pyc and *.c files.
setup.py:
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules=cythonize("*.pyx"),
)
cy1.pyx: (cython)
cdef int timestwo(int x):
return x * 2
cy1.pxd:
cdef int timestwo(int x)
cy3.py: (normal python)
def tripleit(x):
return x*3
go.py:
from cy1 import timestwo
print str(timestwo(5))
Gives me the error: ImportError: cannot import name timestwo
if I change it to:
go.py:
import pyximport; pyximport.install()
import cy1
print str(cy1.timestwo(5))
it tells me: AttributeError: 'module' object has no attribute 'timestwo'
if I take out the cython all together and try to use normal python call from cy3.py:
go.py:
import cy3
print str(cy3.tripeleit(3))
I get: AttributeError: 'module' object has no attribute 'tripeleit'
and last if I do:
go.py:
from cy3 import tripleit
print str(tripeleit(3))
I get:
NameError: name 'tripeleit' is not defined
Sorry if this is super basic but I cannot seem to figure it out.
The problem is that in go.py:
from cy1 import timestwo
print str(timestwo(5))
you are trying to import a function defined as cdef.
To expose this function to Python you have to either use def or cpdef. Possibly you have to keep as cdef in order to cimport from other Cython files, justifying why you also have the pxd file. In that case I usually have a C-like function and a wrapper that can be called from Python.
In this case your cy1.pyx file would look like:
cdef int ctimestwo(int x):
return x * 2
def timestwo(x): # <-- small wrapper to expose ctimestwo() to Python
return ctimestwo(x)
and your cy1.pxd file:
cdef int ctimestwo(int x)
such that you can cimport only the ctimestwo function.
Regarding your second error, you have a typo:
print str(tripeleit(3))
It should be:
print str(tripleit(3))
I am new to cython.
Now, I am trying to import standard c library and define a simple function in pyx file:
from libc.math cimport sin
cdef double f(double x):
return sin(x*x)
I compiled with this file:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules=[
Extension("demo",
["demo.pyx"],
libraries=["m"]) # Unix-like specific
]
setup(
name = "Demos",
cmdclass = {"build_ext": build_ext},
ext_modules = ext_modules
)
and generate a library called demo.so
now I am trying to call this "f" function in a python file:
import demo
print demo.f(2)
The compiler said,
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'f'
Does anyone know, how can I call the function in pyx file? thanks!
Functions defined with cdef cannot be accessed from python. You can write them and use them inside the cython code but they cannot be exposed.
If you want to expose a function either define it with def, or define it using cpdef.
Using def you'll create a normal python function, which means using that function in the cython code might require more conversions then using cdef (and hence more overhead).
Using cpdef cython will generate two functions. One is exactly the function that would be defined using cdef and it will also create a python function that acts as a wrapper for this function. The cython code will use the pure-C version of the function, thus reducing the overhead, and the library will expose the wrapper.
I am having problems compiling cdef-ed types in different packages and I couldn't find an explanation in cython docs.
I have this setup.py in the root of my python src tree:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("flink.pytk.defs.FragIdx",
sources = ["flink/pytk/defs/FragIdx.pyx"]),
Extension("flink.pytk.fragments.STK_idx",
sources = ["flink/pytk/fragments/STK_idx.pyx"])
]
)
FragIdx is a cdef-ed type, defined in flink/pytk/defs/FragIdx.pyx:
cdef class FragIdx:
cdef public FragIdx parent
cdef public FragIdx root
cdef public tuple label
...
And STK_idx is an extension of FragIdx, defined in flink/pytk/fragments/STK_idx.pyx:
from flink.pytk.defs.FragIdx import FragIdx
cdef class STK_idx(FragIdx):
...
When I try to compile using the setup.py listed at the beginning of the post, FragIdx is compiled all right, but when it comes to STK_idx I get the following error message:
flink/pytk/fragments/STK_idx.pyx:5:5: 'FragIdx' is not a type name
Please note that the root directory of my source tree is listed in $PYTHONPATH.
I would really appreciate if anyone could shed any light on this, thanks a lot!
Daniele
Oh, well, for those having a similar problem, it looks like maybe I found the answer.
I was expecting python to automatically scan the symbols compiled into the shared library FragIdx.so, instead it looks like this information must be provided explicitly as a .pxd file (which becomes a C header file after Cython is run).
There are basically two steps involved in the process:
Creation of a definition (.pxd) file for the superclass;
Importing of the the superclass definition via cimport (as opposed to import) in the subclass module.
So, to make it more general.
Suppose that you have defined your cdef-ed type A in module pkg1.mod1. Then you cdef a type B in pkg2.mod2 that subclasses A.
Your directory structure would look something like this:
pkg1/
mod1.pyx
mod1.pxd
pkg2/
mod2.pyx
mod2.pxd
In pkg1/mod1.pxd you would have, say:
cdef class A:
cdef int a
cdef int b
And in pkg1/mod1.pyx you would provide the methods of your class.
In pkg2/mod2.pxd, you would have:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
cdef ... # your attributes here
And again, in pkg2/mod2.pyx you would have to cimport the A symbol again:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
... # your methods here
Interestingly enough, if you just want to use A in your python code, as opposed to using it to define a subtype, the definitions file mod1.pxd is not needed. This is related to the fact that when creating an extension type you need the definitions to be available for the C compiler, whereas you don't have this problem when running python code, but since it is not very intuitive maybe it's important to point it out.
This information is actually available in the Cython docs, though maybe it could be a little bit more explicit.
Hope this information can save some to someone.