I am having problems compiling cdef-ed types in different packages and I couldn't find an explanation in cython docs.
I have this setup.py in the root of my python src tree:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("flink.pytk.defs.FragIdx",
sources = ["flink/pytk/defs/FragIdx.pyx"]),
Extension("flink.pytk.fragments.STK_idx",
sources = ["flink/pytk/fragments/STK_idx.pyx"])
]
)
FragIdx is a cdef-ed type, defined in flink/pytk/defs/FragIdx.pyx:
cdef class FragIdx:
cdef public FragIdx parent
cdef public FragIdx root
cdef public tuple label
...
And STK_idx is an extension of FragIdx, defined in flink/pytk/fragments/STK_idx.pyx:
from flink.pytk.defs.FragIdx import FragIdx
cdef class STK_idx(FragIdx):
...
When I try to compile using the setup.py listed at the beginning of the post, FragIdx is compiled all right, but when it comes to STK_idx I get the following error message:
flink/pytk/fragments/STK_idx.pyx:5:5: 'FragIdx' is not a type name
Please note that the root directory of my source tree is listed in $PYTHONPATH.
I would really appreciate if anyone could shed any light on this, thanks a lot!
Daniele
Oh, well, for those having a similar problem, it looks like maybe I found the answer.
I was expecting python to automatically scan the symbols compiled into the shared library FragIdx.so, instead it looks like this information must be provided explicitly as a .pxd file (which becomes a C header file after Cython is run).
There are basically two steps involved in the process:
Creation of a definition (.pxd) file for the superclass;
Importing of the the superclass definition via cimport (as opposed to import) in the subclass module.
So, to make it more general.
Suppose that you have defined your cdef-ed type A in module pkg1.mod1. Then you cdef a type B in pkg2.mod2 that subclasses A.
Your directory structure would look something like this:
pkg1/
mod1.pyx
mod1.pxd
pkg2/
mod2.pyx
mod2.pxd
In pkg1/mod1.pxd you would have, say:
cdef class A:
cdef int a
cdef int b
And in pkg1/mod1.pyx you would provide the methods of your class.
In pkg2/mod2.pxd, you would have:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
cdef ... # your attributes here
And again, in pkg2/mod2.pyx you would have to cimport the A symbol again:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
... # your methods here
Interestingly enough, if you just want to use A in your python code, as opposed to using it to define a subtype, the definitions file mod1.pxd is not needed. This is related to the fact that when creating an extension type you need the definitions to be available for the C compiler, whereas you don't have this problem when running python code, but since it is not very intuitive maybe it's important to point it out.
This information is actually available in the Cython docs, though maybe it could be a little bit more explicit.
Hope this information can save some to someone.
Related
Using SWIG to generate a Python binding for my C++ project has not been easy but I have finally been able to do so. The only issue is that the generated .py file that houses essentially the class and method definitions of my wrapped C++ code (but callable for Python) is quite large. I basically want to modularize the generated .py file into submodules of relevant classes.
Here is a basic and stripped down sample of what my swig interface file looks like:
%module example
%{
/* these two headers should belong to ModuleOne */
#include "header1.hpp"
#include "header2.hpp"
/* these two headers should belong to ModuleTwo */
#include "header3.hpp"
#include "header4.hpp"
}
%include "header1.hpp"
%include "header2.hpp"
%include "header3.hpp"
%include "header4.hpp"
And from Python importing the package would be done like so:
from example import *
I find this messy as I either need to import each class individually with from example import ClassOne or import the entirety of the module.
How could I go about creating "submodules" of the swig generated .py file allowing me to modularize my project a bit cleaner and import those without necessarily importing the entire package. For example something like:
import example.ModuleOne
import example.ModuleTwo
I think you just need to add an __init__.py file that imports both modules, something like this:
from example.ModuleOne import *
from example.ModuleTwo import *
__all__ = [x for x in dir() if x[0] != '_']
The last line allows your program to use from example import * to import everything.
Edit: I've just read your question a bit more closely and realise you want to import just one of your submodules. You still need an __init__.py file to make your two modules into a package, but it can be empty. Your interface files should include a package declaration, e.g. %module(package="example") ModuleOne.
As part of larger cython project I am getting an error that seems to indicate that maybe the automatic type merging is failing. It is possible I am doing something silly or that the fact that I am structuring things in this way is a bad smell. I think I can get around the issue by using void* and then casts everywhere but this seems quite messy for what should be a simple thing.
It seems similar to this issue, but that involved function pointers.
Wrong signature error using function types in multiple modules
I have put together a minimal example that demonstrates the issue. The files are as follows.
a.pxd
cdef struct A:
int a
cdef A makeA(int a)
a.pyx
cdef struct A:
pass
cdef A makeA(int a):
cdef A sA
sA.a = a
return sA
b.pyx
from cfr.a cimport A, makeA
cdef int get_A_value():
cdef A sA = makeA(5)
return sA.a
If I compile and then try to import b in a jupyter-lab notebook I get the following error:
b.pyx in init b()
TypeError: C function cfr.a.makeA has wrong signature (expected struct __pyx_t_3cfr_1a_A (int), got struct __pyx_t_1a_A (int))
EDIT: It looks like it might be something to do with my directory structure. Note that I do from cfr.a cimport A (as this happens to be the layout of my project). I think I need to do from a import A and get the import to work through the use of my pythonpath.
So it turned out the issue was in the from cfr.a cimport A line. This was somehow meaning that the A type was not being seen as the same type as A in the a.pyx file.
To fix use:
from a cimport A
(You may then get a not found error, but we can fix this with our pythonpath).
In the comments adding a __init__.pxd file was suggested, but adding a blank one did not work for me.
I am trying to compile and run a Cython extension. There are three files in the same directory:
main.pxd
cdef class Function:
cdef object f
main.pyx
cdef class Function:
def __init__(Function self, object f):
if callable(f):
self.f = f
elif type(f) in [staticmethod,classmethod]:
self.f = f
else:
raise TypeError("constructor argument must be callable")
setup.py
# import setuptools
from distutils.core import Extension, setup
from Cython.Build import cythonize
# define an extension that will be cythonized and compiled
ext = Extension(name="Do", sources=["main.pyx"])
setup(ext_modules=cythonize(ext,language_level=3))
I call python setup.py build_ext --inplace. The code compiles without errors. However, when I try to create a new Function object in another Python module, I get the following error:
File "main.pyx", line 4, in Do.Function.__init__
AttributeError: 'Do.Function' object has no attribute 'f'
This tells me that the compiler is not aware of main.pxd. Adding cimport main to the top of main.pyx causes a compiler error.
How do I get Cython to see main.pxd, and include it in the binary?
Python 3.6, Cython 0.29.23, Windows 10
Update:
The Cython docs have an example. In the Sharing Extension Types section there is code for a Shrubbery class. I copied the code exactly into main.pxd and main.pyx. I get a similar error as before: 'Do.Shrubbery' object has no attribute 'width'.
Update:
Added public to cdef in pxd file. Same error occurs.
Problem solved by making the extension name the same as the names of the pyx and pxd files.
I still am having a very hard time figuring this out. Take this simple scenario.. in a file named PyFoo.pyx
cdef extern from "Foo.h" namespace "Foo":
cdef cppclass C_FOO:
...........
cdef class PythonFoo:
C_FOO* pythonFoo
def __cinit__(self):
self.pythonFoo=new CFOO()
I do have a PyFoo.PXD also, that simply does this for a forward decl so other classes can see it
cdef class PythonFoo:
C_FOO * pythonFoo
Now I have another class (in PyBar.pyx) that needs PythonFoo.pythonFoo as a member
cdef extern from "Bar.h" namespace "Bar":
cdef cppclass C_Bar:
ThisFunctionNeeds(C_FOO*)
cimport PythonFoo
cdef class PythonBar:
C_Bar* pythonBar
def __cinit__(self):
self.pythonBar=new C_Bar()
def SomeFunction(self,PythonFoo):
ThisFunctionNeeds(PythonFoo.pythonFoo)
No matter how I try to cimport it, I get "PythonFoo not defined" (Because there is no init in the pxd file I guess) or I have to manually include the Foo.pyx, making this now a member of Bar.
Can someone please show me (with an example please) how this is supposed to work?
Thank you.
EDIT: PythonFoo and PythonBar are in separate modules in the package.
In a Python Tester, I try to use this
import PythonFoo
import PythonBar
foo=PythonFoo()
bar=PythonBar()
bar.SomeFunction(foo)
Here is where I get the "No module named 'PyFoo'
I was able to find why it would not work... The actual directory of the module was not placed, as a matter of fact, the file was placed at a higher directory from the build directory.
I fixed it with a simple
sys.path.append('package_directory')
EDIT: I also found out why this was... I had an __init__.py in the same directory as my setup.py file, so the compiler thought that directory was part of the package. I deleted the __init__.py, and I no longer needed to append the directory.
I'm in the process of writing a Cython wrapper for a complex C++ library. I think I've mostly figured out how to write the necessary .pxd and .pyx files. My problem now is that although my C++ program has about 100 separate namespaces, the namespace of the Cython-compiled python library is totally flat.
For example, if I have this in my .pxd file:
cdef extern from "lm/io/hdf5/SimulationFile.h" namespace "lm::io::hdf5":
cdef cppclass CppHdf5File "lm::io::hdf5::Hdf5File":
...
and this in my .pyx file:
cdef class Hdf5File:
cdef CppHdf5File* thisptr
...
then the Cython-compiled Python library contains a class named Hdf5File. Ideally, I'd like the Python to contain a lm.io.hdf5.Hdf5File class (i.e. a Hdf5File class in a lm.io.hdf5 module). In other words, I'd like it if there was a way to translate the C++ :: scoping operator into the Python . dot operator.
Is there a way to get Cython to play nicely with my existing C++ namespaces?
Suppose your .pyx file is named source.pyx. I would write a setup.py as below:
from setuptools import Extension, setup
from Cython.Build import cythonize
extensions = [
Extension(
name='lm.io.hdf5',
# ^^^^^^^^^^ -- note the name here
sources=[
'path/to/source.pyx',
# other sources like c++ files ...
],
# other options ...
),
]
# Call `setup` as you wish, e.g.:
#setup(
# ext_modules=cythonize(extensions, language_level='3'),
# zip_safe=False,
#)
This will generate lm/io/hdf5.so or like if compilation is successful.
Then in Python, you may import like this:
from lm.io.hdf5 import Hdf5File
Reference: setuptools doc (doc for name field)