I'm in the process of writing a Cython wrapper for a complex C++ library. I think I've mostly figured out how to write the necessary .pxd and .pyx files. My problem now is that although my C++ program has about 100 separate namespaces, the namespace of the Cython-compiled python library is totally flat.
For example, if I have this in my .pxd file:
cdef extern from "lm/io/hdf5/SimulationFile.h" namespace "lm::io::hdf5":
cdef cppclass CppHdf5File "lm::io::hdf5::Hdf5File":
...
and this in my .pyx file:
cdef class Hdf5File:
cdef CppHdf5File* thisptr
...
then the Cython-compiled Python library contains a class named Hdf5File. Ideally, I'd like the Python to contain a lm.io.hdf5.Hdf5File class (i.e. a Hdf5File class in a lm.io.hdf5 module). In other words, I'd like it if there was a way to translate the C++ :: scoping operator into the Python . dot operator.
Is there a way to get Cython to play nicely with my existing C++ namespaces?
Suppose your .pyx file is named source.pyx. I would write a setup.py as below:
from setuptools import Extension, setup
from Cython.Build import cythonize
extensions = [
Extension(
name='lm.io.hdf5',
# ^^^^^^^^^^ -- note the name here
sources=[
'path/to/source.pyx',
# other sources like c++ files ...
],
# other options ...
),
]
# Call `setup` as you wish, e.g.:
#setup(
# ext_modules=cythonize(extensions, language_level='3'),
# zip_safe=False,
#)
This will generate lm/io/hdf5.so or like if compilation is successful.
Then in Python, you may import like this:
from lm.io.hdf5 import Hdf5File
Reference: setuptools doc (doc for name field)
Related
Using SWIG to generate a Python binding for my C++ project has not been easy but I have finally been able to do so. The only issue is that the generated .py file that houses essentially the class and method definitions of my wrapped C++ code (but callable for Python) is quite large. I basically want to modularize the generated .py file into submodules of relevant classes.
Here is a basic and stripped down sample of what my swig interface file looks like:
%module example
%{
/* these two headers should belong to ModuleOne */
#include "header1.hpp"
#include "header2.hpp"
/* these two headers should belong to ModuleTwo */
#include "header3.hpp"
#include "header4.hpp"
}
%include "header1.hpp"
%include "header2.hpp"
%include "header3.hpp"
%include "header4.hpp"
And from Python importing the package would be done like so:
from example import *
I find this messy as I either need to import each class individually with from example import ClassOne or import the entirety of the module.
How could I go about creating "submodules" of the swig generated .py file allowing me to modularize my project a bit cleaner and import those without necessarily importing the entire package. For example something like:
import example.ModuleOne
import example.ModuleTwo
I think you just need to add an __init__.py file that imports both modules, something like this:
from example.ModuleOne import *
from example.ModuleTwo import *
__all__ = [x for x in dir() if x[0] != '_']
The last line allows your program to use from example import * to import everything.
Edit: I've just read your question a bit more closely and realise you want to import just one of your submodules. You still need an __init__.py file to make your two modules into a package, but it can be empty. Your interface files should include a package declaration, e.g. %module(package="example") ModuleOne.
I am trying to compile and run a Cython extension. There are three files in the same directory:
main.pxd
cdef class Function:
cdef object f
main.pyx
cdef class Function:
def __init__(Function self, object f):
if callable(f):
self.f = f
elif type(f) in [staticmethod,classmethod]:
self.f = f
else:
raise TypeError("constructor argument must be callable")
setup.py
# import setuptools
from distutils.core import Extension, setup
from Cython.Build import cythonize
# define an extension that will be cythonized and compiled
ext = Extension(name="Do", sources=["main.pyx"])
setup(ext_modules=cythonize(ext,language_level=3))
I call python setup.py build_ext --inplace. The code compiles without errors. However, when I try to create a new Function object in another Python module, I get the following error:
File "main.pyx", line 4, in Do.Function.__init__
AttributeError: 'Do.Function' object has no attribute 'f'
This tells me that the compiler is not aware of main.pxd. Adding cimport main to the top of main.pyx causes a compiler error.
How do I get Cython to see main.pxd, and include it in the binary?
Python 3.6, Cython 0.29.23, Windows 10
Update:
The Cython docs have an example. In the Sharing Extension Types section there is code for a Shrubbery class. I copied the code exactly into main.pxd and main.pyx. I get a similar error as before: 'Do.Shrubbery' object has no attribute 'width'.
Update:
Added public to cdef in pxd file. Same error occurs.
Problem solved by making the extension name the same as the names of the pyx and pxd files.
I have a header file with struct definitions that I'd like to be able to parse in python. In order to do this I turned to Swig.
Lets say the header file is named "a.h". I first renamed it to "a.c" and added an empty "a.h" file in the same folder.
Next, I added in an "a_wrap.i" file with the following contents.
%module a
%{
/* the resulting C file should be built as a python extension */
#define SWIG_FILE_WITH_INIT
/* Includes the header in the wrapper code */
#include "a.h"
%}
/* Parse the header file to generate wrappers */
%include "a.h"
extern struct a_a;
extern struct a_b;
extern struct a_c;
Next, I wrote a setup.py file as follows :
from distutils.core import setup, Extension
setup(ext_modules=[Extension("_a",
sources=["a.c", "a_wrap.i"])])
Next, I did the build as
python setup.py build_ext --inplace
I finally tried to import it in python
>>> import a # it works, yaay
>>> dir(a)
...
...
I was hoping for a way to access the structs defined in "a.c"(originally a.h). However, I don't seem to be able to find a way to do that. How can I solve this? I'm looking for a way to access the struct's defined in the header file from python.
The global variables a_a, a_b anda_cshould be accessible from [within your SWIG Python module viacvar`]1:
import a
print a.cvar.a_a
print a.cvar.a_b
# etc.
I am new to cython.
Now, I am trying to import standard c library and define a simple function in pyx file:
from libc.math cimport sin
cdef double f(double x):
return sin(x*x)
I compiled with this file:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules=[
Extension("demo",
["demo.pyx"],
libraries=["m"]) # Unix-like specific
]
setup(
name = "Demos",
cmdclass = {"build_ext": build_ext},
ext_modules = ext_modules
)
and generate a library called demo.so
now I am trying to call this "f" function in a python file:
import demo
print demo.f(2)
The compiler said,
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'f'
Does anyone know, how can I call the function in pyx file? thanks!
Functions defined with cdef cannot be accessed from python. You can write them and use them inside the cython code but they cannot be exposed.
If you want to expose a function either define it with def, or define it using cpdef.
Using def you'll create a normal python function, which means using that function in the cython code might require more conversions then using cdef (and hence more overhead).
Using cpdef cython will generate two functions. One is exactly the function that would be defined using cdef and it will also create a python function that acts as a wrapper for this function. The cython code will use the pure-C version of the function, thus reducing the overhead, and the library will expose the wrapper.
I am having problems compiling cdef-ed types in different packages and I couldn't find an explanation in cython docs.
I have this setup.py in the root of my python src tree:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("flink.pytk.defs.FragIdx",
sources = ["flink/pytk/defs/FragIdx.pyx"]),
Extension("flink.pytk.fragments.STK_idx",
sources = ["flink/pytk/fragments/STK_idx.pyx"])
]
)
FragIdx is a cdef-ed type, defined in flink/pytk/defs/FragIdx.pyx:
cdef class FragIdx:
cdef public FragIdx parent
cdef public FragIdx root
cdef public tuple label
...
And STK_idx is an extension of FragIdx, defined in flink/pytk/fragments/STK_idx.pyx:
from flink.pytk.defs.FragIdx import FragIdx
cdef class STK_idx(FragIdx):
...
When I try to compile using the setup.py listed at the beginning of the post, FragIdx is compiled all right, but when it comes to STK_idx I get the following error message:
flink/pytk/fragments/STK_idx.pyx:5:5: 'FragIdx' is not a type name
Please note that the root directory of my source tree is listed in $PYTHONPATH.
I would really appreciate if anyone could shed any light on this, thanks a lot!
Daniele
Oh, well, for those having a similar problem, it looks like maybe I found the answer.
I was expecting python to automatically scan the symbols compiled into the shared library FragIdx.so, instead it looks like this information must be provided explicitly as a .pxd file (which becomes a C header file after Cython is run).
There are basically two steps involved in the process:
Creation of a definition (.pxd) file for the superclass;
Importing of the the superclass definition via cimport (as opposed to import) in the subclass module.
So, to make it more general.
Suppose that you have defined your cdef-ed type A in module pkg1.mod1. Then you cdef a type B in pkg2.mod2 that subclasses A.
Your directory structure would look something like this:
pkg1/
mod1.pyx
mod1.pxd
pkg2/
mod2.pyx
mod2.pxd
In pkg1/mod1.pxd you would have, say:
cdef class A:
cdef int a
cdef int b
And in pkg1/mod1.pyx you would provide the methods of your class.
In pkg2/mod2.pxd, you would have:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
cdef ... # your attributes here
And again, in pkg2/mod2.pyx you would have to cimport the A symbol again:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
... # your methods here
Interestingly enough, if you just want to use A in your python code, as opposed to using it to define a subtype, the definitions file mod1.pxd is not needed. This is related to the fact that when creating an extension type you need the definitions to be available for the C compiler, whereas you don't have this problem when running python code, but since it is not very intuitive maybe it's important to point it out.
This information is actually available in the Cython docs, though maybe it could be a little bit more explicit.
Hope this information can save some to someone.