I still am having a very hard time figuring this out. Take this simple scenario.. in a file named PyFoo.pyx
cdef extern from "Foo.h" namespace "Foo":
cdef cppclass C_FOO:
...........
cdef class PythonFoo:
C_FOO* pythonFoo
def __cinit__(self):
self.pythonFoo=new CFOO()
I do have a PyFoo.PXD also, that simply does this for a forward decl so other classes can see it
cdef class PythonFoo:
C_FOO * pythonFoo
Now I have another class (in PyBar.pyx) that needs PythonFoo.pythonFoo as a member
cdef extern from "Bar.h" namespace "Bar":
cdef cppclass C_Bar:
ThisFunctionNeeds(C_FOO*)
cimport PythonFoo
cdef class PythonBar:
C_Bar* pythonBar
def __cinit__(self):
self.pythonBar=new C_Bar()
def SomeFunction(self,PythonFoo):
ThisFunctionNeeds(PythonFoo.pythonFoo)
No matter how I try to cimport it, I get "PythonFoo not defined" (Because there is no init in the pxd file I guess) or I have to manually include the Foo.pyx, making this now a member of Bar.
Can someone please show me (with an example please) how this is supposed to work?
Thank you.
EDIT: PythonFoo and PythonBar are in separate modules in the package.
In a Python Tester, I try to use this
import PythonFoo
import PythonBar
foo=PythonFoo()
bar=PythonBar()
bar.SomeFunction(foo)
Here is where I get the "No module named 'PyFoo'
I was able to find why it would not work... The actual directory of the module was not placed, as a matter of fact, the file was placed at a higher directory from the build directory.
I fixed it with a simple
sys.path.append('package_directory')
EDIT: I also found out why this was... I had an __init__.py in the same directory as my setup.py file, so the compiler thought that directory was part of the package. I deleted the __init__.py, and I no longer needed to append the directory.
Related
Using SWIG to generate a Python binding for my C++ project has not been easy but I have finally been able to do so. The only issue is that the generated .py file that houses essentially the class and method definitions of my wrapped C++ code (but callable for Python) is quite large. I basically want to modularize the generated .py file into submodules of relevant classes.
Here is a basic and stripped down sample of what my swig interface file looks like:
%module example
%{
/* these two headers should belong to ModuleOne */
#include "header1.hpp"
#include "header2.hpp"
/* these two headers should belong to ModuleTwo */
#include "header3.hpp"
#include "header4.hpp"
}
%include "header1.hpp"
%include "header2.hpp"
%include "header3.hpp"
%include "header4.hpp"
And from Python importing the package would be done like so:
from example import *
I find this messy as I either need to import each class individually with from example import ClassOne or import the entirety of the module.
How could I go about creating "submodules" of the swig generated .py file allowing me to modularize my project a bit cleaner and import those without necessarily importing the entire package. For example something like:
import example.ModuleOne
import example.ModuleTwo
I think you just need to add an __init__.py file that imports both modules, something like this:
from example.ModuleOne import *
from example.ModuleTwo import *
__all__ = [x for x in dir() if x[0] != '_']
The last line allows your program to use from example import * to import everything.
Edit: I've just read your question a bit more closely and realise you want to import just one of your submodules. You still need an __init__.py file to make your two modules into a package, but it can be empty. Your interface files should include a package declaration, e.g. %module(package="example") ModuleOne.
As part of larger cython project I am getting an error that seems to indicate that maybe the automatic type merging is failing. It is possible I am doing something silly or that the fact that I am structuring things in this way is a bad smell. I think I can get around the issue by using void* and then casts everywhere but this seems quite messy for what should be a simple thing.
It seems similar to this issue, but that involved function pointers.
Wrong signature error using function types in multiple modules
I have put together a minimal example that demonstrates the issue. The files are as follows.
a.pxd
cdef struct A:
int a
cdef A makeA(int a)
a.pyx
cdef struct A:
pass
cdef A makeA(int a):
cdef A sA
sA.a = a
return sA
b.pyx
from cfr.a cimport A, makeA
cdef int get_A_value():
cdef A sA = makeA(5)
return sA.a
If I compile and then try to import b in a jupyter-lab notebook I get the following error:
b.pyx in init b()
TypeError: C function cfr.a.makeA has wrong signature (expected struct __pyx_t_3cfr_1a_A (int), got struct __pyx_t_1a_A (int))
EDIT: It looks like it might be something to do with my directory structure. Note that I do from cfr.a cimport A (as this happens to be the layout of my project). I think I need to do from a import A and get the import to work through the use of my pythonpath.
So it turned out the issue was in the from cfr.a cimport A line. This was somehow meaning that the A type was not being seen as the same type as A in the a.pyx file.
To fix use:
from a cimport A
(You may then get a not found error, but we can fix this with our pythonpath).
In the comments adding a __init__.pxd file was suggested, but adding a blank one did not work for me.
A file Globals.h contains the following definition of a constant:
namespace MyNameSpace {
/** Constants **/
constexpr index none = std::numeric_limits<index>::max();
}
... where index is a typedef for uint64_t.
How can I expose it to Cython and Python?
A failed attempt:
cdef extern from "../cpp/Globals.h" namespace "MyNamespace":
cdef index _none "MyNamespace::none"
none = _none
The syntax for exposing (global) constants is similar to the syntax for exposing simple attributes and the syntax for exposing static members, in your example the syntax is almost right except that you need to omit the cdef statement, the cdef statement is only for declaring new variables in Cython, not for adding information about externally declared variables.
cdef extern from "../cpp/Globals.h" namespace "MyNamespace":
index _none "MyNamespace::none"
none = _none
Then you can use none in Python, if your Cython module is named mymodule, the import statement could be
from mymodule import none
if you want to make none available as global name in your Python code.
Let us have script foo.pyx with function in it:
def hello():
cdef int* i = <int *> malloc(sizeof(int))
i[0] = 1
trol(i)
print i
and script with function noo.pyx:
cdef trol(int * i):
i[0] = 42
the question is, how do I now import the trol function from file noo.pyx to foo.pyx, so I can use it in hello function.
This is only model example, but I think, that it illustrate the problem fair enough.
I tried simple
from noo import trol
but that throws "Cannot convert 'int *' to Python object"
Edit: I would better add, that this example will work just fine if I put both functions to same file.
This seems like something obvious to try, but did you try:
from noo cimport trol
If you use import instead of cimport, I think it will try to cast trol as a python function and generate the error you're getting.
The solution eventually was to create additional .pxd file, which something very similar to classic header .h file in C. It stores functions declarations and when cimport is called, it is in this file where it looks for functions and structures.
So to be specific, all I needed to do was to create file noo.pxd containing:
cdef trol(int * i)
and than we can simply cimport this function from foo.pyx by calling
from noo cimport trol
I am having problems compiling cdef-ed types in different packages and I couldn't find an explanation in cython docs.
I have this setup.py in the root of my python src tree:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("flink.pytk.defs.FragIdx",
sources = ["flink/pytk/defs/FragIdx.pyx"]),
Extension("flink.pytk.fragments.STK_idx",
sources = ["flink/pytk/fragments/STK_idx.pyx"])
]
)
FragIdx is a cdef-ed type, defined in flink/pytk/defs/FragIdx.pyx:
cdef class FragIdx:
cdef public FragIdx parent
cdef public FragIdx root
cdef public tuple label
...
And STK_idx is an extension of FragIdx, defined in flink/pytk/fragments/STK_idx.pyx:
from flink.pytk.defs.FragIdx import FragIdx
cdef class STK_idx(FragIdx):
...
When I try to compile using the setup.py listed at the beginning of the post, FragIdx is compiled all right, but when it comes to STK_idx I get the following error message:
flink/pytk/fragments/STK_idx.pyx:5:5: 'FragIdx' is not a type name
Please note that the root directory of my source tree is listed in $PYTHONPATH.
I would really appreciate if anyone could shed any light on this, thanks a lot!
Daniele
Oh, well, for those having a similar problem, it looks like maybe I found the answer.
I was expecting python to automatically scan the symbols compiled into the shared library FragIdx.so, instead it looks like this information must be provided explicitly as a .pxd file (which becomes a C header file after Cython is run).
There are basically two steps involved in the process:
Creation of a definition (.pxd) file for the superclass;
Importing of the the superclass definition via cimport (as opposed to import) in the subclass module.
So, to make it more general.
Suppose that you have defined your cdef-ed type A in module pkg1.mod1. Then you cdef a type B in pkg2.mod2 that subclasses A.
Your directory structure would look something like this:
pkg1/
mod1.pyx
mod1.pxd
pkg2/
mod2.pyx
mod2.pxd
In pkg1/mod1.pxd you would have, say:
cdef class A:
cdef int a
cdef int b
And in pkg1/mod1.pyx you would provide the methods of your class.
In pkg2/mod2.pxd, you would have:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
cdef ... # your attributes here
And again, in pkg2/mod2.pyx you would have to cimport the A symbol again:
from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
... # your methods here
Interestingly enough, if you just want to use A in your python code, as opposed to using it to define a subtype, the definitions file mod1.pxd is not needed. This is related to the fact that when creating an extension type you need the definitions to be available for the C compiler, whereas you don't have this problem when running python code, but since it is not very intuitive maybe it's important to point it out.
This information is actually available in the Cython docs, though maybe it could be a little bit more explicit.
Hope this information can save some to someone.