We Keep Coding

Cython bdist_egg with setuptools creates invalid package

I am attempting to compile a *.pyx file. It uses some definitions and constants inside a __init__.py in the same directory. The project structure is:
setup.py
Foo/__init__.py
Foo/Foo.pyx
and the setup command is as follows:
from setuptools import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy
setup(
cmdclass = {'build_ext': build_ext},
ext_module = [ Extension(name='Foo', sources=['Foo/Foo.pyx']) ],
include_dirs=[numpy.get_include()],
name='Foo',
packages=['Foo'],
zip_safe=True
)
Problem arises when the egg is built and deployed. The resultant egg has the following structure:
Foo.so
Foo.py
Foo/__init__.py
Now, Foo.py contains some dynamic import code that basically imports the *.so file. However, because of the presence of Foo/__init__.py, import Foo attempts to import symbols only from __init__.py, which contains just some constants (all the relevant code is actually in Foo.so).
I've hacked around this issue by pasting all the definitions from __init__.py into Foo.pyx, but I'm trying to figure out what a proper solution might be.
Any advice is appreciated!

I tracked down my problem to an extraneous argument to the setup() command. Judging by the documentation at https://docs.python.org/2/distutils/setupscript.html, I do not need the packages=['Foo'] argument, and in fact that's what's causing it to create the inner Foo package that's messing everything up.

Cannot Import F2PY Generated Module in __init__.py

I am Wrapping a fortran library with f2py, so far it has been successful in terms of getting the functions to work within python
I am at the point of structuring my package and am running into __init__.py issues when using these f2py generated modules. This is related to a package constructed with distutils extensions for f2py
After installing the egg, in an external script I can import the f2py generated module directly
from package.core.module1 import fortran_fn
,but when I place
from module1 import fortran_fn
in package/core/__init__.py in order to refactor the namespace for users,
the package import fails
----> 1 import package.core as core
~/venv/dev/lib/python3.5/site-packages/package-py3.5-linux-x86_64.egg/package/core/__init__.py in <module>()
1
2
----> 3 from module1 import fortran_fn
4
5
ImportError: No module named 'module1'
However, there is one exception, for whatever reason one module in particular, does wish to import itself.
Project Structure:
setup.py
package/
__init__.py
core/
__init__.py
module1.pyf
module2.pyf
...
src/
lib/
module1.f
module2.f
...
*pyf were generated with f2py -h package/core/moduleX.pyf src/lib/moduleX.f -m moduleX
Further,
#core/__init__.py
from module1 import fortran_fn
and,
#setup.py
from setuptools import find_packages
from numpy.distutils.core import Extension
ext1 = Extension(name='package.core.module1',
sources=['package/core/module1.pyf',
'src/lib/module1.f'])
#ext2 = ...
# and others
setup (name="package",
packages = find_packages(),
ext_modules = [ ext1 ]) # ..., ext2, ... ]
Code samples simplified to illustrate the problem, the specific code is available at this github
For some context, I initially tried to have f2py place the fortran into the .core subpackage directly by compiling with -m package.core and the same for name='package.core' for extension, however f2py would complain about missing names.
In this particular structure there is an additional layer where each fortran module is just one function, and I would like to remove the extra submodule layer from the API.

Using a fully qualified name in core/__init__.py will resolve the immediate import problem.
Instead of:
#core/__init__.py
from module1 import fortran_fn
This will work:
#core/__init__.py
from package.core.module1 import fortran_fn
then
> import package.core as c
> c.fortran_fn()
SUCCESS
It appears the f2py extensions do not register as subpackages/submodules when distutils runs. I may be mistaken in that interpretation.

In Python 3 all imports are absolute. To perform relative import do it explicitly:
from .module1 import fortran_fn

Python package Cython module

I have the following package structure:
+ repo/
+ setup.py
+ package/
+ module1/
+ submodule1.py
+ submodule2.pyx
+ module2/
+ submodule3.py
I would like to use submodule2.pyx from submodule1.py by something like:
import submodule2
but I have absolutely no idea how to do this. I tried adding the following lines to my setup.py:
from distutils.core import setup
from setuptools import setup
from Cython.Distutils import build_ext
ext_modules = cythonize(Extension(
"zindex",
sources=["ndmg/graph/zindex.pyx"],
language="c",
))
for e in ext_modules:
e.pyrex_directives = {"boundscheck": False}
setup(
name='ndmg',
ext_modules = ext_modules,
packages=[
'package',
'package.module1',
....
)
but was unsuccessful. All of the tutorials I could find had very very simplified examples, so I am not sure how to include Cython modules in my python package when the rest of the package is all just normal python code. Does anybody have any good examples I could follow, or can somebody tell me what I'm doing wrong?
Thanks in advance!

The name given to cythonize is what Cython will use to call the module and what it will be have to be imported as.
The above setup.py will generate a native extension called zindex and will have to be imported as import zindex even within python files in the zindex package.
Here is an example of how to do this:
from distutils.core import setup
from setuptools import setup
from Cython.Distutils import build_ext
ext_modules = cythonize(Extension(
"ndmg.graph.zindex",
sources=["ndmg/graph/zindex.pyx"],
language="c",
))
<..>
Build and install extension.
In a python file under ndmg/graph/py_index.py you can then do.
from zindex import <..>
to import from the cython module.

Undefined symbol when importing a cython module making use of another cython module

I'm working on wrapping a set of C functions using Cython into a module. I would like to be able to cimport this first module into subsequent Cython-based projects, but I am running into an 'undefined symbol' issue when importing the derivative modules in a python script.
Consider the following minimal working example of two modules developed in separate directories:
# exModA wraps the functions provided in exModAFxn.c
exModA/
__init__.pxd
__init__.py
exModAFxn.c
exModA.pyx
setup.py
# exModB attempts to use the cdef functions from exModA
exModB/
__init__.py
exModB.pyx
setup.py
# The test script attempts to make use of exModB
test.py
exModA/__init__.pxd:
cdef extern void myCMessage()
exModA/__init__.py:
from exModA import *
exModA/exModAFxn.c:
#include <stdio.h>
void myCMessage() { printf( "This is a test!\n" ); }
exModA/exModA.pyx:
cdef extern from "exModAFxn.c" :
void myCMessage()
# Use myCMessage in a python function
def foo() :
myCMessage()
exModA/setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(
name = 'exModA',
ext_modules = cythonize( ['exModA/exModA.pyx'] ),
)
exModB/__init__.py:
from exModB import *
exModB/exModB.pyx:
cimport exModA
# Use a cdef function from exModA in a python function
def bar() :
exModA.myCMessage()
exModB/setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(
name = 'exModB',
ext_modules = cythonize( ['exModB/exModB.pyx'] ),
)
The test.py script is called after compiling the two cython modules.
python extModA/setup.py build_ext --inplace
python extModB/setup.py build_ext --inplace
test.py:
import exModA
exModA.foo() # successfully prints message
import exModB # raises ImportError: undefined symbol: myCMessage
exModB.bar() # we never get to this line :(
The successful printing of the message from the exModA.foo function suggests to me that myCMessage is in fact available, but it isn't found when exModB is imported. I know this problem goes away if I merged exModA and exModB into a single project, but I'd like to avoid that approach if possible -- I'm trying to expose a common set of wrappers for a C library for use in several different applications.

I was able to get this to work by doing the following:
First, specify the exModA library for linking in your exModB/setup.py file as follows:
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension('exModB/exModB',
sources=['exModB/exModB.pyx'],
libraries=['exModA'],
library_dirs=['/home/MyFolder/exModA'],
runtime_library_dirs=['/home/MyFolder/exModA']
)
setup(name='exModB', ext_modules = cythonize(ext))
(Instead of specifying a runtime library directory, I recommend moving exModA.so to a folder on your library search path.)
I also had to rename exModA.so to libexModA.so so that gcc could find it. Do this after building exModA.so, but before building exModB.so:
python exModA/setup.py build_ext --inplace
cp exModA/exModA.so exModA/libexModA.so
python exModB/setup.py build_ext --inplace
You will probably want to tweak this, but it should get you started.

Cython: ImportError: No module named 'myModule': how to call a cython module containing a cimport to another cython nodule?

I'm trying to import a cython module data.pyx into another cython module user.pyx.
Everything compile fine, but when I try to call user.pyx in a python module, I am getting the error 'ImportError: No module named data'.
Everything is in the same directory.
package/
__init__.py #empty
setup.py
data.pxd
data.pyx
user.pyx
My setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension("data", ["data.pyx"]),
Extension("user", ["user.pyx"],include_dirs = ['myPackageDir'])
]
setup(
name = 'app',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
Running the following test.py will raised the error.
import user #this line raised the 'ImportError: No module named data' below
user.doSomething()
The exception I get is
Traceback:
File "test.py", line 1, in <module>
import package.user
File "user.pyx", line 1, in init user (user.c:3384)
ImportError: No module named data
How can I make the import work? Thanks for any help.

I encounter this problem again in an another project. To solve it, here is what I did:
all import and cimport statement must be fully qualified
all the python code must be contained in a rootFolder
the setup.py must be at the same level than the rootFolder
all folder in the rooFolder including the rootFolder must contains a __init__.py
in your setup.py the extension's include_dirs must contains '.'
I created a simple project which illustrates this here.
This page helped me created it.
But my project is simpler and I think it would have helped me a lot if I had it.
My project also illustrate how to automatically build all cython files in a project.

I might be missing something about Cython, but I think it's:
import package.user
user.doSomething()