I am trying to invoke a Cython file in a Python script. I have read this answer and followed the instruction. However, even though I have successfully compiled the C code, PyCharm does not recognize the Cython file 'hello.pyx' while executing the import command, as shown in the screenshot below. What is the remedy?
The Cython file hello.c is generated by setup.py the content of which is shown below.
from distutils.core import setup
from Cython.Build import cythonize
setup(name='Hello world app',
ext_modules=cythonize("hello.pyx")
)
When you compile your module with cythonize, the resulting module will be put into a subfolder. By default, this subfolder is not in the PATHS list. To fix this problem you can just move the resulting file .dll file into the folder where your hello.py and hello.pyx are located.
An alternative approach is to add Extension like so:
from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension
exts = [Extension(name='hello',sources=['hello.pyx'])]
setup(name='Hello world app',
ext_modules=cythonize(exts)
)
And then compile with:
python setup.py build_ext --inplace
This code will put resulting .dll into the same folder as your source files.
I have a question. I would like to distribute my cython-powered packages, but I see no easy way to build them in setup.py. I would like setup.py to:
most importantly: install my package without cython (from pre-generated C files or by installing cython beforehand)
rebuild (run cythonize) package on sdist
not need to hard-code list of my cython modules (just use glob or something)
be able to work without .c files (should not be stored in git) or .pyx (might not be distributed). at least one of those sets will be always present of course.
Currently in my itchy package, I am using this quite complicated code:
import os
from glob import glob
from distutils.command.build_ext import build_ext as _build_ext
from distutils.command.sdist import sdist as _sdist
from distutils.core import setup
from distutils.core import Extension
def generate_extensions():
return [
# Compile cython-generated .c files into importable .so libraries.
Extension(os.path.splitext(name)[0], [name])
for name in C_FILES
]
# In distribution version, there are no pyx files, when you clone package from git, there will be no c files.
CYTHON_FILES = glob('itchy/*.pyx')
C_FILES = glob('itchy/*.c')
extensions = generate_extensions()
class build_ext(_build_ext):
def run(self):
# Compile cython files (.pyx, some of the .py) into .c files if Cython is available.
try:
from Cython.Build import cythonize
if CYTHON_FILES:
cythonize(CYTHON_FILES)
# Update C_FILES in case they were originally missing.
global C_FILES, extensions
C_FILES = glob('itchy/*.c')
extensions = generate_extensions()
else:
print('No .pyx files found, building extensions skipped. Pre-built versions will be used.')
except ImportError:
print('Cython is not installed, building extensions skipped. Pre-built versions will be used.')
assert C_FILES, 'C files have to be present in distribution or Cython has to be installed'
_build_ext.run(self)
class sdist(_sdist):
def run(self):
# Make sure the compiled Cython files in the distribution are up-to-date
self.run_command("build_ext")
_sdist.run(self)
setup(
(...)
ext_modules = extensions,
cmdclass = {
'build_ext': build_ext,
'sdist': sdist,
},
)
Usually done by attempting to import cython and adjusting extensions to either
Build pyx files with cython if cython is present
Build C files if cython is not present
For example:
try:
from Cython.Distutils.extension import Extension
from Cython.Distutils import build_ext
except ImportError:
from setuptools import Extension
USING_CYTHON = False
else:
USING_CYTHON = True
ext = 'pyx' if USING_CYTHON else 'c'
sources = glob('my_module/*.%s' % (ext,))
extensions = [
Extension(source.split('.')[0].replace(os.path.sep, '.'),
sources=[source],
)
for source in sources]
cmdclass = {'build_ext': build_ext} if USING_CYTHON else {}
setup(<..>, ext_modules=extensions, cmdclass=cmdclass)
The source.split stuff is needed as cythonized extension names need to be in the form my_module.ext while glob requires path names like my_module/ext.
See this repository for a real-world example.
You should, however, include .c files in your git repo as well as the distributable otherwise when it comes time to build a distribution the .c files will be re-built and may or may not be the same files as were built on your machine.
They may be built by another version of cython, for example, or on a different platform, producing different code.
Cython is a static compiler - it's recommended to commit the files it produces to repository.
It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.
See Cython documentation on distributing modules.
I have the following package structure:
+ repo/
+ setup.py
+ package/
+ module1/
+ submodule1.py
+ submodule2.pyx
+ module2/
+ submodule3.py
I would like to use submodule2.pyx from submodule1.py by something like:
import submodule2
but I have absolutely no idea how to do this. I tried adding the following lines to my setup.py:
from distutils.core import setup
from setuptools import setup
from Cython.Distutils import build_ext
ext_modules = cythonize(Extension(
"zindex",
sources=["ndmg/graph/zindex.pyx"],
language="c",
))
for e in ext_modules:
e.pyrex_directives = {"boundscheck": False}
setup(
name='ndmg',
ext_modules = ext_modules,
packages=[
'package',
'package.module1',
....
)
but was unsuccessful. All of the tutorials I could find had very very simplified examples, so I am not sure how to include Cython modules in my python package when the rest of the package is all just normal python code. Does anybody have any good examples I could follow, or can somebody tell me what I'm doing wrong?
Thanks in advance!
The name given to cythonize is what Cython will use to call the module and what it will be have to be imported as.
The above setup.py will generate a native extension called zindex and will have to be imported as import zindex even within python files in the zindex package.
Here is an example of how to do this:
from distutils.core import setup
from setuptools import setup
from Cython.Distutils import build_ext
ext_modules = cythonize(Extension(
"ndmg.graph.zindex",
sources=["ndmg/graph/zindex.pyx"],
language="c",
))
<..>
Build and install extension.
In a python file under ndmg/graph/py_index.py you can then do.
from zindex import <..>
to import from the cython module.
I have a package named mypack which inside has a module mymod.py, and
the __init__.py.
For some reason that is not in debate, I need to package this module compiled
(nor .py or .pyc files are allowed). That is, the __init__.py is the only
source file allowed in the distributed compressed file.
The folder structure is:
.
│
├── mypack
│ ├── __init__.py
│ └── mymod.py
├── setup.py
I find that Cython is able to do this, by converting each .py file in a .so library
that can be directly imported with python.
The question is: how the setup.py file must be in order to allow an easy packaging and installation?
The target system has a virtualenv where the package must be installed with
whatever method that allows easy install and uninstall (easy_install, pip, etc are all
welcome).
I tried all that was at my reach. I read setuptools and distutils documentation,
all stackoverflow related questions,
and tried with all kind of commands (sdist, bdist, bdist_egg, etc), with lots
of combinations of setup.cfg and MANIFEST.in file entries.
The closest I got was with the below setup file, that would subclass the bdist_egg
command in order to remove also .pyc files, but that is breaking the installation.
A solution that installs "manually" the files in the venv is
also good, provided that all ancillary files that are included in a proper
installation are covered (I need to run pip freeze in the venv and see
mymod==0.0.1).
Run it with:
python setup.py bdist_egg --exclude-source-files
and (try to) install it with
easy_install mymod-0.0.1-py2.7-linux-x86_64.egg
As you may notice, the target is linux 64 bits with python 2.7.
from Cython.Distutils import build_ext
from setuptools import setup, find_packages
from setuptools.extension import Extension
from setuptools.command import bdist_egg
from setuptools.command.bdist_egg import walk_egg, log
import os
class my_bdist_egg(bdist_egg.bdist_egg):
def zap_pyfiles(self):
log.info("Removing .py files from temporary directory")
for base, dirs, files in walk_egg(self.bdist_dir):
for name in files:
if not name.endswith('__init__.py'):
if name.endswith('.py') or name.endswith('.pyc'):
# original 'if' only has name.endswith('.py')
path = os.path.join(base, name)
log.info("Deleting %s",path)
os.unlink(path)
ext_modules=[
Extension("mypack.mymod", ["mypack/mymod.py"]),
]
setup(
name = 'mypack',
cmdclass = {'build_ext': build_ext,
'bdist_egg': my_bdist_egg },
ext_modules = ext_modules,
version='0.0.1',
description='This is mypack compiled lib',
author='Myself',
packages=['mypack'],
)
UPDATE.
Following #Teyras answer, it was possible to build a wheel as requested in the answer. The setup.py file contents are:
import os
import shutil
from setuptools.extension import Extension
from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils import build_ext
class MyBuildExt(build_ext):
def run(self):
build_ext.run(self)
build_dir = os.path.realpath(self.build_lib)
root_dir = os.path.dirname(os.path.realpath(__file__))
target_dir = build_dir if not self.inplace else root_dir
self.copy_file('mypack/__init__.py', root_dir, target_dir)
def copy_file(self, path, source_dir, destination_dir):
if os.path.exists(os.path.join(source_dir, path)):
shutil.copyfile(os.path.join(source_dir, path),
os.path.join(destination_dir, path))
setup(
name = 'mypack',
cmdclass = {'build_ext': MyBuildExt},
ext_modules = cythonize([Extension("mypack.*", ["mypack/*.py"])]),
version='0.0.1',
description='This is mypack compiled lib',
author='Myself',
packages=[],
include_package_data=True )
The key point was to set packages=[],. The overwriting of the build_ext class run method was needed to get the __init__.py file inside the wheel.
Unfortunately, the answer suggesting setting packages=[] is wrong and may break a lot of stuff, as can e.g. be seen in this question. Don't use it. Instead of excluding all packages from the dist, you should exclude only the python files that will be cythonized and compiled to shared objects.
Below is a working example; it uses my recipe from the question Exclude single source file from python bdist_egg or bdist_wheel. The example project contains package spam with two modules, spam.eggs and spam.bacon, and a subpackage spam.fizz with one module spam.fizz.buzz:
root
├── setup.py
└── spam
├── __init__.py
├── bacon.py
├── eggs.py
└── fizz
├── __init__.py
└── buzz.py
The module lookup is being done in the build_py command, so it is the one you need to subclass with custom behaviour.
Simple case: compile all source code, make no exceptions
If you are about to compile every .py file (including __init__.pys), it is already sufficient to override build_py.build_packages method, making it a noop. Because build_packages doesn't do anything, no .py file will be collected at all and the dist will include only cythonized extensions:
import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize
extensions = [
# example of extensions with regex
Extension('spam.*', ['spam/*.py']),
# example of extension with single source file
Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]
class build_py(build_py_orig):
def build_packages(self):
pass
setup(
name='...',
version='...',
packages=find_packages(),
ext_modules=cythonize(extensions),
cmdclass={'build_py': build_py},
)
Complex case: mix cythonized extensions with source modules
If you want to compile only selected modules and leave the rest untouched, you will need a bit more complex logic; in this case, you need to override module lookup. In the below example, I still compile spam.bacon, spam.eggs and spam.fizz.buzz to shared objects, but leave __init__.py files untouched, so they will be included as source modules:
import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize
extensions = [
Extension('spam.*', ['spam/*.py']),
Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]
cython_excludes = ['**/__init__.py']
def not_cythonized(tup):
(package, module, filepath) = tup
return any(
fnmatch.fnmatchcase(filepath, pat=pattern) for pattern in cython_excludes
) or not any(
fnmatch.fnmatchcase(filepath, pat=pattern)
for ext in extensions
for pattern in ext.sources
)
class build_py(build_py_orig):
def find_modules(self):
modules = super().find_modules()
return list(filter(not_cythonized, modules))
def find_package_modules(self, package, package_dir):
modules = super().find_package_modules(package, package_dir)
return list(filter(not_cythonized, modules))
setup(
name='...',
version='...',
packages=find_packages(),
ext_modules=cythonize(extensions, exclude=cython_excludes),
cmdclass={'build_py': build_py},
)
While packaging as a wheel is definitely what you want, the original question was about excluding .py source files from the package. This is addressed in Using Cython to protect a Python codebase by #Teyras, but his solution uses a hack: it removes the packages argument from the call to setup(). This prevents the build_py step from running which does, indeed, exclude the .py files but it also excludes any data files you want included in the package. (For example my package has a data file called VERSION which contains the package version number.) A better solution would be replacing the build_py setup command with a custom command which only copies the data files.
You also need the __init__.py file as described above. So the custom build_py command should create the __init_.py file. I found that the compiled __init__.so runs when the package is imported so all that is needed is an empty __init__.py file to tell Python that the directory is a module which is ok to import.
Your custom build_py class would look like:
import os
from setuptools.command.build_py import build_py
class CustomBuildPyCommand(build_py):
def run(self):
# package data files but not .py files
build_py.build_package_data(self)
# create empty __init__.py in target dirs
for pdir in self.packages:
open(os.path.join(self.build_lib, pdir, '__init__.py'), 'a').close()
And configure setup to override the original build_py command:
setup(
...
cmdclass={'build_py': CustomBuildPyCommand},
)
I suggest you use the wheel format (as suggested by fish2000). Then, in your setup.py, set the packages argument to []. Your Cython extension will still build and the resulting .so files will be included in the resulting wheel package.
If your __init__.py is not included in the wheel, you can override the run method of build_ext class shipped by Cython and copy the file from your source tree to the build folder (the path can be found in self.build_lib).
This was exactly the sort of problem the Python wheels format – described in PEP 427 – was developed to address.
Wheels are a replacement for Python eggs (which were/are problematic for a bunch of reasons) – they are supported by pip, can contain architecture-specific private binaries (here is one example of such an arrangement) and are accepted generally by the Python communities who have stakes in these kind of things.
Here is one setup.py snippet from the aforelinked Python on Wheels article, showing how one sets up a binary distribution:
import os
from setuptools import setup
from setuptools.dist import Distribution
class BinaryDistribution(Distribution):
def is_pure(self):
return False
setup(
...,
include_package_data=True,
distclass=BinaryDistribution,
)
… in leu of the older (but probably somehow still canonically supported) setuptools classes you are using. It’s very straightforward to make Wheels for your distribution purposes, as outlined – as I recall from experience, either the wheel modules’ build process is somewhat cognizant of virtualenv, or it’s very easy to use one within the other.
In any case, trading in the setuptools egg-based APIs for wheel-based tooling should save you some serious pain, I should think.
I have a Python application to which I recently added a Cython module. Running it from script with pyximport works fine, but I also need an executable version which I build with cx_Freeze.
Trouble is, trying to build it gives me an executable that raises ImportError trying to import the .pyx module.
I modified my setup.py like so to see if I could get it to compile the .pyx first so that cx_Freeze could successfully pack it:
from cx_Freeze import setup, Executable
from Cython.Build import cythonize
setup(name='projectname',
version='0.0',
description=' ',
options={"build_exe": {"packages":["pygame","fx"]},'build_ext': {'compiler': 'mingw32'}},
ext_modules=cythonize("fx.pyx"),
executables=[Executable('main.py',targetName="myproject.exe",base = "Win32GUI")],
requires=['pygcurse','pyperclip','rsa','dill','numpy']
)
... but then all that gives me is No module named fx within cx_Freeze at build-time instead.
How do I make this work?
The solution was to have two separate calls to setup(); one to build fx.pyx with Cython, then one to pack the exe with cx_Freeze. Here's the modified setup.py:
from cx_Freeze import Executable
from cx_Freeze import setup as cx_setup
from distutils.core import setup
from Cython.Build import cythonize
setup(options={'build_ext': {'compiler': 'mingw32'}},
ext_modules=cythonize("fx.pyx"))
cx_setup(name='myproject',
version='0.0',
description='',
options={"build_exe": {"packages":["pygame","fx"]}},
executables=[Executable('main.py',targetName="myproject.exe",base = "Win32GUI")],
requires=['pygcurse','pyperclip','rsa','dill','numpy']
)