I have a large C/C++ project which I usually just build as a standalone executable. It also requires some libraries (system libraries like pthread, but also MKL from intel for example) which I specify when compiling the project.
Now, I want to use some functions from this project in Python. So my first plan was to simply build the project as a static library and then use that to write a cython wrapper. I.e.
Build the c project: icc -c src/main.cpp .. options and stuff.. linking to all libraries needed .. -o main.o
Create a static library (including the libraries I needed in step 1): ar -rc libmain.a main.o ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ... /usr/lib/x86_64-linux-gnu/libpthread.a ...
I use the generated static library to build my cython wrapper
when I try to execute a test python script calling a random function from my C program I get this error ImportError: /home/.../main.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_ok_to_fork
It seems like I need to tell cython to link to the corresponding libraries again or something, but I'm not really sure how to resolve this.. Also I don't want the python project to be dependent on some special libraries that I have installed on my system (that's what I try to achieve by adding all the libraries in step 2), is this possible at all?
My setup.py looks like this
from setuptools import setup, Extension
from Cython.Build import cythonize
import os
os.environ["CC"] = "icc"
os.environ["CXX"] = "icc"
setup(
ext_modules = cythonize([Extension("test", ["test.pyx"], language="c++",
library_dirs=[r'.'], libraries=['main'])])
)
Create a shared library (.so on linux). Python can only support dynamic or shared libraries. The linker is looking for a .so not a .a.
So your setup.py would be remain the same,
from setuptools import setup, Extension
from Cython.Build import cythonize
import os
os.environ["CC"] = "icc"
os.environ["CXX"] = "icc"
setup(
ext_modules = cythonize([Extension("test", ["test.pyx"], language="c++",
library_dirs=[r'.'], libraries=['main'])])
)
Related - Using Cython To Link Python To A Shared Library.
I have a question. I would like to distribute my cython-powered packages, but I see no easy way to build them in setup.py. I would like setup.py to:
most importantly: install my package without cython (from pre-generated C files or by installing cython beforehand)
rebuild (run cythonize) package on sdist
not need to hard-code list of my cython modules (just use glob or something)
be able to work without .c files (should not be stored in git) or .pyx (might not be distributed). at least one of those sets will be always present of course.
Currently in my itchy package, I am using this quite complicated code:
import os
from glob import glob
from distutils.command.build_ext import build_ext as _build_ext
from distutils.command.sdist import sdist as _sdist
from distutils.core import setup
from distutils.core import Extension
def generate_extensions():
return [
# Compile cython-generated .c files into importable .so libraries.
Extension(os.path.splitext(name)[0], [name])
for name in C_FILES
]
# In distribution version, there are no pyx files, when you clone package from git, there will be no c files.
CYTHON_FILES = glob('itchy/*.pyx')
C_FILES = glob('itchy/*.c')
extensions = generate_extensions()
class build_ext(_build_ext):
def run(self):
# Compile cython files (.pyx, some of the .py) into .c files if Cython is available.
try:
from Cython.Build import cythonize
if CYTHON_FILES:
cythonize(CYTHON_FILES)
# Update C_FILES in case they were originally missing.
global C_FILES, extensions
C_FILES = glob('itchy/*.c')
extensions = generate_extensions()
else:
print('No .pyx files found, building extensions skipped. Pre-built versions will be used.')
except ImportError:
print('Cython is not installed, building extensions skipped. Pre-built versions will be used.')
assert C_FILES, 'C files have to be present in distribution or Cython has to be installed'
_build_ext.run(self)
class sdist(_sdist):
def run(self):
# Make sure the compiled Cython files in the distribution are up-to-date
self.run_command("build_ext")
_sdist.run(self)
setup(
(...)
ext_modules = extensions,
cmdclass = {
'build_ext': build_ext,
'sdist': sdist,
},
)
Usually done by attempting to import cython and adjusting extensions to either
Build pyx files with cython if cython is present
Build C files if cython is not present
For example:
try:
from Cython.Distutils.extension import Extension
from Cython.Distutils import build_ext
except ImportError:
from setuptools import Extension
USING_CYTHON = False
else:
USING_CYTHON = True
ext = 'pyx' if USING_CYTHON else 'c'
sources = glob('my_module/*.%s' % (ext,))
extensions = [
Extension(source.split('.')[0].replace(os.path.sep, '.'),
sources=[source],
)
for source in sources]
cmdclass = {'build_ext': build_ext} if USING_CYTHON else {}
setup(<..>, ext_modules=extensions, cmdclass=cmdclass)
The source.split stuff is needed as cythonized extension names need to be in the form my_module.ext while glob requires path names like my_module/ext.
See this repository for a real-world example.
You should, however, include .c files in your git repo as well as the distributable otherwise when it comes time to build a distribution the .c files will be re-built and may or may not be the same files as were built on your machine.
They may be built by another version of cython, for example, or on a different platform, producing different code.
Cython is a static compiler - it's recommended to commit the files it produces to repository.
It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.
See Cython documentation on distributing modules.
I have a package named mypack which inside has a module mymod.py, and
the __init__.py.
For some reason that is not in debate, I need to package this module compiled
(nor .py or .pyc files are allowed). That is, the __init__.py is the only
source file allowed in the distributed compressed file.
The folder structure is:
.
│
├── mypack
│ ├── __init__.py
│ └── mymod.py
├── setup.py
I find that Cython is able to do this, by converting each .py file in a .so library
that can be directly imported with python.
The question is: how the setup.py file must be in order to allow an easy packaging and installation?
The target system has a virtualenv where the package must be installed with
whatever method that allows easy install and uninstall (easy_install, pip, etc are all
welcome).
I tried all that was at my reach. I read setuptools and distutils documentation,
all stackoverflow related questions,
and tried with all kind of commands (sdist, bdist, bdist_egg, etc), with lots
of combinations of setup.cfg and MANIFEST.in file entries.
The closest I got was with the below setup file, that would subclass the bdist_egg
command in order to remove also .pyc files, but that is breaking the installation.
A solution that installs "manually" the files in the venv is
also good, provided that all ancillary files that are included in a proper
installation are covered (I need to run pip freeze in the venv and see
mymod==0.0.1).
Run it with:
python setup.py bdist_egg --exclude-source-files
and (try to) install it with
easy_install mymod-0.0.1-py2.7-linux-x86_64.egg
As you may notice, the target is linux 64 bits with python 2.7.
from Cython.Distutils import build_ext
from setuptools import setup, find_packages
from setuptools.extension import Extension
from setuptools.command import bdist_egg
from setuptools.command.bdist_egg import walk_egg, log
import os
class my_bdist_egg(bdist_egg.bdist_egg):
def zap_pyfiles(self):
log.info("Removing .py files from temporary directory")
for base, dirs, files in walk_egg(self.bdist_dir):
for name in files:
if not name.endswith('__init__.py'):
if name.endswith('.py') or name.endswith('.pyc'):
# original 'if' only has name.endswith('.py')
path = os.path.join(base, name)
log.info("Deleting %s",path)
os.unlink(path)
ext_modules=[
Extension("mypack.mymod", ["mypack/mymod.py"]),
]
setup(
name = 'mypack',
cmdclass = {'build_ext': build_ext,
'bdist_egg': my_bdist_egg },
ext_modules = ext_modules,
version='0.0.1',
description='This is mypack compiled lib',
author='Myself',
packages=['mypack'],
)
UPDATE.
Following #Teyras answer, it was possible to build a wheel as requested in the answer. The setup.py file contents are:
import os
import shutil
from setuptools.extension import Extension
from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils import build_ext
class MyBuildExt(build_ext):
def run(self):
build_ext.run(self)
build_dir = os.path.realpath(self.build_lib)
root_dir = os.path.dirname(os.path.realpath(__file__))
target_dir = build_dir if not self.inplace else root_dir
self.copy_file('mypack/__init__.py', root_dir, target_dir)
def copy_file(self, path, source_dir, destination_dir):
if os.path.exists(os.path.join(source_dir, path)):
shutil.copyfile(os.path.join(source_dir, path),
os.path.join(destination_dir, path))
setup(
name = 'mypack',
cmdclass = {'build_ext': MyBuildExt},
ext_modules = cythonize([Extension("mypack.*", ["mypack/*.py"])]),
version='0.0.1',
description='This is mypack compiled lib',
author='Myself',
packages=[],
include_package_data=True )
The key point was to set packages=[],. The overwriting of the build_ext class run method was needed to get the __init__.py file inside the wheel.
Unfortunately, the answer suggesting setting packages=[] is wrong and may break a lot of stuff, as can e.g. be seen in this question. Don't use it. Instead of excluding all packages from the dist, you should exclude only the python files that will be cythonized and compiled to shared objects.
Below is a working example; it uses my recipe from the question Exclude single source file from python bdist_egg or bdist_wheel. The example project contains package spam with two modules, spam.eggs and spam.bacon, and a subpackage spam.fizz with one module spam.fizz.buzz:
root
├── setup.py
└── spam
├── __init__.py
├── bacon.py
├── eggs.py
└── fizz
├── __init__.py
└── buzz.py
The module lookup is being done in the build_py command, so it is the one you need to subclass with custom behaviour.
Simple case: compile all source code, make no exceptions
If you are about to compile every .py file (including __init__.pys), it is already sufficient to override build_py.build_packages method, making it a noop. Because build_packages doesn't do anything, no .py file will be collected at all and the dist will include only cythonized extensions:
import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize
extensions = [
# example of extensions with regex
Extension('spam.*', ['spam/*.py']),
# example of extension with single source file
Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]
class build_py(build_py_orig):
def build_packages(self):
pass
setup(
name='...',
version='...',
packages=find_packages(),
ext_modules=cythonize(extensions),
cmdclass={'build_py': build_py},
)
Complex case: mix cythonized extensions with source modules
If you want to compile only selected modules and leave the rest untouched, you will need a bit more complex logic; in this case, you need to override module lookup. In the below example, I still compile spam.bacon, spam.eggs and spam.fizz.buzz to shared objects, but leave __init__.py files untouched, so they will be included as source modules:
import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize
extensions = [
Extension('spam.*', ['spam/*.py']),
Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]
cython_excludes = ['**/__init__.py']
def not_cythonized(tup):
(package, module, filepath) = tup
return any(
fnmatch.fnmatchcase(filepath, pat=pattern) for pattern in cython_excludes
) or not any(
fnmatch.fnmatchcase(filepath, pat=pattern)
for ext in extensions
for pattern in ext.sources
)
class build_py(build_py_orig):
def find_modules(self):
modules = super().find_modules()
return list(filter(not_cythonized, modules))
def find_package_modules(self, package, package_dir):
modules = super().find_package_modules(package, package_dir)
return list(filter(not_cythonized, modules))
setup(
name='...',
version='...',
packages=find_packages(),
ext_modules=cythonize(extensions, exclude=cython_excludes),
cmdclass={'build_py': build_py},
)
While packaging as a wheel is definitely what you want, the original question was about excluding .py source files from the package. This is addressed in Using Cython to protect a Python codebase by #Teyras, but his solution uses a hack: it removes the packages argument from the call to setup(). This prevents the build_py step from running which does, indeed, exclude the .py files but it also excludes any data files you want included in the package. (For example my package has a data file called VERSION which contains the package version number.) A better solution would be replacing the build_py setup command with a custom command which only copies the data files.
You also need the __init__.py file as described above. So the custom build_py command should create the __init_.py file. I found that the compiled __init__.so runs when the package is imported so all that is needed is an empty __init__.py file to tell Python that the directory is a module which is ok to import.
Your custom build_py class would look like:
import os
from setuptools.command.build_py import build_py
class CustomBuildPyCommand(build_py):
def run(self):
# package data files but not .py files
build_py.build_package_data(self)
# create empty __init__.py in target dirs
for pdir in self.packages:
open(os.path.join(self.build_lib, pdir, '__init__.py'), 'a').close()
And configure setup to override the original build_py command:
setup(
...
cmdclass={'build_py': CustomBuildPyCommand},
)
I suggest you use the wheel format (as suggested by fish2000). Then, in your setup.py, set the packages argument to []. Your Cython extension will still build and the resulting .so files will be included in the resulting wheel package.
If your __init__.py is not included in the wheel, you can override the run method of build_ext class shipped by Cython and copy the file from your source tree to the build folder (the path can be found in self.build_lib).
This was exactly the sort of problem the Python wheels format – described in PEP 427 – was developed to address.
Wheels are a replacement for Python eggs (which were/are problematic for a bunch of reasons) – they are supported by pip, can contain architecture-specific private binaries (here is one example of such an arrangement) and are accepted generally by the Python communities who have stakes in these kind of things.
Here is one setup.py snippet from the aforelinked Python on Wheels article, showing how one sets up a binary distribution:
import os
from setuptools import setup
from setuptools.dist import Distribution
class BinaryDistribution(Distribution):
def is_pure(self):
return False
setup(
...,
include_package_data=True,
distclass=BinaryDistribution,
)
… in leu of the older (but probably somehow still canonically supported) setuptools classes you are using. It’s very straightforward to make Wheels for your distribution purposes, as outlined – as I recall from experience, either the wheel modules’ build process is somewhat cognizant of virtualenv, or it’s very easy to use one within the other.
In any case, trading in the setuptools egg-based APIs for wheel-based tooling should save you some serious pain, I should think.
I tried to create executable of my File NewExistGUI2.py, where GUI is made using wxpython. The file depends upon other two files localsettings.py and Tryone.py. I referred to py2exe documentation, and created a setup.py file as:
from distutils.core import setup
import py2exe
setup(name = 'python eulexistdb module',
version = '1.0',
description = 'Python eXistdb communicator using eulexistdb module',
author = 'Sarvagya Pant',
py_modules = ['NewExistGUI2','localsettings','Tryone']
)
and compiled the program in command line using
python setup.py py2exe
But I didn't got any .exe file of the main program NewExistGUI2.py in dist folder created. What Should I do now?
I woul recommend you create a module (ExistGUI) with the following structure:
ExistGUI
\_ __init__.py
|_ localsettings.py
|_ Tryone.py
bin
\_ NewExistGUI2.py
Your init.py should have:
from . import localsettings, Tryone
__version__ = 1.0
Your setup.py should look something like:
from setuptools import setup, find_packages
import ExistGUI
import py2exe
setup(
name = 'ExistGUI',
version = ExistGUI.__version__,
console=['bin/NewExistGUI2.py'],
description = 'Python eXistdb communicator using eulexistdb module',
author = 'Sarvagya Pant',
packages= find_packages(),
scripts=['NewExistGUI2.py',],
py_modules = ['localsettings','Tryone'],
include_package_data=True,
zip_safe=False,
)
Then run python setup.py py2exe. Make sure you include any requirements for your module in setup.py. Also, remove the previously generated dist directory, just to be sure.
Hope this helps.
setup.py includes modules in the same directory it is in but leaves out modules in the site.packages directory and in a lib folder I have added to the pythonpath.
Don't tell me it doesn't just import files like the python interpreter. It should if you name them.
programdir-->
programdir-->
datadir
program.py
functions.py
setup.py
setup.py
from distutils.core import setup
setup(name = "program",
version = "1.0",
description = "a program",
author = "me",
author_email = "email#email.com",
url = "http://www.url.com",
py_modules = ["program", "functions", "tweepy", "anothermod", "webbrowser","" ],
data_files = [("data", ["data/intros.txt"])],
long_description = """
Descriptttttionnnn
"""
)
What am I doing wrong?
The setup.py is responsible for bundling and installing YOUR sources, not dependencies. However, you can specify requirements with install_requires (read the manual). This will be used by pip or setuptools to install the given dependencies, but it will not bundle them either.
Julius is using distutils, not setuptools. This means that requires shall be passed to the setup() function rather than install_requires. See the docs:
2.4. Relationships between Distributions and Packages