I have some protocol buffer definitions which need to be built to Python source as part of the pip install process. I've subclassed the setuptools.command.install command in setup.py but I think it's trying to run the Makefile after the package is installed so the sources aren't recognised.
I can't find information about what happens during a pip installation. Can anyone shed any light?
setup.py:
import subprocess
import sys
from setuptools import setup
from setuptools.command.install import install
class Install(install):
"""Customized setuptools install command - builds protos on install."""
def run(self):
protoc_command = ["make", "python"]
if subprocess.call(protoc_command) != 0:
sys.exit(-1)
install.run(self)
setup(
name='myprotos',
version='0.0.1',
description='Protocol Buffers.',
install_requires=[],
cmdclass={
'install': Install,
}
)
Output of $ pip install -vvv .:
Processing /path/to/myprotos
Running setup.py (path:/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py) egg_info for package from file:///path/to/myprotos
Running command python setup.py egg_info
running egg_info
creating pip-egg-info/myprotos.egg-info
writing pip-egg-info/myprotos.egg-info/PKG-INFO
writing top-level names to pip-egg-info/myprotos.egg-info/top_level.txt
writing dependency_links to pip-egg-info/myprotos.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
reading manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
Source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build has version 0.0.1, which satisfies requirement myprotos==0.0.1 from file:///path/to/myprotos
Building wheels for collected packages: myprotos
Running setup.py bdist_wheel for myprotos: started
Destination directory: /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel-
Running command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel- --python-tag cp27
running bdist_wheel
running build
installing to build/bdist.macosx-10.12-x86_64/wheel
running install
# THIS IS MY MAKEFILE RUNNING
Grabbing github.com/google/protobuf...
Building Python protos...
# MAKEFILE COMPLETE
running install_egg_info
running egg_info
creating myprotos.egg-info
writing myprotos.egg-info/PKG-INFO
writing top-level names to myprotos.egg-info/top_level.txt
writing dependency_links to myprotos.egg-info/dependency_links.txt
writing manifest file 'myprotos.egg-info/SOURCES.txt'
reading manifest file 'myprotos.egg-info/SOURCES.txt'
writing manifest file 'myprotos.egg-info/SOURCES.txt'
Copying myprotos.egg-info to build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1-py2.7.egg-info
running install_scripts
creating build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1.dist-info/WHEEL
Running setup.py bdist_wheel for myprotos: finished with status 'done'
Stored in directory: /Users/jds/Library/Caches/pip/wheels/92/0b/37/b5a50146994bc0b6774407139f01d648ba3a9b4853d2719c51
Removing source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build
Successfully built myprotos
Installing collected packages: myprotos
Found existing installation: myprotos 0.0.1
Uninstalling myprotos-0.0.1:
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/DESCRIPTION.rst
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/INSTALLER
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/METADATA
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/RECORD
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/WHEEL
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/metadata.json
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/top_level.txt
Successfully uninstalled myprotos-0.0.1
Successfully installed myprotos-0.0.1
Cleaning up...
Should my Makefile be running early in the process to generate the source files? Do the files need to be there before egg_info runs for example?
If I manually run the Makefile and then install the package then it works.
Update
Here is the structure of my project:
myprotos
├── Makefile
├── README.md
├── document.proto
├── myprotos # Generated by Makefile
│ ├── __init__.py # Generated by Makefile
│ └── proto_pb2.py # Generated by Makefile
└── setup.py
Here is the section of the Makefile which generates the Python source from Potocol Buffer definitions:
python: protoc deps
# the protoc and deps command above just downloads
# the `protoc` binary to a local bin directory
#echo "Building Python protos..."
#mkdir -p "${PYTHON_OUT}"
#touch "${PYTHON_OUT}"/__init__.py
#printf "__all__ = ['proto_pb2']" > "${PYTHON_OUT}"/__init__.py
#PATH="${LOCAL_BINARY_PATH}:$$PATH" protoc \
--proto_path="${BASE}" \
--proto_path="${GOPATH}/src/github.com/google/protobuf/src" \
--python_out="${PYTHON_OUT}/" \
${PROTOS}
OK, there are three things you need to change here:
Add Makefile and document.proto to a new file MANIFEST.in.
Makefile
document.proto
If you do that, the .zip file created by python setup.py sdist (which is also uploaded to PyPI) will contain those files.
You need to run your make command during python setup.py build, not during install. Since you are generating Python code, you will need to change the build_py command here:
import sys
import subprocess
from setuptools import setup
from setuptools.command.build_py import build_py
class Build(build_py):
"""Customized setuptools build command - builds protos on build."""
def run(self):
protoc_command = ["make", "python"]
if subprocess.call(protoc_command) != 0:
sys.exit(-1)
super().run()
setup(
name='buildtest',
version='1.0',
description='Python Distribution Utilities',
packages=['buildtest'],
cmdclass={
'build_py': Build,
}
)
If your Makefile generated machine code, i.e. from C or any other compiled language, you should change the build_ext command:
import sys
import subprocess
from setuptools import setup
from setuptools.command.build_ext import build_ext
class Build(build_ext):
"""Customized setuptools build command - builds protos on build."""
def run(self):
protoc_command = ["make", "python"]
if subprocess.call(protoc_command) != 0:
sys.exit(-1)
super().run()
setup(
name='buildtest',
version='1.0',
description='Python Distribution Utilities',
packages=['buildtest'],
has_ext_modules=lambda: True,
cmdclass={
'build_ext': Build,
}
)
Lastly, you need to tell setuptools to install the resulting package on install by defining an attribute packages in setup():
setup(
...
packages=['myprotos']
)
The reason for deciding to run build_py or build_ext lies in the situation when the two are run:
build_py is also run when creating source distributions, which have to be cross-platform. Compiled extensions are usually not cross-platform, so you cannot compile them in this step.
build_ext is only run when you are creating binary distributions, which are platform-specific. It is OK to compile to platform-specific machine code here.
Related
With the move to the new pyproject.toml system, I was wondering whether there was a way to install packages in editable mode while compiling extensions (which pip install -e . does not do).
So I want pip to:
run the build_ext I configured for Cython and generate my .so files
put them in the local folder
do the rest of the normal editable install
I found some mentions of build_wheel_for_editable on the pip documentation but I could not find any actual example of where this hook should be implemented and what it should look like. (to be honest, I'm not even completely sure this is what I'm looking for)
So would anyone know how to do that?
I'd also happy about any additional explanation as to why pip install . runs build_ext but the editable command does not.
Details:
I don't have a setup.py file anymore; the pyproject.toml uses setuptools and contains
[build-system]
requires = ["setuptools>=61.0", "numpy>=1.17", "cython>=0.18"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
package-dir = {"" = "."}
[tool.setuptools.packages]
find = {}
[tool.setuptools.cmdclass]
build_ext = "_custom_build.build_ext"
The custom build_ext looks like
from setuptools import Extension
from setuptools.command.build_ext import build_ext as _build_ext
from Cython.Build import cythonize
class build_ext(_build_ext):
def initialize_options(self):
super().initialize_options()
if self.distribution.ext_modules is None:
self.distribution.ext_modules = []
extensions = Extension(...)
self.distribution.ext_modules.extend(cythonize(extensions))
def build_extensions(self):
...
super().build_extensions()
It builds a .pyx into .cpp, then adds it with another cpp into a .so.
I created a module that looks like this:
$ tree .
.
├── pyproject.toml
├── setup.py
└── test
└── helloworld.pyx
1 directory, 3 files
My pyproject.toml looks like:
[build-system]
requires = ["setuptools>=61.0", "numpy>=1.17", "cython>=0.18"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
py-modules = ["test"]
[project]
name = "test"
version = "0.0.1"%
My setup.py:
from setuptools import setup
from Cython.Build import cythonize
setup(ext_modules=cythonize("test/helloworld.pyx"))
And helloworld.pyx just contains print("Hello world").
When I do pip install -e ., it builds the cython file as expected.
If you really don't want to have a setup.py at all, I think you'll need to override build_py instead of build_ext, but IMO just having the simple setup.py file isn't a big deal.
I am facing a similar issue as described in: Module found in install mode but not in develop mode using setuptools. But the solution is not applicable as I have multiple package_dir instead of one. Similarly, this problem doesn't not occur when using python3 setup.py install.
My setup.py looks like:
from setuptools import setup
setup(name='my-project',
version='0.1',
description='My project description',
author='Rishab Manocha',
package_dir={'': 'module_a/src', 'module_b': 'module_b/src'},
packages=[
'module_a1',
'module_a2',
'module_b',
'module_b.module_1b',
'module_b.module_2b',
],
)
When running python3 setup.py develop, the .egg-link only links back to the module_a/src and apparently ignores the module_b/src completely. This is the output of command python3 setup.py develop:
running develop
running egg_info
writing module_a/src/my-project.egg-info/PKG-INFO
writing dependency_links to module_a/src/my-project.egg-info/dependency_links.txt
writing top-level names to module_a/src/my-project.egg-info/top_level.txt
reading manifest file 'module_a/src/my-project.egg-info/SOURCES.txt'
writing manifest file 'module_a/src/my-project.egg-info/SOURCES.txt'
running build_ext
Creating /usr/local/lib/python3.7/site-packages/orca-airflow.egg-link (link to module_a/src)
Removing orca-airflow 0.1 from easy-install.pth file
Adding orca-airflow 0.1 to easy-install.pth file```
In my project, I have a single setup.py file that builds multiple modules using the following namespace pattern:
from setuptools import setup
setup(name="testmoduleserver",
packages=["testmodule.server","testmodule.shared"],
namespace_packages=["testmodule"])
setup(name="testmoduleclient",
packages=["testmodule.client","testmodule.shared"],
namespace_packages=["testmodule"])
I am trying to build wheel files for both packages. However, when I do:
python -m pip wheel .
It only ever builds the package for one of the definitions.
Why does only one package get built?
You cannot call setuptools.setup() more than once in your setup.py, even if you want to create several packages out of one codebase.
Instead you need to separate everything out into separate namespace packages, and have one setup.py for each (they all can reside in one Git repository!):
testmodule/
testmodule-client/
setup.py
testmodule/
client/
__init__.py
testmodule-server/
setup.py
testmodule/
server/
__init__.py
testmodule-shared/
setup.py
testmodule/
shared/
__init__.py
And each setup.py contains something along the lines
from setuptools import setup
setup(
name='testmodule-client',
packages=['testmodule.client'],
install_requires=['testmodule-shared'],
...
)
and
from setuptools import setup
setup(
name='testmodule-server',
packages=['testmodule.server'],
install_requires=['testmodule-shared'],
...
)
and
from setuptools import setup
setup(
name='testmodule-shared',
packages=['testmodule.shared'],
...
)
To build all three wheels you then run
pip wheel testmodule-client
pip wheel testmodule-server
pip wheel testmodule-shared
Hello Denizens of the Exchange of Stackness,
I have a library that I'm trying to distribute. I have created a setup.py and run
python setup.py sdist
I see that it creates a .tar.gz file under the dist/ directory, which has all my files and folders that I want in it. However, when I install it on a Windows 8 box (running Python 3.6.5rc1), I don't get any files- only a kivydnd-0.5.dist-info directory. When I install it on Linux (running Fedora 26, Python 2.7), I do see the package's files but I don't see the examples directory.
Can you tell me what I'm doing wrong?
The setup.py is here:
from setuptools import setup, find_packages
setup(
name='kivydnd',
version='0.5',
description='Kivy Drag-n-Drop for Widgets',
url='https://github.com/GreyGnome/KivyDnD',
author='GreyGnome',
author_email='myemail#example.com',
license='Apache License 2.0',
#packages=find_packages('kivydnd'),
packages=['kivydnd'],
zip_safe=False,
scripts=[
'examples/dndexample1.py',
'examples/dndexample2.py',
'examples/dndexample3.py',
'examples/dndexample_copy_draggable.py',
'examples/dndexample_drop_groups.py',
'examples/dndexample_relative_layout.py',
'examples/example_base_classes.py',
'examples/example_base_classes.pyc',
]
)
In my development directory, I perform:
python setup.py sdist
The resulting .tar.gz looks like this; this will also reflect the structure of the directory where I'm doing my development:
kivydnd-0.5/
kivydnd-0.5/setup.py
kivydnd-0.5/PKG-INFO
kivydnd-0.5/examples/
kivydnd-0.5/examples/example_base_classes.pyc
kivydnd-0.5/examples/dndexample1.py
kivydnd-0.5/examples/dndexample_copy_draggable.py
kivydnd-0.5/examples/dndexample3.py
kivydnd-0.5/examples/dndexample_relative_layout.py
kivydnd-0.5/examples/dndexample_drop_groups.py
kivydnd-0.5/examples/dndexample2.py
kivydnd-0.5/examples/example_base_classes.py
kivydnd-0.5/README.md
kivydnd-0.5/RELEASE_NOTES.md
kivydnd-0.5/LICENSE
kivydnd-0.5/kivydnd.egg-info/
kivydnd-0.5/kivydnd.egg-info/top_level.txt
kivydnd-0.5/kivydnd.egg-info/PKG-INFO
kivydnd-0.5/kivydnd.egg-info/not-zip-safe
kivydnd-0.5/kivydnd.egg-info/SOURCES.txt
kivydnd-0.5/kivydnd.egg-info/dependency_links.txt
kivydnd-0.5/setup.cfg
kivydnd-0.5/MANIFEST.in
kivydnd-0.5/kivydnd/
kivydnd-0.5/kivydnd/dnd_storage_singletons.py
kivydnd-0.5/kivydnd/debug_print.py
kivydnd-0.5/kivydnd/__init__.py
kivydnd-0.5/kivydnd/dropdestination.py
kivydnd-0.5/kivydnd/dragndropwidget.py
Here is what happens on Windows 8:
F:\>pip install kivydnd-0.5.tar.gz
Processing f:\kivydnd-0.5.tar.gz
Building wheels for collected packages: kivydnd
Running setup.py bdist_wheel for kivydnd ... done
Stored in directory: C:\Users\schwager\AppData\Local\pip\Cache\wheels\9a\11\cd
\68bfb0d34c7b73ec7e25c6f9c40c5926377747b5951ac2e6ab
Successfully built kivydnd
Installing collected packages: kivydnd
Successfully installed kivydnd-0.5
` c:\users\schwager\python\Lib\site-packages\kivydnd-0.5.dist-info\` I have:
DESCRIPTION.rst
INSTALLER
METADATA
metadata.json
RECORD
top_level.txt
WHEEL
Here is what happens on Linux:
pip install --target=/home/schwager/lib/python kivydnd-0.5.tar.gz
Processing ./kivydnd-0.5.tar.gz
Installing collected packages: kivydnd
Running setup.py install for kivydnd ... done
Successfully installed kivydnd-0.5
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ ls /home/schwager/lib/python
kivydnd kivydnd-0.5-py2.7.egg-info
$ ls -R /home/schwager/lib/python
/home/schwager/lib/python:
kivydnd kivydnd-0.5-py2.7.egg-info
/home/schwager/lib/python/kivydnd:
debug_print.py dnd_storage_singletons.py dragndropwidget.py dropdestination.py __init__.py
debug_print.pyc dnd_storage_singletons.pyc dragndropwidget.pyc dropdestination.pyc __init__.pyc
/home/schwager/lib/python/kivydnd-0.5-py2.7.egg-info:
dependency_links.txt installed-files.txt not-zip-safe PKG-INFO SOURCES.txt top_level.txt
I appears that my setup.py should look like this. The package will get installed under Python's site-packages directory, the examples under <path-to-share>/kivydnd-examples.
from setuptools import setup, find_packages
from codecs import open
from os import path
with open(path.join('.', 'README.md'), encoding='utf-8') as f:
long_description = f.read()
setup(
name='kivydnd',
version='0.5.0',
description='Kivy Drag-n-Drop for Widgets',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://github.com/GreyGnome/KivyDnD',
author='GreyGnome',
author_email='myemail#example.com',
license='Apache License 2.0',
keywords='kivy drag-n-drop',
packages=find_packages(exclude=[]),
data_files=[('share/kivydnd-examples',
[
'examples/dndexample1.py',
'examples/dndexample2.py',
'examples/dndexample3.py',
'examples/dndexample_copy_draggable.py',
'examples/dndexample_drop_groups.py',
'examples/dndexample_relative_layout.py',
'examples/example_base_classes.py',
'examples/example_base_classes.pyc',
]
)],
)
From the following setup.py file, I am trying to create a pure-python wheel from a project that should contain only python 2.7 code.
from setuptools import setup
setup(
name='foo',
version='0.0.1',
description='',
url='',
install_requires=[
'bpython',
'Django==1.8.2',
],
)
However, when I run python setup.py bdist_wheel the wheel file that is generated is platform specific foo-0.0.1-cp27-none-macosx_10_9_x86_64.whl wheel file instead of the expected foo-0.0.1-cp27-none-any.whl. When I try to install this wheel on a different platform it fails saying it is not compatible with this Python.
I there something I need to change about the setup.py file or python interpreter, perhaps, that will allow this wheel to be used on any platform?
The simplistic way is to add --universal to your commandline, as you can see from running python setup.py bdist_wheel --help:
--universal make a universal wheel (default: false)
Alternatively you can add a setup.cfg file next to your setup.py that
takes care of this:
[bdist_wheel]
universal = 1
If you don't like yet another configuration file clobbering your package,
you can just write such a file in your setup.py just before it calls setup() and then remove it after that call returns, this is what I do
in the shared setup.py for all my projects on PyPI e.g. used in ruamel.yaml.
Adding the classifiers field to my setup.py fixed this issue.
from setuptools import setup
setup(
name='foo',
version='0.0.1',
description='',
url='',
classifiers=[
'Programming Language :: Python :: 2.7',
],
install_requires=[
'bpython',
'Django==1.8.2',
],
)
This part of the filename is controlled by the bdist_wheel option called python tag:
python2 setup.py bdist_wheel --help | grep python-tag
--python-tag Python implementation compatibility tag (default: 'py2')
However the default is generally 'py2' (or 'py3' for a python3 runtime), so to get a platform-specific wheel you must have something else in your configuration that is not shown in the question.
Regardless, you can specify the tag explicitly in your setup file:
from setuptools import setup
setup(
name="foo",
version="0.0.1",
...
options={"bdist_wheel": {"python_tag": "cp27"}},
)
This configuration will create a wheel named foo-0.0.1-cp27-none-any.whl.