I am working on a python2 package in which the setup.py contains some custom install commands. These commands actually build some Rust code and output some .dylib files that are moved into the python package.
An important point is that the Rust code is outside the python package.
setuptools is supposed to detect automatically if the python package is pure python or platform specific (if it contains some C extensions for instance).
In my case, when I run python setup.py bdist_wheel, the generated wheel is tagged as a pure python wheel: <package_name>-<version>-py2-none-any.whl.
This is problematic because I need to run this code on different platforms, and thus I need to generated one wheel per platform.
Is there a way, when building a wheel, to force the build to be platform specific ?
Here's the code that I usually look at from uwsgi
The basic approach is:
setup.py
# ...
try:
from wheel.bdist_wheel import bdist_wheel as _bdist_wheel
class bdist_wheel(_bdist_wheel):
def finalize_options(self):
_bdist_wheel.finalize_options(self)
self.root_is_pure = False
except ImportError:
bdist_wheel = None
setup(
# ...
cmdclass={'bdist_wheel': bdist_wheel},
)
The root_is_pure bit tells the wheel machinery to build a non-purelib (pyX-none-any) wheel. You can also get fancier by saying there are binary platform-specific components but no cpython abi specific components.
The modules setuptools, distutils and wheel decide whether a python distribution is pure by checking if it has ext_modules.
If you build an external module on your own, you can still list it in ext_modules so that the building tools know it exists. The trick is to provide an empty list of sources so that setuptools and distutils will not try to build it. For example,
setup(
...,
ext_modules=[
setuptools.Extension(
name='your.external.module',
sources=[]
)
]
)
This solution worked better for me than patching the bdist_wheel command. The reason is that bdist_wheel calls the install command internally and that command checks again for the existence of ext_modules to decide between purelib or platlib install. If you don't list the external module, you end up with the lib installed in a purelib subfolder inside the wheel. That causes problems when using auditwheel repair, which complains about the extensions being installed in a purelib folder.
You can also specify/spoof a specific platform name when building wheels by specifying a --plat-name:
python setup.py bdist_wheel --plat-name=manylinux1_x86_64
Neither the root_is_pure trick nor the empty ext_modules trick worked for me, but after MUCH searching myself, I finally found a working solution in 'pip setup.py bdist_wheel' no longer builds forced non-pure wheels
Basically, you override the 'has_ext_modules' function in the Distribution class, and set distclass to point to the overriding class. At that point, setup.py will believe you have a binary distribution, and will create a wheel with the specific version of python, the ABI, and the current architecture. As suggested by https://stackoverflow.com/users/5316090/py-j:
from setuptools import setup
from setuptools.dist import Distribution
DISTNAME = "packagename"
DESCRIPTION = ""
MAINTAINER = ""
MAINTAINER_EMAIL = ""
URL = ""
LICENSE = ""
DOWNLOAD_URL = ""
VERSION = '1.2'
PYTHON_VERSION = (2, 7)
# Tested with wheel v0.29.0
class BinaryDistribution(Distribution):
"""Distribution which always forces a binary package with platform name"""
def has_ext_modules(foo):
return True
setup(name=DISTNAME,
description=DESCRIPTION,
maintainer=MAINTAINER,
maintainer_email=MAINTAINER_EMAIL,
url=URL,
license=LICENSE,
download_url=DOWNLOAD_URL,
version=VERSION,
packages=["packagename"],
# Include pre-compiled extension
package_data={"packagename": ["_precompiled_extension.pyd"]},
distclass=BinaryDistribution)
I find Anthony Sottile's answer great, but didn't work for me.
My case is that I have to enforce the wheel to be created for x86_64, but any python3, so making root impure actually caused my wheel to be py36-cp36 :(
A better way, IMO, in general is just to use sys.argv:
from setuptools import setup
import sys
sys.argv.extend(['--plat-name', 'x86_64'])
setup(name='example-wheel')
Related
I have a build process that creates a Python wheel using the following command:
python setup.py bdist_wheel
The build process can be run on many platforms (Windows, Linux, py2, py3 etc.) and I'd like to keep the default output names (e.g. mapscript-7.2-cp27-cp27m-win_amd64.whl) to upload to PyPI.
Is there anyway to get the generated wheel's filename (e.g. mapscript-7.2-cp27-cp27m-win_amd64.whl) and save to a variable so I can then install the wheel later on in the script for testing?
Ideally the solution would be cross platform. My current approach is to try and clear the folder, list all files and select the first (and only) file in the list, however this seems a very hacky solution.
setuptools
If you are using a setup.py script to build the wheel distribution, you can use the bdist_wheel command to query the wheel file name. The drawback of this method is that it uses bdist_wheel's private API, so the code may break on wheel package update if the authors decide to change it.
from setuptools.dist import Distribution
def wheel_name(**kwargs):
# create a fake distribution from arguments
dist = Distribution(attrs=kwargs)
# finalize bdist_wheel command
bdist_wheel_cmd = dist.get_command_obj('bdist_wheel')
bdist_wheel_cmd.ensure_finalized()
# assemble wheel file name
distname = bdist_wheel_cmd.wheel_dist_name
tag = '-'.join(bdist_wheel_cmd.get_tag())
return f'{distname}-{tag}.whl'
The wheel_name function accepts the same arguments you pass to the setup() function. Example usage:
>>> wheel_name(name="mydist", version="1.2.3")
mydist-1.2.3-py3-none-any.whl
>>> wheel_name(name="mydist", version="1.2.3", ext_modules=[Extension("mylib", ["mysrc.pyx", "native.c"])])
mydist-1.2.3-cp36-cp36m-linux_x86_64.whl
Notice that the source files for native libs (mysrc.pyx or native.c in the above example) don't have to exist to assemble the wheel name. This is helpful in case the sources for the native lib don't exist yet (e.g. you are generating them later via SWIG, Cython or whatever).
This makes the wheel_name easily reusable in the setup.py script where you define the distribution metadata:
# setup.py
from setuptools import setup, find_packages, Extension
from setup_helpers import wheel_name
setup_kwargs = dict(
name='mydist',
version='1.2.3',
packages=find_packages(),
ext_modules=[Extension(...), ...],
...
)
file = wheel_name(**setup_kwargs)
...
setup(**setup_kwargs)
If you want to use it outside of the setup script, you have to organize the access to setup() args yourself (e.g. reading them from a setup.cfg script or whatever).
This part is loosely based on my other answer to setuptools, know in advance the wheel filename of a native library
poetry
Things can be simplified a lot (it's practically a one-liner) if you use poetry because all the relevant metadata is stored in the pyproject.toml. Again, this uses an undocumented API:
from clikit.io import NullIO
from poetry.factory import Factory
from poetry.masonry.builders.wheel import WheelBuilder
from poetry.utils.env import NullEnv
def wheel_name(rootdir='.'):
builder = WheelBuilder(Factory().create_poetry(rootdir), NullEnv(), NullIO())
return builder.wheel_filename
The rootdir argument is the directory containing your pyproject.toml script.
flit
AFAIK flit can't build wheels with native extensions, so it can give you only the purelib name. Nevertheless, it may be useful if your project uses flit for distribution building. Notice this also uses an undocumented API:
from flit_core.wheel import WheelBuilder
from io import BytesIO
from pathlib import Path
def wheel_name(rootdir='.'):
config = str(Path(rootdir, 'pyproject.toml'))
builder = WheelBuilder.from_ini_path(config, BytesIO())
return builder.wheel_filename
Implementing your own solution
I'm not sure whether it's worth it. Still, if you want to choose this path, consider using packaging.tags before you find some old deprecated stuff or even decide to query the platform yourself. You will still have to fall back to private stuff to assemble the correct wheel name, though.
My current approach to install the wheel is to point pip to the folder containing the wheel and let it search itself:
python -m pip install --no-index --find-links=build/dist mapscript
twine also can be pointed directly at a folder without needing to know the exact wheel name.
I used a modified version of hoefling's solution. My goal was to copy the build to a "latest" wheel file. The setup() function will return an object with all the info you need, so you can find out what it actually built, which seems simpler than the solution above. Assuming you have a variable version in use, the following will get the file name I just built and then copies it.
setup = setuptools.setup(
# whatever options you currently have
)
wheel_built = 'dist/{}-{}.whl'.format(
setup.command_obj['bdist_wheel'].wheel_dist_name,
'-'.join(setup.command_obj['bdist_wheel'].get_tag()))
wheel_latest = wheel_built.replace(version, 'latest')
shutil.copy(wheel_built, wheel_latest)
print('Copied {} >> {}'.format(wheel_built, wheel_latest))
I guess one possible drawback is you have to actually do the build to get the name, but since that was part of my workflow, I was ok with that. hoefling's solution has the benefit of letting you plan the name without doing the build, but it seems more complex.
I have a distutils-style Python package which requires a specific, and quite large, dependency for its build step. Currently, this dependency is specified under the setup_requires argument to distutils.setup. Unfortunately, this means the dependency will be built for any execution of setup.py, including when running setup.py clean. This creates the rather ironic situation of the clean step sometimes causing large amount of code to be compiled.
As I said, this setup dependency is only required for the build step. Is there a way to encode this logic in setup.py so that all commands that do not invoke the build command are run without it?
You can always order the Distribution to fetch some packages explicitly, same way as they will be if you define them in setup_requires. Example with numpy dependency required for build command only:
from distutils.command.build import build as build_orig
from setuptools import setup, find_packages, Command, dist
class build(build_orig):
def run(self):
self.distribution.fetch_build_eggs(['numpy'])
# numpy becomes available after this line. Test it:
import numpy
print(numpy.__version__)
super().run()
setup(
name='spam',
packages=find_packages(),
cmdclass={'build': build,}
...
)
The dependencies are passed the same as they would be defined in setup_requires arg, so version specs are also ok:
self.distribution.fetch_build_eggs(['numpy>=1.13'])
Although I must note that fetching dependencies via setup_requires is usually much slower than installing them via pip (especially when you have some heavy dependencies that must be built from source first), so if you can be sure you will have pip available (or use python3.4 and newer), the approach suggested by phd in his answer will save you time. Fetching eggs via distribution may, however, come handy when building for old python versions or obscure python installations like the system python on MacOS.
if sys.argv[0] == 'build':
kw = {'setup_requires': [req1, req2, …]}
else:
kw = {}
setup(
…,
**kw
)
Another approach to try is override build command with a custom cmdclass:
from setuptools.command.build import build as _build
class build(_build):
def run(self):
subprocess.call(["pip", "install", req1, req2…])
_build.run(self)
setup(
…,
cmdclass={'build': build},
)
and avoid setup_requires at all.
I do this
C:\WINDOWS\system32>python c:\\python34\kissdownloader\setup.py bdist_wheel
and I get this after some other stuff
creating build\bdist.win-amd64\wheel\kissdownloader-1.dist-info\WHEEL
I goto my python root and there's nothing there , no build folder or anything . Is the thing somewhere else ? Oh and I don't have a manifest.in file if that makes any difference .
My setup.py file looks like this :
from setuptools import setup
setup (
name='kissdownloader',
version='1',
description = 'A kisscartoon/kissanime downloader' ,
package_dir={'kissdownloader':'C:\\Python34'},
author='Vriska',
author_email='xyz#gmail.com',
install_requires = ['bs4','cfscrape','requests'],
package_data={'data' : ['C:\Program Files\PhantomJS\phantomjs.exe']},
)
The output is generated in your current working directory. In your case, that's C:\WINDOWS\system32, because that's where you started Python. You'll find a build and dist directory; the latter contains the completed wheel.
If you want the directories to be created next to setup.py, change your directory to c:\python34\kissdownloader\ first. Many a Python project expects you to run setup.py there anyway.
As a side note: I wouldn't bundle the PhantomJS binary in with your project. Instead, require users to install it separately; they may already have it installed anyway, and you may run into legal issues by re-distributing it in your own project without at least a compatible license.
I'm trying to tryout pypi to publish some libraries. So I started with a simple project.
I have the following setup.py:
import os
from distutils.core import setup
setup(
name='event_amd',
packages = ["event_amd"],
description='Port for EventEmitter from nodejs',
version='1.0.7',
author="Borrey Kim",
author_email="borrey#gmail.com",
url="https://bitbucket.org/borreykim/event_amd",
download_url="https://bitbucket.org/borreykim/event_amd/downloads/event_amd-1.0.6.tar.gz",
keywords=['events'],
long_description = """\
This is an initial step to port over EventEmitter of nodejs. This is done with the goal of having libraries that are cross platform so that cross communication is easier, and collected together.
"""
)
I've registered it but: sudo pip install event_amd gives me an error:
DistributionNotFound: No distributions at all found for event-amd
(I'm not sure how event_amd turns to event-amd?)
Also there is no links under (which other projects seem to have ):
https://pypi.python.org/simple/event_amd/
I was wondering if I am doing something wrong in the setup.py or what may be causing this.
Thanks in advance.
You need to upload a source archive after registering the release: python setup.py register sdist upload
I've got a Python package with its setup.py having dependencies declared via the usual way, in install_requires=[...]. One of the packages there, scikits.timeseries, has a setup.py expecting numpy to already be installed, thus, I'd like some way to have numpy installed first. For this case and in general, can the order of dependency installation be controlled? How?
Currently the order in which setup.py pulls down dependencies (as listed in the arg install_requires) seems practically random. Also, in the setup.py setup(...) I tried using the arg:
extras_require={'scikits.timeseries': ['numpy']}
...without success, the order of installing dependencies was unaffected.
I also tried setting up a pip requirements file, but there too, pip's order of installing dependencies didn't match the line-order of the requirements file, so no luck.
Another possibility would be to have a system call near the top of setup.py, to install numpy before the setup(...) call, but I hope there's a better way. Thanks in advance for any help.
If scikits.timeseries needs numpy, then it should declare it as a dependency. If it did, then pip would handle things for you (I'm pretty sure setuptools would, too, but I haven't used it in a long while). If you control scikits.timeseries, then you should fix it's dependency declarations.
Use setup_requires parameter, for instance to install numpy prior scipy put it into setup_requires and add __builtins__.__NUMPY_SETUP__ = False hook to get numpy installed correctly:
setup(
name='test',
version='0.1',
setup_requires=['numpy'],
install_requires=['scipy']
)
def run(self):
__builtins__.__NUMPY_SETUP__ = False
import numpy
Here's a solution which actually works. It's not an overly "pleasant" method to have to resort to, but "desperate times...".
Basically, you have to:
Override the setuptools "install command" class (plus the closely related analogs)
Execute pip from the script via command line statements for which you can enforce the order
The drawbacks to this are:
Pip must be installed. You can't just execute setup.py in an environment without that.
The console output of the initial "prerequisite" installs don't appear for some weird reason. (Perhaps I'll post an update here down the line fixing that...)
The code:
from setuptools import setup
# Override standard setuptools commands.
# Enforce the order of dependency installation.
#-------------------------------------------------
PREREQS = [ "ORDERED-INSTALL-PACKAGE" ]
from setuptools.command.install import install
from setuptools.command.develop import develop
from setuptools.command.egg_info import egg_info
def requires( packages ):
from os import system
from sys import executable as PYTHON_PATH
from pkg_resources import require
require( "pip" )
CMD_TMPLT = '"' + PYTHON_PATH + '" -m pip install %s'
for pkg in packages: system( CMD_TMPLT % (pkg,) )
class OrderedInstall( install ):
def run( self ):
requires( PREREQS )
install.run( self )
class OrderedDevelop( develop ):
def run( self ):
requires( PREREQS )
develop.run( self )
class OrderedEggInfo( egg_info ):
def run( self ):
requires( PREREQS )
egg_info.run( self )
CMD_CLASSES = {
"install" : OrderedInstall
, "develop" : OrderedDevelop
, "egg_info": OrderedEggInfo
}
#-------------------------------------------------
setup (
...
install_requires = [ "UNORDERED-INSTALL-PACKAGE" ],
cmdclass = CMD_CLASSES
)
You can add numpy to setup_requires section:
setup_requires=['numpy'],