I try to create a zipped source python package on a linux distribution without specifying the --formats option to sdist on the command line (using an existing Jenkins pipeline which do not support this option).
In the documentation here, it states:
(assuming you haven’t specified any sdist options in the setup script or config file), sdist creates the archive of the default format for the current platform. The default format is a gzip’ed tar file (.tar.gz) on Unix, and ZIP file on Windows.
But it doesn't say how should you specify sdist options in the setup script?
From the linked documentation previous topic:
The basic syntax of the configuration file is simple:
[command]
option=value
...
where command is one of the Distutils commands (e.g. build_py, install), and option is one of the options that command supports
and later an example for build_ext --inplace
[build_ext]
inplace=1
That means that you must write into the setup.cfg file:
[sdist]
formats=zip
Beware: untested because I have no available Python2...
I'm using setup from setuptools to create a setup.py, and I was wondering if it's possible to change the output directory programmatically to change it from dist/.
I'm aware that you can do this from the command line using the --dist-dir flag, but I want to be able to do from within the setup.py file instead.
Anyone have any ideas?
You need to override code that set the default name:
from distutils.command.bdist import bdist as _bdist
from distutils.command.sdist import sdist as _sdist
dist_dir = 'my-dist-dir'
class bdist(_bdist):
def finalize_options(self):
_bdist.finalize_options(self)
self.dist_dir = dist_dir
class sdist(_sdist):
def finalize_options(self):
_sdist.finalize_options(self)
self.dist_dir = dist_dir
setup(
cmdclass={
'bdist': bdist,
'sdist': sdist,
},
…
)
Other bdist_* commands copy the value from bdist.
I have a build process that creates a Python wheel using the following command:
python setup.py bdist_wheel
The build process can be run on many platforms (Windows, Linux, py2, py3 etc.) and I'd like to keep the default output names (e.g. mapscript-7.2-cp27-cp27m-win_amd64.whl) to upload to PyPI.
Is there anyway to get the generated wheel's filename (e.g. mapscript-7.2-cp27-cp27m-win_amd64.whl) and save to a variable so I can then install the wheel later on in the script for testing?
Ideally the solution would be cross platform. My current approach is to try and clear the folder, list all files and select the first (and only) file in the list, however this seems a very hacky solution.
setuptools
If you are using a setup.py script to build the wheel distribution, you can use the bdist_wheel command to query the wheel file name. The drawback of this method is that it uses bdist_wheel's private API, so the code may break on wheel package update if the authors decide to change it.
from setuptools.dist import Distribution
def wheel_name(**kwargs):
# create a fake distribution from arguments
dist = Distribution(attrs=kwargs)
# finalize bdist_wheel command
bdist_wheel_cmd = dist.get_command_obj('bdist_wheel')
bdist_wheel_cmd.ensure_finalized()
# assemble wheel file name
distname = bdist_wheel_cmd.wheel_dist_name
tag = '-'.join(bdist_wheel_cmd.get_tag())
return f'{distname}-{tag}.whl'
The wheel_name function accepts the same arguments you pass to the setup() function. Example usage:
>>> wheel_name(name="mydist", version="1.2.3")
mydist-1.2.3-py3-none-any.whl
>>> wheel_name(name="mydist", version="1.2.3", ext_modules=[Extension("mylib", ["mysrc.pyx", "native.c"])])
mydist-1.2.3-cp36-cp36m-linux_x86_64.whl
Notice that the source files for native libs (mysrc.pyx or native.c in the above example) don't have to exist to assemble the wheel name. This is helpful in case the sources for the native lib don't exist yet (e.g. you are generating them later via SWIG, Cython or whatever).
This makes the wheel_name easily reusable in the setup.py script where you define the distribution metadata:
# setup.py
from setuptools import setup, find_packages, Extension
from setup_helpers import wheel_name
setup_kwargs = dict(
name='mydist',
version='1.2.3',
packages=find_packages(),
ext_modules=[Extension(...), ...],
...
)
file = wheel_name(**setup_kwargs)
...
setup(**setup_kwargs)
If you want to use it outside of the setup script, you have to organize the access to setup() args yourself (e.g. reading them from a setup.cfg script or whatever).
This part is loosely based on my other answer to setuptools, know in advance the wheel filename of a native library
poetry
Things can be simplified a lot (it's practically a one-liner) if you use poetry because all the relevant metadata is stored in the pyproject.toml. Again, this uses an undocumented API:
from clikit.io import NullIO
from poetry.factory import Factory
from poetry.masonry.builders.wheel import WheelBuilder
from poetry.utils.env import NullEnv
def wheel_name(rootdir='.'):
builder = WheelBuilder(Factory().create_poetry(rootdir), NullEnv(), NullIO())
return builder.wheel_filename
The rootdir argument is the directory containing your pyproject.toml script.
flit
AFAIK flit can't build wheels with native extensions, so it can give you only the purelib name. Nevertheless, it may be useful if your project uses flit for distribution building. Notice this also uses an undocumented API:
from flit_core.wheel import WheelBuilder
from io import BytesIO
from pathlib import Path
def wheel_name(rootdir='.'):
config = str(Path(rootdir, 'pyproject.toml'))
builder = WheelBuilder.from_ini_path(config, BytesIO())
return builder.wheel_filename
Implementing your own solution
I'm not sure whether it's worth it. Still, if you want to choose this path, consider using packaging.tags before you find some old deprecated stuff or even decide to query the platform yourself. You will still have to fall back to private stuff to assemble the correct wheel name, though.
My current approach to install the wheel is to point pip to the folder containing the wheel and let it search itself:
python -m pip install --no-index --find-links=build/dist mapscript
twine also can be pointed directly at a folder without needing to know the exact wheel name.
I used a modified version of hoefling's solution. My goal was to copy the build to a "latest" wheel file. The setup() function will return an object with all the info you need, so you can find out what it actually built, which seems simpler than the solution above. Assuming you have a variable version in use, the following will get the file name I just built and then copies it.
setup = setuptools.setup(
# whatever options you currently have
)
wheel_built = 'dist/{}-{}.whl'.format(
setup.command_obj['bdist_wheel'].wheel_dist_name,
'-'.join(setup.command_obj['bdist_wheel'].get_tag()))
wheel_latest = wheel_built.replace(version, 'latest')
shutil.copy(wheel_built, wheel_latest)
print('Copied {} >> {}'.format(wheel_built, wheel_latest))
I guess one possible drawback is you have to actually do the build to get the name, but since that was part of my workflow, I was ok with that. hoefling's solution has the benefit of letting you plan the name without doing the build, but it seems more complex.
For various not-very-good-but-unfortunately-necessary reasons I'm using a setup.py file to manage some binary assets.
During py setup.py build or install I would like to create a .py file in the "normal" Python package being installed by setup.py which contains some details about these binary assets (their absolute path, version information, etc).
What's the best way to create that file?
For example, I'd like it to work something like this:
$ cd my-python-package
$ py setup.py install
...
Installing version 1.23 of my_binary_assets to /some/path...
...
$ python -c "from my_python_package import binary_asset_version_info as info; print info"
{"path": "/some/path", "version": "1.23"}
(note: I'm using the cmdclass argument to setup(…) to manage the building + installation of the binary assets… I'd just like to know how to create the binary_asset_version_info.py file used in the example)
At first sight, there is a catch-22 in your requirements: The most obvious place to create this .py file would be in the build or build_py command (to get usual distutils operations like byte-compilation), but you want that file to contain the paths to the installed assets, so you’d have to create it during the install step. I see two ways to solve that:
a) Create your info.py file during build_py, and use distutils machinery to get the installation paths of the assets files
b) Create info.py during install and call distutils.util.byte_compile to byte-compile it
I find both ideas distasteful, but well :) Now, do you know how to fill in the file (i.e. get the install paths from distutils)?
When I run
python setup.py sdist
it creates an sdist in my ./dist directory. This includes a "PROJECT-egg.info" file in the zip inside my "dist" folder, which I don't use, but it doesn't hurt me, so I just ignore it.
My question is why does it also create a "PROJECT-egg.info" folder in my project root directory? Can I make it stop creating this? If not, can I just delete it immediately after creating the sdist?
I'm using the 'setup' function imported from setuptools.
WindowsXP, Python2.7, Setuptools 0.6c11, Distribute 0.6.14.
My setup config looks like:
{'author': 'Jonathan Hartley',
'author_email': 'tartley#tartley.com',
'classifiers': ['Development Status :: 1 - Planning',
'Intended Audience :: Developers',
'License :: OSI Approved :: BSD License',
'Operating System :: Microsoft :: Windows',
'Programming Language :: Python :: 2.7'],
'console': [{'script': 'demo.py'}],
'data_files': [('Microsoft.VC90.CRT',
['..\\lib\\Microsoft.VC90.CRT\\Microsoft.VC90.CRT.manifest',
'..\\lib\\Microsoft.VC90.CRT\\msvcr90.dll'])],
'description': 'Utilities for games and OpenGL graphics, built around Pyglet.\n',
'keywords': '',
'license': 'BSD',
'long_description': "blah blah blah",
'name': 'pygpen',
'options': {'py2exe': {'ascii': True,
'bundle_files': 1,
'dist_dir': 'dist/pygpen-0.1-windows',
'dll_excludes': [],
'excludes': ['_imaging_gif',
'_scproxy',
'clr',
'dummy.Process',
'email',
'email.base64mime',
'email.utils',
'email.Utils',
'ICCProfile',
'Image',
'IronPythonConsole',
'modes.editingmodes',
'startup',
'System',
'System.Windows.Forms.Clipboard',
'_hashlib',
'_imaging',
'_multiprocessing',
'_ssl',
'_socket',
'bz2',
'pyexpat',
'pyreadline',
'select',
'win32api',
'win32pipe',
'calendar',
'cookielib',
'difflib',
'doctest',
'locale',
'optparse',
'pdb',
'pickle',
'pyglet.window.xlib',
'pyglet.window.carbon',
'pyglet.window.carbon.constants',
'pyglet.window.carbon.types',
'subprocess',
'tarfile',
'threading',
'unittest',
'urllib',
'urllib2',
'win32con',
'zipfile'],
'optimize': 2}},
'packages': ['pygpen'],
'scripts': ['demo.py'],
'url': 'http://code.google.com/p/edpath/',
'version': '0.1',
'zipfile': None}
This directory is created intentionally as part of the build process for a source distribution. A little gander at the developer guide for setuptools gives you a hint as to why:
But, be sure to ignore any part of the
distutils documentation that deals
with MANIFEST or how it's generated
from MANIFEST.in; setuptools shields
you from these issues and doesn't work
the same way in any case. Unlike the
distutils, setuptools regenerates the
source distribution manifest file
every time you build a source
distribution, and it builds it inside
the project's .egg-info directory, out
of the way of your main project
directory. You therefore need not
worry about whether it is up-to-date
or not.
You may safely delete the directory after your build has completed.
Bonus edit:
I customize the clean command within my setup.py on many of my Python projects to delete *.egg-info, dist, build, and *.pyc and other files. Here's an example of how it's done in setup.py:
import os
from setuptools import setup, Command
class CleanCommand(Command):
"""Custom clean command to tidy up the project root."""
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
os.system('rm -vrf ./build ./dist ./*.pyc ./*.tgz ./*.egg-info')
# Further down when you call setup()
setup(
# ... Other setup options
cmdclass={
'clean': CleanCommand,
}
)
To illustrate, after running python setup.py build on a dummy project called "poop" (Yes, I'm very mature), this happens:
$ python setup.py build
running build
running build_py
creating build
creating build/lib
creating build/lib/poop
copying poop/__init__.py -> build/lib/poop
And now if we run python setup.py clean:
$ python setup.py clean
running clean
removed `./build/lib/poop/__init__.py'
removed directory: `./build/lib/poop'
removed directory: `./build/lib'
removed directory: `./build'
Tada!
The -egg.info folder isn't always a temporary artifact you can delete.
For example, if you use pip install -e YOURPACKAGE for an "editable" install (works via symlink like python setup.py develop so you don't have to re-install a package every time you edit it locally), the -egg.info folder is required at runtime when your package is imported in another source. If it doesn't exist, you will get a DistributionNotFound error.
Note that you can have the PROJECT.egg-info artifacts disappear completely from your sdist.
The command setup.py egg_info will use the source root as the egg base by default, resulting in the PROJECT.egg-info directory being packaged into the sdist.
You can configure the egg base by passing the option --egg-base.
This will create the PROJECT.egg-info directory somewhere else, leaving it out of your source distribution completely. You might also use a setup.cfg to set that property.
The following command to create a sdist without a PROJECT.egg-info works for me:
python setup.py egg_info --egg-base /tmp sdist
Or in a setup.cfg:
[egg_info]
egg_base = /tmp
Pythons packaging and build system is broken imho. So there are many hacks and workarounds for things that one would belive work out of the box.
However, the "cleanest" hack I found for deleting the *.egg-info is using the normal clean --all switch along with the egg_info to place the *.egg-info file in a subfolder that will be cleaned by the clean command. Here an example:
In your setup.cfg use something like this:
[egg_info]
egg_base = ./build/lib
where ./build/lib is a folder that clean --all will delete. Then when building your project with setuptools use the clean command with the --all flag, e.g.
python setup.py bdist_wheel clean --all
if you want to build a source bundle as well just make sure to build bdist_wheel before sdist so the build/lib folder exists, e.g.:
python setup.py bdist_wheel sdist clean --all
An alternative solution to jathanism's method could be the usage of the egg_info hook. I use it to clean up before each build:
from pathlib import Path
from setuptools import setup
from setuptools.command.egg_info import egg_info
here = Path(__file__).parent.resolve()
class CleanEggInfo(egg_info):
def run(self):
shutil.rmtree(here.joinpath('build'), ignore_errors=True)
shutil.rmtree(here.joinpath('dist'), ignore_errors=True)
for d in here.glob('*.egg-info'):
shutil.rmtree(d, ignore_errors=True)
egg_info.run(self)
setup(
cmdclass={
'egg_info': CleanEggInfo,
}
)