We Keep Coding

Overwrite to-be-installed source file (without overwriting the orignal file in the source directory)

I want to encode the git hash of the lastest commit at the time of installing as the output of a function in my library.
I am using a structure
.git/
setup.py
|- mylib/
| - __init__.py
| - git.py
whereby git.py contains a dummy, unconfigured, function
def git():
return None
At the time of installing I want to overwrite git.py such that it returns the hash of the latest git commit, e.g.:
def git():
return "2d04d1a10f81d24183df7622c95398d60106dfff"
(whereby the hash is the output of git rev-parse HEAD at the time of installing).
So my question:
How can I overwrite the to-be-installed git.py, using setuptools, without overwriting the file in the source directory?
Current work-around setup.py
import subprocess
git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).strip().decode('UTF-8')
cmd = '''def git():
return "{0:s}"
'''.format(git_hash)
os.rename('mylib/git.py', 'mylib/git.py.bak')
with open('mylib/git.py', 'w') as f:
f.write(cmd)
setup(
name = 'mylib',
version = __version__,
# ...
packages = find_packages(),
)
os.rename('mylib/git.py.bak', 'mylib/git.py')
Which has the down-sight of temporarily changing git.py in the source directory. I'd rather leave that file completely untouched.

With the help of #webknjaz on setuptools' git repository (see issue) I found a solution using setuptools_scm that works:
File structure
setup.py
|- mylib/
| - __init__.py
| - _version.py
setup.py
setup(
name = 'mylib',
packages = find_packages(),
use_scm_version = {'write_to': 'mylib/_version.py'},
setup_requires = ['setuptools_scm'],
)
init.py
from ._version import *
_version.py (dummy, overwritten at install-time)
version = "None"
version_tuple = (0, 0, 0, "None", "None")

Specify per-file-ignores with pyproject.toml and flake8

I am using flake8 (with flakehell but that should not interfere) and keep its configuration in a pyproject.toml file. I want to add a per-file-ignores config but nothing works and there is no documentation on how it is supposed to be formatted in a toml file.
Flake8 docs show only the 'native' config file format:
per-file-ignores =
project/__init__.py:F401
setup.py:E121
other_project/*:W9
There is no description / example for pyproject.toml.
I tried:
per-file-ignores=["file1.py:W0621", "file2.py:W0621"]
and
per-file-ignores={"file1.py" = "W0621", "file2.py" = "W0621"}
both of which silently fail and have no effect (the warning is still raised).
What is the proper syntax for per-file-ignores setting in flake8/flakehell while using pyproject.toml?

flake8 does not have support for pyproject.toml, only .flake8, setup.cfg, and tox.ini
disclaimer: I am the flake8 maintainer

Currently, pyproject-flake8 enables you to write your flake8 settings on pyproject.toml like this.
# pyproject.toml
[tool.flake8]
exclude = ".venv"
max-complexity = 10
max-line-length = 100
extend-ignore = """
W503,
E203,
E701,
"""
per-file-ignores = """
__init__.py: F401
./src/*: E402
"""

How to get access to long_description in setup after installation using pip?

In the Python packaging sampleproject there is a long description for the module, given in setup.py using:
# Get the long description from the README file
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
long_description = f.read()
...
setup(
...
# This is an optional longer description of your project that represents
# the body of text which users will see when they visit PyPI.
#
# Often, this is the same as your README, so you can just read it in from
# that file directly (as we have already done above)
#
# This field corresponds to the "Description" metadata field:
# https://packaging.python.org/specifications/core-metadata/#description-optional
long_description=long_description, # Optional
How can the user get access to this descprtion, maybe in different format, after the module is installed with pip install sampleproject?

import email
from pkg_resources import get_distribution
pkgInfo = get_distribution(package_name).get_metadata('PKG-INFO')
print(email.message_from_string(pkgInfo)['Description'])
package_name must be your distribution name. pkgInfo is a string that contains all metadata for the package, so I use email to parse it and return Description as a header. A little dirty trick.

How do I sync values in setup.py / install_requires with Pipfile / packages

If you work on a project that uses both setup.py and Pipfile you often find the same values in: Pipfile/[packages] and setup.py/install_requires.
Does anyone know how I can tell Pipfile to use values from setup.py/install_requires for [packages]?

Within your setup.py:
Define a function to read a section:
def locked_requirements(section):
"""Look through the 'Pipfile.lock' to fetch requirements by section."""
with open('Pipfile.lock') as pip_file:
pipfile_json = json.load(pip_file)
if section not in pipfile_json:
print("{0} section missing from Pipfile.lock".format(section))
return []
return [package + detail.get('version', "")
for package, detail in pipfile_json[section].items()]
Within the setup function return the list from the default section:
setup(
# ...snip...
install_requires=locked_requirements('default'),
# ...snip...
)
IMPORTANT NOTE: include Pipfile.lock within the MANIFEST.in like:
include Pipfile.lock

Create different distribution types with setup.py

Given the following (demonstration) project layout:
MyProject/
README
LICENSE
setup.py
myproject/
... # packages
extrastuff/
... # some extra data
How (and where) do I declare different distribution types? Especially I need these two options:
A distribution containing only the source
A distribution containing the source and all data files under (extrastuff)
Ideally, how do I declare the upper two configuration whereas the second one depends on the first one?

I've implemented something like this before ... the sdist command can be extended to handle additional command line arguments and to manipulate the data files based on these. If you run python setup.py sdist --help, it'll include your custom command line arguments in the help, which is nice. Use the following recipe:
from distutils import log
from distutils.core import setup
from distutils.command.sdist import sdist
class CustomSdist(sdist):
user_options = [
('packaging=', None, "Some option to indicate what should be packaged")
] + sdist.user_options
def __init__(self, *args, **kwargs):
sdist.__init__(self, *args, **kwargs)
self.packaging = "default value for this option"
def get_file_list(self):
log.info("Chosen packaging option: {self.packaging}".format(self=self))
# Change the data_files list here based on the packaging option
self.distribution.data_files = list(
('folder', ['file1', 'file2'])
)
sdist.get_file_list(self)
if __name__ == "__main__":
setup(
name = "name",
version = "version",
author = "author",
author_email = "author_email",
url = "url",
py_modules = [
# ...
],
packages = [
# ...
],
# data_files = default data files for commands other than sdist if you wish
cmdclass={
'sdist': CustomSdist
}
)

You could extend setup.py to additionally include some custom command line parsing. You could then catch a custom argument and strip it out so that it won't affect setuptools.
You can access the command line argument in sys.argv. As for modifying the call to setuptools.setup(), I recommend creating a dictionary of arguments to pass, modifying the dictionary based on the command line arguments, and then calling setup() using the **dict notation, like so:
from setuptools import setup
import sys
basic = {'name': 'my program'}
extra = {'bonus': 'content'}
if '--extras' in sys.argv:
basic.update(extra)
sys.argv.remove('--extras')
setup(**basic)
For more thorough command line parsing you could also use the getopt module, or the newer argparse module if you're only targeting Python 2.7 and higher.
EDIT: I also found a section in the distutils documentation titled Creating a new Distutils command. That may also be a helpful resource.