For traditional Python projects with a setup.py, there are various ways of ensuring that the version string does not have to be repeated throughout the code base. See PyPA's guide on "Single-sourcing the package version" for a list of recommendations.
Many are trying to move away from setup.py to setup.cfg (probably under the influence of PEP517 and PEP518; setup.py was mostly used declaratively anyway, and when there was logic in setup.py, it was probably for the worse.) This means that most the suggestions won't work anymore since setup.cfg cannot contain "code".
How can I single-source the package version for Python projects that use setup.cfg?
There are a couple of ways to do this (see below for the project structure used in these examples):
1.
setup.cfg
[metadata]
version = 1.2.3.dev4
src/my_top_level_package/__init__.py
import importlib.metadata
__version__ = importlib.metadata.version('MyProject')
2.
setup.cfg
[metadata]
version = file: VERSION.txt
VERSION.txt
1.2.3.dev4
src/my_top_level_package/__init__.py
import importlib.metadata
__version__ = importlib.metadata.version('MyProject')
3.
setup.cfg
[metadata]
version = attr: my_top_level_package.__version__
src/my_top_level_package/__init__.py
__version__ = '1.2.3.dev4'
And more...
There are probably other ways to do this, by playing with different combinatons.
References:
https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html
https://docs.python.org/3/library/importlib.metadata.html
Structure assumed in the previous examples is as follows...
MyProject
├── setup.cfg
├── setup.py
└── src
└── my_top_level_package
└── __init__.py
setup.py
#!/usr/bin/env python3
import setuptools
if __name__ == '__main__':
setuptools.setup(
# see 'setup.cfg'
)
setup.cfg
[metadata]
name = MyProject
# See above for the value of 'version = ...'
[options]
package_dir =
= src
packages = find:
[options.packages.find]
where = src
$ cd path/to/MyProject
$ python3 setup.py --version
1.2.3.dev4
$ python3 -m pip install .
# ...
$ python3 -c 'import my_top_level_package; print(my_top_level_package.__version__)'
1.2.3.dev4
$ python3 -V
Python 3.6.9
$ python3 -m pip list
Package Version
------------- ----------
MyProject 1.2.3.dev4
pip 20.0.2
pkg-resources 0.0.0
setuptools 45.2.0
wheel 0.34.2
zipp 3.0.0
Related
Due to the console message of setup.py install is deprecated, I am in the middle of upgrading my existing setup.py install to the recommended setup.cfg with build
My existing setup.py looks something like
from setuptools import setup
setup(
name='pybindsample',
version='0.1.0',
packages=[''],
package_data={'': ['pybindsample.so']},
has_ext_modules=lambda: True,
)
My current translation looks like:
setup.cfg
[metadata]
name = pybindsample
version = 0.1.0
[options]
packages = .
[options.package_data]
. = pybindsample.so
pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
My question is how can I translate has_ext_modules=lambda: True? has_ext_modules=lambda: True is from the solution here. Without this, after executing python3 -m build --wheel the file name of the generated wheel will become pybindsample-0.1.0-py3-none-any.whl, whereas my old python3 setup.py bdist_wheel will generate wheel with file name pybindsample-0.1.0-cp39-cp39-macosx_11_0_x86_64.whl. I have attempted
setup.cfg
[metadata]
name = pybindsample
version = 0.1.0
[options]
packages = .
has_ext_modules=lambda: True,
[options.package_data]
. = pybindsample.so
but it still generates pybindsample-0.1.0-py3-none-any.whl, I also attempted
setup.cfg
[metadata]
name = pybindsample
version = 0.1.0
[options]
packages = .
[options.package_data]
. = pybindsample.so
[bdist_wheel]
python-tag = c39
plat-name = macosx_11_0_x86_64
py-limited-api = c39
this generates pybindsample-0.1.0-cp39-none-macosx_11_0_x86_64.whl, and I couldn't figure out why the abi tag is still none.
What is the right way to configure setuptools with setup.cfg to include platform name, python tag, and ABI tag?
Consider the following minimal Python project with setuptools packaging and a "pyproject.toml" file (see setuptools Build System Support):
> tree myproject
myproject
|-- myproject
| `-- __init__.py
|-- pyproject.toml
|-- setup.cfg
`-- setup.py
"setup.py" is only a minimal dummy file to enable support for editable installs, as described here:
from setuptools import setup
if __name__ == '__main__':
setup()
When performing an editable install (pip install -e) to a virtualenv, everything works as expected:
> ls venv/lib/python3.9/site-packages | grep myproject
myproject.egg-link
> cat venv/lib/python3.9/site-packages/easy-install.pth
/myproject
> python3
>>> import myproject
Hello world!
The same is true for a non-editable system-wide install:
> ls /usr/local/lib/python3.9/dist-packages | grep myproject
myproject
myproject-1.0.dist-info
> python3
>>> import myproject
Hello world!
For an editable system-wide install, however, pip succeeds but does not result in a usable module:
> ls /usr/local/lib/python3.9/dist-packages | grep myproject
(No output)
> python3
>>> import myproject
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'myproject'
I'm aware that there were some issues in the past with "pyproject.toml" and editable installs. However, these appear to be fixed since 2019.
I have a relatively recent Debian Bullseye system with pip 20.3.4 and setuptools 52.0.0.
There also is PEP 660, which has not been implemented by setuptools yet. However, the dummy "setup.py" file (see above) should work around that limitation.
This is a Debian-specific problem.
Having a "pyproject.toml" file (i.e. a PEP 518 package), enables build isolation by default. On Debian, that results in editable installs ending up in "/usr/lib/python3.9/site-packages" instead of "/usr/local/lib/python3.9/dist-packages":
> ls /usr/lib/python3.9/site-packages
easy-install.pth myproject.egg-link
> cat /usr/lib/python3.9/site-packages/easy-install.pth
/myproject
However, "/usr/lib/python3.9/site-packages" is not part of the default Debian Python module paths.
I reported this behavior as Debian bug #1004149. Until it has been fixed, one can work around it by using pip install --no-build-isolation -e.
Problem Statement: when I install my pip package, a specific file inside the package get coped to Temp directory
Approach:
My Package directory Sturcture is following:
my-app/
├─ app/
│ ├─ __init__.py
│ ├─ __main__.py
├─ folder-with-extra-stuff/
│ ├─ __init__.py
│ ├─ file_I_want_to_cppy.tar.gz
├─ setup.py
├─ MANIFEST.in
I'm tweaking my setup.py file to do the job. Following is my setup.py
#!/usr/bin/env python
from setuptools import setup, find_packages
from setuptools.command.install import install
import os
import sys
import shutil
rootDir = os.path.abspath(os.path.dirname(__file__))
def run_custom_install():
print("--------Start running custom command -------")
temp_dir = r'c:\temp' if sys.platform == "win32" else r'/tmp'
temp_col_dir = temp_dir + os.sep + 'dump'
os.makedirs(temp_dir, exist_ok=True)
os.makedirs(temp_col_dir, exist_ok=True)
print("----------locate the zip file ---------------")
ColDirTests = os.path.abspath(os.path.join(rootDir, 'my-app','folder-with-extra-stuff'))
_src_file = os.path.join(ColDirTests , 'file_I_want_to_cppy.tar.gz ')
print(f"******{_src_file}**********")
if os.path.exists(_src_file):
print(f"-----zip file has been located at {_src_file}")
shutil.copy(_src_file, temp_col_dir)
else:
print("!!!!Couldn't locate the zip file for transfer!!!!")
class CustomInstall(install):
def run(self):
print("***********Custom run from install********")
install.run(self)
run_custom_install()
ver = "0.0.0"
setup(
name='my_pkg',
version=ver,
packages=find_packages(),
python_requires='>=3.6.0',
install_requires = getRequirements(),
include_package_data= True,
cmdclass={
'install' : CustomInstall,
}
)
MANIFEST.in
include README.md
include file_I_want_to_cppy.tar.gz
recursive-include my-app *
global-exclude *.pyc
include requirements.txt
prune test
Testing build:
> python setup.py bdist_wheel
It is working during build. I can see there is a directory formed C:\temp\dump and file_I_want_to_cppy.tar.gz inside it. But when I release the package in pip and try to install it from pip, the folder remains Empty!
Any idea what I might be doing wrong here?
After a lot of research I have figure out how to resolve this issue. Let me summarize my findings, it might be helpful for other who wants to do post_pip_install processing.
setup.py
Different options to install package: 1) pip install pkg_name, 2) python -m setup.py sdist
If you want to make them work in either ways, need to have install, egg_info and develop all 3 options repeated as shown in setup.py
If you create *.whl file by python -m setup.py bdist_wheel , post pip install processing won't be executed! Please upload .tar.gz format generated usingsdist to PyPi/Artifacts to make post pip install processing work. Again, Please note: It will not work when installing from a binary wheel
upload the pip package: twine upload dist/*.tar.gz
from setuptools import setup, find_packages
from setuptools.command.install import install
from setuptools.command.egg_info import egg_info
from setuptools.command.develop import develop
rootDir = os.path.abspath(os.path.dirname(__file__))
def run_post_processing():
print("--------Start running custom command -------")
# One can Run any Post Processing here that will be executed post pip install
class PostInstallCommand(install):
def run(self):
print("***********Custom run from install********")
install.run(self)
run_post_processing()
class PostEggCommand(egg_info):
def run(self):
print("***********Custom run from Egg********")
egg_info.run(self)
run_post_processing()
class PostDevelopCommand(develop):
def run(self):
print("***********Custom run from Develop********")
develop.run(self)
run_post_processing()
ver = "0.0.0"
setup(
name='my_pkg',
version=ver,
packages=find_packages(),
python_requires='>=3.6.0',
install_requires = getRequirements(),
include_package_data= True,
cmdclass={
'install' : PostInstallCommand,
'egg_info': PostEggCommand,
'develop': PostDevelopCommand
}
)
Few More Things from my research:
If you want to do pre-processing instead of post-processing, need to move install.run(self) at the end
while pip installing, if you want to see custom messages of pre/post instllation, use -vvv. Example: pip install -vvv my_pkg
I am packaging a python project which has the following directory structure:
toingpkg/
src/
subtoingpkg1/
subsubtoingpkg1/
...
__init__.py
__init__.py
subtoingpkg2/
...
subtoingpkg2.py
__init__.py
toingpkg.py
__init__.py
setup.cfg
pyproject.toml
My setup.cfg is as follows:
[metadata]
name = toingpkg
...
classifiers =
Programming Language :: Python :: 3
...
[options]
package_dir =
= src
packages = find_namespace:
python_requires = >=3.6
install_requires =
requests
pytz
[options.packages.find]
where=src
And my pyproject.toml is as follows:
[build-system]
requires = [
"setuptools>=42",
"wheel"
]
build-backend = "setuptools.build_meta"
When I build my package using python3 -m build as mentioned in the docs, I get a whl file in my dist folder but it does not include the files in the root of the src directory.
So when I do a pip3 install dist/toingpackage-xxx-.whl, the package gets installed (shows up in pip3 list) but I cannot do a:
>>> import toingpkg
I get:
Traceback (most recent call last): File "", line 1, in
ModuleNotFoundError: No module named 'toingpkg'
I also tried specifing all the subpackages manually but got the same result. My python environment is 3.8.5, setuptools 45.2.0.
What am I doing wrong?
I wonder if as well as .deb packages for example, it is possible in my setup.py I configure the dependencies for my package, and run:
$ sudo python setup.py install
They are installed automatically. Already researched the internet but all I found out just leaving me confused, things like "requires", "install_requires" and "requirements.txt"
Just create requirements.txt in your lib folder and add all dependencies like this:
gunicorn
docutils>=0.3
lxml==0.5a7
Then create a setup.py script and read the requirements.txt in:
import os
lib_folder = os.path.dirname(os.path.realpath(__file__))
requirement_path = lib_folder + '/requirements.txt'
install_requires = [] # Here we'll get: ["gunicorn", "docutils>=0.3", "lxml==0.5a7"]
if os.path.isfile(requirement_path):
with open(requirement_path) as f:
install_requires = f.read().splitlines()
setup(name="mypackage", install_requires=install_requires, [...])
The execution of python setup.py install will install your package and all dependencies. Like #jwodder said it is not mandatory to create a requirements.txt file, you can just set install_requires directly in the setup.py script. But writing a requirements.txt file is a best practice.
In the setup function call, you also have to set version, packages, author, etc, read the doc for a complete example: https://docs.python.org/3/distutils/setupscript.html
You package dir will look like this:
├── mypackage
│ ├── mypackage
│ │ ├── __init__.py
│ │ └── mymodule.py
│ ├── requirements.txt
│ └── setup.py
Another possible solution
try:
# for pip >= 10
from pip._internal.req import parse_requirements
except ImportError:
# for pip <= 9.0.3
from pip.req import parse_requirements
def load_requirements(fname):
reqs = parse_requirements(fname, session="test")
return [str(ir.req) for ir in reqs]
setup(name="yourpackage", install_requires=load_requirements("requirements.txt"))
You generate egg information from your setup.py, then you use the requirements.txt from these egg information:
$ python setup.py egg_info
$ pip install -r <your_package_name>.egg-info/requires.txt
In Python 3.4+, it is possible to use the Path class from pathlib, to do effectively the same thing as #hayj answer.
from pathlib import Path
import setuptools
...
def get_install_requires() -> List[str]:
"""Returns requirements.txt parsed to a list"""
fname = Path(__file__).parent / 'requirements.txt'
targets = []
if fname.exists():
with open(fname, 'r') as f:
targets = f.read().splitlines()
return targets
...
setuptools.setup(
...
install_requires=get_install_requires(),
...
)