I have a program that uses dateutil from the package index. I would like to have setup.py check for for its presence and try to get it using easy_install if it is not there.
The documentation for distutils seems to indicate that this can be done using the requires keyword in setup(), but when I try, it installs on a system without dateutil without giving a warning or installing the required package.
The only thing I could find on google was this blog post about the same issue which did not have any answer either.
Am I using distutils wrong? Do I need to subclass distutils.command.install and do the checking/installing myself?
Automatic downloading of dependencies is a feature introduced by setuptools which is a third-party add-on to distutils, in particular, the install_requires argument it adds. See the setuptools documentation for more information.
Another option is to use requirements.txt file with pip rather than using easy_install as a package installer. pip has now become the recommended installer; see the Python Packaging User Guide for more information.
Update [2015-01]: The previous version of this answer referred to the distribute fork of setuptools. The distribute fork has since been merged back into a newer active setuptools project. distribute is now dead and should no longer be used. setuptools and pip are now very actively maintained and support Python 3.
The argument install_requires in setup function from distutils work for me well, only if I create sdist distributive, like: python setup.py sdist
Related
Any documentation I've found about this topic mentions that the "only" requirement to build a deb package is to have a correct setup.py (and requirements.txt). For instance in dh-virtualenv tutorial, stdeb documentation and the Debian's library style guide for python.
But nowadays new (amazing) tools like poetry allow to develop (and upload to PyPI) python projects without any setup.py (this file and several others including requirements.txt are all replaced by pyproject.toml). I believe flit allows this too.
I have developed a python project managed by poetry and would like to package it for Ubuntu/Debian. I guess, as a workaround I can still write a setup.py file that would take its values from pyproject.toml and a requirements.txt file (written by hand using values from poetry.lock).
But, is there a way to do this without any setup.py file?
setuptools, and the setup.py file that it requires, has been the de-facto packaging standard in python for the longest time. The new package managers you mention were enabled by the introduction of PEP 517 and PEP 518 (or read this for a high-level description on the topic), which provide a standardized way of specifying the build backend without the need of a setup.py (and the ensuing hen-egg problem where you already need setuptools to correctly parse it).
Anyway, it's all still very fresh, and the linux packaging community hasn't caught up yet. I found no recent discussion regarding debian packages, but the rpm side sums it up neatly over here.
So, the short answer is to just wait a while, and google debian packaging pep517 support every now and then.
Until then, you can use dephell to generate the setup.py, and poetry to generate the requirements.txt for you as a workaround to keep using the existing tools:
dephell deps convert --from=poetry --to=setuppy
poetry export -f requirements.txt -o requirements.txt
And, during the build, tell your pyproject.tom that you plan to use setuptools for the build instead of poetry:
[build-system]
requires = ["setuptools >= 40.6.0", "wheel"]
build-backend = "setuptools.build_meta"
Background
I was about to try Python package downloaded from GitHub, and realized that it did not have a setup.py, so I could not install it with
pip install -e <folder>
Instead, the package had a pyproject.toml file which seems to have very similar entries as the setup.py usually has.
What I found
Googling lead me into PEP-518 and it gives some critique to setup.py in Rationale section. However, it does not clearly tell that usage of setup.py should be avoided, or that pyproject.toml would as such completely replace setup.py.
Questions
Is the pyproject.toml something that is used to replace setup.py? Or should a package come with both, a pyproject.toml and a setup.py?
How would one install a project with pyproject.toml in an editable state?
Yes, pyproject.toml is the specified file format of PEP 518 which contains the build system requirements of Python projects.
This solves the build-tool dependency chicken and egg problem, i.e. pip can read pyproject.toml and what version of setuptools or wheel one may need.
If you need a setup.py for an editable install, you could use a shim in setup.py:
#!/usr/bin/env python
import setuptools
if __name__ == "__main__":
setuptools.setup()
pyproject.toml is the new unified Python project settings file that replaces setup.py.
Editable installs still need a setup.py: import setuptools; setuptools.setup()
To use pyproject.toml, run python -m pip install .
Then, if the project is using poetry instead of pip, you can install dependencies (into %USERPROFILE%\AppData\Local\pypoetry\Cache\virtualenvs) like this:
poetry install
And then run dependencies like pytest:
poetry run pytest tests/
And pre-commit (uses .pre-commit-config.yaml):
poetry run pre-commit install
poetry run pre-commit run --all-files
What is it for?
Currently there are multiple packaging tools being popular in Python community and while setuptools still seems to be prevalent it's not a de facto standard anymore. This situation creates a number of hassles for both end users and developers:
For setuptools-based packages installation from source / build of a distribution can fail if one doesn't have setuptools installed;
pip doesn't support the installation of packages based on other packaging tools from source, so these tools had to generate a setup.py file to produce a compatible package. To build a distribution package one has to install the packaging tool first and then use tool-specific commands;
If package author decides to change the packaging tool, workflows must be changed as well to use different tool-specific commands.
pyproject.toml is a new configuration file introduced by PEP 517 and PEP 518 to solve these problems:
... think of the (rough) steps required to produce a built artifact for a project:
The source checkout of the project.
Installation of the build system.
Execute the build system.
This PEP [518] covers step #2. PEP 517 covers step #3 ...
Any tool can also extend this file with its own section (table) to accept tool-specific options, but it's up to them and not required.
PEP 621 suggests using pyproject.toml to specify package core metadata in static, tool-agnostic way. Which backends currently support this is shown in the following table:
enscons
flit_core
hatchling
pdm-pep517
poetry-core
setuptools
0.26.0+
3.2+
0.3+
0.3.0+
Issue #3332
61.0.0+
Does it replace setup.py?
For setuptools-based packages pyproject.toml is not strictly meant to replace setup.py, but rather to ensure its correct execution if it's still needed. For other packaging tools – yes, it is:
Where the build-backend key exists, this takes precedence and the source tree follows the format and conventions of the specified backend (as such no setup.py is needed unless the backend requires it). Projects may still wish to include a setup.py for compatibility with tools that do not use this spec.
How to install a package in editable mode?
Originally "editable install" was a setuptools-specific feature and as such it was not supported by PEP 517. Later on PEP 660 extended this concept to packages using pyproject.toml.
There are two possible conditions for installing a package in editable mode using pip:
Modern:
Both the frontend (pip) and a backend must support PEP 660.
pip supports it since version 21.3;
Legacy:
Packaging tool must provide a setup.py file which supports the develop command.
Since version 21.1 pip can also install packages using only setup.cfg file in editable mode.
The following table describes the support of editable installs by various backends:
enscons
flit_core
hatchling
pdm-pep517
poetry-core
setuptools
0.28.0+
3.4+
0.3+
0.8.0+
1.0.8+
64.0.0+
Answering this part only, as the rest has nicely been explained by others:
How would one install a project with pyproject.toml in an editable state?
Solution
Since the release of poetry-core v1.0.8 in Feb 2022 you can do this:
a) you need this entry in your pyproject.toml:
[build-system]
requires = ["poetry-core>=1.0.8"]
build-backend = "poetry.core.masonry.api"
b) run:
pip install -e .
Sources
https://github.com/python-poetry/poetry/issues/34#issuecomment-1054626460
pyproject.toml can declare the files in your python package and all the metadata for it that will show in PyPi.
A tool like flit can process the pyproject.toml file into a package that can be uploaded to PyPi or installed with pip.
Other tools use pyproject.toml for other purposes. For example, pytest stores information about where to find and how to run tests, and instructions to pytest about modifying pythonpath (sys.path) before running the tests. Many IDEs can use this to help developers conveniently run tests.
The importlib_resources backport for Python < 3.7 of the importlib.resources standard library module has the following section in the setup.cfg file:
[options]
python_requires = >=2.7,!=3.0,!=3.1,!=3.2,!=3.3
setup_requires =
setuptools
wheel
install_requires =
pathlib2; python_version < '3'
typing; python_version < '3.5'
packages = find:
Why does setup_requires include setuptools? This does not seem to make sense since:
the first line of the setup.py file imports setuptools, so by the time the setup function is called and reads the setup.cfg file that instructs to install setuptools it is already too late to install setuptools:
from setuptools import setup
setup()
setuptools is already installed on any fresh Python installation (well, only tested on Windows 10 and MacOS 10.15 with Python 3.8.0):
$ python -V
Python 3.8.0
$ pip list
Package Version
---------- -------
pip 19.2.3
setuptools 41.2.0
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
No, setuptools should not be included in setup_requires, according to PEP 518 (bold emphasis mine):
Setuptools tried to solve this with a setup_requires argument to its
setup() function [3]. This solution has a number of issues, such as:
No tooling (besides setuptools itself) can access this information without executing the setup.py, but setup.py can't be executed without having these items installed.
While setuptools itself will install anything listed in this, they won't be installed until during the execution of the setup() function, which means that the only way to actually use anything added here is through increasingly complex machinations that delay the import and usage of these modules until later on in the execution of the setup() function.
This cannot include setuptools itself nor can it include a replacement to setuptools, which means that projects such as numpy.distutils are largely incapable of utilizing it and projects cannot take advantage of newer setuptools features until their users naturally upgrade the version of setuptools to a newer one.
The items listed in setup_requires get implicitly installed whenever you execute the setup.py but one of the common ways that the setup.py is executed is via another tool, such as pip, who is already managing dependencies. This means that a command like pip install spam might end up having both pip and setuptools downloading and installing packages and end users needing to configure both tools (and for setuptools without being in control of the invocation) to change settings like which repository it installs from. It also means that users need to be aware of the discovery rules for both tools, as one may support different package formats or determine the latest version differently.
The accepted answer is mostly correct, but where PEP 518 says.
[The setup_requires mechanism] cannot include setuptools itself...
It's technically incorrect, and as importlib_resources demonstrates, it can actually include setuptools. The problem is that including setuptools in setup_requires serves mostly as documentation. It declares that setuptools is a build requirement (required to run setup.py), but it won't be capable of satisfying that requirement if it's not already satisfied.
But, the presence of setuptools in setup_requires is technically correct and does serve the purpose of declaring the requirement and asking setuptools to verify that the requirement is in fact installed (alongside other setup-time requirements).
It is, however, just a legacy artifact and doesn't provide that much value, and as can be seen in the question and answers, it does lead to confusion. The recommended, proper, approach is to use PEP 517 and 518 declarations and builders, but that part of the ecosystem hasn't matured yet, so setuptools vestiges will remain. Try not to let them bother you.
Why does setup_requires includes setuptools? This does not seem to make sense
Does not make sense at all. On the other hand it doesn't hamper anything so why not?
Newbie here. I've created my first Python package and I managed to register it on Pypi, as well as upload the tar.gz file. Now whenever I want to run:
pip install myPackage
I get this error in console:
Could not find a version that satisfies the requirement myPackage (from versions: 1.0dev)
No distributions matching the version for flashCardStudy
Storing debug log for failure in /Users/xxx/Library/Logs/pip.log
I believe this is because my version is development version I guess? So yeah, I can install it by adding --pre argument but what I'd really like is to turn it into a normal version so to speak.
I've tried figuring out how to do it and looking at some docs but I can't still figure it out. In my setup.py my version is set to '1.0' so I don't see where to problem is. If anyone wants to have a look at the file, here it is.
So I found the problem. I used utility called Paster which generates package structure, including setup.py and setup.cfg files among others. My utility hasn't been updated in a while and meanwhile submission rules to PyPi have changed. It now requires certain setup.py structure and unless it passes via pip, it's labeled as development version - which pip does not install without --pre argument.
So I just went to PyPi pages and looked at setup.py tutorial, did it their way and now it works.
Does anybody encounter this warning when executing python setup.py install of a PyPI package?
install_requires defines what the package requires. A lot of PyPI packages have this option. How can it be an "unknown distribution option"?
python setup.py uses distutils which doesn't support install_requires. setuptools does, also distribute (its successor), and pip (which uses either) do. But you actually have to use them. I.e. call setuptools through the easy_install command or pip install.
Another way is to import setup from setuptools in your setup.py, but this not standard and makes everybody wanting to use your package have to have setuptools installed.
This was the first result on my google search, but had no answer.
I found that upgrading setuptools resolved the issue for me (and pip for good measure)
pip install --upgrade pip
pip install --upgrade setuptools
Hope this helps the next person to find this link!
I'm on a Mac with python 2.7.11. I have been toying with creating extremely simple and straightforward projects, where my only requirement is that I can run python setup.py install, and have setup.py use the setup command, ideally from distutils. There are literally no other imports or code aside from the kwargs to setup() other than what I note here.
I get the error when the imports for my setup.py file are:
from distutils.core import setup
When I use this, I get warnings such as
/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'
warnings.warn(msg)
If I change the imports (and nothing else) to the following:
from distutils.core import setup
import setuptools # noqa
The warnings go away.
Note that I am not using setuptools, just importing it changes the behavior such that it no longer emits the warnings. For me, this is the cause of a truly baffling difference where some projects I'm using give those warnings, and some others do not.
Clearly, some form of monkey-patching is going on, and it is affected by whether or not that import is done. This probably isn't the situation for everyone researching this problem, but for the narrow environment in which I'm working, this is the answer I was looking for.
This is consistent with the other (community) comment, which says that distutils should monkeypatch setuptools, and that they had the problem when installing Ansible. Ansible appears to have tried to allow installs without having setuptools in the past, and then went back on that.
https://github.com/ansible/ansible/blob/devel/setup.py
A lot of stuff is up in the air... but if you're looking for a simple answer for a simple project, you should probably just import setuptools.
ATTENTION! ATTENTION! Imperfect answer ahead. To get the "latest memo" on the state of packaging in the Python universe, read this fairly detailed essay.
I have just ran into this problem when trying to build/install ansible. The problem seems to be that distutils really doesn't support install_requires. Setuptools should monkey-patch distutils on-the-fly, but it doesn't, probably because the last release of setuptools is 0.6c11 from 2009, whereas distutils is a core Python project.
So even after manually installing the setuptools-0.6c11-py2.7.egg running setup.py only picks up distutils dist.py, and not the one from site-packages/setuptools/.
Also the setuptools documentation hints to using ez_setup and not distutils.
However, setuptools is itself provided by distribute nowadays, and that flavor of setup() supports install_requires.
This is a warning from distutils, and is a sign that you do not have setuptools installed.
Installing it from http://pypi.python.org/pypi/setuptools will remove the warning.
In conclusion:
distutils doesn't support install_requires or entry_points, setuptools does.
change from distutils.core import setup in setup.py to from setuptools import setup or refactor your setup.py to use only distutils features.
I came here because I hadn't realized entry_points was only a setuptools feature.
If you are here wanting to convert setuptools to distutils like me:
remove install_requires from setup.py and just use requirements.txt with pip
change entry_points to scripts (doc) and refactor any modules relying on entry_points to be full scripts with shebangs and an entry point.
sudo apt-get install python-dev # for python2.x installs
sudo apt-get install python3-dev # for python3.x installs
It will install any missing headers. It solved my issue
As far as I can tell, this is a bug in setuptools where it isn't removing the setuptools specific options before calling up to the base class in the standard library: https://bitbucket.org/pypa/setuptools/issue/29/avoid-userwarnings-emitted-when-calling
If you have an unconditional import setuptools in your setup.py (as you should if using the setuptools specific options), then the fact the script isn't failing with ImportError indicates that setuptools is properly installed.
You can silence the warning as follows:
python -W ignore::UserWarning:distutils.dist setup.py <any-other-args>
Only do this if you use the unconditional import that will fail completely if setuptools isn't installed :)
(I'm seeing this same behaviour in a checkout from the post-merger setuptools repo, which is why I'm confident it's a setuptools bug rather than a system config problem. I expect pre-merge distribute would have the same problem)
I've now seen this in legacy tools using Python2.7, where a build (like a Dockerfile) installs an unpinned dependancy, for example pytest. PyTest has dropped Python 2.7 support, so you may need to specify version < the new package release.
Or bite the bullet and convert that app to Python 3 if that is viable.
It works fine if you follow the official documentation:
import setuptools
setuptools.setup(...)
Source: https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py