The importlib_resources backport for Python < 3.7 of the importlib.resources standard library module has the following section in the setup.cfg file:
[options]
python_requires = >=2.7,!=3.0,!=3.1,!=3.2,!=3.3
setup_requires =
setuptools
wheel
install_requires =
pathlib2; python_version < '3'
typing; python_version < '3.5'
packages = find:
Why does setup_requires include setuptools? This does not seem to make sense since:
the first line of the setup.py file imports setuptools, so by the time the setup function is called and reads the setup.cfg file that instructs to install setuptools it is already too late to install setuptools:
from setuptools import setup
setup()
setuptools is already installed on any fresh Python installation (well, only tested on Windows 10 and MacOS 10.15 with Python 3.8.0):
$ python -V
Python 3.8.0
$ pip list
Package Version
---------- -------
pip 19.2.3
setuptools 41.2.0
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
No, setuptools should not be included in setup_requires, according to PEP 518 (bold emphasis mine):
Setuptools tried to solve this with a setup_requires argument to its
setup() function [3]. This solution has a number of issues, such as:
No tooling (besides setuptools itself) can access this information without executing the setup.py, but setup.py can't be executed without having these items installed.
While setuptools itself will install anything listed in this, they won't be installed until during the execution of the setup() function, which means that the only way to actually use anything added here is through increasingly complex machinations that delay the import and usage of these modules until later on in the execution of the setup() function.
This cannot include setuptools itself nor can it include a replacement to setuptools, which means that projects such as numpy.distutils are largely incapable of utilizing it and projects cannot take advantage of newer setuptools features until their users naturally upgrade the version of setuptools to a newer one.
The items listed in setup_requires get implicitly installed whenever you execute the setup.py but one of the common ways that the setup.py is executed is via another tool, such as pip, who is already managing dependencies. This means that a command like pip install spam might end up having both pip and setuptools downloading and installing packages and end users needing to configure both tools (and for setuptools without being in control of the invocation) to change settings like which repository it installs from. It also means that users need to be aware of the discovery rules for both tools, as one may support different package formats or determine the latest version differently.
The accepted answer is mostly correct, but where PEP 518 says.
[The setup_requires mechanism] cannot include setuptools itself...
It's technically incorrect, and as importlib_resources demonstrates, it can actually include setuptools. The problem is that including setuptools in setup_requires serves mostly as documentation. It declares that setuptools is a build requirement (required to run setup.py), but it won't be capable of satisfying that requirement if it's not already satisfied.
But, the presence of setuptools in setup_requires is technically correct and does serve the purpose of declaring the requirement and asking setuptools to verify that the requirement is in fact installed (alongside other setup-time requirements).
It is, however, just a legacy artifact and doesn't provide that much value, and as can be seen in the question and answers, it does lead to confusion. The recommended, proper, approach is to use PEP 517 and 518 declarations and builders, but that part of the ecosystem hasn't matured yet, so setuptools vestiges will remain. Try not to let them bother you.
Why does setup_requires includes setuptools? This does not seem to make sense
Does not make sense at all. On the other hand it doesn't hamper anything so why not?
Related
accordingly to my research the following should work:
from setuptools import setup
from setuptools import find_packages
...
REQUIRES_INSTALL = [
'spacy==2.3.2',
'tensorflow==1.14.0',
'Keras==2.2.4',
'keras-contrib#git+https://github.com/keras-team/keras-contrib.git#egg=keras-contrib',
'en-core-web-sm#https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.0/en_core_web_sm-2.3.0.tar.gz#egg=en-core-web-sm'
]
...
setup(
name=NAME,
version=VERSION,
description=DESCRIPTION,
install_requires=REQUIRES_INSTALL,
...
)
When building a wheel or egg, everything is fine: python setup.py bdist_wheel.
But when trying to install the package (whl or egg) with pip install -U dist/mypack-....whl.
I get:
ERROR: Could not find a version that satisfies the requirement keras-contrib (from mypack==0.3.5) (from versions: none)
ERROR: No matching distribution found for keras-contrib (from mypack==0.3.5)
...
ERROR: Could not find a version that satisfies the requirement en-core-web-sm (from mypack==0.3.5) (from versions: none)
ERROR: No matching distribution found for en-core-web-sm (from mypack==0.3.5)
I have tried to same via setup.cfg but still no luck.
As reference - all these dependency are working when installing them first from requirments.txt and then installing the wheel.
spacy==2.3.2
tensorflow==1.14.0
Keras==2.2.4
keras-contrib#git+https://github.com/keras-team/keras-contrib.git#egg=keras-contrib
en-core-web-sm#https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.0/en_core_web_sm-2.3.0.tar.gz#egg=en-core-web-sm
pip install -r requirements.txt
pip install -U dist/mypack-....whl
But this is not clean way, since a wheel should be self contained.
Thank you for any hint!
Environment
Python: 3.7.0
Pip: 20.2.4
setuptools: 50.3.2
Some time ago it was possible to define a single requirements.txt or similar containing both specs for PyPI packages and links to repositories and archives.
That demanded to parse the requirements.txt and split them into "requirements" and "dependencies", where "requirements" would contain definitions of PyPI packages and "dependencies" — links.
Setuptools has different args for setup() for these: install_requires and dependency_links.
And it really worked: one was able to define a requirements.txt and install a package both as python setup.py install and as pip install .. Moreover, it was possible to install just dependencies via pip install -r requirements.txt. All ways worked and allowed to have a single place to define all requirements including non-PyPI links.
However, support of dependency_links arg was dropped by pip since v19. And here's the weird part: it is not dropped by setuptools. But there's more.
As of today, pip:
Supports only install_requires.
Prefers PEP 508 notation for dependencies in definitions of packages (install_requires) and standalone requirements.txt or similar.
Aborts installation of packages, which contain links in their install_requires.
Your definitions of dependencies mix 2 notations: prefixes like keras-contrib# are from PEP 508 and #egg= parts are from setuptools links notation.
This is not an issue: pip will ignore "eggs" as names are already defined before #.
I believe the installation of the package via pip works fine, i.e.:
pip install .
However, the issues will arise if the package is installed via setuptools, i.e.:
python setup.py install
setuptools does not understand PEP 508 notation and ignores links in install_requires. As of today, to make setuptools following links, both install_requires and dependency_links have to be used, e.g.:
setup(
...
install_requires=[
...
"keras_contrib==2.0.8",
...
],
dependency_links=[
"https://github.com/keras-team/keras-contrib/tarball/master#egg=keras_contrib-2.0.8",
...
],
)
Here are several tricky points:
A single dependency is defined in 2 places: a package name in install_requires and a link in dependency_links to resolve the package dependency.
The link is not git+https://.../....git, but it's a link to an archive: https://.../tarball/....
Egg name is in snake_case, not in dash-case. While it's possible to use dash-case, this will not allow specifying the version.
Version in install_requires is delimited via == and in dependency_links — via -.
It's possible to omit the version. But the only viable use case for that is if the package is not present in PyPI and is rarely updated. If the package is present in PyPI, but an unpublished version is needed, then the version must be specified.
And here's the bummer: fixing links for setuptools will break pip, as PEP 508 does not allow to specify versions. Keeping keras-contrib==x.y.z # ... in install_requires will make pip to search for the package keras-contrib==x.y.z, where ==x.y.z is not a version, but a part of the name. At the same time, not specifying a version will make setuptools to grab the latest version available at PyPI, not at the link from dependency_links.
In your case neither keras-contrib nor en-core-web-sm are present at PyPI, so using keras_contrib#git+https://... + dependency_links without version specified might work.
Otherwise, stick to pip install . and avoid using python setup.py install if the package depends on links.
See also:
PEP 508
PEP508: why either version requirement or URL but not both?
How can I make setuptools install a package that's not on PyPI?
pip install dependency links
pip3 setup.py install_requires PEP 508 git URL for private repo
Why is dependency links in setup.py deprecated?
Changing PEP 508 URLs in setup.py doesn't reinstall the dependency
Updating remote links with new URLs for PEP508 functionality
Requirements using PEP 508 direct references ignore the URL
Suggest alternatives for --process-dependency-links
Un-deprecate --process-dependency-links until an alternative is implemented
Changes to the pip dependency resolver in 20.3 (2020)
Trivia: several issues on GitHub are still open and PEP 508 is still in Active state since 2015. Digging around source code would reveal that setuptools is a wrapper around Python's distutils. setuptools is not a part of Python's stdlib, but the docs of distutils imply stdlib docs will be removed after docs of setuptools will be updated. At the same time pip is already bundled with Python's installations as Python's module. And yet we have pipfiles, pipenv, poetry, conda, pipx, pip-tools, shiv, spack, and the rest. Looks a bit overwhelming.
My python package footools needs html5lib via install_requires in setup.py.
setup.py develop fails
Installing via setup.py develop fails:
cd src/footools/
python setup.py develop
Processing dependencies for footools==2016.205
Searching for html5lib==0.9999999
Reading https://source.example.com/pypi/simple/html5lib/
Download error on https://source.example.com/pypi/simple/html5lib/:
[Errno 185090050] _ssl.c:354: error:0B084002:x509
certificate routines:X509_load_cert_crl_file:system lib --
Some packages may not be found!
Couldn't find index page for 'html5lib' (maybe misspelled?)
pip works
But direct download works:
bar#workdevel123:~/src/footools> pip install html5lib==0.9999999
/home/bar/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79:
InsecurePlatformWarning: A true SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately
and may cause certain SSL connections to fail.
For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Collecting html5lib==0.9999999
/home/bar/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79:
InsecurePlatformWarning: A true SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately and
may cause certain SSL connections to fail.
For more information,
see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Downloading https://source.example.com/pypi/packages/html5lib-0.9999999.tar.gz
Requirement already satisfied (use --upgrade to upgrade):
six in /usr/lib/python2.7/site-packages (from html5lib==0.9999999)
Installing collected packages: html5lib
Running setup.py install for html5lib
Successfully installed html5lib-0.9999999
Questions
What is the difference between these two methods?
Why are they different?
What is the correct way to install a dependency in python?
setup.py
The setup.py is not special:
import setuptools
setuptools.setup(
name='foo',
version='2016.210',
long_description=open('README.txt').read(),
packages=setuptools.find_packages(),
install_requires=[
# about twenty packages before this line
'html5lib==0.9999999'
],
include_package_data=True,
entry_points={
'console_scripts': [
'foo=foo.utils.bar:main',
],
},
)
python setup.py develop, or setuptools in general uses easy_install to satisfy dependencies which in turn uses urllib2 whereas pip uses requests. See here for easy_install vs pip.
pip is more modern and among other things has capability to uninstall packages and complies with PEP 438 -- Transitioning to release-file hosting on PyPI. You can achieve the same thing as python setup.py develop with pip install -e src/footools/, note if the project path is in the current directory use, ./footools.
The requests package bundles CA certs in the package itself, python -c 'import pip;print(pip.download.requests.certs.where())'.
setuptools uses system installed CA certs python -c 'from setuptools import ssl_support;print(ssl_support.cert_paths)'.
You have to update system installed CA certs using tools like update-ca-certificates for Ubuntu to either update CA certs automatically or download from https://curl.haxx.se/docs/caextract.html and install into one of the paths shown by setuptools or set setuptools.ssl_support.cert_paths to an empty sequence like [] and do pip install certifi.
Calling setuptools.ssl_support.find_ca_bundle() will reveal the location of CA certs.
setuptools is a collection of enhancements to the Python distutils (for Python 2.6 and up) that allow developers to more easily build and distribute Python packages, especially ones that have dependencies on other packages.
So, among other stuff, you can create packages that can be upload to Pypi, and later installed using pip (therefore distibuting your module).
That said, they actually should not be that different in the installation part. You are running the develop mode, so maybe you have to fiddle a bit with the directories or fix the authorization error.
In the Development Mode the project is deployed into a staging area (in some way similar to the process of a virtual environment)
deployment is done in such a way that changes to the project source are immediately available in the staging area(s), without needing to run a build or install step after each change.
Meaning also everything will be available for that python interpreter. It can be later unstaged.
I noticed that html5lib is being fetched from different places: /pypi/simple/in one case and /pypi/packages/ in the other.
dependency_links
A list of strings naming URLs to be searched when satisfying dependencies. These links will be used if needed to install packages specified by setup_requires or tests_require.
Back to the problem I think it is most probably the ssl issue, since in pip it handled graciously (i.e., nice warning and there is some kind of workaround), but the same does not happen with setuptools. If there is an error in the request that is not handled then Couldn't find index page for 'html5lib' could be because of that.
What is the difference between these two methods?
Nothing terribly important to you as a user, aside from their different user interfaces. They are two stops on the scenic historical train ride of Python package management. There were others along the way.
Why are they different?
Back in the day, Python didn't ship with a package management system. Third party solutions filled the void. They were designed by different people at different moments in time. If you look at other programming languages, you sometimes see similar stories; sometimes you see happier stories; sometimes more tragic.
What is the correct way to install a dependency in python?
Both of these methods are technically correct. Pip is the more modern method, and in my experience it is both more popular and handier to work with. As of Python 3.4 and up, Pip has been included in the CPython distribution and is officially 'preferred'. So you can see which way the wind is blowing.
I ran into a problem when installing a package which depended on python-daemon. I ultimately traced it to the latest version of the package python-daemon (2.0.3) released yesterday. Testing in a virtual environment on an Ubuntu 14.04 machine and issuing the following commands:
(venv) $ pip list
argparse (1.2.1)
pip (1.5.6)
setuptools (3.6)
wsgiref (0.1.2)
(venv) $ pip install redis
... works fine ....
(venv) $ pip install python-daemon
...
snip
...
File "/home/pwj/.virtualenvs/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load
['__name__'])
ImportError: No module named version
(venv)02:15 PM tmp$ pip list
argparse (1.2.1)
lockfile (0.10.2)
pip (1.5.6)
python-daemon (2.0.3)
setuptools (3.6)
wsgiref (0.1.2)
So the install of python-daemon seemed to work but something affected pip or setuptools because other packages (celery, flask), I try to install with pip after this gives me the same traceback:
...
snip
...
File "/home/pwj/.virtualenvs/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load
['__name__'])
ImportError: No module named version
If I uninstall python-daemon with pip things again and packages that weren't installing now install fine. Has anyone else come across this or something similar with a different project? My solution was to pip install the previous version
(venv) $ pip install python-daemon==2.0.2
... works ...
but was wondering what might be causing such an error.
(This behaviour is corrected in python-daemon version 2.0.4 and later.)
There are two sides to this:
Setuptools assumes it is the centre of everything.
Version 2.0.3 of python-daemon doesn't take that into account.
A more detailed explanation: There is some complex code using Docutils involved in the python-daemon build process, that isn't needed after install and isn't part of the library code.
It's too complex to leave in the un-importable (and therefore not-unit-testable) setup.py, so that build code is shunted to a separate testable module, version (in the file version.py), which itself uses Docutils.
But then the setup.py has a circular dependency: How to import version, when Docutils isn't yet installed? How to use Setuptools to ensure Docutils is installed, when running setup.py to completion will need version? All the feasible solutions are ugly and confusing.
The approach taken in ‘python-daemon’ 2.0.3 is to declare Docutils required for setup, and declare a Setuptools entry point for the work that needs version. That way setup.py gets to install Docutils before any of the entry points that will use version.
But now we come to the first point, that Setuptools arrogates itself as the centre of everything. By declaring an entry point, setup.py has modified every Setuptools action thereafter, and every package will fail if it can't find the entry points. And, since most of them don't have version or the specified functions in that module, they crash Setuptools.
What is essentially a bug to be fixed, reveals a poorly-understood corner case in Setuptools. So I'm voting your question up.
There doesn't seem to be a good solution to this: having modules available for setup.py but ensuring requirements are met first. Setuptools assumes it is the only build system needed to satisfy all dependencies for everything, and when that assumption fails it's very difficult to get around.
Thanks to the Python Packaging Authority folks, and the distutils-sig forum, for explaining this to me.
I have a program that uses dateutil from the package index. I would like to have setup.py check for for its presence and try to get it using easy_install if it is not there.
The documentation for distutils seems to indicate that this can be done using the requires keyword in setup(), but when I try, it installs on a system without dateutil without giving a warning or installing the required package.
The only thing I could find on google was this blog post about the same issue which did not have any answer either.
Am I using distutils wrong? Do I need to subclass distutils.command.install and do the checking/installing myself?
Automatic downloading of dependencies is a feature introduced by setuptools which is a third-party add-on to distutils, in particular, the install_requires argument it adds. See the setuptools documentation for more information.
Another option is to use requirements.txt file with pip rather than using easy_install as a package installer. pip has now become the recommended installer; see the Python Packaging User Guide for more information.
Update [2015-01]: The previous version of this answer referred to the distribute fork of setuptools. The distribute fork has since been merged back into a newer active setuptools project. distribute is now dead and should no longer be used. setuptools and pip are now very actively maintained and support Python 3.
The argument install_requires in setup function from distutils work for me well, only if I create sdist distributive, like: python setup.py sdist
Does anybody encounter this warning when executing python setup.py install of a PyPI package?
install_requires defines what the package requires. A lot of PyPI packages have this option. How can it be an "unknown distribution option"?
python setup.py uses distutils which doesn't support install_requires. setuptools does, also distribute (its successor), and pip (which uses either) do. But you actually have to use them. I.e. call setuptools through the easy_install command or pip install.
Another way is to import setup from setuptools in your setup.py, but this not standard and makes everybody wanting to use your package have to have setuptools installed.
This was the first result on my google search, but had no answer.
I found that upgrading setuptools resolved the issue for me (and pip for good measure)
pip install --upgrade pip
pip install --upgrade setuptools
Hope this helps the next person to find this link!
I'm on a Mac with python 2.7.11. I have been toying with creating extremely simple and straightforward projects, where my only requirement is that I can run python setup.py install, and have setup.py use the setup command, ideally from distutils. There are literally no other imports or code aside from the kwargs to setup() other than what I note here.
I get the error when the imports for my setup.py file are:
from distutils.core import setup
When I use this, I get warnings such as
/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'
warnings.warn(msg)
If I change the imports (and nothing else) to the following:
from distutils.core import setup
import setuptools # noqa
The warnings go away.
Note that I am not using setuptools, just importing it changes the behavior such that it no longer emits the warnings. For me, this is the cause of a truly baffling difference where some projects I'm using give those warnings, and some others do not.
Clearly, some form of monkey-patching is going on, and it is affected by whether or not that import is done. This probably isn't the situation for everyone researching this problem, but for the narrow environment in which I'm working, this is the answer I was looking for.
This is consistent with the other (community) comment, which says that distutils should monkeypatch setuptools, and that they had the problem when installing Ansible. Ansible appears to have tried to allow installs without having setuptools in the past, and then went back on that.
https://github.com/ansible/ansible/blob/devel/setup.py
A lot of stuff is up in the air... but if you're looking for a simple answer for a simple project, you should probably just import setuptools.
ATTENTION! ATTENTION! Imperfect answer ahead. To get the "latest memo" on the state of packaging in the Python universe, read this fairly detailed essay.
I have just ran into this problem when trying to build/install ansible. The problem seems to be that distutils really doesn't support install_requires. Setuptools should monkey-patch distutils on-the-fly, but it doesn't, probably because the last release of setuptools is 0.6c11 from 2009, whereas distutils is a core Python project.
So even after manually installing the setuptools-0.6c11-py2.7.egg running setup.py only picks up distutils dist.py, and not the one from site-packages/setuptools/.
Also the setuptools documentation hints to using ez_setup and not distutils.
However, setuptools is itself provided by distribute nowadays, and that flavor of setup() supports install_requires.
This is a warning from distutils, and is a sign that you do not have setuptools installed.
Installing it from http://pypi.python.org/pypi/setuptools will remove the warning.
In conclusion:
distutils doesn't support install_requires or entry_points, setuptools does.
change from distutils.core import setup in setup.py to from setuptools import setup or refactor your setup.py to use only distutils features.
I came here because I hadn't realized entry_points was only a setuptools feature.
If you are here wanting to convert setuptools to distutils like me:
remove install_requires from setup.py and just use requirements.txt with pip
change entry_points to scripts (doc) and refactor any modules relying on entry_points to be full scripts with shebangs and an entry point.
sudo apt-get install python-dev # for python2.x installs
sudo apt-get install python3-dev # for python3.x installs
It will install any missing headers. It solved my issue
As far as I can tell, this is a bug in setuptools where it isn't removing the setuptools specific options before calling up to the base class in the standard library: https://bitbucket.org/pypa/setuptools/issue/29/avoid-userwarnings-emitted-when-calling
If you have an unconditional import setuptools in your setup.py (as you should if using the setuptools specific options), then the fact the script isn't failing with ImportError indicates that setuptools is properly installed.
You can silence the warning as follows:
python -W ignore::UserWarning:distutils.dist setup.py <any-other-args>
Only do this if you use the unconditional import that will fail completely if setuptools isn't installed :)
(I'm seeing this same behaviour in a checkout from the post-merger setuptools repo, which is why I'm confident it's a setuptools bug rather than a system config problem. I expect pre-merge distribute would have the same problem)
I've now seen this in legacy tools using Python2.7, where a build (like a Dockerfile) installs an unpinned dependancy, for example pytest. PyTest has dropped Python 2.7 support, so you may need to specify version < the new package release.
Or bite the bullet and convert that app to Python 3 if that is viable.
It works fine if you follow the official documentation:
import setuptools
setuptools.setup(...)
Source: https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py