accordingly to my research the following should work:
from setuptools import setup
from setuptools import find_packages
...
REQUIRES_INSTALL = [
'spacy==2.3.2',
'tensorflow==1.14.0',
'Keras==2.2.4',
'keras-contrib#git+https://github.com/keras-team/keras-contrib.git#egg=keras-contrib',
'en-core-web-sm#https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.0/en_core_web_sm-2.3.0.tar.gz#egg=en-core-web-sm'
]
...
setup(
name=NAME,
version=VERSION,
description=DESCRIPTION,
install_requires=REQUIRES_INSTALL,
...
)
When building a wheel or egg, everything is fine: python setup.py bdist_wheel.
But when trying to install the package (whl or egg) with pip install -U dist/mypack-....whl.
I get:
ERROR: Could not find a version that satisfies the requirement keras-contrib (from mypack==0.3.5) (from versions: none)
ERROR: No matching distribution found for keras-contrib (from mypack==0.3.5)
...
ERROR: Could not find a version that satisfies the requirement en-core-web-sm (from mypack==0.3.5) (from versions: none)
ERROR: No matching distribution found for en-core-web-sm (from mypack==0.3.5)
I have tried to same via setup.cfg but still no luck.
As reference - all these dependency are working when installing them first from requirments.txt and then installing the wheel.
spacy==2.3.2
tensorflow==1.14.0
Keras==2.2.4
keras-contrib#git+https://github.com/keras-team/keras-contrib.git#egg=keras-contrib
en-core-web-sm#https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.0/en_core_web_sm-2.3.0.tar.gz#egg=en-core-web-sm
pip install -r requirements.txt
pip install -U dist/mypack-....whl
But this is not clean way, since a wheel should be self contained.
Thank you for any hint!
Environment
Python: 3.7.0
Pip: 20.2.4
setuptools: 50.3.2
Some time ago it was possible to define a single requirements.txt or similar containing both specs for PyPI packages and links to repositories and archives.
That demanded to parse the requirements.txt and split them into "requirements" and "dependencies", where "requirements" would contain definitions of PyPI packages and "dependencies" — links.
Setuptools has different args for setup() for these: install_requires and dependency_links.
And it really worked: one was able to define a requirements.txt and install a package both as python setup.py install and as pip install .. Moreover, it was possible to install just dependencies via pip install -r requirements.txt. All ways worked and allowed to have a single place to define all requirements including non-PyPI links.
However, support of dependency_links arg was dropped by pip since v19. And here's the weird part: it is not dropped by setuptools. But there's more.
As of today, pip:
Supports only install_requires.
Prefers PEP 508 notation for dependencies in definitions of packages (install_requires) and standalone requirements.txt or similar.
Aborts installation of packages, which contain links in their install_requires.
Your definitions of dependencies mix 2 notations: prefixes like keras-contrib# are from PEP 508 and #egg= parts are from setuptools links notation.
This is not an issue: pip will ignore "eggs" as names are already defined before #.
I believe the installation of the package via pip works fine, i.e.:
pip install .
However, the issues will arise if the package is installed via setuptools, i.e.:
python setup.py install
setuptools does not understand PEP 508 notation and ignores links in install_requires. As of today, to make setuptools following links, both install_requires and dependency_links have to be used, e.g.:
setup(
...
install_requires=[
...
"keras_contrib==2.0.8",
...
],
dependency_links=[
"https://github.com/keras-team/keras-contrib/tarball/master#egg=keras_contrib-2.0.8",
...
],
)
Here are several tricky points:
A single dependency is defined in 2 places: a package name in install_requires and a link in dependency_links to resolve the package dependency.
The link is not git+https://.../....git, but it's a link to an archive: https://.../tarball/....
Egg name is in snake_case, not in dash-case. While it's possible to use dash-case, this will not allow specifying the version.
Version in install_requires is delimited via == and in dependency_links — via -.
It's possible to omit the version. But the only viable use case for that is if the package is not present in PyPI and is rarely updated. If the package is present in PyPI, but an unpublished version is needed, then the version must be specified.
And here's the bummer: fixing links for setuptools will break pip, as PEP 508 does not allow to specify versions. Keeping keras-contrib==x.y.z # ... in install_requires will make pip to search for the package keras-contrib==x.y.z, where ==x.y.z is not a version, but a part of the name. At the same time, not specifying a version will make setuptools to grab the latest version available at PyPI, not at the link from dependency_links.
In your case neither keras-contrib nor en-core-web-sm are present at PyPI, so using keras_contrib#git+https://... + dependency_links without version specified might work.
Otherwise, stick to pip install . and avoid using python setup.py install if the package depends on links.
See also:
PEP 508
PEP508: why either version requirement or URL but not both?
How can I make setuptools install a package that's not on PyPI?
pip install dependency links
pip3 setup.py install_requires PEP 508 git URL for private repo
Why is dependency links in setup.py deprecated?
Changing PEP 508 URLs in setup.py doesn't reinstall the dependency
Updating remote links with new URLs for PEP508 functionality
Requirements using PEP 508 direct references ignore the URL
Suggest alternatives for --process-dependency-links
Un-deprecate --process-dependency-links until an alternative is implemented
Changes to the pip dependency resolver in 20.3 (2020)
Trivia: several issues on GitHub are still open and PEP 508 is still in Active state since 2015. Digging around source code would reveal that setuptools is a wrapper around Python's distutils. setuptools is not a part of Python's stdlib, but the docs of distutils imply stdlib docs will be removed after docs of setuptools will be updated. At the same time pip is already bundled with Python's installations as Python's module. And yet we have pipfiles, pipenv, poetry, conda, pipx, pip-tools, shiv, spack, and the rest. Looks a bit overwhelming.
Related
This is an extension of SO setup.py ignores full path dependencies, instead looks for "best match" in pypi
I am trying to write setup.py to install a proprietary package from a .tar.gz file on an internal web site. Unfortunately for me the prop package name duplicates a public package in the public PyPI, so I need to force install of the proprietary package at a specific version. I'm building a docker image from a Debian-Buster base image, so pip, setuptools and tox are all freshly installed, the image brings python 3.8 and pip upgrades itself to version 21.2.4.
Solution 1 - dependency_links
I followed the instructions at the post linked above to put the prop package in install_requires and dependency_links. Here are the relevant lines from my setup.py:
install_requires=["requests", "proppkg==70.1.0"],
dependency_links=["https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"]
Installation is successful in Debian-Buster if I run python3 setup.py install in my package directory. I see the proprietary package get downloaded and installed.
Installation fails if I run pip3 install . also tox (version 3.24.4) fails similarly. In both cases, pip shows a message "Looking in indexes" then fails with "ERROR: Could not find a version that satisfies the requirement".
Solution 2 - PEP 508
Studying SO answer pip ignores dependency_links in setup.py which states that dependency_links is deprecated, I started over, revised setup.py to have:
install_requires=[
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
Installation is successful in Debian-Buster if I run pip3 install . in my package directory. Pip shows a message "Looking in indexes" but still downloads and installs the proprietary package successfully.
Installation fails in Debian-Buster if I run python3 setup.py install in my package directory. I see these messages:
Searching for proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0
..
Reading https://pypi.org/simple/proppkg/
..
error: Could not find suitable distribution for Requirement.parse(...).
Tox also fails in this scenario as it installs dependencies.
Really speculating now, it almost seems like there's an ordering issue. Tox invokes pip like this:
python -m pip install --exists-action w .tox/.tmp/package/1/te-0.3.5.zip
In that output I see "Collecting proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0" as the first step. That install fails because it fails to import package requests. Then tox continues collecting other dependencies. Finally tox reports as its last step "Collecting requests" (and that succeeds). Do I have to worry about ordering of install steps?
I'm starting to think that maybe the proprietary package is broken. I verified that the prop package setup.py has requests in its install_requires entry. Not sure what else to check.
Workaround solution
My workaround is installing the proprietary package in the docker image as a separate step before I install my own package, just by running pip3 install https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz. The setup.py has the PEP508 URL in install_requires. Then pip and tox find the prop package in the pip cache, and work fine.
Please suggest what to try for the latest pip and tox, or if this is as good as it gets, thanks in advance.
Update - add setup.py
Here's a (slightly sanitized) version of my package's setup.py
from setuptools import setup, find_packages
def get_version():
"""
read version string
"""
version_globals = {}
with open("te/version.py") as fp:
exec(fp.read(), version_globals)
return version_globals['__version__']
setup(
name="te",
version=get_version(),
packages=find_packages(exclude=["tests.*", "tests"]),
author="My Name",
author_email="email#mycompany.com",
description="My Back-End Server",
entry_points={"console_scripts": [
"te-be=te.server:main"
]},
python_requires=">=3.7",
install_requires=["connexion[swagger-ui]",
"Flask",
"gevent",
"redis",
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
package_data={"te": ["openapi_te.yml"]},
include_package_data=True, # read MANIFEST.in
)
The importlib_resources backport for Python < 3.7 of the importlib.resources standard library module has the following section in the setup.cfg file:
[options]
python_requires = >=2.7,!=3.0,!=3.1,!=3.2,!=3.3
setup_requires =
setuptools
wheel
install_requires =
pathlib2; python_version < '3'
typing; python_version < '3.5'
packages = find:
Why does setup_requires include setuptools? This does not seem to make sense since:
the first line of the setup.py file imports setuptools, so by the time the setup function is called and reads the setup.cfg file that instructs to install setuptools it is already too late to install setuptools:
from setuptools import setup
setup()
setuptools is already installed on any fresh Python installation (well, only tested on Windows 10 and MacOS 10.15 with Python 3.8.0):
$ python -V
Python 3.8.0
$ pip list
Package Version
---------- -------
pip 19.2.3
setuptools 41.2.0
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
No, setuptools should not be included in setup_requires, according to PEP 518 (bold emphasis mine):
Setuptools tried to solve this with a setup_requires argument to its
setup() function [3]. This solution has a number of issues, such as:
No tooling (besides setuptools itself) can access this information without executing the setup.py, but setup.py can't be executed without having these items installed.
While setuptools itself will install anything listed in this, they won't be installed until during the execution of the setup() function, which means that the only way to actually use anything added here is through increasingly complex machinations that delay the import and usage of these modules until later on in the execution of the setup() function.
This cannot include setuptools itself nor can it include a replacement to setuptools, which means that projects such as numpy.distutils are largely incapable of utilizing it and projects cannot take advantage of newer setuptools features until their users naturally upgrade the version of setuptools to a newer one.
The items listed in setup_requires get implicitly installed whenever you execute the setup.py but one of the common ways that the setup.py is executed is via another tool, such as pip, who is already managing dependencies. This means that a command like pip install spam might end up having both pip and setuptools downloading and installing packages and end users needing to configure both tools (and for setuptools without being in control of the invocation) to change settings like which repository it installs from. It also means that users need to be aware of the discovery rules for both tools, as one may support different package formats or determine the latest version differently.
The accepted answer is mostly correct, but where PEP 518 says.
[The setup_requires mechanism] cannot include setuptools itself...
It's technically incorrect, and as importlib_resources demonstrates, it can actually include setuptools. The problem is that including setuptools in setup_requires serves mostly as documentation. It declares that setuptools is a build requirement (required to run setup.py), but it won't be capable of satisfying that requirement if it's not already satisfied.
But, the presence of setuptools in setup_requires is technically correct and does serve the purpose of declaring the requirement and asking setuptools to verify that the requirement is in fact installed (alongside other setup-time requirements).
It is, however, just a legacy artifact and doesn't provide that much value, and as can be seen in the question and answers, it does lead to confusion. The recommended, proper, approach is to use PEP 517 and 518 declarations and builders, but that part of the ecosystem hasn't matured yet, so setuptools vestiges will remain. Try not to let them bother you.
Why does setup_requires includes setuptools? This does not seem to make sense
Does not make sense at all. On the other hand it doesn't hamper anything so why not?
Looking as solutions in the past such as
pip ignores dependency_links in setup.py, this configuration should work.
Relevant content of my setup.py
packages=find_packages(),
dependency_links=['http://github.com/koji-project/koji/tarball/master#egg=koji'],
install_requires=['jira', 'PyYAML', 'requests', 'psycopg2',
'elasticsearch', 'beanbag', 'pyzabbix', 'enum34',
'beautifulsoup4', 'pytz', 'koji'],
tests_require=['flake8', 'autopep8', 'mock'],
include_package_data=True,
cmdclass={'test': setupTestRequirements}
The only thing I can think of is that my url is invalid. I don't see why it would be since it is of version 1.14.0.
Upon running pip install . I get.
Could not find a version that satisfies the requirement koji (from MARs==0.17.10) (from versions: ) No matching distribution found for koji (from MARs==0.17.10)
Upon running python setup.py develop --user, the output doesn't mention Koji
Your configuration is correct. However the problem lies elsewhere. Take a look at the koji repo on github: the project has no setup.py committed. As long as there's no setup.py script, neither pip nor setuptools (via setup.py install/setup.py develop) won't be able to install your project because they won't be able to install koji dependency as it is no valid python package at all.
Update:
The problem with koji repo on github is that it is only a mirror of the actual dev repo located on Fedora Pagure and is not synced with the upstream. So the correct answer is to use the real development repository instead of the github mirror:
dependency_links=['git+https://pagure.io/koji.git#egg=koji-1.14.0']
Easy peasy. :-)
Original answer (obsolete, only if you want to install from kojis repo mirror on Github):
I see two ways out of this situation:
Forking
fork koji on github
write your own setup.py script or copy it somewhere (see below for more info), commit and push
adapt the URL in dependency_links in your project's setup.py.
For testing, I prepared a fork of koji with a setup script; if I use its URL instead of the upstream repo, the installation succeeds. I also tagged my own "release" with koji-1.14.0.post1 to distinct the version with the setup script from the vanilla ones. Example setup.py with the new dependency:
from setuptools import setup, find_packages
setup(
name='spam',
version='0.1',
author='nobody',
author_email='nobody#nowhere.com',
url='www.example.com',
packages=[],
dependency_links=['https://github.com/hoefling/koji/tarball/master#egg=koji-1.14.0.post1'],
install_requires=['koji==1.14.0.post1'],
)
Testing the installation with pip yields:
$ pip install . --process-dependency-links
Obtaining file:///home/hoefling/python/spam
DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting koji==1.14.0.post1 (from spam==0.1)
Downloading https://github.com/hoefling/koji/tarball/master (1.4MB)
100% |████████████████████████████████| 1.4MB 759kB/s
Collecting pyOpenSSL (from koji==1.14.0.post1->spam==0.1)
Using cached pyOpenSSL-17.5.0-py2.py3-none-any.whl
Collecting pycurl (from koji==1.14.0.post1->spam==0.1)
Using cached pycurl-7.43.0.1.tar.gz
...
Installing collected packages: six, idna, asn1crypto, pycparser, cffi,
cryptography, pyOpenSSL, pycurl, python-dateutil, chardet, certifi,
urllib3, requests, pykerberos, requests-kerberos, rpm-py-installer,
koji, spam
Running setup.py install for rpm-py-installer ... done
Running setup.py install for koji ... done
Running setup.py install for spam ... done
Successfully installed asn1crypto-0.23.0 certifi-2017.11.5 cffi-1.11.2
chardet-3.0.4 cryptography-2.1.4 idna-2.6 koji-1.14.0.post1 pyOpenSSL-17.5.0
pycparser-2.18 pycurl-7.43.0.1 pykerberos-1.1.14 python-dateutil-2.6.1
requests-2.18.4 requests-kerberos-0.11.0 rpm-py-installer-0.5.0 six-1.11.0
spam-0.1 urllib3-1.22
Installed packages look good:
$ pip list
Package Version
----------------- ------------
asn1crypto 0.23.0
certifi 2017.11.5
cffi 1.11.2
chardet 3.0.4
cryptography 2.1.4
idna 2.6
koji 1.14.0.post1
pip 9.0.1
pycparser 2.18
pycurl 7.43.0.1
pykerberos 1.1.14
pyOpenSSL 17.5.0
python-dateutil 2.6.1
requests 2.18.4
requests-kerberos 0.11.0
rpm-py-installer 0.5.0
rpm-python 4.11.3
setuptools 38.2.4
six 1.11.0
spam 0.1
urllib3 1.22
wheel 0.30.0
The downside of this method is the additional overhead you get in maintaining the fork until the setup script is merged into upstream. This includes testing and eventually adapting koji's setup.py in your fork each time you want to sync the upstream updates. I would probably create a separate branch with the setup script committed there, sync the fork as usual and then rebase the branch on top of fork's master, but if you are used to another update strategy, stick to it.
Use koji package from TestPyPI
Actually, I found some koji wheels of the most recent version on TestPyPI. This is also the place where I got the setup.py for the fork above - I downloaded the source tar, unpacked it and copied the setup script. This means that the koji devs are looking into distributing the project via PyPI and are working on the setup script, but didn't commit it yet. While they are working on it, you can use the testing package index as the workaround. This way, you will not build the package from sources, taking the wheel instead that koji devs built and uploaded:
setup(
...
dependency_links=['https://testpypi.python.org/pypi/koji'],
install_requires=['koji'],
)
The downsides of this method are:
You don't know if the koji package from TestPyPI is installable at all. Even if it is, there's no guarantee that the installed code will work as intended (although it should). When you have the fork, you can always fix the setup script yourself - here you are doomed if the wheel file has errors.
Packages on TestPyPI are removed on a regular basis. From the docs:
Note: The database for TestPyPI may be periodically pruned, so it is not unusual for user accounts to be deleted.
Last note
You can of course combine the two workarounds and use both URLs in dependency_links:
setup(
...
dependency_links=[
'https://testpypi.python.org/pypi/koji',
'https://github.com/hoefling/tarball/master#egg=koji-1.14.0.post1',
],
install_requires=['koji'],
)
This way, if the package is not found on TestPyPI, it will be built from your fork.
Last note 2
You will probably need to install some additional system packages; at least for my system CentOS Linux release 7.3.1611 (Core) I had to install curl-devel to satisfy pycurl.
Here is my setup.py:
setup(
...
install_requires=['GEDThriftStubs'],
dependency_links=['git+ssh://user#git.server.com/ged-thrift-stubs.git#egg=GEDThriftStubs'],
...)
Then I create package:
python setup.py sdist
Then I try to install it:
pip install file://path/package-0.0.1.tar.gz
And get this in terminal:
Downloading/unpacking GEDThriftStubs (from package==0.0.1)
Could not find any downloads that satisfy the requirement GEDThriftStubs (from package==0.0.1)
No distributions at all found for GEDThriftStubs (from package==0.0.1)
And in pip.log messages like this:
Skipping link git+ssh://user#git.server.com/ged-thrift-stubs.git#egg=GEDThriftStubs; wrong project name (not gedthriftstubs)
And I don't have anywhere in my project that exact name "gedthriftstubs", if it matters.
But this works fine:
pip install git+ssh://user#git.server.com/ged-thrift-stubs.git#egg=GEDThriftStubs
Try:
$ pip install --process-dependency-links file://path/package-0.0.1.tar.gz
Note that this tag is removed from pip in pip 1.6. See this article on pip.pypa.io for more information.
In pip 1.5 processing dependency links was deprecated and it was removed completely in pip 1.6.
There's also a lengthy discussion ( issue #1519 ) regarding pip & dependency links
If that doesn't work, you may also need to add a version suffix on your link, like this:
git+ssh://user#git.server.com/ged-thrift-stubs.git#egg=GEDThriftStubs-0.0.1
where 0.0.1 is the version specified in the setup.py of ged-thrift-stubs
It's a similar question to How can I make setuptools install a package that's not on PyPI? but not the same.
As I would like to use the forked version of some package, setuptools ignore the dependency link (as it has the same version number).
Is there a way to force using the link from the dependency_links? Or is the only way to change the version number in the forked repo?
requires = [
...
'pyScss==1.1.3'
...
dependencies = [
'https://github.com/nadavshatz/pyScss/zipball/master#egg=pyScss-1.1.3'
]
Update
Weird, apparently it works if this package is the only one in the required list, that is not installed yet. If there's another missing package it will download it from pypi.
I believe you can just use dependency_links as described in that question:
from setuptools import setup
setup(name = 'mypkg',
version = '0.0.1',
description = 'Foo',
author = 'bar',
author_email = 'bar#example.com',
install_requires = ['pyScss==1.1.3'],
dependency_links = [
'https://github.com/nadavshatz/pyScss/zipball/master#egg=pyScss-1.1.3'
]
)
Tested using python setup.py develop
You probably want to rename the egg to emphasize it's a fork http://www.python.org/dev/peps/pep-0386/
Outside of the setup.py you can enforce this locally using requirements.txt and pip. Whilst this won't make your package depend on the fork you can easily document it as the way to install.
$ cat requirements.txt
https://github.com/nadavshatz/pyScss/zipball/master#egg=pyScss-1.1.3
$ pip install -r requirements.txt
I ended up doing something very similar to the answer in stackoverflow.com/a/17442663/368102.
I need a requests-file github package that name-conflicts with a different requests-file package in PyPi. They both have a version 1.0, and the PyPi version has some higher versions.
The workaround in my ias_tools/setup.py looks like this:
setup(
...
install_requires=[
'requests-file<=99.99',
],
dependency_links=[
'https://github.com/jvantuyl/requests-file/archive/b0a7b34af6e287e07a96bc7e89bac3bc855323ae.zip#egg=requests-file-99.99'
]
)
In my case, I'm using pip so I also had to use --process-dependency-links:
% pip install --process-dependency-links ./ias_tools
You are using pip version 6.0.6, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Processing ./ias_tools
DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting requests-file<=99.99 (from ias-tools==0.1)
Downloading https://github.com/jvantuyl/requests-file/archive/b0a7b34af6e287e07a96bc7e89bac3bc855323ae.zip
Requirement already satisfied (use --upgrade to upgrade): requests>=1.1.0 in ./venv/lib/python2.7/site-packages (from requests-file<=99.99->ias-tools==0.1)
Installing collected packages: ias-tools, requests-file
Running setup.py install for ias-tools
Running setup.py install for requests-file
Successfully installed ias-tools-0.1 requests-file-1.0
I'm not too worried about the deprecation notice, as a pull request was submitted to pip to deprecate the deprecation (after a discussion about it).