This question already has answers here:
Pip install from pypi works, but from testpypi fails (cannot find requirements)
(2 answers)
Closed 2 years ago.
How can I publish a package on PyPI such that all dependencies are automatically installed, rather than manually by the user.
I specify the dependencies in setup.py with install_requires as follows:
setuptools.setup(name='myPackage',
version='1.0',
packages=setuptools.find_packages(),
include_package_data=True,
classifiers=[
'Programming Language :: Python :: 3',
'Operating System :: OS Independent',
'Topic :: Scientific/Engineering :: Bio-Informatics'
],
install_requires=['numpy', 'pandas', 'sklearn'],
python_requires='>=3'
)
And I have a requirements.txt file which is included in my MANIFEST.in:
numpy==1.15.4
sklearn==0.20.1
pandas==0.23.4
However, after publishing on test.pypi when I try to install the package, I get the following error:
Could not find a version that satisfies the requirement numpy (from myPackage==1.0.0) (from versions: )
No matching distribution found for sklearn (from myPackage==1.0.0)
This means that PyPI does not install the numpy dependency.
How do I enable automatic installation of my dependencies?
Should I use a virtual environment when building and publishing the package? How do I do this?
P.S. I am entirely new to this so I will appreciate explicit code or links to simple tutorial pages. Thank you.
You can specify multiple indexes via --extra-index-url. Point it to TestPyPI so your package is pulled from there, the deps from PyPI:
$ pip install myPackage --extra-index-url=https://test.pypi.org/simple/
However, the real root for the issue is that you have included the wrong dist name for the scikit-learn package. Replace sklearn with scikit-learn:
setup(
...,
install_requires=['numpy', 'pandas', 'scikit-learn'],
)
This is an unfortunate (and known) downside to TestPyPI: The issue is that sklearn does not exist on TestPyPI, and by installing your package from there, you are telling pip to look for dependencies there as well.
Instead, you should publish to PyPI instead, and use a pre-release version so as not to pollute your versions. You can delete these pre-releases from the project later.
I realized that installing packages from test.PyPI does not install all packages, since some of these packages are hosted on PyPI and not test.PyPI.
When I published the package on PyPI as a pre-release version (1.0a1), instead on test.PyPI, the dependencies were correctly installed. Hence, the problem was purely with test.PyPI.
This is my approach.
I like to use a requirements.txt file instead of putting dependencies in install_requires because it's easier during dev to run:
$ pip install -r requirements.txt
To have pip install dependencies automatically, I include at the top of setup.py before setuptools.setup():
requirements = []
with open('requirements.txt', 'r') as fh:
for line in fh:
requirements.append(line.strip())
Then in setuptools.setup():
install_requires = requirements
To install:
pip install --index-url https://test.pypi.org/simple/ --upgrade --no-cache-dir --extra-index-url=https://pypi.org/simple/ <<package name>>
--index-url is telling pip to use the test version of pypi.
--upgrade forces an upgrade if a previous version is installed.
--no-cache-dir resolves issues with caching if doing a very quick re-release (pip doesn't pick up the new version)
--extra-index tells pip to look in the prod version of pypi if it can't find the required package in test (i.e. solves problems of dependencies not being available in test)
Your install_requires should be of the form
...
install_requires=["numpy==1.15.4",
"sklearn==0.20.1",
"pandas==0.23.4"]
...
You can also use >= instead of == to allow for more recent versions of those libraries.
Related
This is an extension of SO setup.py ignores full path dependencies, instead looks for "best match" in pypi
I am trying to write setup.py to install a proprietary package from a .tar.gz file on an internal web site. Unfortunately for me the prop package name duplicates a public package in the public PyPI, so I need to force install of the proprietary package at a specific version. I'm building a docker image from a Debian-Buster base image, so pip, setuptools and tox are all freshly installed, the image brings python 3.8 and pip upgrades itself to version 21.2.4.
Solution 1 - dependency_links
I followed the instructions at the post linked above to put the prop package in install_requires and dependency_links. Here are the relevant lines from my setup.py:
install_requires=["requests", "proppkg==70.1.0"],
dependency_links=["https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"]
Installation is successful in Debian-Buster if I run python3 setup.py install in my package directory. I see the proprietary package get downloaded and installed.
Installation fails if I run pip3 install . also tox (version 3.24.4) fails similarly. In both cases, pip shows a message "Looking in indexes" then fails with "ERROR: Could not find a version that satisfies the requirement".
Solution 2 - PEP 508
Studying SO answer pip ignores dependency_links in setup.py which states that dependency_links is deprecated, I started over, revised setup.py to have:
install_requires=[
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
Installation is successful in Debian-Buster if I run pip3 install . in my package directory. Pip shows a message "Looking in indexes" but still downloads and installs the proprietary package successfully.
Installation fails in Debian-Buster if I run python3 setup.py install in my package directory. I see these messages:
Searching for proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0
..
Reading https://pypi.org/simple/proppkg/
..
error: Could not find suitable distribution for Requirement.parse(...).
Tox also fails in this scenario as it installs dependencies.
Really speculating now, it almost seems like there's an ordering issue. Tox invokes pip like this:
python -m pip install --exists-action w .tox/.tmp/package/1/te-0.3.5.zip
In that output I see "Collecting proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0" as the first step. That install fails because it fails to import package requests. Then tox continues collecting other dependencies. Finally tox reports as its last step "Collecting requests" (and that succeeds). Do I have to worry about ordering of install steps?
I'm starting to think that maybe the proprietary package is broken. I verified that the prop package setup.py has requests in its install_requires entry. Not sure what else to check.
Workaround solution
My workaround is installing the proprietary package in the docker image as a separate step before I install my own package, just by running pip3 install https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz. The setup.py has the PEP508 URL in install_requires. Then pip and tox find the prop package in the pip cache, and work fine.
Please suggest what to try for the latest pip and tox, or if this is as good as it gets, thanks in advance.
Update - add setup.py
Here's a (slightly sanitized) version of my package's setup.py
from setuptools import setup, find_packages
def get_version():
"""
read version string
"""
version_globals = {}
with open("te/version.py") as fp:
exec(fp.read(), version_globals)
return version_globals['__version__']
setup(
name="te",
version=get_version(),
packages=find_packages(exclude=["tests.*", "tests"]),
author="My Name",
author_email="email#mycompany.com",
description="My Back-End Server",
entry_points={"console_scripts": [
"te-be=te.server:main"
]},
python_requires=">=3.7",
install_requires=["connexion[swagger-ui]",
"Flask",
"gevent",
"redis",
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
package_data={"te": ["openapi_te.yml"]},
include_package_data=True, # read MANIFEST.in
)
accordingly to my research the following should work:
from setuptools import setup
from setuptools import find_packages
...
REQUIRES_INSTALL = [
'spacy==2.3.2',
'tensorflow==1.14.0',
'Keras==2.2.4',
'keras-contrib#git+https://github.com/keras-team/keras-contrib.git#egg=keras-contrib',
'en-core-web-sm#https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.0/en_core_web_sm-2.3.0.tar.gz#egg=en-core-web-sm'
]
...
setup(
name=NAME,
version=VERSION,
description=DESCRIPTION,
install_requires=REQUIRES_INSTALL,
...
)
When building a wheel or egg, everything is fine: python setup.py bdist_wheel.
But when trying to install the package (whl or egg) with pip install -U dist/mypack-....whl.
I get:
ERROR: Could not find a version that satisfies the requirement keras-contrib (from mypack==0.3.5) (from versions: none)
ERROR: No matching distribution found for keras-contrib (from mypack==0.3.5)
...
ERROR: Could not find a version that satisfies the requirement en-core-web-sm (from mypack==0.3.5) (from versions: none)
ERROR: No matching distribution found for en-core-web-sm (from mypack==0.3.5)
I have tried to same via setup.cfg but still no luck.
As reference - all these dependency are working when installing them first from requirments.txt and then installing the wheel.
spacy==2.3.2
tensorflow==1.14.0
Keras==2.2.4
keras-contrib#git+https://github.com/keras-team/keras-contrib.git#egg=keras-contrib
en-core-web-sm#https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.0/en_core_web_sm-2.3.0.tar.gz#egg=en-core-web-sm
pip install -r requirements.txt
pip install -U dist/mypack-....whl
But this is not clean way, since a wheel should be self contained.
Thank you for any hint!
Environment
Python: 3.7.0
Pip: 20.2.4
setuptools: 50.3.2
Some time ago it was possible to define a single requirements.txt or similar containing both specs for PyPI packages and links to repositories and archives.
That demanded to parse the requirements.txt and split them into "requirements" and "dependencies", where "requirements" would contain definitions of PyPI packages and "dependencies" — links.
Setuptools has different args for setup() for these: install_requires and dependency_links.
And it really worked: one was able to define a requirements.txt and install a package both as python setup.py install and as pip install .. Moreover, it was possible to install just dependencies via pip install -r requirements.txt. All ways worked and allowed to have a single place to define all requirements including non-PyPI links.
However, support of dependency_links arg was dropped by pip since v19. And here's the weird part: it is not dropped by setuptools. But there's more.
As of today, pip:
Supports only install_requires.
Prefers PEP 508 notation for dependencies in definitions of packages (install_requires) and standalone requirements.txt or similar.
Aborts installation of packages, which contain links in their install_requires.
Your definitions of dependencies mix 2 notations: prefixes like keras-contrib# are from PEP 508 and #egg= parts are from setuptools links notation.
This is not an issue: pip will ignore "eggs" as names are already defined before #.
I believe the installation of the package via pip works fine, i.e.:
pip install .
However, the issues will arise if the package is installed via setuptools, i.e.:
python setup.py install
setuptools does not understand PEP 508 notation and ignores links in install_requires. As of today, to make setuptools following links, both install_requires and dependency_links have to be used, e.g.:
setup(
...
install_requires=[
...
"keras_contrib==2.0.8",
...
],
dependency_links=[
"https://github.com/keras-team/keras-contrib/tarball/master#egg=keras_contrib-2.0.8",
...
],
)
Here are several tricky points:
A single dependency is defined in 2 places: a package name in install_requires and a link in dependency_links to resolve the package dependency.
The link is not git+https://.../....git, but it's a link to an archive: https://.../tarball/....
Egg name is in snake_case, not in dash-case. While it's possible to use dash-case, this will not allow specifying the version.
Version in install_requires is delimited via == and in dependency_links — via -.
It's possible to omit the version. But the only viable use case for that is if the package is not present in PyPI and is rarely updated. If the package is present in PyPI, but an unpublished version is needed, then the version must be specified.
And here's the bummer: fixing links for setuptools will break pip, as PEP 508 does not allow to specify versions. Keeping keras-contrib==x.y.z # ... in install_requires will make pip to search for the package keras-contrib==x.y.z, where ==x.y.z is not a version, but a part of the name. At the same time, not specifying a version will make setuptools to grab the latest version available at PyPI, not at the link from dependency_links.
In your case neither keras-contrib nor en-core-web-sm are present at PyPI, so using keras_contrib#git+https://... + dependency_links without version specified might work.
Otherwise, stick to pip install . and avoid using python setup.py install if the package depends on links.
See also:
PEP 508
PEP508: why either version requirement or URL but not both?
How can I make setuptools install a package that's not on PyPI?
pip install dependency links
pip3 setup.py install_requires PEP 508 git URL for private repo
Why is dependency links in setup.py deprecated?
Changing PEP 508 URLs in setup.py doesn't reinstall the dependency
Updating remote links with new URLs for PEP508 functionality
Requirements using PEP 508 direct references ignore the URL
Suggest alternatives for --process-dependency-links
Un-deprecate --process-dependency-links until an alternative is implemented
Changes to the pip dependency resolver in 20.3 (2020)
Trivia: several issues on GitHub are still open and PEP 508 is still in Active state since 2015. Digging around source code would reveal that setuptools is a wrapper around Python's distutils. setuptools is not a part of Python's stdlib, but the docs of distutils imply stdlib docs will be removed after docs of setuptools will be updated. At the same time pip is already bundled with Python's installations as Python's module. And yet we have pipfiles, pipenv, poetry, conda, pipx, pip-tools, shiv, spack, and the rest. Looks a bit overwhelming.
This question already has answers here:
Pip install from pypi works, but from testpypi fails (cannot find requirements)
(2 answers)
Closed 2 years ago.
I created a Python package, uploaded it to TestPyPI, and try to install it to a new virtual environment. pip cannot install the required packages, but if I try to install the same packages with pip install -r requirements.txt, it works.
I created my setup.py file following the guidelines listed in Packaging Python Projects. I uploaded my package to TestPyPI and try to install it to a brand new virtual environment with Python 3.6.4 and pip 19.1.1.
Here is my install instruction: pip install -i https://test.pypi.org/simple/ my-package-name==0.0.1
Here is what I have in setuptools.setup call (only related stuff):
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent',
],
install_requires=[
"requests",
"jsonpickle",
"pandas",
"matplotlib",
"seaborn",
"Pillow"
]
It starts by successfull installing matplotlib, but I get the following error later:
ERROR: Could not find a version that satisfies the requirement jsonpickle (from my-package-name==0.0.1) (from versions: none)
ERROR: No matching distribution found for jsonpickle (from my-package-name==0.0.1)
I also tried with specifying package versions, but then I cannot install any packages.
As I have told, I can install the packages just fine by using pip install -r requirements.txt command.
This is an unfortunate (and known) downside to TestPyPI: The issue is that jsonpickle does not exist on TestPyPI, and by installing your package from there, you are telling pip to look for dependencies there as well.
Instead, you should publish to PyPI instead, and use a pre-release version so as not to pollute your versions. You can delete these pre-releases from the project later.
it's better if you create a requirements.txt file and add the the package you require in that file and in setup.py open that file get the package in list and add that in setup.py
eg
requirements.txt
requests
jsonpickle
pandas
matplotlib
seaborn
Pillow
and in setup.py
with open('requirements.txt') as f:
requirements = f.read().splitlines()
...
setup(
name='package name',
version='version',
install_requires=requirements,
...
)
why this way ? -> By this way, you only have to maintain requirements.txt and no duplication of same in setup.py
I ran into a problem when installing a package which depended on python-daemon. I ultimately traced it to the latest version of the package python-daemon (2.0.3) released yesterday. Testing in a virtual environment on an Ubuntu 14.04 machine and issuing the following commands:
(venv) $ pip list
argparse (1.2.1)
pip (1.5.6)
setuptools (3.6)
wsgiref (0.1.2)
(venv) $ pip install redis
... works fine ....
(venv) $ pip install python-daemon
...
snip
...
File "/home/pwj/.virtualenvs/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load
['__name__'])
ImportError: No module named version
(venv)02:15 PM tmp$ pip list
argparse (1.2.1)
lockfile (0.10.2)
pip (1.5.6)
python-daemon (2.0.3)
setuptools (3.6)
wsgiref (0.1.2)
So the install of python-daemon seemed to work but something affected pip or setuptools because other packages (celery, flask), I try to install with pip after this gives me the same traceback:
...
snip
...
File "/home/pwj/.virtualenvs/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load
['__name__'])
ImportError: No module named version
If I uninstall python-daemon with pip things again and packages that weren't installing now install fine. Has anyone else come across this or something similar with a different project? My solution was to pip install the previous version
(venv) $ pip install python-daemon==2.0.2
... works ...
but was wondering what might be causing such an error.
(This behaviour is corrected in python-daemon version 2.0.4 and later.)
There are two sides to this:
Setuptools assumes it is the centre of everything.
Version 2.0.3 of python-daemon doesn't take that into account.
A more detailed explanation: There is some complex code using Docutils involved in the python-daemon build process, that isn't needed after install and isn't part of the library code.
It's too complex to leave in the un-importable (and therefore not-unit-testable) setup.py, so that build code is shunted to a separate testable module, version (in the file version.py), which itself uses Docutils.
But then the setup.py has a circular dependency: How to import version, when Docutils isn't yet installed? How to use Setuptools to ensure Docutils is installed, when running setup.py to completion will need version? All the feasible solutions are ugly and confusing.
The approach taken in ‘python-daemon’ 2.0.3 is to declare Docutils required for setup, and declare a Setuptools entry point for the work that needs version. That way setup.py gets to install Docutils before any of the entry points that will use version.
But now we come to the first point, that Setuptools arrogates itself as the centre of everything. By declaring an entry point, setup.py has modified every Setuptools action thereafter, and every package will fail if it can't find the entry points. And, since most of them don't have version or the specified functions in that module, they crash Setuptools.
What is essentially a bug to be fixed, reveals a poorly-understood corner case in Setuptools. So I'm voting your question up.
There doesn't seem to be a good solution to this: having modules available for setup.py but ensuring requirements are met first. Setuptools assumes it is the only build system needed to satisfy all dependencies for everything, and when that assumption fails it's very difficult to get around.
Thanks to the Python Packaging Authority folks, and the distutils-sig forum, for explaining this to me.
It's a similar question to How can I make setuptools install a package that's not on PyPI? but not the same.
As I would like to use the forked version of some package, setuptools ignore the dependency link (as it has the same version number).
Is there a way to force using the link from the dependency_links? Or is the only way to change the version number in the forked repo?
requires = [
...
'pyScss==1.1.3'
...
dependencies = [
'https://github.com/nadavshatz/pyScss/zipball/master#egg=pyScss-1.1.3'
]
Update
Weird, apparently it works if this package is the only one in the required list, that is not installed yet. If there's another missing package it will download it from pypi.
I believe you can just use dependency_links as described in that question:
from setuptools import setup
setup(name = 'mypkg',
version = '0.0.1',
description = 'Foo',
author = 'bar',
author_email = 'bar#example.com',
install_requires = ['pyScss==1.1.3'],
dependency_links = [
'https://github.com/nadavshatz/pyScss/zipball/master#egg=pyScss-1.1.3'
]
)
Tested using python setup.py develop
You probably want to rename the egg to emphasize it's a fork http://www.python.org/dev/peps/pep-0386/
Outside of the setup.py you can enforce this locally using requirements.txt and pip. Whilst this won't make your package depend on the fork you can easily document it as the way to install.
$ cat requirements.txt
https://github.com/nadavshatz/pyScss/zipball/master#egg=pyScss-1.1.3
$ pip install -r requirements.txt
I ended up doing something very similar to the answer in stackoverflow.com/a/17442663/368102.
I need a requests-file github package that name-conflicts with a different requests-file package in PyPi. They both have a version 1.0, and the PyPi version has some higher versions.
The workaround in my ias_tools/setup.py looks like this:
setup(
...
install_requires=[
'requests-file<=99.99',
],
dependency_links=[
'https://github.com/jvantuyl/requests-file/archive/b0a7b34af6e287e07a96bc7e89bac3bc855323ae.zip#egg=requests-file-99.99'
]
)
In my case, I'm using pip so I also had to use --process-dependency-links:
% pip install --process-dependency-links ./ias_tools
You are using pip version 6.0.6, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Processing ./ias_tools
DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting requests-file<=99.99 (from ias-tools==0.1)
Downloading https://github.com/jvantuyl/requests-file/archive/b0a7b34af6e287e07a96bc7e89bac3bc855323ae.zip
Requirement already satisfied (use --upgrade to upgrade): requests>=1.1.0 in ./venv/lib/python2.7/site-packages (from requests-file<=99.99->ias-tools==0.1)
Installing collected packages: ias-tools, requests-file
Running setup.py install for ias-tools
Running setup.py install for requests-file
Successfully installed ias-tools-0.1 requests-file-1.0
I'm not too worried about the deprecation notice, as a pull request was submitted to pip to deprecate the deprecation (after a discussion about it).