We Keep Coding

Pip package version conflicts despite seemingly matching ranges

When using pip install -r requirements.txt, I get ERROR: Cannot install -r requirements.txt (line 3), [...] because these package versions have conflicting dependencies..
And further:
The conflict is caused by:
tensorflow 2.11.0 depends on protobuf<3.20 and >=3.9.2
tensorboard 2.11.0 depends on protobuf<4 and >=3.9.2
wandb 0.13.5 depends on protobuf!=4.0.*, !=4.21.0, <5 and >=3.12.0
I don't see any conflicts in these ranges - every version in [3.12.0, 3.20) should be fine. Can someone explain the problem?
Update: As a workaround, I removed all version restrictions and only specified the names of the libraries in the requirements.txt file. Now it works. But I still don't see a problem with the above ranges, so I'll leave the question open.

I would suggest that, rather than using a range of versions, use a specific version you know works. That way, there won't be any problems.
I think that one of the versions of the dependencies is incompatible with the main module, and since it is within the range of versions you ask for, pip tries to intall it and fails to do so since it is incompatible.
Also, pip normally handles dependencies automatically.

Pip wheel collects 2 versions of a package then pip install gets a conflict

We use a pipeline that first uses pip wheel to collect all the packages that are needed in the project and then it creates a docker image that calls to pip install on the collected wheels.
The issue I am encountering is that when calling pip wheel, pip is collecting 2 different versions of a package. This has started occurring once a new version of the package is available.
The project has a requirement for an internal library ecs-deployer==10.1.2 and that library has in turn a requirement in the form of: elb-listener>=3.2.1+25,<4
The relevant output of pip wheel with the verbose option says:
Collecting elb-listener>=3.2.1+25,<4
Created temporary directory: /tmp/pip-unpack-zr930807
File was already downloaded /home/user/path/dist/elb_listener-3.2.2+26-py3-none-any.whl
Added elb-listener>=3.2.1+25,<4 from https://internal-repository.com/path/elb_listener/3.2.2%2B26/elb_listener-3.2.2%2B26-py3-none-any.whl#md5=foo (from ecs-deployer==10.1.2->service==1.0.0) to build tracker '/tmp/pip-req-tracker-1tz9t5ls'
Removed elb-listener>=3.2.1+25,<4 from https://internal-repository.com/path/elb_listener/3.2.2%2B26/elb_listener-3.2.2%2B26-py3-none-any.whl#md5=blabla (from ecs-deployer==10.1.2->service==1.0.0) to build tracker '/tmp/pip-req-tracker-1tz9t5ls'
And also:
Collecting elb-listener>=3.2.1+25,<4
Created temporary directory: /tmp/pip-unpack-yfnxim_u
File was already downloaded /home/user/path/dist/elb_listener-3.2.3+27-py3-none-any.whl
Added elb-listener>=3.2.1+25,<4 from https://internal-repository.com/path/elb_listener/3.2.3%2B27/elb_listener-3.2.3%2B27-py3-none-any.whl#md5=bar (from ecs-deployer==10.1.2->service==1.0.0) to build tracker '/tmp/pip-req-tracker-1tz9t5ls'
Then when the pip install is called I get this:
ERROR: Cannot install elb-listener 3.2.2+26 (from /opt/elb_listener-3.2.2+26-py3-none-any.whl) and cad-aws-elb-listener-target-group-builder 3.2.3+27 (from /opt/elb_listener-3.2.3+27-py3-none-any.whl) because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested elb-listener 3.2.2+26 (from /opt/elb_listener-3.2.2+26-py3-none-any.whl)
The user requested elb-listener 3.2.3+27 (from /opt/elb_listener-3.2.3+27-py3-none-any.whl)
We use pip 20.2.3 with the option --use-feature=2020-resolver
Is it normal that pip wheel collects several versions of the same package?
If so, can I indicate in any way to either pip wheel to only collect one of the versions or to pip install to only use the latest version?
If not, is there any way to solve this problem? I guess changing the requirement to elb-listener>=3.2.1+27,<4 would solve it, but we don't have direct access to that library and it would take a while for other team to change it.

As per #sinoroc comment, upgrading the python to 3.10 and pip version to 21.2.4 solved this particular issue.

As far as I understood, "local version identifiers" such as 3.2.1+25 are far from usual, apparently they are not meant to be used anywhere public (like PyPI), and that might be the reason for all the trouble here. I am really not sure how well they are supported by Python packaging tools and maybe they confuse the dependency resolution.
Local version identifiers SHOULD NOT be used when publishing upstream projects to a public index server, but MAY be used to identify private builds created directly from the project source. Local version identifiers SHOULD be used by downstream projects when releasing a version that is API compatible with the version of the upstream project identified by the public version identifier, but contains additional changes (such as bug fixes). As the Python Package Index is intended solely for indexing and hosting upstream projects, it MUST NOT allow the use of local version identifiers.
-- "Local version identifiers" section of _PEP 440

Error when installing django on windows using `pip install django`

I get an error when installing django using pip command
ERROR: After October 2020 you may experience errors when installing or updating
packages. This is because pip will change the way that it resolves dependency conflicts.
We recommend you use --use-feature=2020-resolver to test your packages with the
new resolver before it becomes the default.
drf-yasg 1.17.0 requires six>=1.10.0, but you'll have six 1.9.0 which is incompatible.
can anyone help?
Thanks in advance

According to this announcement, pip will introduce a new dependency resolver in October 2020, which will be more robust but might break some existing setups. Therefore they are suggesting users to try running their pip install scripts at least once (in dev mode) with this option: --use-feature=2020-resolver to anticipate any potential issue before the new resolver becomes the default in October 2020 with pip version 20.3.
On behalf of the PyPA, I am pleased to announce that we have just
released pip 20.2, a new version of pip. You can install it by running
python -m pip install --upgrade pip.
The highlights for this release are:
The beta of the next-generation dependency resolver is available
Faster installations from wheel files Improved handling of wheels
containing non-ASCII file contents Faster pip list using parallelized
network operations Installed packages now contain metadata about
whether they were directly requested by the user (PEP 376’s REQUESTED
file) The new dependency resolver is off by default because it is not
yet ready for everyday use. The new dependency resolver is
significantly stricter and more consistent when it receives
incompatible instructions, and reduces support for certain kinds of
constraints files, so some workarounds and workflows may break. Please
test it with the --use-feature=2020-resolver flag. Please see our
guide on how to test and migrate, and how to report issues. We are
preparing to change the default dependency resolution behaviour and
make the new resolver the default in pip 20.3 (in October 2020).

How to resolve conflicts between dependencies in Plone?

I am writing an add-on for Plone 5.1. I have added dependency to my add-on's setup.py install_requires parameter. I have also added my add-on (esperonus.vimeo) to Plone's buildout.cfg. However, upon running buildout, I get this error:
ryselis#ryselis-All-Series:~/Plone/zinstance$ bin/buildout -c develop.cfg
mr.developer: Queued 'esperonus.vimeo_plone' for checkout.
mr.developer: Filesystem package 'esperonus.vimeo_plone' doesn't need a checkout.
Develop: '/home/ryselis/Plone/zinstance/src/esperonus.vimeo_plone'
Updating _mr.developer.
Installing instance.
Version and requirements information containing certifi:
[versions] constraint on certifi: 2017.11.5
Requirement of tuspy==0.2.1: certifi==2017.7.27.1
Requirement of requests>=2.4.0: certifi>=2017.4.17
While:
Installing instance.
Error: The requirement ('certifi==2017.7.27.1') is not allowed by your [versions] constraint (2017.11.5)
It looks like VimeoPy requires different version of certifi than tuspy (I guess this is Plone's dependency?). Is there anything I could do to add VimeoPy to my add-on?

This usually means that you have pinned version in your buildout configuration which are di8ffer from what a packages want's to install. You can remove the pinnings from the buildout configuration accept you have a reason for it.
New buildout versions usually have an option called "update-versions-file" which points to the buildout config file with the list of versions. Buildout will add all picked versions, which where not pinned and add them for you, so that you can rebuild this configuration later with exact the same versions.
But these pinnings are save to remove, especially if you have packages requiring diffent versions.

Installing via `setup.py develop` fails - pip works

My python package footools needs html5lib via install_requires in setup.py.
setup.py develop fails
Installing via setup.py develop fails:
cd src/footools/
python setup.py develop
Processing dependencies for footools==2016.205
Searching for html5lib==0.9999999
Reading https://source.example.com/pypi/simple/html5lib/
Download error on https://source.example.com/pypi/simple/html5lib/:
[Errno 185090050] _ssl.c:354: error:0B084002:x509
certificate routines:X509_load_cert_crl_file:system lib --
Some packages may not be found!
Couldn't find index page for 'html5lib' (maybe misspelled?)
pip works
But direct download works:
bar#workdevel123:~/src/footools> pip install html5lib==0.9999999
/home/bar/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79:
InsecurePlatformWarning: A true SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately
and may cause certain SSL connections to fail.
For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Collecting html5lib==0.9999999
/home/bar/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79:
InsecurePlatformWarning: A true SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately and
may cause certain SSL connections to fail.
For more information,
see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Downloading https://source.example.com/pypi/packages/html5lib-0.9999999.tar.gz
Requirement already satisfied (use --upgrade to upgrade):
six in /usr/lib/python2.7/site-packages (from html5lib==0.9999999)
Installing collected packages: html5lib
Running setup.py install for html5lib
Successfully installed html5lib-0.9999999
Questions
What is the difference between these two methods?
Why are they different?
What is the correct way to install a dependency in python?
setup.py
The setup.py is not special:
import setuptools
setuptools.setup(
name='foo',
version='2016.210',
long_description=open('README.txt').read(),
packages=setuptools.find_packages(),
install_requires=[
# about twenty packages before this line
'html5lib==0.9999999'
],
include_package_data=True,
entry_points={
'console_scripts': [
'foo=foo.utils.bar:main',
],
},
)

python setup.py develop, or setuptools in general uses easy_install to satisfy dependencies which in turn uses urllib2 whereas pip uses requests. See here for easy_install vs pip.
pip is more modern and among other things has capability to uninstall packages and complies with PEP 438 -- Transitioning to release-file hosting on PyPI. You can achieve the same thing as python setup.py develop with pip install -e src/footools/, note if the project path is in the current directory use, ./footools.
The requests package bundles CA certs in the package itself, python -c 'import pip;print(pip.download.requests.certs.where())'.
setuptools uses system installed CA certs python -c 'from setuptools import ssl_support;print(ssl_support.cert_paths)'.
You have to update system installed CA certs using tools like update-ca-certificates for Ubuntu to either update CA certs automatically or download from https://curl.haxx.se/docs/caextract.html and install into one of the paths shown by setuptools or set setuptools.ssl_support.cert_paths to an empty sequence like [] and do pip install certifi.
Calling setuptools.ssl_support.find_ca_bundle() will reveal the location of CA certs.

setuptools is a collection of enhancements to the Python distutils (for Python 2.6 and up) that allow developers to more easily build and distribute Python packages, especially ones that have dependencies on other packages.
So, among other stuff, you can create packages that can be upload to Pypi, and later installed using pip (therefore distibuting your module).
That said, they actually should not be that different in the installation part. You are running the develop mode, so maybe you have to fiddle a bit with the directories or fix the authorization error.
In the Development Mode the project is deployed into a staging area (in some way similar to the process of a virtual environment)
deployment is done in such a way that changes to the project source are immediately available in the staging area(s), without needing to run a build or install step after each change.
Meaning also everything will be available for that python interpreter. It can be later unstaged.
I noticed that html5lib is being fetched from different places: /pypi/simple/in one case and /pypi/packages/ in the other.
dependency_links
A list of strings naming URLs to be searched when satisfying dependencies. These links will be used if needed to install packages specified by setup_requires or tests_require.
Back to the problem I think it is most probably the ssl issue, since in pip it handled graciously (i.e., nice warning and there is some kind of workaround), but the same does not happen with setuptools. If there is an error in the request that is not handled then Couldn't find index page for 'html5lib' could be because of that.

What is the difference between these two methods?
Nothing terribly important to you as a user, aside from their different user interfaces. They are two stops on the scenic historical train ride of Python package management. There were others along the way.
Why are they different?
Back in the day, Python didn't ship with a package management system. Third party solutions filled the void. They were designed by different people at different moments in time. If you look at other programming languages, you sometimes see similar stories; sometimes you see happier stories; sometimes more tragic.
What is the correct way to install a dependency in python?
Both of these methods are technically correct. Pip is the more modern method, and in my experience it is both more popular and handier to work with. As of Python 3.4 and up, Pip has been included in the CPython distribution and is officially 'preferred'. So you can see which way the wind is blowing.