I'm using pip with virtualenv to package and install some Python libraries.
I'd imagine what I'm doing is a pretty common scenario. I'm the maintainer on several libraries for which I can specify the dependencies explicitly. A few of my libraries are dependent on third party libraries that have transitive dependencies over which I have no control.
What I'm trying to achieve is for a pip install on one of my libraries to download/install all of its upstream dependencies. What I'm struggling with in the pip documentation is if/how requirements files can do this on their own or if they're really just a supplement to using install_requires.
Would I use install_requires in all of my libraries to specify dependencies and version ranges and then only use a requirements file to resolve a conflict and/or freeze them for a production build?
Let's pretend I live in an imaginary world (I know, I know) and my upstream dependencies are straightforward and guaranteed to never conflict or break backward compatibility. Would I be compelled to use a pip requirements file at all or just let pip/setuptools/distribute install everything based on install_requires?
There are a lot of similar questions on here, but I couldn't find any that were as basic as when to use one or the other or using them both together harmoniously.
My philosophy is that install_requires should indicate a minimum of what you need. It might include version requirements if you know that some versions will not work; but it shouldn't have version requirements where you aren't sure (e.g., you aren't sure if a future release of a dependency will break your library or not).
Requirements files on the other hand should indicate what you know does work, and may include optional dependencies that you recommend. For example you might use SQLAlchemy but suggest MySQL, and so put MySQLdb in the requirements file).
So, in summary: install_requires is to keep people away from things that you know don't work, while requirements files to lead people towards things you know do work. One reason for this is that install_requires requirements are always checked, and cannot be disabled without actually changing the package metadata. So you can't easily try a new combination. Requirements files are only checked at install time.
here's what I put in my setup.py:
# this grabs the requirements from requirements.txt
REQUIREMENTS = [i.strip() for i in open("requirements.txt").readlines()]
setup(
.....
install_requires=REQUIREMENTS
)
The Python Packaging User Guide has a page about this topic, I highly recommend you read it:
install_requires vs Requirements files
Summary:
install_requires is there to list the dependencies of the package that absolutely must be installed for the package to work. It is not meant to pin the dependencies to specific versions, but ranges are accepted, for example install_requires=['django>=1.8']. install_requires is observed by pip install name-on-pypi and other tools.
requirements.txt is just a text file, that you can choose to run pip install -r requirements.txt against. It's meant to have versions of all dependencies and subdependencies pinned, like this: django==1.8.1. You can create one using pip freeze > requirements.txt. (Some services, like Heroku, automatically run pip install -r requirements.txt for you.) pip install name-on-pypi does not look at requirements.txt, only at install_requires.
I only ever use a setup.py and install_requires because there is only one place to look at. It is just as powerful as having a requirements file and there is no duplication to maintain.
Related
I have dependency_links in my setup.py:
...
dependency_links = ['http://github.com/robot-republic/python-s3/tarball/master.tar.gz#egg=python-s3'],
...
But it doesn't work. However install_requires works fine.
Maybe there are another method to set up git repo as required for setup.py?
This answer should help. In a nutshell, you need to specify the version (or "dev") for the #egg=python-s3 so it looks like #egg=python-s3-1.0.0.
Updates based on #Cerin's comment:
Pip 1.5.x has a flag to enable dependency-links processing: --process-dependency-links. I haven't tested it because I agree with the point below.
This discussion seems to indicate that using dependency-links for pip is a bad practice. Although this feature was enlisted for deprecation, it's not anymore. There's a valid use case for private packages.
since pip version 18.1 PEP 508 URL is supported. That means you don't need the deprecated dependency_links anymore.
You write the dependency directly in the install_requires list instead.
The example from #Chad looks like this:
setup(
name='yourpackage',
version='1.7.5',
packages=[],
url='',
license='',
author='',
author_email='',
description='',
install_requires=[
'somepackage==1.2.0',
'repo # https://github.com/user/archive/master.zip#egg=repo-1.0.0',
'anotherpackage==4.2.1'
],
)
To install your package you can simply write:
pip install yourpackage
(without --process-dependency-links)
I realize this is an old question, but, just in case you find yourself here like I did, this is what worked for me.
I've got a package on GitHub (not registered with pypi) that relies on other GitHub (non-pypi) packages. I spent an inordinate amount of time trying to figure out how to get pip to handle this correctly. I will include what I did to fix it here.
Putting dependencies in a requirements.txt file is the preferred method of listing dependencies. However, you also need to populate install_requires in setup. It was at this stage that I ran into a roadblock with pip not wanting to install dependencies from GitHub.
Most places, including answers to this question, tell you to populate the dependency_links section of setup. However, you also need to populate the install_requires field with the name of the package referenced in dependency_links.
For example, if your requirements.txt contains the following.
somepackage==1.2.0
https://github.com/user/repo/tarball/master#egg=repo-1.0.0
anotherpackage==4.2.1
Then, your setup call should look like this:
setup(
name='yourpackage',
version='1.7.5',
packages=[],
url='',
license='',
author='',
author_email='',
description='',
install_requires=[
'somepackage==1.2.0',
'repo==1.0.0',
'anotherpackage==4.2.1'
],
dependency_links=[
'https://github.com/user/repo/tarball/master#egg=repo-1.0.0'
]
)
Ok, so now we've got our package configured; installing it is the next task. This is where I spent a lot of time. I could not figure out why specifying dependency_links apparently did nothing. The trick is that in some cases, you need to set the allow-all-external (can be more specific) flag for pip. For example:
pip install git+https://github.com/user/anotherrepo.git
--process-dependency-links --allow-all-external
You're done and it works!
DISCLAIMER: dependency_links and the flags process-dependency-links and allow-all-external are deprecated, so they will be removed soon. In the time I spent, I could not locate a better, prefered method and still have pip function properly.
A couple of notes on some issues I found, in particular for installing from private repos.
Installing from pip & setuptools have some subtle differences; but this way should work for both.
from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']
setup(
# ...
install_requires='package',
dependency_links = [
'git+https://{github_token}#github.com/user/{package}.git/#{version}#egg={package}-0'
.format(github_token=github_token, package=package, version=master)
]
A couple of notes here:
For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
You need to include some version number (here is 0) at the end of the link, even if there's no package on PyPI. This has to be a actual number, not a word.
You need to preface with git+ to tell setuptools it's to clone the repo, rather than pointing at a zip / tarball
version can be a branch, a tag, or a commit hash
You need to supply --process-dependency-links if installing from pip
First upgrade your pip version as this is a new syntax.
pip install pip --upgrade
Then do the following:
install_requires=[
'bleualign-git # https://github.com/rsennrich/Bleualign/archive/<commit-hash or branch-name>.zip#egg=bleualign-git-1.0.0'
]
Version 1.0.0 and name "bluealign-git" are randomly chosen.
The version number is necessary.
We advisedly used the name "bleualign-git" to distinguish it from the main repository version.
Hope this helps.
Comments:
Good answer (upvoted). Also want to mention that the actual result would likely depend on the platform and/or pip version, though. I've seen when it works, or seemingly works but did not really pull the dependency from the specified link, or simply being rejected. So I would probably just use this as a short time workaround only. Stick with the mainstream way whenever possible.
None of the above solutions worked for me in their respective explicit forms. I am adding this as another solution for certain specific cases. OP has a tarball in their remote repo, but I ended up here for a related case: only source code in the remote repo.
I have a dependency that is a remote private repository on Azure DevOps. I would like to use pip to install a local python package and also automatically install the remote dependency. The remote repository is a python package with a setup.py and some source code. The repo doesn't have build artifacts like eggs/wheels/tarballs, just .py files. I am able to manually install the remote dependency with pip install git+https://..., hence I suspected correctly that the remote repository doesn't need to contain eggs/wheels/tarballs.
Here is the relevant line of my setup.py file
setup.py
...
install_requires=["packagename # git+https://dev.azure.com/.../_git/packagename"]
I started working with Python. I've added requirements.txt and setup.py to my project. But, I am still confused about the purpose of both files. I have read that setup.py is designed for redistributable things and that requirements.txt is designed for non-redistributable things. But I am not certain this is accurate.
How are those two files truly intended to be used?
requirements.txt:
This helps you to set up your development environment.
Programs like pip can be used to install all packages listed in the file in one fell swoop. After that you can start developing your python script. Especially useful if you plan to have others contribute to the development or use virtual environments.
This is how you use it:
pip install -r requirements.txt
It can be produced easily by pip itself:
pip freeze > requirements.txt
pip automatically tries to only add packages that are not installed by default, so the produced file is pretty minimal.
setup.py:
This helps you to create packages that you can redistribute.
The setup.py script is meant to install your package on the end user's system, not to prepare the development environment as pip install -r requirements.txt does. See this answer for more details on setup.py.
The dependencies of your project are listed in both files.
The short answer is that requirements.txt is for listing package requirements only. setup.py on the other hand is more like an installation script. If you don't plan on installing the python code, typically you would only need requirements.txt.
The file setup.py describes, in addition to the package dependencies, the set of files and modules that should be packaged (or compiled, in the case of native modules (i.e., written in C)), and metadata to add to the python package listings (e.g. package name, package version, package description, author, ...).
Because both files list dependencies, this can lead to a bit of duplication. Read below for details.
requirements.txt
This file lists python package requirements. It is a plain text file (optionally with comments) that lists the package dependencies of your python project (one per line). It does not describe the way in which your python package is installed. You would generally consume the requirements file with pip install -r requirements.txt.
The filename of the text file is arbitrary, but is often requirements.txt by convention. When exploring source code repositories of other python packages, you might stumble on other names, such as dev-dependencies.txt or dependencies-dev.txt. Those serve the same purpose as dependencies.txt but generally list additional dependencies of interest to developers of the particular package, namely for testing the source code (e.g. pytest, pylint, etc.) before release. Users of the package generally wouldn't need the entire set of developer dependencies to run the package.
If multiplerequirements-X.txt variants are present, then usually one will list runtime dependencies, and the other build-time, or test dependencies. Some projects also cascade their requirements file, i.e. when one requirements file includes another file (example). Doing so can reduce repetition.
setup.py
This is a python script which uses the setuptools module to define a python package (name, files included, package metadata, and installation). It will, like requirements.txt, also list runtime dependencies of the package. Setuptools is the de-facto way to build and install python packages, but it has its shortcomings, which over time have sprouted the development of new "meta-package managers", like pip. Example shortcomings of setuptools are its inability to install multiple versions of the same package, and lack of an uninstall command.
When a python user does pip install ./pkgdir_my_module (or pip install my-module), pip will run setup.py in the given directory (or module). Similarly, any module which has a setup.py can be pip-installed, e.g. by running pip install . from the same folder.
Do I really need both?
Short answer is no, but it's nice to have both. They achieve different purposes, but they can both be used to list your dependencies.
There is one trick you may consider to avoid duplicating your list of dependencies between requirements.txt and setup.py. If you have written a fully working setup.py for your package already, and your dependencies are mostly external, you could consider having a simple requirements.txt with only the following:
# requirements.txt
#
# installs dependencies from ./setup.py, and the package itself,
# in editable mode
-e .
# (the -e above is optional). you could also just install the package
# normally with just the line below (after uncommenting)
# .
The -e is a special pip install option which installs the given package in editable mode. When pip -r requirements.txt is run on this file, pip will install your dependencies via the list in ./setup.py. The editable option will place a symlink in your install directory (instead of an egg or archived copy). It allows developers to edit code in place from the repository without reinstalling.
You can also take advantage of what's called "setuptools extras" when you have both files in your package repository. You can define optional packages in setup.py under a custom category, and install those packages from just that category with pip:
# setup.py
from setuptools import setup
setup(
name="FOO"
...
extras_require = {
'dev': ['pylint'],
'build': ['requests']
}
...
)
and then, in the requirements file:
# install packages in the [build] category, from setup.py
# (path/to/mypkg is the directory where setup.py is)
-e path/to/mypkg[build]
This would keep all your dependency lists inside setup.py.
Note: You would normally execute pip and setup.py from a sandbox, such as those created with the program virtualenv. This will avoid installing python packages outside the context of your project's development environment.
For the sake of completeness, here is how I see it in 3 4 different angles.
Their design purposes are different
This is the precise description quoted from the official documentation (emphasis mine):
Whereas install_requires (in setup.py) defines the dependencies for a single project, Requirements Files are often used to define the requirements for a complete Python environment.
Whereas install_requires requirements are minimal, requirements files often contain an exhaustive listing of pinned versions for the purpose of achieving repeatable installations of a complete environment.
But it might still not easy to be understood, so in next section, there come 2 factual examples to demonstrate how the 2 approaches are supposed to be used, differently.
Their actual usages are therefore (supposed to be) different
If your project foo is going to be released as a standalone library (meaning, others would probably do import foo), then you (and your downstream users) would want to have a flexible declaration of dependency, so that your library would not (and it must not) be "picky" about what exact version of YOUR dependencies should be. So, typically, your setup.py would contain lines like this:
install_requires=[
'A>=1,<2',
'B>=2'
]
If you just want to somehow "document" or "pin" your EXACT current environment for your application bar, meaning, you or your users would like to use your application bar as-is, i.e. running python bar.py, you may want to freeze your environment so that it would always behave the same. In such case, your requirements file would look like this:
A==1.2.3
B==2.3.4
# It could even contain some dependencies NOT strickly required by your library
pylint==3.4.5
In reality, which one do I use?
If you are developing an application bar which will be used by python bar.py, even if that is "just script for fun", you are still recommended to use requirements.txt because, who knows, next week (which happens to be Christmas) you would receive a new computer as a gift, so you would need to setup your exact environment there again.
If you are developing a library foo which will be used by import foo, you have to prepare a setup.py. Period.
But you may still choose to also provide a requirements.txt at the same time, which can:
(a) either be in the A==1.2.3 style (as explained in #2 above);
(b) or just contain a magical single .
.
The latter is essentially using the conventional requirements.txt habit to document your installation step is pip install ., which means to "install the requirements based on setup.py" while without duplication. Personally I consider this last approach kind of blurs the line, adds to the confusion, but it is nonetheless a convenient way to explicitly opt out for dependency pinning when running in a CI environment. The trick was derived from an approach mentioned by Python packaging maintainer Donald in his blog post.
Different lower bounds.
Assuming there is an existing engine library with this history:
engine 1.1.0 Use steam
...
engine 1.2.0 Internal combustion is invented
engine 1.2.1 Fix engine leaking oil
engine 1.2.2 Fix engine overheat
engine 1.2.3 Fix occasional engine stalling
engine 2.0.0 Introducing nuclear reactor
You follow the above 3 criteria and correctly decided that your new library hybrid-engine would use a setup.py to declare its dependency engine>=1.2.0,<2, and then your separated application reliable-car would use requirements.txt to declare its dependency engine>=1.2.3,<2 (or you may want to just pin engine==1.2.3). As you see, your choice for their lower bound number are still subtly different, and neither of them uses the latest engine==2.0.0. And here is why.
hybrid-engine depends on engine>=1.2.0 because, the needed add_fuel() API was first introduced in engine 1.2.0, and that capability is the necessity of hybrid-engine, regardless of whether there might be some (minor) bugs inside such version and been fixed in subsequent versions 1.2.1, 1.2.2 and 1.2.3.
reliable-car depends on engine>=1.2.3 because that is the earliest version WITHOUT known issues, so far. Sure there are new capabilities in later versions, i.e. "nuclear reactor" introduced in engine 2.0.0, but they are not necessarily desirable for project reliable-car. (Your yet another new project time-machine would likely use engine>=2.0.0, but that is a different topic, though.)
TL;DR
requirements.txt lists concrete dependencies
setup.py lists abstract dependencies
A common misunderstanding with respect to dependency management in Python is whether you need to use a requirements.txt or setup.py file in order to handle dependencies.
The chances are you may have to use both in order to ensure that dependencies are handled appropriately in your Python project.
The requirements.txt file is supposed to list the concrete dependencies. In other words, it should list pinned dependencies (using the == specifier). This file will then be used in order to create a working virtual environment that will have all the dependencies installed, with the specified versions.
On the other hand, the setup.py file should list the abstract dependencies. This means that it should list the minimal dependencies for running the project. Apart from dependency management though, this file also serves the package distribution (say on PyPI).
For a more comprehensive read, you can read the article requirements.txt vs setup.py in Python on TDS.
Now going forward and as of PEP-517 and PEP-518, you may have to use a pyproject.toml in order to specify that you want to use setuptools as the build-tool and an additional setup.cfg file to specify the details.
For more details you can read the article setup.py vs setup.cfg in Python.
I need to include Python packages available via public Github repositories along with my Python (2.7) package. My package should be installable via pip using setup.py.
So far, this could be done using dependency_links in the setup.py file:
setuptools.setup(
name="my_package",
version="1.0",
install_requires=[
"other_package==1.2"
],
dependency_links=[
"https://github.com/user/other_package/tarball/master#egg=other_package-1.2"
]
)
This still works when the package gets installed with the --process-dependency-links flag, but the dependency_links functionality seems to be deprecated, since:
pip install git+https://github.com/user/my_package#master#egg=my_package-1.0 --process-dependency-links
gives me the following warning:
DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Is there an alternative way to include git dependencies in the setup.py file with support for pip installation?
Edit (10/17/2016) to clarify my use case:
Let's say I find a bug in other_package. I fork the respective repo on Github, fix the bug and make a pull request. My pull request doesn't get immediately accepted (or never will be because the package is no longer actively maintained). I would like to distribute my_package together with my fork of other_package and want users to be able to pip install my_package without any further knowledge about the details of this requirement and without having to provide any additional flags upon installation. Users of my_package should further be able to include my_package as a requirement in their own custom packages.
How can this be achieved bearing compatibly with different modes of installation (wheels, eggs, develop, ...) in mind?
I ran into this exact issue (found a bug in someone else's project that mine depended on, made a pull request but didn't have time to wait for them to merge).
I solved it by adding this line to install_requires:
'my-package # https://github.com/user/my-package/archive/master.tar.gz'
Personally, I would avoid including git repositories as dependencies. In the scenarios you describe, I see two options.
Where Package is Unmaintained
If a package is unmaintained, you can either fork the project and distribute your own version, or you can distribute the forked code as as a submodule of your own code (i.e. include the external dependency directly in your distributable package)
Personally I prefer distributing my own version.
Where the package has yet to include your bugfix
In this case, I would distribute the fixed code as a part of your package until such a time as the bug is fixed.
I have a private library called some-library (actual names have been changed) with a setup file looking somewhat like this:
setup(
name='some-library',
// Omitted some less important stuff here...
install_requires=[
'some-git-dependency',
'another-git-dependency',
],
dependency_links=[
'git+ssh://git#github.com/my-organization/some-git-dependency.git#egg=some-git-dependency',
'git+ssh://git#github.com/my-organization/another-git-dependency.git#egg=another-git-dependency',
],
)
All of these Git dependencies may be private, so installation via HTTP is not an option. I can use python setup.py install and python setup.py develop in some-library's root directory without problems.
However, installing over Git doesn't work:
pip install -vvv -e 'git+ssh://git#github.com/my-organization/some-library.git#1.4.4#egg=some-library'
The command fails when it looks for some-git-dependency, mistakenly assumes it needs to get the dependency from PyPI and then fails after concluding it's not on PyPI. My first guess was to try re-running the command with --process-dependency-links, but then this happened:
Cannot look at git URL git+ssh://git#github.com/my-organization/some-git-dependency.git#egg=some-git-dependency
Could not find a version that satisfies the requirement some-git-dependency (from some-library) (from versions: )
Why is it producing this vague error? What's the proper way to pip install a package with Git dependencies that might be private?
What's the proper way to pip install a package with Git dependencies that might be private?
Two options
Use dependency_links as you do. See below for details.
Along side the dependency_links in your setup.py's, use a special dependency-links.txt that collects all the required packages. Then add this package in requirements.txt. That's my recommendend option as explained below.
# dependency-links.txt
git+ssh://...#tag#egg=package-name1
git+ssh://...#tag#egg=package-name2
# requirements.txt (per deployed application)
-r dependency-links.txt
While option 2 adds some extra burden on package management, namely keeping dependency-links.txt up to date, it makes installing packages a lot easier because you can' forget to add the --process-dependency-link option on pip install.
Perhaps more importantly, using dependency-links.txt you get to specify the exact version to be installed on deployment, which is want you want in a CI/CD environment - nothing is more risky than to install some version. From a package maintainer's perspective however it is common and considered good practice to specify a minimum version, such as
# setup.py in a package
...
install_requires = [ 'foo>1.0', ... ]
That's great because it makes your packages work nicely with other packages that have similar dependencies yet possibly on different versions. However, in a deployed application this can still cause mayhem if there are conflicting requirements among packages. E.g. package A is ok with foo>1.0, package B wants foo<=1.5 and the most recent version is foo==2.0. Using dependency-links.txt you can be specific, applying one version for all packages:
# dependency-links.txt
foo==1.5
The command fails when it looks for some-git-dependency,
To make it work, you need to add --process-dependency-links for pip to recognize the dependency to github, e.g.
pip install --process-dependency-links -r private-requirements.txt
Note since pip 8.1.0 you can add this option to requirements.txt. On the downside it gets applied to all packages installed and may have unintended consequences. That said, I find using dependency-links.txt is a safer and more manageable solution.
All of these Git dependencies may be private
There are three options:
Add collaborators on each of the required packages' repositories. These collaborators need to have their ssh keys setup with github for this to work. Then use git+ssh://...
Add a deploy key to each of the repositories. The downside here is that you need to distribute the corresponding private key to all the machines that need to deploy. Again use git+ssh://...
Add a personal access token on the github account that holds the private repositories. Then you can use git+https://accesstoken#github.com/... The downside is that the access token will have read + write access to all repositories, public and private, on the respective github account. On the plus side distributing and managing per-repository private keys is no longer necessary, and cycling the key is a lot simpler. In an all-inhouse environment where every dev has access to all repositories I have found this to be the most efficient, hassle-free way for everybody. YMMV
This should work for private repositories as well:
dependency_links = [
'git+ssh://git#github.com/my-organization/some-git-dependency.git#master#egg=some-git-dependency',
'git+ssh://git#github.com/my-organization/another-git-dependency.git#master#egg=another-git-dependency'
],
You should use git+git when url with #egg, like this:
-e git+git#repo.some.la:foo/my-repo.git#egg=my-repo
Use git+ssh in production without #egg, but you can specify #version or branch #master
git+ssh://git#repo.some.la/foo/my-repo.git#1.1.6
for work with app versions use git tagging Git Basics - Tagging
If I refer to "pip install dependency links", you would not refer to the GitHub repo itself, but to the tarball image associated to that GitHub repo:
dependency_links=[
'git+ssh://git#github.com/my-organization/some-git-dependency/tarball/master/#egg=some-git-dependency',
'git+ssh://git#github.com/my-organization/another-git-dependency/tarball/master/#egg=another-git-dependency',
],
with "some-git-dependency" being the name *and version of the dependency.
"Cannot look at git URL git+ssh://git#github.com/my-organization/some-git-dependency.git#egg=some-git-dependency" means pip cannot fetch an html page from this url to look for direct download links in the page, i.e, pip doesn't recognize the URL as a vcs checkout, because maybe some discrepancy between the requirement specifier and the fragment part in the vcs url.
In the case of a VCS checkout, you should also append #egg=project-version in order to identify for what package that checkout should be used.
Be sure to escape any dashes in the name or version by replacing them with underscores.
Check Dependencies that aren’t in PyPI
replace - with an _ in the package and version string.
git+ssh://git#github.com/my-organization/some-git-dependency.git#egg=some_git_dependency
and --allow-all-external may be useful.
I was reading the docs and couldn't wrap my head around it:
Constraints files are requirements files that only control which
version of a requirement is installed, not whether it is installed or
not. Their syntax and contents is nearly identical to Requirements
Files.
There is one key difference: Including a package in a constraints file
does not trigger installation of the package.
So does that mean I need to requirement files first then run constraint?
Need to have both requirements.txt and constraints.txt?
Or just -c requirements.txt?
Could someone please explain in plain English that what the new PIP feature: Constraints Files does?
Update: pip 20.3, released Nov 30, 2020, introduced a new resolver that fixes some design issues. It also reimplements the constraints feature. It is now difficult or impossible to use the feature the way I describe below. I do not understand the new constraints implementation, and I no longer use it. I've had success with pip-compile from pip-tools. I specify top-level requirements in requirements.in, and pip-compile generates a requirements.txt with specific versions an package hashes. See requirements.in and requirements.txt in the ichnaea project for a working example.
Original answer below, for pip 20.2 and previous
I think a constraints file is a good way to keep your "true" requirements separate from your full install list.
It's good practice to fully specify package versions in your requirements file. For example, if you are installing django-allauth with Django LTS, pin it to the latest releases (as of my answer):
Django==1.8.12
django-allauth==0.25.2
When you install the package, it ends up installing some required packages as well. So, you add those as well, so that everyone gets the same versions of packages:
Django==1.8.12
django-allauth==0.25.2
oauthlib==1.0.3
python-openid==2.2.5
requests==2.9.1
requests-oauthlib==0.6.1
And then you get the bug report "Doesn't work under Python 3". Oops, python-openid is Python 2 only, and python3-openid is used instead, further requiring defusedxml:
Django==1.8.12
django-allauth==0.25.2
oauthlib==1.0.3
python-openid==2.2.5 ; python_version < '3.0'
python3-openid==3.0.10 ; python_version >= '3.0'
defusedxml==0.4.1 ; python_version >= '3.0'
requests==2.9.1
requests-oauthlib==0.6.1
Now requirements.txt is getting ugly, and it's hard to see the "requirements" of Django and django-allauth in the mess.
Here is a requirements.txt that refers to a constraints file:
-c constraints.txt
Django==1.8.12
django-allauth==0.25.2
And constraints.txt with a helpful comment:
# django-allauth requirements
oauthlib==1.0.3
python-openid==2.2.5
python3-openid==3.0.10
defusedxml==0.4.1
requests==2.9.1
requests-oauthlib==0.6.1
No Python classifiers are needed, because constraints are only installed if a package requires them, and are ignored otherwise. Additionally, if a package stops requiring another package 2 years down the road, fresh installs will stop installing it.
I think this, plus some comments, is a useful way to communicate what packages you are using for the project, and which ones are included because they are dependencies.
I think it gets even more useful if you are using pip 8.x's hash-checking mode, which requires specifying versions for dependencies-of-dependencies. If you go down that path, I recommend hashin to help you manage the hashes. See browsercompat for a complicated requirements setup using constraints and hashes.
From the book "Secret Recipes of the Python Ninja"
Constraints files differ from requirements files in one key way: putting a package in the constraints file does not cause the package to be installed, whereas a requirements file will install all packages listed. Constraints files are simply requirements files that control which version of a package will be installed but provide no control over the actual installation.
Let say you have requirements.txt file with following code
# requirements.txt
pandas
and constraints.txt
# constraints.txt
# math / science / graph stuff
bokeh==0.11.1
numpy==1.10.4
pandas==0.17.1
scipy==0.17.0
openpyxl==2.3.3
patsy==0.4.1
matplotlib==1.5.1
ggplot==0.6.8
seaborn==0.7.0
scikit-learn==0.17
executing
pip install -c constraints.txt
will install all packages from requirements.txt and using constraints.txt file for version constraint.
Constraint files are very useful when you provide production environments with optimized package compilations for the productive infrastructure.
We provide such environments in docker containers with versions of numpy, scipy, tensorflow, opencv, ... optimized for our production servers. In order to ensure that the requirements files developers set up to make the environment reproducible for them don't overwrite these optimized installations, we use constraint files. They are written as part of the build process. pip is made aware of the constraints by setting the PIP_CONSTRAINT environment variable.
Before, developers always had to make sure their requirements line up with what's available in the prod containers, which can be a pain given that some minor version numbers iterate quite quickly.
I haven't used constrains files myself, so I may be wrong, but here's how I understand it from the documentation.
Let's say, you have multiple projects which you install using pip. Some of them depend on a popular package foobar directly, some indirectly (via their dependencies), some don't depend on it at all.
One day you decide to change foobar code a bit, to solve some problem related to your environment, so you go ahead and create a local copy of foobar. Now you want to ensure that all your projects use this local copy, but you don't want to spend time figuring out which of them depend on foobar and then editing their requirements.txt or setup.py files.
Here comes constraints file, where you point out to your local copy of foobar and use it for all your projects, not caring which of them need foobar, because constrains file only applies when the project being installed actually depends on foobar (whether directly or indirectly).
There's an even better example of possible usage on reddit.