We Keep Coding

Where does pip download .whl files?

I'd like to install a certain python package with pip but because of the proxy I am sitting behind pip cannot connect to the internet.
So my question is: Where does pip look for .whl files in order to download them? Can't I just use my browser (which can connect to the internet just fine) to download the .whl file? Installing the package with the downloaded .whl file would be not a problem then.

pip searches the Python package index (PyPI), each package lists downloads (including wheels, if there are any) with a direct download link on the page. Package pages have the form of https://pypi.python.org/pypi/<package_name> or https://pypi.python.org/pypi/<package_name>/<version> for specific versions.
If you can only download wheels manually with your browser, it doesn't matter where you put the wheel file. Just install the wheel file directly:
pip install path/to/wheel.whl
However, pip supports downloading over a proxy just fine:
pip --proxy username:password#proxy_server:proxy_port install ...
See the --proxy command line switch documentation. You can add the proxy setting to a pip configuration file so you don't have to set it on the command line each time, or by setting environment variables; see the Using a Proxy Server section in the Pip User Guide.

How to get an URL pip is using to download the file:
Get JSON from https://pypi.python.org/pypi/package_name/json
parse releases part, select the latest release
go through available files (usually there are more than one), taking your platform into account (e.g. x32 vs x64, Windows or Linux version, installed Python etc)
use url property
E.g.:
import requests
package = requests.get("https://pypi.python.org/pypi/pandas/json").json()
max_ver = max(package["releases"].keys())
# ... check compatibility
file = get_file_idx(package['releases'][max_ver])
urllib.urlretrieve(file["url"])

Could not find a version that satisfies the requirement Pillow==2.7.0

I have a requirements.txt file with the line below
Pillow==2.7.0
I download the requirements for offline mode like this and put it in vendor director
pip install --download vendor -r requirements.txt
This downloads the file Pillow-2.7.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel to my vendor directory
But when I try to deploy (push) my app I get the error below:
Collecting Pillow==2.7.0 (from -r requirements.txt (line 4))
Could not find a version that satisfies the requirement Pillow==2.7.0 (from -r requirements.txt (line 4)) (from versions: )
No matching distribution found for Pillow==2.7.0 (from -r requirements.txt (line 4))
I am deploying the app to Cloud Foundry using the Python Buildpack following this documentation http://docs.cloudfoundry.org/buildpacks/python/index.html#vendoring

UPDATE: Another post points out how you can specify to recent versions of pip a flag not to pull binaries (which I'm assuming is the issue). If that resolves the issues then you should pick his post as solution. I'd also poke cloudfoundry just needing to point this out.
The --download option just downloads the file for the dependency, but not actually install it (that might have contributed to why the flag was deprecated, to get rid of this confusion).
pip download replaces the --download option to pip install, which is now deprecated and will be removed in pip 10.
It's the same as the newer:
pip download ....
https://pip.pypa.io/en/stable/reference/pip_download/
Now assuming your cloud provider is reading the requirements.txt and looking themselves in your vendor folder to install (a-not-so-wise-approach), you'd potentially have all kinds of issues with any binaries, symbolic links, etc... unless the environment you're deploying to matches the local environment. Part of the idea with package systems like pip and a requirements file is so different platforms can pull down the needed libraries for their specific architecture, OS, etc...
This approach has so many things could go wrong...
For example, your mac probably is using a case insensitive file system. If the cloudfoundry is using linux, that's case-sensitive. On mac, this wouldn't matter if they saved it all lower case, but might be an issue when copying the files to a linux system expecting to match a capitlized "P".

In example at http://docs.cloudfoundry.org/buildpacks/python/index.html#vendoring you can see the following comment
vendors all the pip *.tar.gz into vendor/
*.tar.gz packages are source packages whereas recent versions of pip download binary packages by default. To force pip to download only source packages use --no-binary :all: If you want to download binary packages you have to specify target platform as per this remark from pip download documentation:
pip download with the --platform, --python-version,
--implementation, and --abi options provides the ability to fetch
dependencies for an interpreter and system other than the ones that
pip is running on. --only-binary=:all: is required when using any of
these options. It is important to note that these options all default
to the current system/interpreter, and not to the most restrictive
constraints (e.g. platform any, abi none, etc). To avoid fetching
dependencies that happen to match the constraint of the current
interpreter (but not your target one), it is recommended to specify
all of these options if you are specifying one of them.

How do prevent pip and easy_install from removing the temporary directories?

I need to debug some failures to install packages using pip and it seems that once I get the error pip is removing all the temporary files, preventing me from looking inside and finding what happened.
The gist below contains the error that I get but I do want to find-out which config file is generating this error and all the files from tmp are removed by the time the script ends.
https://gist.github.com/ssbarnea/b373062dd45de92735c7482b2735c5fb

From my experience with pip 19.0.3, it is sufficient to use the --no-clean option, which is available to both pip install and pip wheel.
It can also be specified in the respective sections of the pip configuration file:
[install]
no-clean = true
[wheel]
no-clean = true

How to change default install location for pip

I'm trying to install Pandas using pip, but I'm having a bit of trouble. I just ran sudo pip install pandas which successfully downloaded pandas. However, it did not get downloaded to the location that I wanted. Here's what I see when I use pip show pandas:
---
Name: pandas
Version: 0.14.0
Location: /Library/Python/2.7/site-packages/pandas-0.14.0-py2.7-macosx-10.9-intel.egg
Requires: python-dateutil, pytz, numpy
So it is installed. But I was confused when I created a new Python Project and searched under System Libs/lib/python for pandas, because it didn't show up. Some of the other packages that I've downloaded in the past did show up, however, so I tried to take a look at where those were. Running pip show numpy (which I can import with no problem) yielded:
---
Name: numpy
Version: 1.6.2
Location: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
Requires:
Which is in a completely different directory. For the sake of confirming my error, I ran pip install pyquery to see where it would be downloaded to, and got:
Name: pyquery
Version: 1.2.8
Location: /Library/Python/2.7/site-packages
Requires: lxml, cssselect
So the same place as pandas...
How do I change the default download location for pip so that these packages are downloaded to the same location that numpy is in?
Note: There were a few similar questions that I saw when searching for a solution, but I didn't see anything that mentioned permanently changing the default location.

According to pip documentation at
http://pip.readthedocs.org/en/stable/user_guide/#configuration
You will need to specify the default install location within a pip.ini file, which, also according to the website above is usually located as follows
On Unix and Mac OS X the configuration file is: $HOME/.pip/pip.conf
On Windows, the configuration file is: %HOME%\pip\pip.ini
The %HOME% is located in C:\Users\Bob on windows assuming your name is Bob
On linux the $HOME directory can be located by using cd ~
You may have to create the pip.ini file when you find your pip directory. Within your pip.ini or pip.config you will then need to put (assuming your on windows) something like
[global]
target=C:\Users\Bob\Desktop
Except that you would replace C:\Users\Bob\Desktop with whatever path you desire. If you are on Linux you would replace it with something like /usr/local/your/path
After saving the command would then be
pip install pandas
However, the program you install might assume it will be installed in a certain directory and might not work as a result of being installed elsewhere.

You can set the following environment variable:
PIP_TARGET=/path/to/pip/dir
https://pip.pypa.io/en/stable/user_guide/#environment-variables

Open Terminal and type:
pip config set global.target /Users/Bob/Library/Python/3.8/lib/python/site-packages
except instead of
/Users/Bob/Library/Python/3.8/lib/python/site-packages
you would use whatever directory you want.

Follow these steps
pip config set global.target D:\site-packages to change install path
or py -m pip config --user --editor notepad edit
[global]
target = D:\site-packages
set environment variable to use download import xxx
PIP_TARGET=site-packages
PYTHONPATH=site-packages
3.pip config unset global.target, to upgrade pip py -m pip install --upgrade pip

#Austin's answer is outdated, here for more up-to-date solution:
According to pip documentation at
https://pip.pypa.io/en/stable/topics/configuration/
You will need to specify the default install location within a configuration file, which, also according to the website above is usually located as follows
Mac OS
$HOME/Library/Application Support/pip/pip.conf if directory $HOME/Library/Application Support/pip exists else $HOME/.config/pip/pip.conf.
The legacy “per-user” configuration file is also loaded, if it exists: $HOME/.pip/pip.conf.
The $HOME folder can be located by navigating to ~/ (cmd+shift+G in Finder; cmd+shift+. to show hidden files).
Windows
%APPDATA%\pip\pip.ini
The legacy “per-user” configuration file is also loaded, if it exists: %HOME%\pip\pip.ini
The %HOME% is located in C:\Users\Bob on windows assuming your username is Bob
Unix
$HOME/.config/pip/pip.conf, which respects the XDG_CONFIG_HOME environment variable.
The legacy “per-user” configuration file is also loaded, if it exists: $HOME/.pip/pip.conf.
On linux the $HOME directory can be located by using cd ~
You may have to create the configuration file when you find your pip directory. Put something like
[global]
target = /Library/Frameworks/Python.framework/Versions/Current/lib/python3.10/site-packages/
if you are on a Mac. Except that you would replace /Library/Frameworks/Python.framework/Versions/Current/lib/python3.10/site-packages/ with whatever path you desire. If you are on Linux you would replace it with something like /usr/local/your/path
After saving the command would then be
pip install pandas
However, the program you install might assume it will be installed in a certain directory and might not work as a result of being installed elsewhere.
Please note that
pip3 install pandas
might be the solution if your packages gets installed in the Python2 folder vs Python3.

How to state in requirements.txt a direct github source

I've installed a library using the command
pip install git+git://github.com/mozilla/elasticutils.git
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like
-f git+git://github.com/mozilla/elasticutils.git
elasticutils==0.7.dev
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported.
Does anybody have a solution for my problem?

Normally your requirements.txt file would look something like this:
package-one==1.9.4
package-two==3.7.1
package-three==1.0.1
...
To specify a Github repo, you do not need the package-name== convention.
The examples below update package-two using a GitHub repo. The text between # and # denotes the specifics of the package.
Specify commit hash (41b95ec in the context of updated requirements.txt):
package-one==1.9.4
git+https://github.com/path/to/package-two#41b95ec#egg=package-two
package-three==1.0.1
Specify branch name (master):
git+https://github.com/path/to/package-two#master#egg=package-two
Specify tag (0.1):
git+https://github.com/path/to/package-two#0.1#egg=package-two
Specify release (3.7.1):
git+https://github.com/path/to/package-two#releases/tag/v3.7.1#egg=package-two
Note that #egg=package-two is not a comment here, it is to explicitly state the package name
This blog post has some more discussion on the topic.

“Editable” packages syntax can be used in requirements.txt to import packages from a variety of VCS (git, hg, bzr, svn):
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
Also, it is possible to point to particular commit:
-e git://github.com/mozilla/elasticutils.git#000b14389171a9f0d7d713466b32bc649b0bed8e#egg=elasticutils

requirements.txt allows the following ways of specifying a dependency on a package in a git repository as of pip 7.0:1
[-e] git+git://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+https://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+ssh://git.myproject.org/SomeProject#egg=SomeProject
-e git+git#git.myproject.org:SomeProject#egg=SomeProject (deprecated as of Jan 2020)
For Github that means you can do (notice the omitted -e):
git+git://github.com/mozilla/elasticutils.git#egg=elasticutils
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here's my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Documentation
1 https://pip.readthedocs.org/en/stable/reference/pip_install/#git
2 https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support

First, install with git+git or git+https, in any way you know. Example of installing kronok's branch of the brabeion project:
pip install -e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion
Second, use pip freeze > requirements.txt to get the right thing in your requirements.txt. In this case, you will get
-e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion-master
Third, test the result:
pip uninstall brabeion
pip install -r requirements.txt

Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you'd probably want to publish your module in it's own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.

None of these answers worked for me. The only thing that worked was:
git+https://github.com/path_to_my_project.git
No "e", no double "git" and no previous installs necessary.

Github has zip endpoints that in my opinion are preferable to using the git protocol. The advantages are:
You don't have to specify #egg=<project name>
Git doesn't need to be installed in your environment, which is nice for containerized environments
It works much better with pip hashing and caching
The URL structure is easier to remember and more discoverable
You usually want requirements.txt entries to look like this, e.g. without the -e prefix:
https://github.com/org/package/archive/1a58aa586efd4bca37f2cfb9d9348958986aab6c.tar.gz
To install from main branch:
https://github.com/org/package/archive/main.tar.gz
There is also an equivalent .zip endpoint, but it was reported in a comment that always using the .tar.gz endpoint avoids problems with unicode package names.

It seems like this is also a valid format:
gym-tictactoe # git+https://github.com/haje01/gym-tictactoe.git#84e22fc28fe192ba0040bdd56a697f63d3d4a3d5
If you do a pip install "git+https://github.com/haje01/gym-tictactoe.git", then look at what got installed by running pip freeze, you will see the package described in this format and can copy and paste into requirements.txt.

I'm finding that it's kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04's package manager) to actually install the thing I tell it to install. I'm posting this answer to save anyone's time who runs into this problem.
Putting this into a requirements.txt file failed:
git+git://github.com/myname/myrepo.git#my-branch#egg=eggname
By "failed" I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
git+git://github.com/myname/myrepo.git#d27d07c9e862feb939e56d0df19d5733ea7b4f4d#egg=eggname

For private repositories, I found that these two work fine for me:
pip install https://${GITHUB_TOKEN}#github.com/owner/repo/archive/main.tar.gz
Where main.tar.gz refers to the main branch of your repo and can be replaced with other branch names. For more information and using the more recent Github API see here:
pip install https://${GITHUB_TOKEN}#api.github.com/repos/owner/repo/tarball/master
If you have git installed and available, then
pip install git+https://${GITHUB_TOKEN}#github.com/owner/repo.git#main
achieves the same, and it also allows for some more flexibility by appending #branch or #tag or #commit-hash. That approach, however, actually clones the repo into a local temp folder which can take a noticeable amount of time.
You can use the URLs in your requirements.txt, too.