I have a requirements.txt file with several dependencies listed.
Whenever I try pip install -r requirements.txt in a brand new system, this will usually fail when some dependency is not met (see for example: here and here) This happens specially with the matplotlib package.
After downloading the entire package (in the case of matplotlib it's about 50Mb) and failing to install it, I go and fix the issue and then attempt to install the package again.
pip does not seem to be smart enough to realize it just downloaded that package and automatically re-use that same file (perhaps because it keeps no copy by default?) so the package will be downloaded entirely again.
To get around this issue I can follow the instructions given here and use:
pip install --download=/path/to/packages -r requirements.txt
to first download all packages and then:
pip install --no-index --find-links=/path/to/packages -r requirements.txt
to install all the packages using the locally stored files.
My question: is there a smart command that includes both these directives? I'm after a single line I can run repeatedly so that it will use stored copies of the packages if they exist or download them if they don't, and in this last case, store those copies to some location so they can be re-used later on if needed.
How can I keep track of the packages when I install them using pip inside a virtualenv?
It seems like a mess now; if I install package A, it automatically install its dependancies; B, C and D. Then I decide to use package N instead which installs its dependancies as well.
Now when I remove package A, its dependancies are not automatically removed.
How I can keep my virtualenv clean? Is there a tool to check for unused packages and remove them?
To remove a package:
pip uninstall package_name
To get list of packages required by any given package (using pip):
pip show package_name
This will show you the packages that are required for it to run, and also the packages that require your package for them to run.
So the best way to uninstall a package with all its dependency packages is to run pip show package_name first to see the list of its dependency packages and then uninstall it along with its dependency packages one by one. For example:
pip show package_name
pip uninstall package_name
pip uninstall dependency_package_1
pip uninstall dependency_package_2
...etc
Making virtualenvs is relatively cheap. You could just create a new virtualenv whenever you get into this situation and run your pip install again.
Not very elegant, but it gets the job done. Of course you need to be maintaining some requirements file for the pip install and it will go faster if you have some local index or cache for pip.
To get a clean environment, create a new one. Some pip hooks can help you on this path:
pip freeze to get list of installed packages and their versions, wich can later be used with
-r <file> to install list of packages, stated in a requirements file
--build <dir> to place builds in a specific directory
--no-clean to not clean up build directories
later you can use those builds with --no-download
--no-deps to not install dependencies
Alternative way is to name each dependency of your project in your "setup.py" or "requirements.txt". Exercise setup.py or pip install cat requirements.txt multiple times with virtualenv in order to run your application successfully. After that, manually add the new dependency to one of the files to keep your dependency in sync.
Is it possible to install packages using pip from the local filesystem?
I have run python setup.py sdist for my package, which has created the appropriate tar.gz file. This file is stored on my system at /srv/pkg/mypackage/mypackage-0.1.0.tar.gz.
Now in a virtual environment I would like to install packages either coming from pypi or from the specific local location /srv/pkg.
Is this possible?
PS
I know that I can specify pip install /srv/pkg/mypackage/mypackage-0.1.0.tar.gz. That will work, but I am talking about using the /srv/pkg location as another place for pip to search if I typed pip install mypackage.
What about::
pip install --help
...
-e, --editable <path/url> Install a project in editable mode (i.e. setuptools
"develop mode") from a local project path or a VCS url.
eg, pip install -e /srv/pkg
where /srv/pkg is the top-level directory where 'setup.py' can be found.
I am pretty sure that what you are looking for is called --find-links option.
You can do
pip install mypackage --no-index --find-links file:///srv/pkg/mypackage
From the installing-packages page you can simply run:
pip install /srv/pkg/mypackage
where /srv/pkg/mypackage is the directory, containing setup.py.
Additionally1, you can install it from the archive file:
pip install ./mypackage-1.0.4.tar.gz
1
Although noted in the question, due to its popularity, it is also included.
I am installing pyfuzzybut is is not in PyPI; it returns the message: No matching distribution found for pyfuzzy.
I tried the accepted answer
pip install --no-index --find-links=file:///Users/victor/Downloads/pyfuzzy-0.1.0 pyfuzzy
But it does not work either and returns the following error:
Ignoring indexes: https://pypi.python.org/simple
Collecting pyfuzzy
Could not find a version that satisfies the requirement pyfuzzy (from versions: )
No matching distribution found for pyfuzzy
At last , I have found a simple good way there: https://pip.pypa.io/en/latest/reference/pip_install.html
Install a particular source archive file.
$ pip install ./downloads/SomePackage-1.0.4.tar.gz
$ pip install http://my.package.repo/SomePackage-1.0.4.zip
So the following command worked for me:
pip install ../pyfuzzy-0.1.0.tar.gz.
Hope it can help you.
This is the solution that I ended up using:
import pip
def install(package):
# Debugging
# pip.main(["install", "--pre", "--upgrade", "--no-index",
# "--find-links=.", package, "--log-file", "log.txt", "-vv"])
pip.main(["install", "--upgrade", "--no-index", "--find-links=.", package])
if __name__ == "__main__":
install("mypackagename")
raw_input("Press Enter to Exit...\n")
I pieced this together from pip install examples as well as from Rikard's answer on another question. The "--pre" argument lets you install non-production versions. The "--no-index" argument avoids searching the PyPI indexes. The "--find-links=." argument searches in the local folder (this can be relative or absolute). I used the "--log-file", "log.txt", and "-vv" arguments for debugging. The "--upgrade" argument lets you install newer versions over older ones.
I also found a good way to uninstall them. This is useful when you have several different Python environments. It's the same basic format, just using "uninstall" instead of "install", with a safety measure to prevent unintended uninstalls:
import pip
def uninstall(package):
response = raw_input("Uninstall '%s'? [y/n]:\n" % package)
if "y" in response.lower():
# Debugging
# pip.main(["uninstall", package, "-vv"])
pip.main(["uninstall", package])
pass
if __name__ == "__main__":
uninstall("mypackagename")
raw_input("Press Enter to Exit...\n")
The local folder contains these files: install.py, uninstall.py, mypackagename-1.0.zip
An option --find-links does the job and it works from requirements.txt file!
You can put package archives in some folder and take the latest one without changing the requirements file, for example requirements:
.
└───requirements.txt
└───requirements
├───foo_bar-0.1.5-py2.py3-none-any.whl
├───foo_bar-0.1.6-py2.py3-none-any.whl
├───wiz_bang-0.7-py2.py3-none-any.whl
├───wiz_bang-0.8-py2.py3-none-any.whl
├───base.txt
├───local.txt
└───production.txt
Now in requirements/base.txt put:
--find-links=requirements
foo_bar
wiz_bang>=0.8
A neat way to update proprietary packages, just drop new one in the folder
In this way you can install packages from local folder AND pypi with the same single call: pip install -r requirements/production.txt
PS. See my cookiecutter-djangopackage fork to see how to split requirements and use folder based requirements organization.
Assuming you have virtualenv and a requirements.txt file, then you can define inside this file where to get the packages:
# Published pypi packages
PyJWT==1.6.4
email_validator==1.0.3
# Remote GIT repo package, this will install as django-bootstrap-themes
git+https://github.com/marquicus/django-bootstrap-themes#egg=django-bootstrap-themes
# Local GIT repo package, this will install as django-knowledge
git+file:///soft/SANDBOX/python/django/forks/django-knowledge#egg=django-knowledge
To install only from local you need 2 options:
--find-links: where to look for dependencies. There is no need for the file:// prefix mentioned by others.
--no-index: do not look in pypi indexes for missing dependencies (dependencies not installed and not in the --find-links path).
So you could run from any folder the following:
pip install --no-index --find-links /srv/pkg /path/to/mypackage-0.1.0.tar.gz
If your mypackage is setup properly, it will list all its dependencies, and if you used pip download to download the cascade of dependencies (ie dependencies of depencies etc), everything will work.
If you want to use the pypi index if it is accessible, but fallback to local wheels if not, you can remove --no-index and add --retries 0. You will see pip pause for a bit while it is try to check pypi for a missing dependency (one not installed) and when it finds it cannot reach it, will fall back to local. There does not seem to be a way to tell pip to "look for local ones first, then the index".
Having requirements in requirements.txt and egg_dir as a directory
you can build your local cache:
$ pip download -r requirements.txt -d eggs_dir
then, using that "cache" is simple like:
$ pip install -r requirements.txt --find-links=eggs_dir
What you need is --find-links of pip install.
-f, --find-links If a url or path to an html file, then parse for links to archives. If a local path or
file:// url that's a directory, then look for archives in the directory listing.
In my case, after python -m build, tar.gz package (and whl file) are generated in ./dist directory.
pip install --no-index -f ./dist YOUR_PACKAGE_NAME
Any tar.gz python package in ./dist can be installed by this way.
But if your package has dependencies, this command will prompt error.
To solve this, you can either pip install those deps from official pypi source, then add --no-deps like this
pip install --no-index --no-deps -f ./dist YOUR_PACKAGE_NAME
or copy your deps packages to ./dist directory.
I've been trying to achieve something really simple and failed miserably, probably I'm stupid.
Anyway, if you have a script/Dockerfile which download a python package zip file (e.g. from GitHub) and you then want to install it you can use the file:/// prefix to install it as shown in the following example:
$ wget https://example.com/mypackage.zip
$ echo "${MYPACKAGE_MD5} mypackage.zip" | md5sum --check -
$ pip install file:///.mypackage.zip
NOTE: I know you could install the package straight away using pip install https://example.com/mypackage.zip but in my case I wanted to verify the checksum (never paranoid enough) and I failed miserably when trying to use the various options that pip provides/the #md5 fragment.
It's been surprisingly frustrating to do something so simple directly with pip. I just wanted to pass a checksum and have pip verify that the zip was matching before installing it.
I was probably doing something very stupid but in the end I gave up and opted for this. I hope it helps others trying to do something similar.
In my case, it was because this library depended on another local library, which I had not yet installed. Installing the dependency with pip, and then the dependent library, solved the issue.
If you want to install one local package (package A) to be used inside another local project/package (B) this is quite simple. All you need is to CD to (B) and call:
pip install /path/to/package(A)
Of course you will need to first compile the package (A) with:
sudo python3 ./setup.py install
And, each time you change package A, just run again setup.py in package (A) then pip install ... inside the using project/package (B)
Just add directory on pip command
pip install mypackage file:/location/in/disk/mypackagename.filetype
What's the proper way to install pip packages to a virtualenv using cached packages? I've tried setting --timeout=360 and --use-mirrors, but pypi performance is so flakey, it routinely hangs or disconnects, making deployments a crapshoot.
So, my plan was to use the --download-cache option to pre-download all package archives (but not install them), e.g.:
pip install --upgrade --timeout=120 --use-mirrors --no-install --download-cache /usr/local/pip/cache -r pip-requirements.txt
and then specify this cache location during the install into the virtualenv, e.g.:
virtualenv /usr/local/myapp/.env
. /usr/local/myapp/.env/bin/activate; pip install --no-download --download-cache /usr/local/pip/cache -r pip-requirements.txt
deactivate
Creating the cache was very frustrating, because pip seems to nondeterministically save the downloaded package to the cache dir. Specifically, it refused to cache Django==1.4.0, saying "that requirement has already been met", so I had to use the --force-reinstall.
Installing using the cache is currently where I'm stuck. Running the above command gives me the error:
Could not install requirement Django==1.4.0 (from -r pip-requirements.txt (line 1)) because source folder /usr/local/myapp/.env/build/Django does not exist (perhaps --no-download was used without first running an equivalent install with --no-install?)
Storing complete log in /home/chris/.pip/pip.log
What does this mean?
The files:
http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FD%2FDjango%2FDjango-1.4.tar.gz
http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FD%2FDjango%2FDjango-1.4.tar.gz.content-type
definitely exist in the cache directory, and I used the --no-install option.
What am I doing wrong?
The problem seems to be that --download-cache can only be used to specify the cache for downloading, not for installing. Therefore pip is still looking at /usr/local/myapp/.env/build/Django instead of /usr/local/pip/cache. Have you tried moving
pip install --upgrade --timeout=120 --use-mirrors --no-install --download-cache /usr/local/pip/cache -r pip-requirements.txt
to after the creation of virtualenv? I wonder if that would help.
You may also want to try to export PIP_DOWNLOAD_CACHE and see if it works without using --download-cache.
Here is what I find that works:
pip install --no-install --use-mirrors --download=/DIRNAME PKGNAME
pip install --find-links=file:///DIRNAME --no-index --index-url=file:///dev/null PKGNAME
Now, actually, here is the tool for I would use instead of all the above and it solves all of the problems much more elegantly and reliably: pip2pi by David Wolever.
From the docs:
pip2pi builds a PyPI-compatible package repository from pip requirements
pip2pi allows you to create your own PyPI index by using two simple commands:
To mirror a package and all of its requirements, use pip2tgz:
$ cd /tmp/; mkdir package/
$ pip2tgz packages/ httpie==0.2
...
$ ls packages/
Pygments-1.5.tar.gz
httpie-0.2.0.tar.gz
requests-0.14.0.tar.gz
To build a package index from the previous directory:
$ ls packages/
bar-0.8.tar.gz
baz-0.3.tar.gz
foo-1.2.tar.gz
$ dir2pi packages/
$ find packages/
/httpie-0.2.0.tar.gz
/Pygments-1.5.tar.gz
/requests-0.14.0.tar.gz
/simple
/simple/httpie
/simple/httpie/httpie-0.2.0.tar.gz
/simple/Pygments
/simple/Pygments/Pygments-1.5.tar.gz
/simple/requests
/simple/requests/requests-0.14.0.tar.gz
To install from the index you built in step 2., you can simply use:
pip install --index-url=file:///tmp/packages/simple/ httpie==0.2
Bonus: you can even mirror your own index to a remote host with pip2pi.
Is there a way to configure easy_install to avoid having to download the files again when an installation fails?
Update 13 years later: easy_install was removed from Python in January 2021. The python package manager is pip, it caches downloaded packages.
pip (http://pypi.python.org/pypi/pip/) is a drop-in replacement for the easy_install tool and can do that.
Just run easy_install pip and set an environment variable PIP_DOWNLOAD_CACHE to the path you want pip to store the files.
Note that the cache won't work with dependencies that checkout from a source code repository (like svn/git/hg/bzr).
Then use pip install instead of easy_install
Here is my solution using pip, managing even installation of binary packages and usable on both, Linux and Windows. And as requested, it will limit download from PyPi to the minimum, and as extra bonus, on Linux, it allows to speed up repeated installation of packages usually requiring compilation to a fraction of a second.
Setup takes few steps, but I thing, it is worth to do.
Create pip config file
Create pip configuration file (on linux: ~/.pip/pip.conf, on Windows %HOME%\pip\pip.ini)
My one has this content:
[global]
download-cache = /home/javl/.pip/cache
find-links = /home/javl/.pip/packages
[install]
use-wheel = yes
[wheel]
wheel-dir = /home/javl/.pip/packages
Populating cache dir - goes automatically
The cache dir will get cached version of data downloaded from pypi each time, pip attempts to get some package from pypi. It is easy to get it there (no special care needed), but note, that from pip point of view, these are just cashed data downloaded from PyPi, not packages, so in case you use an option --no-index, it will not work.
pip download to populate packages dir
The packages dir is place to put real package files to. E.g. for my favorite package plac, I would do:
$ pip download --dest ~/.pip/packages plac
and the plac package file would appear in that dir. You may even use -r requirements.txt file to do this for multiple packages at once.
These packages are used even with $ pip install --no-index <something>.
Prevent repeated compilation of the same package on Linux
E.g. lxml package requires compliation, and download and compile may take from 45 seconds to minutes. Using wheel format, you may save here a lot.
Install wheel tool, if you do not have it yet:
$ pip install wheel
Create the wheel for lxml (assuming, you have managed to install lxml in past - it requires some libs in the system to be installed):
$ pip wheel lxml
This goes over download, compile, but finally results in lxml whl file being in packages dir.
Since then
$ pip install lxml
or even faster
$ pip install --no-index lxml
will take fraction of a second, as it uses wheel formatted package.
Prepare wheel package from Window setup exe package
(note: this can be prepared even on Linux machine, there is no compilation, only some repacking from exe file into whl.)
download the exe form of the package from pypi, e.g:
$ wget https://pypi.python.org/packages/2.7/l/lxml/lxml-3.2.3.win32-py2.7.exe#md5=14ab978b7f0a3382719b65a1ca938d33
$ dir
lxml-3.2.3.win32-py2.7.exe
convert it to whl
$ wheel convert lxml-3.2.3.win32-py2.7.exe
$ dir
lxml-3.2.3.win32-py2.7.exe
lxml-3.2.3-cp27-none-win32.whl
Test it:
$ pip install lxml
or
$ pip install --no-index lxml
shall be very quick.
Note, that wheel convert can do exactly the same conversion for egg formatted packages.
Let easy_install and setup.py install reuse your packages dir
easy_install and $ python setup.py install do not seem to offer download cache, but allow to use packages we have in our packages dir.
To do so, edit config file for these two tools:
On Linux: $HOME/.pydistutils.cfg
On Windows: %HOME%\pydistutils.cfg
In my case I have here in /home/javl/.pydistutils.cfg:
[easy_install]
find_links = /home/javl/.pip/packages
This config may help even some cases of pip install calls, when pip attempts to install a package, declaring dependency on other ones. As it delegates this task to setup.py call, without the .pydistutils.cfg config it would download the file from PyPi.
Unfortunately, wheel format is not supported in this case (as far as I am aware of).