PIP: Installing only the dependencies - python

I have a script that creates a virtualenv, installs distribute and pip in it and then optionally clones a git repo.
Now I have the project I will be working on, installed. But its dependencies are not installed. How can I make pip install all the dependencies as if I have issued a pip install MyApp?
EDIT: Appareantly my question is a duplicate of this one.
Not exactly sure but pip install -e . seems to do what I want without too many extra stuff lying around. I'd prefer if my code wasn't linked from site-packages though.


If your dependencies are defined in the setup.py file, you can first dump them to an external file using:
python setup.py egg_info
This will list all your dependencies in YOUR_PROJECT.egg-info/requires.txt file. Then you can install them using pip:
pip install -r *.egg-info/requires.txt
to delete what you just created:
rm -rf *.egg-info/
to save some time copy pasting:
python setup.py egg_info
pip install -r *.egg-info/requires.txt
rm -rf *.egg-info/


In my package root issuing pip install -e . installs dependencies.


To install your project's dependencies (i.e. install_requires + extra_requires) you have to extract your dependencies using setuptools egg-info and then install the filtered list of the combined dependencies:
python setup.py egg_info
pip install `grep -v '^\[' *.egg-info/requires.txt`


You should use the pip requirements file.
Essentially, place all your requirements, one in each line in a file and pass that to pip using the command
pip install -r requirements.txt
What more, if you have a standard environment, pip can actually dump such a file from existing installs using the command:
pip freeze
You can put the file thus generated directly into the pip requirements, and call the previous command from your deployment script.
Pretty cool, isnt it? :)


You can use pip-tools to create a requirements.txt that only contains the dependencies of your package:
$ pip-compile -o requirements.txt setup.py
Note that the command above only works if you do not already have a requirements.txt file. If you happen to have one already, just delete it.
Using the generated requirements.txt you can then run pip to install the dependencies:
$ pip install -r requirements.txt
Bonus 1:
The requirements.txt will include comments that indicate where the regarding dependency originates from.
Bonus 2:
If you have have an extras_require section for optional dependencies in your setup.py that looks e.g. like this:
...
extras_require={
"development": [
"wheel",
"debugpy",
"pytest",
],
},
...
You can create the requirements.txt including the optional dependencies by using:
$ pip-compile -o requirements.txt --extra development setup.py

Related

How to install packages listed in `install_requires` from setup.py without installing the main package?

Is there any way to install all of the packages that are listed in install_requires of setup.py without installing the main package?
The list looks like:
install_requires=[
'cassandra-driver==3.23.0',
# 'kafka-python==1.4.7',
# 'parmap==1.2.3',
'imblearn==0.0',
# 'imbalanced-learn==0.3.3',
'toolz==0.9.0',
]
Is there any way to install directly from the setup file without having to separate the packages and use pip?

When looking at
pip install --help
you can see that you can either install all packages from the file itself with the help of
python -m pip install -r requirements.txt
or you can do it in the command line like this:
pip install package1 package2 package3
Maybe have a look at the Pip-Documentary: https://pip.pypa.io/en/stable/user_guide/

Why use pip install requirements.txt instead of pip install ./?

I'm not very experienced with dependency management in Python but it seems that all of the dependencies in requirements.txt could alternatively be placed in the setup.py file under the install_requires field as follows:
setup(
...
install_requires=['numpy=1.2.3', 'pandas=1.2.3']
...
)
Since pip install ./ using a setup.py file also provides lots of additional functionality compared to pip install -r requirements.txt, what is the use for the latter? Would a project ever have a valid reason to use both?

pip install . installs your custom package with all its dependencies. pip install -r requirements.txt installs only the dependencies.

You could also add -e . to your requirements.txt and install your package with pip install -r requirements.txt

What does pip install . (dot) mean?

I have a shell script whose last line is:
pip install .
What does it do?
pip install <package-name> installs the specified package
pip install -r requirements.txt installs all packages specified in requirements.txt
But I am not sure what the above command does.

Explicitly, pip install . will execute the setup.py file in the current directory (which will usually load a requirements.txt file).

"Install the project found in the current directory".
This is just a specific case of pip install /path/to-source/tree.
To quote the the pip install documentation describing this usage:
pip install [options] [-e] <local project path> ...

What does " -r " do in pip install -r requirements.txt

I looked up how to install multiple packages from a requirements document using pip. The answers were mostly:
pip install -r requirements.txt
What does the -r do though? I can't find an answer for this and it isn't listed when I run pip help.

Instead of pip --help, look into pip install --help:
-r, --requirement Install from the given requirements
file. This option can be used multiple
times.
Also see these documentation paragraphs:
pip install
Requirements Files.

-r will search for requirement file.
pip install --help
will help you !!

May, 2022 Update:
If you run this command below without "-r":
pip install requirements.txt
You will get this error below:
ERROR: Could not find a version that satisfies the requirement requirements.txt (from versions: none)
HINT: You are attempting to install a package literally named "requirements.txt" (which cannot exist). Consider using the '-r' flag to install the packages listed in requirements.txt
ERROR: No matching distribution found for requirements.txt
Because "pip" tries to install the package "requirements.txt" instead of installing the packages listed in "requirements.txt". Of cource, the package "requirements.txt" doesn't exist in PyPI while for example, the packages "django" and "pillow" exist in PyPI:
pip install django
pip install pillow
So, to install the packages listed in "requirements.txt", you must need "-r";
pip install -r requirements.txt
You can check what "-r" means by running the command below:
pip install --help
-r, --requirement Install from the given requirements file. This option can be used multiple times.

In your case pip install -r requirements.txt will install the libraries listed in your requirements.txt file.

pip install requirements.txt
Above statement looks for a python package named requirements.txt. No such package exists. Your intention is that pip install opens the txt and reads the packages from there. The -r allows pip install to open requirements.txt and install the packages inside of it instead.

Upgrade python packages from requirements.txt using pip command

How do I upgrade all my python packages from requirements.txt file using pip command?
tried with below command
$ pip install --upgrade -r requirements.txt
Since, the python packages are suffixed with the version number (Django==1.5.1) they don't seem to upgrade. Is there any better approach than manually editing requirements.txt file?
EDIT
As Andy mentioned in his answer packages are pinned to a specific version, hence it is not possible to upgrade packages through pip command.
But, we can achieve this with pip-tools using the following command.
$ pip-review --auto
this will automatically upgrade all packages from requirements.txt (make sure to install pip-tools using pip install command).

I already answered this question here. Here's my solution:
Because there was no easy way for upgrading package by package, and updating the requirements.txt file, I wrote this pip-upgrader which also updates the versions in your requirements.txt file for the packages chosen (or all packages).
Installation
pip install pip-upgrader
Usage
Activate your virtualenv (important, because it will also install the new versions of upgraded packages in current virtualenv).
cd into your project directory, then run:
pip-upgrade
Advanced usage
If the requirements are placed in a non-standard location, send them as arguments:
pip-upgrade path/to/requirements.txt
If you already know what package you want to upgrade, simply send them as arguments:
pip-upgrade -p django -p celery -p dateutil
If you need to upgrade to pre-release / post-release version, add --prerelease argument to your command.
Full disclosure: I wrote this package.

you can try:
pip install --upgrade --force-reinstall -r requirements.txt
You can also ignore installed package and install the new one :
pip install --ignore-installed -r requirements.txt

No. Your requirements file has been pinned to specific versions. If your requirements are set to that version, you should not be trying to upgrade beyond those versions. If you need to upgrade, then you need to switch to unpinned versions in your requirements file.
Example:
lxml>=2.2.0
This would upgrade lxml to any version newer than 2.2.0
lxml>=2.2.0,<2.3.0
This would upgrade lxml to the most recent version between 2.2.0 and 2.3.0.

I suggest freezing all of your dependencies in order to have predictable builds.
When doing that, you can update all dependencies at once like this:
sed -i '' 's/[~=]=/>=/' requirements.txt
pip install -U -r requirements.txt
pip freeze | sed 's/==/~=/' > requirements.txt
Having done the above, test your project with the new set of packages and eventually commit the requirements.txt file to the repository while still allowing for installing hot-fixes.

Another solution is to use the upgrade-requirements package
pip install upgrade-requirements
And then run :
upgrade-requirements
It will upgrade all the packages that are not at their latest versions, and also create an updated requirements.txt at the end.

Fixing dependencies to a specific version is the recommended practice.
Here's another solution using pur to keep the dependencies fresh!
Give pur your requirements.txt file and it will auto update all your high-level packages to the latest versions, keeping your original formatting and comments in-place.
For example, running pur on the example requirements.txt updates the packages to the currently available latest versions:
$ pur -r requirements.txt
Updated flask: 0.9 -> 0.10.1
Updated sqlalchemy: 0.9.10 -> 1.0.12
Updated alembic: 0.8.4 -> 0.8.6
All requirements up-to-date.
As pur never modifies your environment or installed packages, it's extremely fast and you can safely run it without fear of corrupting your local virtual environment. Pur separates updating your requirements.txt file from installing the updates. So you can use pur, then install the updates in separate steps.

I've just had to do the same... used this small one-liner to do the job:
packages=$(cat requirements.txt | sed 's/==.*//g'); echo $packages | xargs pip3 install -U; freeze=$(pip3 freeze); for p in $(echo $packages); do echo $freeze | grep -E "^${p}==" >> requirements.new; done
which:
packages=$(cat requirements.txt | sed 's/==.*//g') creates a list of the current packages names in requirements.txt (removing the version).
echo $packages | xargs pip3 install -U then passes all of the packages as arguments to pip3 to upgrade.
freeze=$(pip3 freeze); Gets all of the current package versions in the format required for requirements.txt
for p in $(echo $packages) then iterates through the package names
echo $freeze | grep -E "^${p}==" >> requirements.new gets the package version line from the pip freeze output which matches the package and writes to new requirements.txt
This has the added benefit of preserving the ordering of the original requirements.txt. :)
Hope this helps!

The second answer is the most useful but what I wanted to do is lock some packages while having others at the latest version (e.g. youtube-dl).
An example requirements.txt would look like this (~ means compatible):
Pillow==6.2.2
requests~=2.22.0
youtube_dl
Then in the terminal, use the command pip install --upgrade -r requirements.txt
This ensures that Pillow will stay at 6.2.2, requests will be upgraded to the latest 2.22.x (if available), and the latest version of youtube-dl will be installed if not already.

Since I couldn't do that using bash, I wrote a python module to create a new requirements file with no versions and use it:
data = open('requirements-prod.pip', 'r')
data2 = open('requirements-prod-no-version.pip', 'w')
for line in data.readlines():
new_line = line[:line.index('==')]
data2.write(new_line + '\n')
data2.flush()
Then install the libs from the new file pip install -U -r requirements-prod-no-version.pip
Finally freeze the versions to the original file pip freeze > requirements-prod.pip

More robust solution is IMO to use a dependency management such as poetry, https://python-poetry.org which comes with an exhaustive dependency resolver.

I guess the simplest solution is creating the requirements.txt with:
pip freeze | sed 's/==/>=/' > requirements.txt

You can use below command on Linux and Mac:
cat requirements.txt | cut -f1 -d= | xargs pip install -U

1) To upgrade pip installed files from reqs.txt
add the >= in replacement of ==
this will tell pip to install lib greater than or equal to the version you are requesting, here by installing the most to-date version of requested library
1.a) **My answer for thread ** By adding py -m pip install -r reqs.txt to a daily restart... or something of the nature you can update your installed libs.
Summed up by Andy Perfectly
-My reason For entering this thread was to find information on how to update virtual env base pip (usually 10.0.03 for me??)
in-hopes of satisfying an issue of which have I was able to derive one of two solutions
A. creation of venv || B. Installation of Required libs
Thanks to Andy I have satisfied need B
By adding pip >= requested version in reqs.txt
upon instantiation of new virtual-Environment || re-instantiation of previous Venv
py -m venv devenv
to setup new dev env
devenv\scripts\activate.bat
to activate dev env
python -m pip install -r requirenments.txt
to install base libs
yeilds output
Collecting pip >= 20.0.2 (from -r requirenments.txt (line 1))
Using cached >https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Found existing installation: pip 10.0.1
Uninstalling pip-10.0.1:
Successfully uninstalled pip-10.0.1
Successfully installed pip-20.0.2
Sorry for the Brain Dump, Hopes this helps someone :)
🤳 Austin 👨‍🎤🚀🥊

If you install anything in your django project and after installation you want to update your requirement file this command can update you requirement.txt file
pip freeze > requirements.txt
if your requirement file not exist in you project you can use this command for make new requirement.txt file
pip freeze > requirements.txt

With pip-tools you have a basic requirements.in with desired dependencies and a requirements.txt file with pinned versions. pip-tools then generates the pinned versions automatically, which makes handling the whole process including upgrading your dependencies a lot easier.
# requirements.in
django
and the autogenerated requirements.txt (to pin all dependencies)
$ pip-compile requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile requirements.in
#
asgiref==3.2.3
# via django
django==3.0.3
# via -r requirements.in
pytz==2019.3
# via django
sqlparse==0.3.0
# via django
If you use that workflow, which I can highly recommend, it's
pip-compile --upgrade
which generates the requirements.txt with the latest versions.

I edit the requirements.txt as below and run $sh ./requirements.txt
pip install -U amqp;
pip install -U appdirs;
pip install -U arrow;
pip install -U Babel;
pip install -U billiard;
pip install -U celery;
pip install -U Django;
pip install -U django-cors-headers;
pip install -U django-crispy-forms;
pip install -U django-filter;
pip install -U django-markdown-deux;
pip install -U django-pagedown;
pip install -U django-timezone-field;
pip install -U djangorestframework;
pip install -U fcm-django;
pip install -U flower;
pip install -U gunicorn;
pip install -U kombu;
pip install -U Markdown;
pip install -U markdown2;
pip install -U packaging;

Categories