Keep order of installation in pip freeze - python

Quick question.
Is there a way to ensure that pip freeze > requirements.txt keeps the order in which the packages were installed? This is an issue for me because I continuously get something like this in requirements.txt:
matplotlib==1.1.1
numpy==1.6.2
So an error occurs when I try to install using pip install -r requirements.txt because numpy is a dependency of matplotlib, so I have to install manually numpy first and then rerun pip install -r requirements.txt
Is there any fix on that?
UPDATE: In response to mechmind, I installed matplotlib and numpy in Ubuntu 12.04 using pip with virtualenv --distribute myenv. After installation, I got this freeze file:
argparse==1.2.1
distribute==0.6.28
matplotlib==1.1.1
numpy==1.6.2
wsgiref==0.1.2
Then when I try to reinstall in another virtual environment I get the following error:
REQUIRED DEPENDENCIES
numpy: no
* You must install numpy 1.4 or later to build
* matplotlib.
So maybe it's dependent on the system.
Thanks!

Just tried pip with numpy and matplotlib and pip correctly resolved dependency checks - numpy built first.
Tried on old stock pip from ubuntu 10.10.
EDIT: After playing with pip and virtualenv, i realized that dependency check actually works only when that dependencies was discovered, i.e. when package was installed, removed and installed again.
So actual solution will involve reordering of packages in requrements file (for simple case when there are only two packages with wrong order, you can just reverse requirements file: sort -r | xargs pip install

Related

How do I prevent pip automatically installing supporting packages?

I am using python 3.6.0 within a venv. I would like to "pip install" matplotlib==2.0.0, however when I do this, pip seems to automatically grab the newest versions of all other required supporting packages for matplotlib. i.e. cycler 0.11.0, pyparsing==3.0.7, etc. These latest supporting package versions do not seem to work with the older version of matplotlib and it throws errors when attempting to import matplotlib.
How do I install matplotlib without pip attempting to install all its supporting packages automatically?
My current temporary solution is to go back and manually install each package before installing matplotlib but I'm sure I will run into this issue again so would like to find a better solution.
Pip has a built-in feature:
pip install matplotlib --no-dependencies
To exclude specific, you can put it in requirements file and pass it:
pip install --no-deps -r requirements.txt

Docker pip install odd with requirements.txt [duplicate]

I have a repository with an inflated requirements.txt file that I'd like to clean up. Using pipreqs I've set my requirements.txt to be a minimal set of packages need for my repository. To test this, I setup a virtualenv to install the packages and then run all my unit tests to make sure they're satisfactory.
virtualenv temp_venv --no-site-packages
source temp_venv/bin/activate
pip install -r requirements.txt
Which runs fine, but I see that a whole bunch of extra packages are collected and installed. Why? Are these identified as needed by required packages, and thus installed? If so, should I then include them in the requirements.txt?
Yes. The packages are dependencies of your dependencies​.
But no, you should not specify them directly. Automatic tools know to download dependencies recursively and it would significantly add to maintenance overhead.
This might have been because of the dependencies of your written libs in requirements.txt. For ex: if you have written scipy as requirement numpy will also be installed because scipy is dependent on numpy.
Well, for me the above answers were not the case. Pip install was installing extra packages not in requirements.txt . The solution was:
Run conda create -n venv_name and conda activate venv_name, where venv_name is the name of your virtual environment.
Run conda install pip. This will install pip to your venv directory.
Then run pip install -r requeriments.txt
The above answer was adapted from here: Using Pip to install packages to Anaconda Environment

Cannot upgrade packages using pip inside virtualenv

I wanted to be able to access all of my site packages from another installation of Python, so I created a virtual environment in this way:
venv my_project --system-site-packages
I noticed that my version of Keras was outdated, so from within my virtualenv, I executed:
pip install keras
which worked without an issue. I'm using pip version 9.0.1
I'm trying to run a python program that uses TensorFlow, but when I run it, I get an error:
ImportError: No module named tensorboard.plugins
I googled around and found that I needed to upgrade TensorFlow. I tried several commands:
(my_project/) user#GPU5:~/spatial/zero_padded/powerlaw$ pip install tensorflow
The above gives me a 'requirement already satisfied' error.
$ pip install --target=~/spatial/zero_padded/powerlaw/my_project/ --upgrade tensorflow
Collecting tensorflow
Could not find a version that satisfies the requirement tensorflow (from versions: )
No matching distribution found for tensorflow
The output of which python:
/user/spatial/zero_padded/powerlaw/my_project/bin/python
I think my PYTHONPATH is the first line in this:
(my_project/) user#GPU5:~/spatial/zero_padded/powerlaw/my_project$ python -c "import sys; print '\n'.join(sys.path)"
/user/spatial/zero_padded/powerlaw/my_project
/opt/enthought/canopy-1.5.1/appdata/canopy-1.5.1.2730.rh5-x86_64/lib/python27.zip
/opt/enthought/canopy-1.5.1/appdata/canopy-1.5.1.2730.rh5-x86_64/lib/python2.7
/opt/enthought/canopy-1.5.1/appdata/canopy-1.5.1.2730.rh5-x86_64/lib/python2.7/plat-linux2
/opt/enthought/canopy-1.5.1/appdata/canopy-1.5.1.2730.rh5-x86_64/lib/python2.7/lib-tk
/opt/enthought/canopy-1.5.1/appdata/canopy-1.5.1.2730.rh5-x86_64/lib/python2.7/lib-old
/opt/enthought/canopy-1.5.1/appdata/canopy-1.5.1.2730.rh5-x86_64/lib/python2.7/lib-dynload
/user/spatial/zero_padded/powerlaw/my_project/lib/python2.7/site-packages
/user/pkgs/enthought/canopy-1.5.1/lib/python2.7/site-packages
/user/pkgs/enthought/canopy-1.5.1/lib/python2.7/site-packages/PIL
/opt/enthought/canopy-1.5.1/appdata/canopy-1.5.1.2730.rh5-x86_64/lib/python2.7/site-packages
How do I upgrade TensorFlow inside my virtualenv?
Pretty sure that all you need to do is run pip install with -U to upgrade the package inside the virtualenv:
(my_project/) user#GPU5:~/spatial/zero_padded/powerlaw$ pip install -U tensorflow
-U is just shorthand for --upgrade. But, you should really go ahead and create a dependencies file for yourself called requirements.txt that lives in the project root and specify version numbers there.
e.g.,
tensorflow==1.2.0
And that makes it easier to install all requirements
pip install -r requirements.txt
The best way to do it is install the dependencies outside de vm, and create a new one I'm afraid to say that. Because doing upgrades is different than installing

pip install with brute force (no prompts)

Is there a way to install packages with pip to avoid the need to repeatedly delete files like:
pip can't proceed with requirement 'Flask-Restless==0.13.1 (from -r requirements.txt (line 2))' due to a pre-existing build directory.
location: /private/var/folders/0k/t9lwmd2j1212pxydpr6l596h0000gq/T/pip_build_jacob/Flask-Restless
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
I'm on round 4 of doing this and have no idea how long it may take to get through.
Looking at pip --help isn't helpful and man pip returns nothing.
As it has already been mentioned it's better to use virtualenv in order to avoid python package chaos on your system and install the python packages only for particular projects.
However, in your particular case you can try the following in the terminal:
pip uninstall flask-restless
Then try to run the installation again:
pip install -r requirements.txt
The options to consider during installation:
--force-reinstall
--ignore-installed
--no-deps
Add these options to the end of pip install -r requirements.txt to play with them and see if they can help.
Using
--force-reinstall
may solve your issue.
I would also recommend considering using a virtualenv for each project you are working on.
https://virtualenv.pypa.io/en/stable/
You can then activate the virtual environment for that project and pip
pip install -r requrements.txt
will install dependencies for that project in the virtual environment instead of globally. This will reduce the odds of having weird conflicts like you are having and if you do have an issue you can blow away the virtualenv and reinstall just the dependencies for that project without borking your global packages.

Pip install to custom target directory and exclude specific dependencies

I'm looking for a method to use pip or similiar to install a list of python packages to a custom target directory (ex./mypath/python/pkgs/ ), but also exclude/blacklist specific dependencies.
I want to exclude specific dependencies since they are already met from a different install path (e.g. an anaconda install). I don't have the privilege of adding packages to the default python installation (nor do I want to).
I'm currently use the -r and -t options of pip. But have not found a way to exclude specific packages.
A pip command like this is would be ideal:
pip install --log pip.log -r req.txt -t /mypath/pypkgs/ --exclude exclude.txt
--no-deps is not an option since I need some of the dependencies.
I'm currently pursuing a python script to do pip installs that include dependencies I don't need via:
pip install --log pip.log -r req.txt -t /mypath/python/pkgs/
and then (automatically) remove the unneeded dependencies after the pip install finishes.
I hoping some combination of pip commands can achieve what I'm looking for some straightforward away. I'm using pip 7.1.2. Thanks!
Similar, yet I'm not upgrading and want to specify a target path:
pip: upgrade package without upgrading particular dependency
Faced with a similar problem, and using a running bash shell I managed to exclude specific packages with
pip install $(grep -ivE "pkg1|pkg2|pkg3" requirements.txt)
where pkg1 and so on are the names of the packages to exclude.
I think this essentially can be achieved in several steps, assuming you're using virtualenv or similar...
If you first do a normal pip freeze > requirements.txt you'll get all of the transitive dependencies (e.g. not excluding anything.)
Now edit requirements.txt, remove the packages you want to exclude...
Finally, in a new environment do pip install -r requirements.txt -t ... --no-deps. Voila: you've installed the dependencies you wanted while excluding specific ones.
You can use pip install --no-deps with pip check to exclude specific packages.
A real example of mine is when I installed paddleocr on Jetson Nano, pip kept installing python-opencv for me, which has already installed by myself without using pip, but pip cannot detect it. To stop pip installing python-opencv for me, here are the steps
use pip install --no-deps paddleocr to install paddleocr without its dependencies
use pip check to list unresolved dependencies, reformat the output so that it can be read by pip install -r, then remove the packages you don't want to install (it is python-opencv in my case), and save it to a file named fix-deps.txt
use pip install --no-deps -r fix-deps.txt to install unresolved dependencies
repeat steps 2 and 3 until the output of pip check only contains the packages you don't want to install.
pip install --no-deps and pip check are very useful commands that allow you to resolve the dependencies by yourself when pip cannot do it right for you. The shortcoming of this solution is the output of pip check is designed for humans, it cannot be used by pip install -r directly, so you have to reformat the output manually or use the awk command.
PR and issue have been created for the pip community to make the output of pip check suitable for the pip install to read, but for some reason, they don't think it is a good idea. so I guess this is the best we can do now.
Refs:
https://github.com/pypa/pip/pull/10108
https://github.com/pypa/pip/issues/10066#issuecomment-872638361
An approach that takes into account sub-dependencies is to first install the exclude.txt environment, then the req.txt environment, then check the diff and finally install that into the target directory.
Example using virtualenv that will work on GNU/Linux:
virtualenv tmpenv
source tmpenv/bin/activate
pip install -r exclude.txt
pip freeze > exclude-with-deps.txt
pip install -r req.txt
pip freeze > req-with-deps.txt
comm -13 exclude-with-deps.txt req-with-deps.txt > final-req.txt
pip install -r final-req.txt --no-deps -t pypkgs
Quick Answer:
pip freeze | grep -vFxf pip_ignore.txt > requirements.txt
Will ignore whatever you save in pip_ignore.txt
This is good if you already know what you're going to ignore. However if you're about to install something that you know you'll want to ignore, you're going to want to be able to handle things a bit more easily. In that scenario what I do is pip freeze > before.txt then I install whatever packages it is, and then pip freeze | grep -vFxf before.txt > new_stuff_to_ignore.txt. You now have a list of the stuff you just installed. You can add it to your pip_ignore.txt and then finally do pip freeze | grep -vFxf pip_ignore.txt > requirements.txt
Be mindfudl that some packages in your requirements may share dependencies with packages you want to ignore. These dependencies might end up in your pip_ignore.txt.

Categories