Pip install to custom target directory and exclude specific dependencies - python

I'm looking for a method to use pip or similiar to install a list of python packages to a custom target directory (ex./mypath/python/pkgs/ ), but also exclude/blacklist specific dependencies.
I want to exclude specific dependencies since they are already met from a different install path (e.g. an anaconda install). I don't have the privilege of adding packages to the default python installation (nor do I want to).
I'm currently use the -r and -t options of pip. But have not found a way to exclude specific packages.
A pip command like this is would be ideal:
pip install --log pip.log -r req.txt -t /mypath/pypkgs/ --exclude exclude.txt
--no-deps is not an option since I need some of the dependencies.
I'm currently pursuing a python script to do pip installs that include dependencies I don't need via:
pip install --log pip.log -r req.txt -t /mypath/python/pkgs/
and then (automatically) remove the unneeded dependencies after the pip install finishes.
I hoping some combination of pip commands can achieve what I'm looking for some straightforward away. I'm using pip 7.1.2. Thanks!
Similar, yet I'm not upgrading and want to specify a target path:
pip: upgrade package without upgrading particular dependency


Faced with a similar problem, and using a running bash shell I managed to exclude specific packages with
pip install $(grep -ivE "pkg1|pkg2|pkg3" requirements.txt)
where pkg1 and so on are the names of the packages to exclude.


I think this essentially can be achieved in several steps, assuming you're using virtualenv or similar...
If you first do a normal pip freeze > requirements.txt you'll get all of the transitive dependencies (e.g. not excluding anything.)
Now edit requirements.txt, remove the packages you want to exclude...
Finally, in a new environment do pip install -r requirements.txt -t ... --no-deps. Voila: you've installed the dependencies you wanted while excluding specific ones.


You can use pip install --no-deps with pip check to exclude specific packages.
A real example of mine is when I installed paddleocr on Jetson Nano, pip kept installing python-opencv for me, which has already installed by myself without using pip, but pip cannot detect it. To stop pip installing python-opencv for me, here are the steps
use pip install --no-deps paddleocr to install paddleocr without its dependencies
use pip check to list unresolved dependencies, reformat the output so that it can be read by pip install -r, then remove the packages you don't want to install (it is python-opencv in my case), and save it to a file named fix-deps.txt
use pip install --no-deps -r fix-deps.txt to install unresolved dependencies
repeat steps 2 and 3 until the output of pip check only contains the packages you don't want to install.
pip install --no-deps and pip check are very useful commands that allow you to resolve the dependencies by yourself when pip cannot do it right for you. The shortcoming of this solution is the output of pip check is designed for humans, it cannot be used by pip install -r directly, so you have to reformat the output manually or use the awk command.
PR and issue have been created for the pip community to make the output of pip check suitable for the pip install to read, but for some reason, they don't think it is a good idea. so I guess this is the best we can do now.
Refs:
https://github.com/pypa/pip/pull/10108
https://github.com/pypa/pip/issues/10066#issuecomment-872638361


An approach that takes into account sub-dependencies is to first install the exclude.txt environment, then the req.txt environment, then check the diff and finally install that into the target directory.
Example using virtualenv that will work on GNU/Linux:
virtualenv tmpenv
source tmpenv/bin/activate
pip install -r exclude.txt
pip freeze > exclude-with-deps.txt
pip install -r req.txt
pip freeze > req-with-deps.txt
comm -13 exclude-with-deps.txt req-with-deps.txt > final-req.txt
pip install -r final-req.txt --no-deps -t pypkgs


Quick Answer:
pip freeze | grep -vFxf pip_ignore.txt > requirements.txt
Will ignore whatever you save in pip_ignore.txt
This is good if you already know what you're going to ignore. However if you're about to install something that you know you'll want to ignore, you're going to want to be able to handle things a bit more easily. In that scenario what I do is pip freeze > before.txt then I install whatever packages it is, and then pip freeze | grep -vFxf before.txt > new_stuff_to_ignore.txt. You now have a list of the stuff you just installed. You can add it to your pip_ignore.txt and then finally do pip freeze | grep -vFxf pip_ignore.txt > requirements.txt
Be mindfudl that some packages in your requirements may share dependencies with packages you want to ignore. These dependencies might end up in your pip_ignore.txt.

Related

Docker pip install odd with requirements.txt [duplicate]

I have a repository with an inflated requirements.txt file that I'd like to clean up. Using pipreqs I've set my requirements.txt to be a minimal set of packages need for my repository. To test this, I setup a virtualenv to install the packages and then run all my unit tests to make sure they're satisfactory.
virtualenv temp_venv --no-site-packages
source temp_venv/bin/activate
pip install -r requirements.txt
Which runs fine, but I see that a whole bunch of extra packages are collected and installed. Why? Are these identified as needed by required packages, and thus installed? If so, should I then include them in the requirements.txt?

Yes. The packages are dependencies of your dependencies​.
But no, you should not specify them directly. Automatic tools know to download dependencies recursively and it would significantly add to maintenance overhead.

This might have been because of the dependencies of your written libs in requirements.txt. For ex: if you have written scipy as requirement numpy will also be installed because scipy is dependent on numpy.

Well, for me the above answers were not the case. Pip install was installing extra packages not in requirements.txt . The solution was:
Run conda create -n venv_name and conda activate venv_name, where venv_name is the name of your virtual environment.
Run conda install pip. This will install pip to your venv directory.
Then run pip install -r requeriments.txt
The above answer was adapted from here: Using Pip to install packages to Anaconda Environment

How to ignore my development project for installing a package with pip

I uploaded my package to testpypi, and installed it via:
pip install -i https://test.pypi.org/simple/ myporj==0.1.6
However it refuse to install it by saying:
Requirement already satisfied: myproj==0.1.6 in ./projs/myproj (0.1.6)
I guess I may add the project in editable mode:
pip install --editable .
However, I know want to disable it. I tried:
python setup.py develop --uninstall
But it has no effect.

It may be worth creating a separate env (Virtual Environments) for the installation.
Here are some articles on this subject:
https://docs.python.org/3/tutorial/venv.html
https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment
Or does it need to be installed in the same place?
You can try to find your package pip search myporj or pip list for show all packages.
And uninstall it later pip uninstall myporj (it may require the right of sudo in linux) then install again.
Maybe you may need --no-cache-dir option to ignore the cache during installation. Here is more details: https://pip.pypa.io/en/stable/reference/pip_install/#caching

Removing All Packages With Defunct Dependees in PIP

I am wondering if there is a way to uninstall packages in PIP including those that are not listed in the requirements.txt but which were installed as dependencies of those that are.
For example, suppose I have Django==2.1 line in requirements.txt. When running pip install -r requirements.txt, the above will instruct PIP to install many extra packages on which Django depends.
However, if I then execute pip uninstall -r requirements.txt, the Django package will be uninstalled, but PIP will retain many of its now unused dependencies.
My question is how to go about cleaning those up nicely. Is there a way to make PIP preserve and consider history explicitly? If the thing which forced PIP to install a package is being uninstalled, it appears that we should also be able to flag it to wipe its now defunct dependencies.

Take a look at pipdeptree Python package and pipdeptree --reverse some_package command in particular.

The easiest option is to use pip-autoremove. After installing it via pip, you can simply call the following from the command line:
pip-autoremove Django
Which uninstalls Django and its unused dependencies including those not listed in requirements.txt.

pip install with brute force (no prompts)

Is there a way to install packages with pip to avoid the need to repeatedly delete files like:
pip can't proceed with requirement 'Flask-Restless==0.13.1 (from -r requirements.txt (line 2))' due to a pre-existing build directory.
location: /private/var/folders/0k/t9lwmd2j1212pxydpr6l596h0000gq/T/pip_build_jacob/Flask-Restless
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
I'm on round 4 of doing this and have no idea how long it may take to get through.
Looking at pip --help isn't helpful and man pip returns nothing.

As it has already been mentioned it's better to use virtualenv in order to avoid python package chaos on your system and install the python packages only for particular projects.
However, in your particular case you can try the following in the terminal:
pip uninstall flask-restless
Then try to run the installation again:
pip install -r requirements.txt
The options to consider during installation:
--force-reinstall
--ignore-installed
--no-deps
Add these options to the end of pip install -r requirements.txt to play with them and see if they can help.

Using
--force-reinstall
may solve your issue.
I would also recommend considering using a virtualenv for each project you are working on.
https://virtualenv.pypa.io/en/stable/
You can then activate the virtual environment for that project and pip
pip install -r requrements.txt
will install dependencies for that project in the virtual environment instead of globally. This will reduce the odds of having weird conflicts like you are having and if you do have an issue you can blow away the virtualenv and reinstall just the dependencies for that project without borking your global packages.

How do I update a Python package?

I'm running Ubuntu 9:10 and a package called M2Crypto is installed (version is 0.19.1). I need to download, build and install the latest version of the M2Crypto package (0.20.2).
The 0.19.1 package has files in a number of locations including (/usr/share/pyshared and /usr/lib/pymodules.python2.6).
How can I completely uninstall version 0.19.1 from my system before installing 0.20.2?

The best way I've found is to run this command from terminal
sudo pip install [package_name] --upgrade
sudo will ask to enter your root password to confirm the action.
Note: Some users may have pip3 installed instead. In that case, use
sudo pip3 install [package_name] --upgrade

You might want to look into a Python package manager like pip. If you don't want to use a Python package manager, you should be able to download M2Crypto and build/compile/install over the old installation.

To automatically upgrade all the outdated packages (that were installed using pip), just run the script bellow,
pip install $(pip list --outdated | awk '{ print $1 }') --upgrade
Here, pip list --outdated will list all the out dated packages and then we pipe it to awk, so it will print only the names.
Then, the $(...) will make it a variable and then, everything is done auto matically. Make sure you have the permissions. (Just put sudo before pip if you're confused)
I would write a script named, pip-upgrade
The code is bellow,
#!/bin/bash
sudo pip install $(pip list --outdated | awk '{ print $1 }') --upgrade
Then use the following lines of script to prepare it:
sudo chmod +x pip-upgrade
sudo cp pip-upgrade /usr/bin/
Then, just hit pip-upgrade and voila!

Via windows command prompt, run: pip list --outdated
You will get the list of outdated packages.
Run: pip install [package] --upgrade
It will upgrade the [package] and uninstall the previous version.
To update pip:
py -m pip install --upgrade pip
Again, this will uninstall the previous version of pip and will install the latest version of pip.

Method 1: Upgrade manually one by one
pip install package_name -U
Method 2: Upgrade all at once (high chance rollback if some package fail to upgrade
pip install $(pip list --outdated --format=columns |tail -n +3|cut -d" " -f1) --upgrade
Method 3: Upgrade one by one using loop
for i in $(pip list --outdated --format=columns |tail -n +3|cut -d" " -f1); do pip install $i --upgrade; done

I think the best one-liner is:
pip install --upgrade <package>==<version>

Open Command prompt or terminal and use below syntax
pip install --upgrade [package]==[specific version or latest version]
For Example
pip install --upgrade numpy==1.19.1

How was the package originally installed? If it was via apt, you could just be able to do apt-get remove python-m2crypto
If you installed it via easy_install, I'm pretty sure the only way is to just trash the files under lib, shared, etc..
My recommendation in the future? Use something like pip to install your packages. Furthermore, you could look up into something called virtualenv so your packages are stored on a per-environment basis, rather than solely on root.
With pip, it's pretty easy:
pip install m2crypto
But you can also install from git, svn, etc repos with the right address. This is all explained in the pip documentation

pip install -U $(pip list --outdated | awk 'NR>2 {print $1}')

In Juptyer notebook, a very simple way is
!pip install <package_name> --upgrade
So, you just need to replace with the actual package name.

How can I completely uninstall version 0.19.1 from my system before
installing 0.20.2?
In order to uninstall M2Crypto use
pip uninstall M2Crypto
I need to download, build and install the latest version of the
M2Crypto package (0.20.2).
In order to install the latest version, one can use PyPi
pip install M2Crypto
To install the version 20.2 (an outdated one), run
pip install M2Crypto==0.20.2
Assuming one just wants to upgrade
pip install M2Crypto --upgrade # Or pip install M2Crypto -U
Notes:
Depending on one's Python version (here's how to find the version) one may use a different pip command. Let's say one is working with Python 3.7, instead of just using pip, one might use pip3.7.
Using sudo is considered unsafe.
Nowadays there are better practices to manage the development system, such as: virtual environments or development containers. The development containers allow one to put the entire development environment (be it modules, VS Code extensions, npm libraries,...) inside a Docker container. When the project comes to an end, one closes the container. There's no need to keep all of those requirements around in the computer for no reason. If you feel like reading more about it: Visual Studio Docs, Github.

Get all the outdated packages and create a batch file with the following
commands
pip install xxx --upgrade for each outdated packages

I.e.:
python -m pip install --proxy <proxyserver_name>:<port#> <pkg_name>
Remember to export the variables after setting them, to make them available to the outer shell session.
Windows:
Add to environment variables:
set HTTP_PROXY=<proxyserver_name>:<port#>
You might have to install the full python package first

Categories