I am updating all my packages with pip with
import pip
from subprocess import call
for dist in pip.get_installed_distributions():
call("pip install --upgrade " + dist.project_name, shell=True)
but it takes forever since many more packages are installed than I actually need.
Can I delete all unused packages with pip?
Of course you can delete any package you want. The real question is should you?
Is this in a virtualenv or is it system wide python? If it's system wide, just because your project isn't using those packages doesn't mean no other apps rely on them. If its a virtualenv, I would recommend creating a temp virtualenv, and installing only the packages that you know your project depends on. This way you can see which packages were installed as dependencies of packages you require and you can remove the ones that are not needed anymore.
Related
I have pip3, installed via the yum install of python3-pip.
I've done a pip3 global install of some modules I need, but python3 can't find them to import. After a little investigation I see that pip3 installed the modules to /usrlib/python3.6/site-packages/pip/_vendor/
The problem is that python3 doesn't seem to know to look at pip/_vendor, it only finds modules directly installed under site-package. If I just copy the modules from .../site-package/pip/_vendor to .../site-package everything works fine.
The issue doesn't appear to be related to file permissions or ability to read the modules.
I'm wondering how I configure either pip to install directly to site-package or python3 to understand how to look in the pip/_vendor location.
I'm configuring this all with ansible and would like as module an option as possible. For instance I could manually use an argument to tell pip3 to install to the folder I want, but I don't want to hardcode the exact site-package directory if I don't have to.
I recommend starting over with pip by downloading and running get-pip.py. This will not only install the latest version of pip, but it will also install packages to a Python-readable location (the version of Python you use to run get-pip.py).
As an aside, I would avoid installing packages system-wide unless there is a specific need for them. At the very least, you should be installing them as a regular user, and even better you should be using a virtualenv.
I tried to install the Twilio module:
sudo -H pip install twilio
And I got this error:
Installing collected packages: pyOpenSSL
Found existing installation: pyOpenSSL 0.13.1
Cannot uninstall 'pyOpenSSL'. It is a distutils installed project and
thus we cannot accurately determine which files belong to it which
would lead to only a partial uninstall.
Anyone know how to uninstall pyOpenSSL?
This error means that this package's metadata doesn't include a list of files that belong to it. Most probably, you have installed this package via your OS' package manager, so you need to use that rather than pip to update or remove it, too.
See e.g. Upgrading to pip 10: It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. · Issue #5247 · pypa/pip for one such example where the package was installed with apt.
Alternatively, depending on your needs, it may be more productive to not use your system Python and/or its global environment but create a private Python installation and/or environment. There are many options here including virtualenv, venv, pyenv, pipenv and installing Python from source into
/usr/local or $HOME/$HOME/.local (or /opt/<whatever>).
Finally, I must comment on the often-suggested (e.g. at pip 10 and apt: how to avoid "Cannot uninstall X" errors for distutils packages) --ignore-installed pip switch.
It may work (potentially for a long enough time for your business needs), but may just as well break things on the system in unpredictable ways. One thing is sure: it makes the system's configuration unsupported and thus unmaintainable -- because you have essentially overwritten files from your distribution with some other arbitrary stuff. E.g.:
If the new files are binary incompatible with the old ones, other software from the distribution built to link against the originals will segfault or otherwise malfunction.
If the new version has a different set of files, you'll end up with a mix of old and new files which may break dependent software as well as the package itself.
If you change the package with your OS' package manager later, it will overwrite pip-installed files, with similarly unpredictable results.
If there are things like configuration files, differences in them between the versions can also lead to all sorts of breakage.
I had the same error and was able to resolve using the following steps:
pip install --ignore-installed pyOpenSSL
This will install the package with latest version and then if you try to install,
pip install twilio
It will work.
Generally, for similar errors, use this format:
pip install --ignore-installed [package name]==[package version]
I just had this error and the only way I was able to resolve it was by manually deleting the offending directory from site-packages.
After doing this you may need to reinstall the packages with --force-reinstall.
Reading the above comments, I understood that package a was installed with conda and the new package b that I was trying to install using pip was causing problems. I was lucky that package b had conda support so using conda to install package b solved the problem.
In my case, I was installing a package from internal git using the following command:
python -m pip install package.whl --force
I was doing this because I didn't want to explicitly uninstall the previous version and just replace it with a newer version. But what it also does is install all the dependencies again. I was getting the error in one of those packages. Removing --force fixed the problem.
I want to add, having --ignore-installed also worked for me. And removing --force is essentially doing the same thing in my case.
I have been given access to a University Data Center to deploy an Image Analysis python project. The server has Python 2.7 and 3.5 installed and I can see that it is missing packages like numpy, theano and keras which I have used in my code as additional libraries.
The problem at hand is, that I do not have access to install anything, or run commands like pip install or apt-get install, and cannot copy anything to the original site-packages location in my server.
But I can copy files into my userspace, and I tried to:
- clone numpy and its prerequisites, and all the additional packages I need into a folder called site-packages.
- add this path to my sys.path, but it gives me errors like "cannot import multiarray"
I'm new to Linux, and my question is: can I copy package files into a Linux system and provide this path to my PYTHONPATH to run the code?
I believe you are looking for:
pip install --user package_name
You might also need to investigate compiling some packages from their source code, but this will depend on the package.
From the user guide more on pip install --user:
pip install --user follows four rules:
When globally installed packages are on the python path, and they
conflict with the installation requirements, they are ignored, and not
uninstalled.
When globally installed packages are on the python path,
and they satisfy the installation requirements, pip does nothing, and
reports that requirement is satisfied (similar to how global packages
can satisfy requirements when installing packages in a
--system-site-packages virtualenv).
pip will not perform a --user install in a --no-site-packages > virtualenv (i.e. the default kind of
virtualenv), due to the user site not being on the python path. The
installation would be pointless.
In a --system-site-packages
virtualenv, pip will not install a package that conflicts with a
package in the virtualenv site-packages. The --user installation would
lack sys.path precedence and be pointless.
Edit: If pip itself is not installed then you can read up here: https://pip.pypa.io/en/stable/installing/
There are a number of resources that compare and contrast the advantages and disadvantages of using apt-get and pip to install, update, and uninstall python packages.
What I cannot find is a resource that indicates what happens if something that is installed by one package manager is updated or uninstalled by the other.
When I run pip list, it lists a lot of packages that are installed, most of which on my system were installed by apt-get and not pip.
So, are these two package managers able to manage packages installed by the other? Or, is pip able to manage a package installed by apt-get, but then apt-get is messed up afterwards. Is apt-get able to manage a package installed by pip?
I would recommend to try and avoid using two (or more) package managers at the same time. It's not very likely that they will cooperate correctly and smoothly.
If possible, pick one of them and use it. Combine them only if you really need to. Usually you don't.
There are ways of avoiding conflicts such as
pip install --user <package> which installs the package into the user's directory only
virtualenv which allows you to have packages installed per application/project - this is a very good idea since various projects might need different versions of the same package and it's easy to move such project do a different computer etc.
venv - Python 3 has a built-in support for virtual environments
How can I keep track of the packages when I install them using pip inside a virtualenv?
It seems like a mess now; if I install package A, it automatically install its dependancies; B, C and D. Then I decide to use package N instead which installs its dependancies as well.
Now when I remove package A, its dependancies are not automatically removed.
How I can keep my virtualenv clean? Is there a tool to check for unused packages and remove them?
To remove a package:
pip uninstall package_name
To get list of packages required by any given package (using pip):
pip show package_name
This will show you the packages that are required for it to run, and also the packages that require your package for them to run.
So the best way to uninstall a package with all its dependency packages is to run pip show package_name first to see the list of its dependency packages and then uninstall it along with its dependency packages one by one. For example:
pip show package_name
pip uninstall package_name
pip uninstall dependency_package_1
pip uninstall dependency_package_2
...etc
Making virtualenvs is relatively cheap. You could just create a new virtualenv whenever you get into this situation and run your pip install again.
Not very elegant, but it gets the job done. Of course you need to be maintaining some requirements file for the pip install and it will go faster if you have some local index or cache for pip.
To get a clean environment, create a new one. Some pip hooks can help you on this path:
pip freeze to get list of installed packages and their versions, wich can later be used with
-r <file> to install list of packages, stated in a requirements file
--build <dir> to place builds in a specific directory
--no-clean to not clean up build directories
later you can use those builds with --no-download
--no-deps to not install dependencies
Alternative way is to name each dependency of your project in your "setup.py" or "requirements.txt". Exercise setup.py or pip install cat requirements.txt multiple times with virtualenv in order to run your application successfully. After that, manually add the new dependency to one of the files to keep your dependency in sync.