I just re-installed python & virtualenv after accidentally getting everything messy. I noticed that my requirements.txt in my newly created environment automatically includes wheel==0.24.0. Is this normal, and what is it? I understand that virtualenv should create new environments without any system-installed packages.
I came across this issue when setting up virtualenv in windows.
To avoid wheel from being installed you can use the --no-wheel option.
virtualenv venv --no-wheel
To my understanding wheel is used handle a compressed format of a python package (.whl) that can be used by pip to speed up the install time. https://wheel.readthedocs.org/en/latest/
Probably one of the itens listed in your requirements.txt needs wheel package installed to work.
Although it's a common (and good) pratice list the fundamental python packages of a project inside requirements.txt, not all of the packages the eventually will be installed are there.
Your pip log may helps you to point what package required wheel to be installed.
Related
For some context, my project uses python version 3.8 and has some packages in a requirements.txt file. I'm trying to upgrade the python version to 3.10 and also all the packages in my requirements.txt file to the latest possible version such that there are no conflicts between dependencies. Is there a way to do this?
It is a bit hard to say what will work best for you since you've given no info on your OS, and that thing you need is done differently on macOS and Windows.
If you're on macOS (this one could also work for Linux I guess), the best way to manage python versions is with pyenv and to update python packages with pip-review, both of which you can install via brew.
So first, you want to create a list of all your packages installed. To do that, you type pip3 freeze > requirements.txt in the terminal. The command creates a txt file in your home folder. In the file, you can see the names of all your packages installed, and it will look like this:
astroid==2.11.6
async-generator==1.10
attrs==21.4.0
autopep8==1.6.0
beautifulsoup4==4.11.1
Secondly, to update your python version, you'll need to type pyen install -l in the terminal to get a list of all currently available versions of python. To install the one you need, let's say it's the most recent at the moment 3.10.5, type pyenv install 3.10.5. To set this version as a default one, type pyenv global 3.10.5. If you don't need your previous version of python, just type pyenv uninstall and the number of the version you want to delete after a space (it's really just like installing one).
Thirdly, to move all your previous packages in the newly installed python version, just type pip3 install -r requirements.txt in the terminal. The command will get the packages of the same versions you had before.
Lastly, to upgrade the packages to the most recent version, type pip-review --auto. It will fetch the latest versions available and install them with no more commands from you.
Now here's why I think pip-review works better than anything else I've tried so far: in case some dependencies are broken (i. e. a package you are using needs another package of an older version than you've just upgraded to), it will point that out in the terminal, and you could get the right one.
yes, you can do that just you need to update your requirements file.
and run the requirements file. it will automatically update your all files.
You can update to Python3.10 and drop the previous requirements.txt into your project root directory. From there you can run pip install -r requirements.txt to install all the packages.
You can also run pip install -U <package-name> to update any package to the latest version.
Also, to re-generate a new requirements.txt file, run pip freeze > requirements.txt.
Pretty sure there is no such thing as automatically updating dependencies in such way that there are no conflicts. Some dependencies may have changes that are not backward compatible and a complete test of all your project's features will be necessary whenever you change the version of anything.
There is also an open-source project called Renovate that can help you maintain your packages and update them to recent versions
I tried to install the Twilio module:
sudo -H pip install twilio
And I got this error:
Installing collected packages: pyOpenSSL
Found existing installation: pyOpenSSL 0.13.1
Cannot uninstall 'pyOpenSSL'. It is a distutils installed project and
thus we cannot accurately determine which files belong to it which
would lead to only a partial uninstall.
Anyone know how to uninstall pyOpenSSL?
This error means that this package's metadata doesn't include a list of files that belong to it. Most probably, you have installed this package via your OS' package manager, so you need to use that rather than pip to update or remove it, too.
See e.g. Upgrading to pip 10: It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. · Issue #5247 · pypa/pip for one such example where the package was installed with apt.
Alternatively, depending on your needs, it may be more productive to not use your system Python and/or its global environment but create a private Python installation and/or environment. There are many options here including virtualenv, venv, pyenv, pipenv and installing Python from source into
/usr/local or $HOME/$HOME/.local (or /opt/<whatever>).
Finally, I must comment on the often-suggested (e.g. at pip 10 and apt: how to avoid "Cannot uninstall X" errors for distutils packages) --ignore-installed pip switch.
It may work (potentially for a long enough time for your business needs), but may just as well break things on the system in unpredictable ways. One thing is sure: it makes the system's configuration unsupported and thus unmaintainable -- because you have essentially overwritten files from your distribution with some other arbitrary stuff. E.g.:
If the new files are binary incompatible with the old ones, other software from the distribution built to link against the originals will segfault or otherwise malfunction.
If the new version has a different set of files, you'll end up with a mix of old and new files which may break dependent software as well as the package itself.
If you change the package with your OS' package manager later, it will overwrite pip-installed files, with similarly unpredictable results.
If there are things like configuration files, differences in them between the versions can also lead to all sorts of breakage.
I had the same error and was able to resolve using the following steps:
pip install --ignore-installed pyOpenSSL
This will install the package with latest version and then if you try to install,
pip install twilio
It will work.
Generally, for similar errors, use this format:
pip install --ignore-installed [package name]==[package version]
I just had this error and the only way I was able to resolve it was by manually deleting the offending directory from site-packages.
After doing this you may need to reinstall the packages with --force-reinstall.
Reading the above comments, I understood that package a was installed with conda and the new package b that I was trying to install using pip was causing problems. I was lucky that package b had conda support so using conda to install package b solved the problem.
In my case, I was installing a package from internal git using the following command:
python -m pip install package.whl --force
I was doing this because I didn't want to explicitly uninstall the previous version and just replace it with a newer version. But what it also does is install all the dependencies again. I was getting the error in one of those packages. Removing --force fixed the problem.
I want to add, having --ignore-installed also worked for me. And removing --force is essentially doing the same thing in my case.
I tried to install the Twilio module:
sudo -H pip install twilio
And I got this error:
Installing collected packages: pyOpenSSL
Found existing installation: pyOpenSSL 0.13.1
Cannot uninstall 'pyOpenSSL'. It is a distutils installed project and
thus we cannot accurately determine which files belong to it which
would lead to only a partial uninstall.
Anyone know how to uninstall pyOpenSSL?
This error means that this package's metadata doesn't include a list of files that belong to it. Most probably, you have installed this package via your OS' package manager, so you need to use that rather than pip to update or remove it, too.
See e.g. Upgrading to pip 10: It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. · Issue #5247 · pypa/pip for one such example where the package was installed with apt.
Alternatively, depending on your needs, it may be more productive to not use your system Python and/or its global environment but create a private Python installation and/or environment. There are many options here including virtualenv, venv, pyenv, pipenv and installing Python from source into
/usr/local or $HOME/$HOME/.local (or /opt/<whatever>).
Finally, I must comment on the often-suggested (e.g. at pip 10 and apt: how to avoid "Cannot uninstall X" errors for distutils packages) --ignore-installed pip switch.
It may work (potentially for a long enough time for your business needs), but may just as well break things on the system in unpredictable ways. One thing is sure: it makes the system's configuration unsupported and thus unmaintainable -- because you have essentially overwritten files from your distribution with some other arbitrary stuff. E.g.:
If the new files are binary incompatible with the old ones, other software from the distribution built to link against the originals will segfault or otherwise malfunction.
If the new version has a different set of files, you'll end up with a mix of old and new files which may break dependent software as well as the package itself.
If you change the package with your OS' package manager later, it will overwrite pip-installed files, with similarly unpredictable results.
If there are things like configuration files, differences in them between the versions can also lead to all sorts of breakage.
I had the same error and was able to resolve using the following steps:
pip install --ignore-installed pyOpenSSL
This will install the package with latest version and then if you try to install,
pip install twilio
It will work.
Generally, for similar errors, use this format:
pip install --ignore-installed [package name]==[package version]
I just had this error and the only way I was able to resolve it was by manually deleting the offending directory from site-packages.
After doing this you may need to reinstall the packages with --force-reinstall.
Reading the above comments, I understood that package a was installed with conda and the new package b that I was trying to install using pip was causing problems. I was lucky that package b had conda support so using conda to install package b solved the problem.
In my case, I was installing a package from internal git using the following command:
python -m pip install package.whl --force
I was doing this because I didn't want to explicitly uninstall the previous version and just replace it with a newer version. But what it also does is install all the dependencies again. I was getting the error in one of those packages. Removing --force fixed the problem.
I want to add, having --ignore-installed also worked for me. And removing --force is essentially doing the same thing in my case.
I struggle understanding the pipreqs behaviour.
In my project, I use a virtualenv (Pipreqs is installed globally). I have a requirements.txt file that I first wrote manually, then I did a pip freeze and now I have way more packages than I want (because there are dependencies of dependencies...).
So I found pipreqs to get a requirements file with only the packages that matter.
I activate my virtualenv then pipreqs --savepath requirements2.txt --use-local.
I use --use-local cause I want the currently installed versions of my packages.
And I end up with a file with only one package (not even the version used in my project)...
Without --use-local it seems to retrieve the correct number of packages but the versions are the "up-to-date" ones, not the ones in my project.
Am I doing something wrong ?
Thanks for the help
Is it possible to re-check the dependencies of packages installed with pip? That is, suppose we have a working environment. Then, one of the packages changes (gets upgraded, etc). Is there a command one can run to make to make sure that the dependency tree is still sound and does not have conflicts?
Nowadays python -m pip check should do the trick.
Seems to have been added as early as pip 9.0.0 released on 2016-11-02.
It's not part of pip, but there is a tool you can use called pip-conflict-checker. Just install it through pip and run pipconflictchecker to get a dump of all the conflicts. pipdeptree could also help here.
You might also be interested in reading this article about dealing with pip dependency issues. The article also discusses the two tools I mentioned above along with strategies to fix broken dependencies.
In recent pip versions using pip install -r requirements.txt will fail if you have any conflict in your dependencies specified in requirements.txt.