We Keep Coding

Conflict resolution in pip vs. conda

I assume both pip and conda, despite differences, are package managers and check for consistency of packages installed in an environment! In my case though, I have a list of requirements.txt, on top of python=3.6. In my conda virtual environment, I installed them one-by-one. The strange thing is that when locating some packages in anaconda.org channels and installing them with conda install, conda complains! An example is when I tried to install statistics=1.0.3.5, and I got this message on terminal:
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
- statistics=1.0.3.5 -> python[version='2.7.*|<3|>=2.7,<2.8.0a0']
Your python: python=3.6
However, when I did it with pip, it worked!
Why is that?
Am I going to bump into a problem down the road with this package?
I read this Stackoverflow post about the difference between pip and conda and tried to understand it from the doc (Although not that successful).

When working with conda virtual environments, installing packages with pip should be the last resort. If a package isn't available through the default channel, try installing from conda-forge first.
The difference between conda and pip is huge (not to mention virtual environments): Conda aims to install a consistent set of packages - which results in an optimization problem - while pip just installs dependencies, no matter if that is in conflict with any previously installed package.
However, since you are writing unit tests with your code you'll immediatly realize if you bump into a problem.

Install prebuilt packages from conda-forge (e.g. cartopy) using poetry without relying on conda (using only the channel)

I'm testing poetry and I was wondering if it is possible to install prebuilt packages from conda-forge, as cartopy without relying on conda (so keeping a 100% poetry process). I googled this a bit but the only way I found is to install poetry within a conda venv using pip and then installing from conda-forge using conda and then tweaking poetry files to make it aware of the conda venv so that the TOML is written properly.
Packages like cartopy are a pain to install if not from a prebuilt version, if possible I'd change my conda stack to poetry stack if something like poetry add [?conda-forge?] cartopy works
Thanks.

Not currently possible. Conda is a generic package manager, not just a Python package manager. Furthermore, there is no dedicated metadata in Conda packages to discriminate whether or not they are Python packages, which I think would be a prerequisite for Poetry being able to determine whether the Conda package is even valid for installation. Hence, what OP requests cannot be a thing, or at least would it be a major undertaking to make it one.
However, others have requested similar features, so someone hopeful for such functionality could subscribe to notifications on those, or follow the Feature Roadmap.

Cannot find package on Anaconda navigator after installing it using pip

I followed the instructions here: Can't find package on Anaconda Navigator. What to do next?
I clicked Open terminal from environment on Anaconda navigator, and then used "pip3 install lmfit" in the terminal. But after installing the lmfit package using pip3, I still cannot find it in the conda list. What should I do?

The Problem
At the time of this question, Conda builds of pip had only just started including a pip3 entrypoint,1 therefore pip3 is very likely referring to a non-Conda version of Python and that is where the package was installed. Try checking which pip3 to find out where it went.
Recommendation
Conda First
Generally, it is preferable to use Conda to install packages in Conda environments, and in this case the package is available via the Conda Forge channel:
conda install -c conda-forge lmfit
Contrary to M. Newville's answer, this recommendation to prefer Conda packages is not about benefiting Conda developers, but instead a rule of thumb to help users avoid creating unstable or unreproducible environments. More info about the risks of mixing pip install and conda install can be found in the post "Using Pip in a Conda Environment".
Nevertheless, the comment that not all packages (and specifically lmfit) are found in the default repository and this complicates installation by requiring resorting to third-party channels is a good point. In fact, because third-parties are free to use different build stacks there are known problems with mixing packages built by Anaconda and those from Conda Forge. However, these issues tend to be rare and limited to compiled code. Additionally, adding trusted channels to a configuration and setting channel priorities heuristically solves the issue.
As for risks in using third-party channels, arbitrary Anaconda Cloud user channels are risky: one should only source packages from channels you trust (just like anything else one installs). Conda Forge in particular is well-reputed and all feedstocks are freely available on GitHub. Moreover, many Python package builds on Conda Forge are simply wrappers around the PyPI build of the package.
PyPI Last
Sometimes it isn't possible to avoid using PyPI. When one must resort to installing from PyPI, it is better practice to use the pip entrypoint from an activate environment, rather than pip3, since only some Conda builds of pip include pip3. For example,
conda activate my_env
pip install lmfit
Again, following the recommendations in "Using Pip in a Conda Environment", one should operate under the assumption that any subsequent calls to conda (install|upgrade|remove) in the environment could have undefined behavior.
PyPI Only
For the sake of completeness, I will note that a stable way of using Conda that is consistent with the recommendations is to limit Conda to the role of environment creation and use pip for all package installation.
This strategy is perhaps the least burden on the Python-only user, who doesn't want to deal with things like finding the Conda-equivalent package name or searching non-default channels. However, its applicability seems limited to Python-only environments, since other libraries may still need to resort to conda install.
[1]: Conda Forge and Anaconda started consistently including pip3 entrypoints for the pip module after version 20.2.

Installing a pure Python package, such as lmfit with the correct version of pip install lmfit should be fine.
Conda first is recommended to make the life of the conda maintainers and packagers easier, not the user's life. FWIW, I maintain both kinds of packages,
and there is no reason to recommend conda install lmfit over pip install lmfit.
In fact, lmfit is not in the default anaconda repository so that installing it requires going to a third-party conda channel such as conda-forge. That adds complexity and risk to your conda environment.
Really, pip install lmfit should be fine.

Fail to uninstall distutils installed project [duplicate]

I tried to install the Twilio module:
sudo -H pip install twilio
And I got this error:
Installing collected packages: pyOpenSSL
Found existing installation: pyOpenSSL 0.13.1
Cannot uninstall 'pyOpenSSL'. It is a distutils installed project and
thus we cannot accurately determine which files belong to it which
would lead to only a partial uninstall.
Anyone know how to uninstall pyOpenSSL?

This error means that this package's metadata doesn't include a list of files that belong to it. Most probably, you have installed this package via your OS' package manager, so you need to use that rather than pip to update or remove it, too.
See e.g. Upgrading to pip 10: It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. · Issue #5247 · pypa/pip for one such example where the package was installed with apt.
Alternatively, depending on your needs, it may be more productive to not use your system Python and/or its global environment but create a private Python installation and/or environment. There are many options here including virtualenv, venv, pyenv, pipenv and installing Python from source into
/usr/local or $HOME/$HOME/.local (or /opt/<whatever>).
Finally, I must comment on the often-suggested (e.g. at pip 10 and apt: how to avoid "Cannot uninstall X" errors for distutils packages) --ignore-installed pip switch.
It may work (potentially for a long enough time for your business needs), but may just as well break things on the system in unpredictable ways. One thing is sure: it makes the system's configuration unsupported and thus unmaintainable -- because you have essentially overwritten files from your distribution with some other arbitrary stuff. E.g.:
If the new files are binary incompatible with the old ones, other software from the distribution built to link against the originals will segfault or otherwise malfunction.
If the new version has a different set of files, you'll end up with a mix of old and new files which may break dependent software as well as the package itself.
If you change the package with your OS' package manager later, it will overwrite pip-installed files, with similarly unpredictable results.
If there are things like configuration files, differences in them between the versions can also lead to all sorts of breakage.

I had the same error and was able to resolve using the following steps:
pip install --ignore-installed pyOpenSSL
This will install the package with latest version and then if you try to install,
pip install twilio
It will work.

Generally, for similar errors, use this format:
pip install --ignore-installed [package name]==[package version]

I just had this error and the only way I was able to resolve it was by manually deleting the offending directory from site-packages.
After doing this you may need to reinstall the packages with --force-reinstall.

Reading the above comments, I understood that package a was installed with conda and the new package b that I was trying to install using pip was causing problems. I was lucky that package b had conda support so using conda to install package b solved the problem.

In my case, I was installing a package from internal git using the following command:
python -m pip install package.whl --force
I was doing this because I didn't want to explicitly uninstall the previous version and just replace it with a newer version. But what it also does is install all the dependencies again. I was getting the error in one of those packages. Removing --force fixed the problem.
I want to add, having --ignore-installed also worked for me. And removing --force is essentially doing the same thing in my case.

pip cannot uninstall <package>: "It is a distutils installed project"

I tried to install the Twilio module:
sudo -H pip install twilio
And I got this error:
Installing collected packages: pyOpenSSL
Found existing installation: pyOpenSSL 0.13.1
Cannot uninstall 'pyOpenSSL'. It is a distutils installed project and
thus we cannot accurately determine which files belong to it which
would lead to only a partial uninstall.
Anyone know how to uninstall pyOpenSSL?

This error means that this package's metadata doesn't include a list of files that belong to it. Most probably, you have installed this package via your OS' package manager, so you need to use that rather than pip to update or remove it, too.
See e.g. Upgrading to pip 10: It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. · Issue #5247 · pypa/pip for one such example where the package was installed with apt.
Alternatively, depending on your needs, it may be more productive to not use your system Python and/or its global environment but create a private Python installation and/or environment. There are many options here including virtualenv, venv, pyenv, pipenv and installing Python from source into
/usr/local or $HOME/$HOME/.local (or /opt/<whatever>).
Finally, I must comment on the often-suggested (e.g. at pip 10 and apt: how to avoid "Cannot uninstall X" errors for distutils packages) --ignore-installed pip switch.
It may work (potentially for a long enough time for your business needs), but may just as well break things on the system in unpredictable ways. One thing is sure: it makes the system's configuration unsupported and thus unmaintainable -- because you have essentially overwritten files from your distribution with some other arbitrary stuff. E.g.:
If the new files are binary incompatible with the old ones, other software from the distribution built to link against the originals will segfault or otherwise malfunction.
If the new version has a different set of files, you'll end up with a mix of old and new files which may break dependent software as well as the package itself.
If you change the package with your OS' package manager later, it will overwrite pip-installed files, with similarly unpredictable results.
If there are things like configuration files, differences in them between the versions can also lead to all sorts of breakage.

I had the same error and was able to resolve using the following steps:
pip install --ignore-installed pyOpenSSL
This will install the package with latest version and then if you try to install,
pip install twilio
It will work.

Generally, for similar errors, use this format:
pip install --ignore-installed [package name]==[package version]

I just had this error and the only way I was able to resolve it was by manually deleting the offending directory from site-packages.
After doing this you may need to reinstall the packages with --force-reinstall.

Reading the above comments, I understood that package a was installed with conda and the new package b that I was trying to install using pip was causing problems. I was lucky that package b had conda support so using conda to install package b solved the problem.

In my case, I was installing a package from internal git using the following command:
python -m pip install package.whl --force
I was doing this because I didn't want to explicitly uninstall the previous version and just replace it with a newer version. But what it also does is install all the dependencies again. I was getting the error in one of those packages. Removing --force fixed the problem.
I want to add, having --ignore-installed also worked for me. And removing --force is essentially doing the same thing in my case.