I followed the instructions here: Can't find package on Anaconda Navigator. What to do next?
I clicked Open terminal from environment on Anaconda navigator, and then used "pip3 install lmfit" in the terminal. But after installing the lmfit package using pip3, I still cannot find it in the conda list. What should I do?
The Problem
At the time of this question, Conda builds of pip had only just started including a pip3 entrypoint,1 therefore pip3 is very likely referring to a non-Conda version of Python and that is where the package was installed. Try checking which pip3 to find out where it went.
Recommendation
Conda First
Generally, it is preferable to use Conda to install packages in Conda environments, and in this case the package is available via the Conda Forge channel:
conda install -c conda-forge lmfit
Contrary to M. Newville's answer, this recommendation to prefer Conda packages is not about benefiting Conda developers, but instead a rule of thumb to help users avoid creating unstable or unreproducible environments. More info about the risks of mixing pip install and conda install can be found in the post "Using Pip in a Conda Environment".
Nevertheless, the comment that not all packages (and specifically lmfit) are found in the default repository and this complicates installation by requiring resorting to third-party channels is a good point. In fact, because third-parties are free to use different build stacks there are known problems with mixing packages built by Anaconda and those from Conda Forge. However, these issues tend to be rare and limited to compiled code. Additionally, adding trusted channels to a configuration and setting channel priorities heuristically solves the issue.
As for risks in using third-party channels, arbitrary Anaconda Cloud user channels are risky: one should only source packages from channels you trust (just like anything else one installs). Conda Forge in particular is well-reputed and all feedstocks are freely available on GitHub. Moreover, many Python package builds on Conda Forge are simply wrappers around the PyPI build of the package.
PyPI Last
Sometimes it isn't possible to avoid using PyPI. When one must resort to installing from PyPI, it is better practice to use the pip entrypoint from an activate environment, rather than pip3, since only some Conda builds of pip include pip3. For example,
conda activate my_env
pip install lmfit
Again, following the recommendations in "Using Pip in a Conda Environment", one should operate under the assumption that any subsequent calls to conda (install|upgrade|remove) in the environment could have undefined behavior.
PyPI Only
For the sake of completeness, I will note that a stable way of using Conda that is consistent with the recommendations is to limit Conda to the role of environment creation and use pip for all package installation.
This strategy is perhaps the least burden on the Python-only user, who doesn't want to deal with things like finding the Conda-equivalent package name or searching non-default channels. However, its applicability seems limited to Python-only environments, since other libraries may still need to resort to conda install.
[1]: Conda Forge and Anaconda started consistently including pip3 entrypoints for the pip module after version 20.2.
Installing a pure Python package, such as lmfit with the correct version of pip install lmfit should be fine.
Conda first is recommended to make the life of the conda maintainers and packagers easier, not the user's life. FWIW, I maintain both kinds of packages,
and there is no reason to recommend conda install lmfit over pip install lmfit.
In fact, lmfit is not in the default anaconda repository so that installing it requires going to a third-party conda channel such as conda-forge. That adds complexity and risk to your conda environment.
Really, pip install lmfit should be fine.
Related
I'm testing poetry and I was wondering if it is possible to install prebuilt packages from conda-forge, as cartopy without relying on conda (so keeping a 100% poetry process). I googled this a bit but the only way I found is to install poetry within a conda venv using pip and then installing from conda-forge using conda and then tweaking poetry files to make it aware of the conda venv so that the TOML is written properly.
Packages like cartopy are a pain to install if not from a prebuilt version, if possible I'd change my conda stack to poetry stack if something like poetry add [?conda-forge?] cartopy works
Thanks.
Not currently possible. Conda is a generic package manager, not just a Python package manager. Furthermore, there is no dedicated metadata in Conda packages to discriminate whether or not they are Python packages, which I think would be a prerequisite for Poetry being able to determine whether the Conda package is even valid for installation. Hence, what OP requests cannot be a thing, or at least would it be a major undertaking to make it one.
However, others have requested similar features, so someone hopeful for such functionality could subscribe to notifications on those, or follow the Feature Roadmap.
Since conda install and pip install in many cases do essentially the same thing, what would be the best option? Is there a case when someone should stick to pip install only? Symmetrical, is there a case when one should stick to conda install only? Is there a way to shoot in one's foot by using both conda and pip install in a single environment?
If both approaches are essentially the same and don't contradict each other there should be no reason to stick solely to one of them but not to the other.
Don't mix conda install and pip install within conda environment. Probably, decide to use conda or virtualenv+piponce and for all. And here is how you decide which one suits you best:
Conda installs various (not only python) conda-adopted packages within conda environment. It gets your environments right if you are into environments.
Pip installs python packages within Python environment (virtualenv is one of them). It gets your python packages installed right.
Safe way to use conda: don't rush for the latest stuff and stick to the available packages and you'll be fine.
Safe way to use pip+virtualenv: if you see a dependency issue or wish to remove and clean up after package - don't. Just burn the house, abandon your old environment and create a new one. One command line and 2-5 minutes later things gonna be nice and tidy again.
Pip is the best tool for installing Python packages among the two of them. Since pip packages normally come out first and only later are adopted for conda (by conda staff or contributors). Chances are, after updating or installing the latest version of Python some of the packages would only be available through pip. And the latest freshest versions of packages would only be available in pip. And mixing pip and conda packages together can be a nightmare (at least if you want to utilize conda's advantages).
Conda is the best when it comes to managing dependencies and replicating environments. When uninstalling a package conda can properly clean up after itself and has better control over conflicting dependency versions. Also, conda can export environment config and, if the planets are right at the moment and the new machine is not too different, replicate that environment somewhere else. Also, conda can have larger control over the environment and can, for example, have a different version of Python installed inside of it (virtualenv - only the Python available in the system). You can always create a conda package when you have no freedom of choosing what to use.
Some relevant facts:
Conda takes more space and time to setup
Conda might be better if you don't have admin rights on the system
Conda will help when you have no system Python
virtualenv+pip will free you up of knowing lots of details like that
Some outdated notions:
Conda used to be better for novice developers back in the day (2012ish). There is no usability gap anymore
Conda was linked to Continuum Analytics too much. Now Conda itself is open source, the packages - not so much.
Depends on the complexity of your environment really.
Using pip for a few simple packages should not generate any issues.
Using more pip installs raises the question "Why not use a pip venv then?".
If you're not doing anything major, you might be able to have a mix of pip and conda installs.
There is an extensive explanation why mixing them can be a bad idea here: Using Pip in a Conda Environment.
I am using Mac. I am wondering is it possible to have 2 versions of tensor flow co-existing in my computer? I pip installed tensorflow-1.13 and tensor flow-1.8 through two python virtual env. However, there seem to be some problems ...
How do I find out the corresponding c++ tensor flow library in my Mac? Where are they installed? Thanks!
Yes, you can do this with virtual environments: each virtual environment will contain a different version of TensorFlow, and you can switch from one to the other easily. There are many solutions to create virtual environments, but some of the most popular are:
conda
virtualenv
pipenv
Conda is a general-purpose, cross-platform package manager, mostly used with Python, but it can also install many other software packages. A conda environment includes everything, including Python itself, and the system binaries for the libraries you use. So you can have different conda environments with different versions of Python, and different versions of every package you want, including TensorFlow, and any C++ library your code relies on. You can install Anaconda, which is a bundle that includes Conda + Python + many scientific libraries. Or you can install miniconda which includes the bare minimum to run conda.
Virtualenv is a python library which allows you to create virtual environments strictly for Python.
pipenv is also a python library that seems to be gaining a lot of momentum right now, and includes a lot of the functionality of virtualenv.
If you are a beginner, I would recommend going with conda. You will usually run into less issues.
First, download and install either Anaconda or Miniconda.
Next, create a virtual environment:
conda create --name myenv
Then activate this virtual environment:
conda activate myenv
Now you can install all the libraries you need:
conda install whatever-library-you-need
However, not all libraries are available in conda. For example, TensorFlow 2.0 is not there yet (as of May 13th 2019). But that's okay, you can also use pip!
pip install --pre tensorflow
This will install TF 2.0 alpha.
You can then create another environment and install a different version of TF.
You can read more about the interaction between Conda and Pip on the web, but the short story is that they work well together as long as you use pip last. In short, install everything you can with conda, and finish with pip.
After searching and not finding, I must ask here:
How does conda env work under the hood, meaning, how does anaconda handle environments?
To clarify, I would like an answer or a reference to questions like:
What is kept in the envs/myenv folder?
What happens upon activate myenv?
What happens upon conda install ...?
Where can i find such information?
Conda envs
Basically, conda environments replicate the structure of your system, meaning it will store /bin, /lib, /etc, /var, among other directories. This is more obvious for unix systems, but the same concept is true under windows (DLLs, libs, Scripts, ...).
More details in the official documentation.
Conda install
The idea is that conda install PACKAGE will fetch a precompiled package from a channel (a conda packages repository), and install it under this system-like structure. Instead of relying on system dependencies, conda will install all dependencies of this package under the environment structure, using only conda packages.
Thus installing the same package at a given time point under different systems should result in reliably identical installs.
This is a way to standardize binaries, and it is only achieved by precompiling every package against given versions of libraries, which are shipped as dependencies of the conda environment. For instance, conda-forge and bioconda channels rely on cloud-based CI/CD pipelines to compile all packages on identical and completely clean system images.
Conda also stores metadata about these packages (version, build number, dependencies, license,...) so it is able to solve pretty complex dependency trees and avoid packages/libraries incompatibilities. It is the Solving... step each time you execute conda install.
Conda activate
Then when you conda activate ENV, conda prepends the environment root $CONDA_PREFIX/bin to PATH, so that all executables installed in the environment will be found by the system (and will overload system-wide install of the same executable).
You can imagine it like temporarily replacing the system executables with those from the environment.
More
This a very basic explanation, not 100% accurate, and certainly not complete. If you want to learn more, go read the documentation, experiment with conda, and maybe have an in-depth look to how Conda-forge and Bioconda do build packages, as everything is hosted on github.
Does conda allow you to install a dependency into an environment as a development dependency?
I'm thinking of something like how bower does this with --save-dev
AFAICT, no, it does not. This repo represents work around options that might be useful elsewhere:
https://github.com/dazza-codes/conda_container
In short, it supplements a conda install with subsequent pip installs from a requirements.txt and/or a requirements.dev file. Since there can be inconsistencies in conda vs. pip packages (like different name variants etc.), there are use cases for having a combination of conda and pip. Also conda can support a pip array in an environment.yml file but the version specs for conda vs. pip packages are not compatible. Liberal use of pip check is recommended for any combination of packages from different packaging systems.