I've been using Anaconda for a few years now, but since I started using a Mac with a M1 processor I had to deal with a bunch of problem with the installation of some packages, which left me a little confused about some basic concepts.
For example, I was trying to install Tensorflow, and it turns out the proper way is to install miniforge, and get Tensorflow from the conda-forge channel (which is the default for miniforge), as explained here.
Then, I was wondering whether I could do the same using Anaconda/Miniconda...set up the conda-forge channel as default, and install Tensorflow (or any other arm-compatible package), but I've been told it's not possible
So, now I'm trying to understand how this all works.
If a Tensorflow version compatible with M1 processors exists in the conda-forge channel (and it does exist), why can't I install it by using Anaconda/Miniconda, after configuring it to use said channel? To phrase it in another way, what is the difference between Anaconda/Miniconda and Miniforge, other than the channels they look into for packages (and, as I understand, some licenses)?
Here there is a similar question, but the answers don't seem to address my main concern (why Anaconda/Miniconda with conda-forge as default channel is different than miniforge).
It's not impossible, but you'll have to jump through hoops to get it done.
First, if you have an Anaconda installation, you can't install conda-forge packages into the base environment consistently, because the anaconda package in the base environment of Anaconda will conflict with packages from conda-forge.
Second, since Anaconda is only x86_64 at the moment, you can only install it via Rossetta emulation. After that, you need to tell conda that you need arm64 compatible packages by setting the env variable CONDA_SUBDIR.
CONDA_SUBDIR=osx-arm64 conda create -n native numpy -c conda-forge
will get you a new env with native arm64 packages. However if you want to update this env, you have to prefix all your conda commands with CONDA_SUBDIR=osx-arm64.
To fix this permanently, you can do the following
conda activate native
conda config --env --set subdir osx-arm64
which will make conda use osx-arm64 for this environment.
Related
I'm setting up a separate environment to work in. However, whenever I try running spyder, I receive this error:
/Users/user/anaconda3/envs/myenv/bin/pythonw: line 3: 84700 Segmentation fault: 11 /Users/user/anaconda3/envs/myenv/python.app/Contents/MacOS/python "$#"
I have no idea what this means. The commands I used to get to this point were:
conda create -n myenv python
conda activate myenv
conda config --add channels conda-forge
conda install -c conda-forge spyder
spyder
I have anaconda navigator installed, and spyder runs fine in the root environment. I do receive the statement "link image0 hasn't been detected!" every time I run it.
I've always used spyder and anaconda, and I've never had these issues before
I was dealing with this same problem for several weeks and spent many days trying to fix it. Of the countless operations I attempted, I finally got spyder to successfully run from its own conda environment without throwing the segmentation fault error you've referenced.
With a fresh install of anaconda3 on macOS 10.14.6, I created a conda environment from the command line just as you’ve shown:
conda create -n myenv python
conda activate myenv
A comment about:
conda create -n myenv python
Note 1: if you list “spyder” to install during the environment creation there’s no need to include “python” as a package unless you wish to have a specific version. For example, “python=2.7.9”, or “python=3.10.*”. Otherwise, python gets added automatically when the environment is created.
Next,
conda config --add channels conda-forge
instead of setting the conda configuration to use the conda-forge channel and instead of including it as a channel when installing spyder into your conda environment, simply install spyder using the virgin (unmodified) default channel configuration by only typing:
conda install spyder
Now when you run spyder from the command line within your active environment it should open as expected.
As previously indicated, you can install spyder during the environment creation:
conda create -n myenv spyder
Then you can activate the environment to test everything:
conda activate myenv
spyder
Note 2: the official spyder documentation recommends including the -c conda-forge channel, but this was the entry that I removed to finally avoid the segmentation fault. I suspect that although the community repository at conda-forge keeps the most up-to-date versions, there are conflicts not being resolved at creation of the new local environment.
Furthermore, you may notice that the method described above installs spyder version 5.3.3 and spyder-kernels version 2.3.3, both of which are not the most up-to-date. You could check this by running conda list from the command line and then searching for the corresponding entries.
However, if you try to update spyder by calling conda update spyder, the system will tell you that all packages are currently installed. This is because the virgin channel defaults are looking at the anaconda repository, not conda-forge. My suggestion for right now is not to attempt updating spyder in your conda environment because it will update other packages as dependencies that will result in breaking the application again.
In case you’re curious, to get the most recent version of spyder based on the procedure I've described above would be by setting the channel as follows: conda update spyder -c conda-forge. Depending on your OS, this will install spyder 5.4.1 and spyder-kernels 2.4.1 (as of 16-Jan-23), but it will also remove and update several other packages, hence causing compatibility issues.
UPDATE:
My key takeaway is to remove the -c conda-forge from the installation line of code to prevent the segmentation fault when using conda. Everything else is up to the user. Installing with pip, brew, or any other method is discretionary but could also result in more trouble than it’s worth for the already frustrated user.
If you run conda create -n myenv spyder at the environment creation or conda install spyder after creating the environment, the packages pyqt and qtpy are installed alongside spyder because they are dependencies and not explicitly needed (unless specifying specific versions). If you run pip install spyder, the packages pyqt5 and qtpy get installed. (as of 16-Jan-23)
Caution:
(with reservation because if you can follow instructions and you are both okay with and the only one affected by the consequences of your choices when things go wrong, I support you to thoughtfully experiment and explore if you so desire)
According to the docs, using pip to install packages within the conda environment is reserved for experienced users. A big reason for this caution is because pip and conda don’t communicate and so package versions and dependencies can become incompatible if the user is not aware of what’s happening. It should also be pointed out that if and when using pip, one should generally install all the desired conda packages prior to installing a package with pip. Problems usually present themselves when updating, or installing with conda after pip. When this becomes necessary the general advice is to start over and install new packages as well as updated versions in a new environment.
Installing with brew is unsupported by the spyder devs. This can be referenced from the spyder documentation.
I just started a new job, and they gave me a Mac pro to work with. I was reading about the new M1 processor and everything, I found this article particularly useful.
However, I still have one doubt about it. If I didn’t misunderstand, in the article they suggest using either Anaconda (with Rosetta2 translation), OR to install Miniforge, which sets conda-forge as default and only channel, in order to download packages compiled specifically for the M1. I found different tutorials, and they all suggest using Miniforge to install packages on the M1.
My question is…can’t I just use Anaconda, and set conda-forge as the only installation channel, to get M1-compiled packages, like it would do with Miniforge? Or am I misunderstanding something?
Thanks!
Update
I stand corrected. A bit of explanation: In each channel, there exist subdirs for different OS and architectures. When you install conda, then it will automatically look in the correct subdir for all channels that you configure. The difference between a conda that was installed for arm and x64 will be in what subdirs it looks. E.g. for my conda on a linux machine (installed through miniconda) where I added the conda-forge channel manually, the list of channels actually looks like this:
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
On your MAC, it will look similar for your 64bit installation of anaconda/miniconda. Now to the difference between miniforge and anaconda/miniconda: The first is already preconfigured to use the conda-forge channel, the latter ones can be configured to to the same. So far, no difference. The sole difference will be that you can only install x64 versions of anaconda and miniconda, but miniforge can be installed as an arm64 installation, which will look into different sub-directories of conda-forge, i.e. conda-forge/osx-arm64.
You can however modify which sub-directory is considered when a conda command is run. I have no experience on how stable that works though. This can be achieved by setting the environment variable CONDA_SUBDIR=osx-arm64 which you can set each time before running a specific command, or you just set it using the conda env config command for your environment. You will have to make sure that all packages in an env will be installed while the CONDA_SUBDIR=osx-arm64 variable is set though.
Old
There are currently only 64 bit installers for anaconda. All that come along it will be 64bit. All modules installed will be targeted for 64 bit and conda will download 64bit modules. You cannot install arm modules to a 64 bit conda installation. You will need to decide:
"Convenience" (very subjective) of anaconda, i.e. getting a bunch of pre installed packages targeted for 64 bit
An installation specific for your architecture with possible speed and other advantages
I followed the instructions here: Can't find package on Anaconda Navigator. What to do next?
I clicked Open terminal from environment on Anaconda navigator, and then used "pip3 install lmfit" in the terminal. But after installing the lmfit package using pip3, I still cannot find it in the conda list. What should I do?
The Problem
At the time of this question, Conda builds of pip had only just started including a pip3 entrypoint,1 therefore pip3 is very likely referring to a non-Conda version of Python and that is where the package was installed. Try checking which pip3 to find out where it went.
Recommendation
Conda First
Generally, it is preferable to use Conda to install packages in Conda environments, and in this case the package is available via the Conda Forge channel:
conda install -c conda-forge lmfit
Contrary to M. Newville's answer, this recommendation to prefer Conda packages is not about benefiting Conda developers, but instead a rule of thumb to help users avoid creating unstable or unreproducible environments. More info about the risks of mixing pip install and conda install can be found in the post "Using Pip in a Conda Environment".
Nevertheless, the comment that not all packages (and specifically lmfit) are found in the default repository and this complicates installation by requiring resorting to third-party channels is a good point. In fact, because third-parties are free to use different build stacks there are known problems with mixing packages built by Anaconda and those from Conda Forge. However, these issues tend to be rare and limited to compiled code. Additionally, adding trusted channels to a configuration and setting channel priorities heuristically solves the issue.
As for risks in using third-party channels, arbitrary Anaconda Cloud user channels are risky: one should only source packages from channels you trust (just like anything else one installs). Conda Forge in particular is well-reputed and all feedstocks are freely available on GitHub. Moreover, many Python package builds on Conda Forge are simply wrappers around the PyPI build of the package.
PyPI Last
Sometimes it isn't possible to avoid using PyPI. When one must resort to installing from PyPI, it is better practice to use the pip entrypoint from an activate environment, rather than pip3, since only some Conda builds of pip include pip3. For example,
conda activate my_env
pip install lmfit
Again, following the recommendations in "Using Pip in a Conda Environment", one should operate under the assumption that any subsequent calls to conda (install|upgrade|remove) in the environment could have undefined behavior.
PyPI Only
For the sake of completeness, I will note that a stable way of using Conda that is consistent with the recommendations is to limit Conda to the role of environment creation and use pip for all package installation.
This strategy is perhaps the least burden on the Python-only user, who doesn't want to deal with things like finding the Conda-equivalent package name or searching non-default channels. However, its applicability seems limited to Python-only environments, since other libraries may still need to resort to conda install.
[1]: Conda Forge and Anaconda started consistently including pip3 entrypoints for the pip module after version 20.2.
Installing a pure Python package, such as lmfit with the correct version of pip install lmfit should be fine.
Conda first is recommended to make the life of the conda maintainers and packagers easier, not the user's life. FWIW, I maintain both kinds of packages,
and there is no reason to recommend conda install lmfit over pip install lmfit.
In fact, lmfit is not in the default anaconda repository so that installing it requires going to a third-party conda channel such as conda-forge. That adds complexity and risk to your conda environment.
Really, pip install lmfit should be fine.
I used homebrew to install graph-tool on a macOS 10.14 using the command brew install graph-tool, but I cannot import it in python (that is installed through anaconda in /opt/anaconda3/bin/python). How can I make Python able to import graph-tool?
Prefer Conda Packages
If you want to use something in a Conda environment, then the best practice is to install through Conda. In this case, the graph-tool package is available through Conda Forge and can be installed with
conda install -c conda-forge graph-tool
Personally, since I started using Conda in 2016, I've found no reason to use Homebrew or MacPorts. There might be some exceptions, but this package isn't one.
conda install not working
Sometimes the packages you already have installed can conflict with the ones you want to add. This is especially the case when it comes to massive envs like the base in Anaconda. Best practice for using Conda is to avoid installing in base and instead create envs for each project or project type. This also helps you avoid breaking code you've already written because sometimes installs can trigger a version change in a package you were using.
In this case, you could make a new env, say foo, with
conda create --name foo -c conda-forge graph-tool
You can list any additional packages you know you'll need after graph-tool.
I am using Mac. I am wondering is it possible to have 2 versions of tensor flow co-existing in my computer? I pip installed tensorflow-1.13 and tensor flow-1.8 through two python virtual env. However, there seem to be some problems ...
How do I find out the corresponding c++ tensor flow library in my Mac? Where are they installed? Thanks!
Yes, you can do this with virtual environments: each virtual environment will contain a different version of TensorFlow, and you can switch from one to the other easily. There are many solutions to create virtual environments, but some of the most popular are:
conda
virtualenv
pipenv
Conda is a general-purpose, cross-platform package manager, mostly used with Python, but it can also install many other software packages. A conda environment includes everything, including Python itself, and the system binaries for the libraries you use. So you can have different conda environments with different versions of Python, and different versions of every package you want, including TensorFlow, and any C++ library your code relies on. You can install Anaconda, which is a bundle that includes Conda + Python + many scientific libraries. Or you can install miniconda which includes the bare minimum to run conda.
Virtualenv is a python library which allows you to create virtual environments strictly for Python.
pipenv is also a python library that seems to be gaining a lot of momentum right now, and includes a lot of the functionality of virtualenv.
If you are a beginner, I would recommend going with conda. You will usually run into less issues.
First, download and install either Anaconda or Miniconda.
Next, create a virtual environment:
conda create --name myenv
Then activate this virtual environment:
conda activate myenv
Now you can install all the libraries you need:
conda install whatever-library-you-need
However, not all libraries are available in conda. For example, TensorFlow 2.0 is not there yet (as of May 13th 2019). But that's okay, you can also use pip!
pip install --pre tensorflow
This will install TF 2.0 alpha.
You can then create another environment and install a different version of TF.
You can read more about the interaction between Conda and Pip on the web, but the short story is that they work well together as long as you use pip last. In short, install everything you can with conda, and finish with pip.