Recover list of packages installed when Conda environment is broken - python

I borked my Conda environment.
I've spent hours debugging, and think it's time to just wipe it from my MacOS and start fresh. Is there a way to recover the list of packages installed in each environment? The nature of the issue is such that simple commands like conda list and conda config --show-sources (etc., etc.) do not work...

Related

Segmentation fault: 11 when trying to run Spyder in conda environment

I'm setting up a separate environment to work in. However, whenever I try running spyder, I receive this error:
/Users/user/anaconda3/envs/myenv/bin/pythonw: line 3: 84700 Segmentation fault: 11 /Users/user/anaconda3/envs/myenv/python.app/Contents/MacOS/python "$#"
I have no idea what this means. The commands I used to get to this point were:
conda create -n myenv python
conda activate myenv
conda config --add channels conda-forge
conda install -c conda-forge spyder
spyder
I have anaconda navigator installed, and spyder runs fine in the root environment. I do receive the statement "link image0 hasn't been detected!" every time I run it.
I've always used spyder and anaconda, and I've never had these issues before
I was dealing with this same problem for several weeks and spent many days trying to fix it. Of the countless operations I attempted, I finally got spyder to successfully run from its own conda environment without throwing the segmentation fault error you've referenced.
With a fresh install of anaconda3 on macOS 10.14.6, I created a conda environment from the command line just as you’ve shown:
conda create -n myenv python
conda activate myenv
A comment about:
conda create -n myenv python
Note 1: if you list “spyder” to install during the environment creation there’s no need to include “python” as a package unless you wish to have a specific version. For example, “python=2.7.9”, or “python=3.10.*”. Otherwise, python gets added automatically when the environment is created.
Next,
conda config --add channels conda-forge
instead of setting the conda configuration to use the conda-forge channel and instead of including it as a channel when installing spyder into your conda environment, simply install spyder using the virgin (unmodified) default channel configuration by only typing:
conda install spyder
Now when you run spyder from the command line within your active environment it should open as expected.
As previously indicated, you can install spyder during the environment creation:
conda create -n myenv spyder
Then you can activate the environment to test everything:
conda activate myenv
spyder
Note 2: the official spyder documentation recommends including the -c conda-forge channel, but this was the entry that I removed to finally avoid the segmentation fault. I suspect that although the community repository at conda-forge keeps the most up-to-date versions, there are conflicts not being resolved at creation of the new local environment.
Furthermore, you may notice that the method described above installs spyder version 5.3.3 and spyder-kernels version 2.3.3, both of which are not the most up-to-date. You could check this by running conda list from the command line and then searching for the corresponding entries.
However, if you try to update spyder by calling conda update spyder, the system will tell you that all packages are currently installed. This is because the virgin channel defaults are looking at the anaconda repository, not conda-forge. My suggestion for right now is not to attempt updating spyder in your conda environment because it will update other packages as dependencies that will result in breaking the application again.
In case you’re curious, to get the most recent version of spyder based on the procedure I've described above would be by setting the channel as follows: conda update spyder -c conda-forge. Depending on your OS, this will install spyder 5.4.1 and spyder-kernels 2.4.1 (as of 16-Jan-23), but it will also remove and update several other packages, hence causing compatibility issues.
UPDATE:
My key takeaway is to remove the -c conda-forge from the installation line of code to prevent the segmentation fault when using conda. Everything else is up to the user. Installing with pip, brew, or any other method is discretionary but could also result in more trouble than it’s worth for the already frustrated user.
If you run conda create -n myenv spyder at the environment creation or conda install spyder after creating the environment, the packages pyqt and qtpy are installed alongside spyder because they are dependencies and not explicitly needed (unless specifying specific versions). If you run pip install spyder, the packages pyqt5 and qtpy get installed. (as of 16-Jan-23)
Caution:
(with reservation because if you can follow instructions and you are both okay with and the only one affected by the consequences of your choices when things go wrong, I support you to thoughtfully experiment and explore if you so desire)
According to the docs, using pip to install packages within the conda environment is reserved for experienced users. A big reason for this caution is because pip and conda don’t communicate and so package versions and dependencies can become incompatible if the user is not aware of what’s happening. It should also be pointed out that if and when using pip, one should generally install all the desired conda packages prior to installing a package with pip. Problems usually present themselves when updating, or installing with conda after pip. When this becomes necessary the general advice is to start over and install new packages as well as updated versions in a new environment.
Installing with brew is unsupported by the spyder devs. This can be referenced from the spyder documentation.

Is there a way to list all python virtual environments created using the venv module?

Conda allows me to list all virtual environments as shown here. The commands are:
conda info --envs OR conda env list
I want to do that using pip. Does pip have any option to list all virtual environments created by me ? I have created a virtual environment on my desktop but I cannot figure out a way to list it along with the base environment.
No, not in an easy way.
Python virtual environments created with venv, virtualenv and most other python-only virtual environments can be stored anywhere on the disk. And AFAIK they are not indexed, they are truly isolated (after all you can just remove venv directory and be done with it, you don't need to do anything special). They are also unmanaged by an environment manager. So that would require entire disk scan. Which potentially can be done (you can search for all Python executables for example) but is rather painful.
It works with miniconda because miniconda manages other packages and files that it installs, so it places venvs in concrete path, e.g. /home/username/miniconda/envs/.
One indirect way is to run below command and see the directories where the python executable resides. Some of the paths will be python virtual environments and you can verify it by listing the files in those paths.
$ whereis python
You can use conda to create and manage venv and virtualenv environments and other packages installed using pip.
First create a conda environment with CONDA AND PIP installed into it, e.g.,
conda create --name core --channel conda-forge python=3.9 conda pip
Here I created the conda environment named "core" and installed Python 3.9, conda, and pip into it. So now the 'core' conda environment functions like an administrative environment shell. By installing conda into the conda environment, conda will track packages installed by pip into that environment. You must use "pip install" INSIDE this new conda environment, so conda will index and track those pip package installations. However, conda will still not index and centrally manage the venv environments, like it does for its own conda environments.
Here is a very good detailed guide that explains how and why to use conda and pip for virtual environments. It covers the important aspect of using conda and pip together.
Why You Need Python Environments and How to Manage Them with Conda
Create virtual environments for python with conda

Anaconda install used all space, how to free up space again?

I created a conda environment and started installing anaconda, but quickly noticed the size of the library was too big and stopped the install, after having used up around 2 GB of space.
I removed the environment with all installed packages using
conda env remove -n myenv
and then ran
conda clean --all
but my system is still running out of space. I could not free the ~2GB that got used during the anaconda install.
How can I proceed to restore that space?
You need to completely uninstall Anaconda if you want to free up that space. Follow the instructions on the Anaconda documentation
here to uninstall it.

What does conda env do under the hood?

After searching and not finding, I must ask here:
How does conda env work under the hood, meaning, how does anaconda handle environments?
To clarify, I would like an answer or a reference to questions like:
What is kept in the envs/myenv folder?
What happens upon activate myenv?
What happens upon conda install ...?
Where can i find such information?
Conda envs
Basically, conda environments replicate the structure of your system, meaning it will store /bin, /lib, /etc, /var, among other directories. This is more obvious for unix systems, but the same concept is true under windows (DLLs, libs, Scripts, ...).
More details in the official documentation.
Conda install
The idea is that conda install PACKAGE will fetch a precompiled package from a channel (a conda packages repository), and install it under this system-like structure. Instead of relying on system dependencies, conda will install all dependencies of this package under the environment structure, using only conda packages.
Thus installing the same package at a given time point under different systems should result in reliably identical installs.
This is a way to standardize binaries, and it is only achieved by precompiling every package against given versions of libraries, which are shipped as dependencies of the conda environment. For instance, conda-forge and bioconda channels rely on cloud-based CI/CD pipelines to compile all packages on identical and completely clean system images.
Conda also stores metadata about these packages (version, build number, dependencies, license,...) so it is able to solve pretty complex dependency trees and avoid packages/libraries incompatibilities. It is the Solving... step each time you execute conda install.
Conda activate
Then when you conda activate ENV, conda prepends the environment root $CONDA_PREFIX/bin to PATH, so that all executables installed in the environment will be found by the system (and will overload system-wide install of the same executable).
You can imagine it like temporarily replacing the system executables with those from the environment.
More
This a very basic explanation, not 100% accurate, and certainly not complete. If you want to learn more, go read the documentation, experiment with conda, and maybe have an in-depth look to how Conda-forge and Bioconda do build packages, as everything is hosted on github.

Cleaning up system env of conda packages

I forgot to activate a new conda environment,I created before, so I started installing the packages. Now my system is littered with packages. How do I clean this up? I only want my conda packages to be in the base environment and the ones I create.
I checked
How to clean local python environment of conda installs
but that is of no help to me.
Do you have pip installed? If so, run pip freeze and save the output to a file. Review the file, make sure there are no packages listed that you DO NOT WANT to remove. Now, write a script to read in that file to a list, only keeping the package names. Have python loop over that list and uninstall each entry.

Categories