conda env remove -n envname does not delete the actual folder? - python

I have a question regarding conda environments. Whenever I try to remove an environment in command prompt with conda env remove -n envname, the actual folder remains in the environments folder. Thus, if I try to recreate the environment with the same name, command prompt gives me an error saying that a folder already exists there and whether or not I would like to proceed.
$ conda create --name test
WARNING: A directory already exists at the target location 'C:\...\Anaconda3\envs\test'
but it is not a conda environment.
Continue creating environment (y/[n])?
I'm not sure whether that is safe, so at this point I manually delete the folder. Is this something that is viable? Is leaving the folder there normal conda behavior? Is there a solution to allow conda to just automatically delete the folder?
Thanks so much!

Ok, I think I figured out the issue. Before you try to remove the env, make sure that:
The env isn't still attached to any running processes. For example, if you'd used it in VS Code, make sure you close VS Code (that might assume you're using the Anaconda Extension pack, not sure, haven't tried without it).
Make sure you use conda deactivate to deactivate the env you're about to remove.
If the env (or maybe more correctly, the folders where it was stored) are still referenced by a running process or you haven't deactivated the env, conda env remove will act like it's removing the env and it won't show up in conda env list, but it will still be possible to activate it and the folders will remain.

Related

How to get `conda env list` to hide an environment without removing it?

I have a virtual environment in a folder which I have not yet activated. Running conda env list will not list this environment.
However, after I have activated the environment with Conda for the first time, now every time I run conda env list this environment will be included.
Is there a way to ignore the environment in conda env list, without removing it (since this removes the whole folder)?
I want to keep the folder without removing it and without Conda listing it every time.
AFAIK, there is no way through configuration or otherwise out-of-the-box options to designate specific environments to be ignored by the conda env list command. However, if we look under the hood at how Conda generates this list, we can at least implement a workaround.
Conda User Environment Tracking
Conda tracks environments in two ways:
Environments located in any of the envs_dirs directories are automatically discovered. You can check conda config --show envs_dirs to see which directories that includes. Such environments will not be ignorable in conda env list without altering the internals of how conda-env works (i.e., you'd have to fork the conda code and add new functionality).
Any time a user activates an environment, it gets added to a user-specific tracking file, namely,
~/.conda/environments.txt
where ~ is the user home directory. Purging this file of environments you wish to hide should remove the environment from the conda env list output, at least until it is activated again.
Automated Workaround Example
If you'd like a more automated purging, you could include a line in your shell initialization file (e.g., .bash_profile) to remove entries from this file using something like sed and a regex to match the environments you wish to omit.
As a concrete example of this, I frequently encounter this problem as Snakemake user. Snakemake automatically generates Conda environments and uses them to run code in reproducible (-ish) contexts. These environments all get generated under a .snakemake/ directory and eventually start to dominate my conda env list output. This is an absolute bother, since I never intend to manually activate any of these environments, plus they are all named by hashes, so it is practically impossible to recognize their contents by prefix.
To automatically purge these, I can add the following to my .bashrc or .bash_profile:
sed -i '/\.snakemake/d' ~/.conda/environments.txt
This will still lead to transiently showing these auto-generated environments temporarily, but they'll get purged every time a new shell launches. Hopefully, such transient cases aren't a major bother, otherwise I imagine more creative solutions to this are also workable, e.g., triggering the purge operation whenever the file is altered.

MLFlow Projects can't find conda executable

I am following the tutorial on MLFlow website. I was able to run the train.py and mlflow ui worked fine. Packaging the project tries to use env variable MLFLOW_CONDA_HOME but can't find conda.
I have tried setting the variable to the path of anaconda3/condabin but it doesn't seem to find my executable. This is the error I get:
ERROR mlflow.cli: === Could not find Conda executable at /anaconda3/condabin\bin/conda. Ensure Conda is installed as per the inst
ructions at https://conda.io/docs/user-guide/install/index.html. You can also configure MLflow to look for a specific Conda executable by setting the MLFLOW_CONDA_HOME environment variable
to the path of the Conda executable ===
Adding \bin/conda at the end of my path seems to be the problem, I am not sure why mlflow is doing it. I even tried setting it to my python.exe in my conda env, but no luck. I can't find bin/conda folder in my Anaconda folder anywhere.
I resolved this by running it from Anaconda Prompt. Make sure mlflow is installed in anaconda first as well, nothing else. But the problem then is that it's not well compatible on windows, you would need to split into two steps, activate the conda environment and then run with --no-conda as mentioned here https://github.com/mlflow/mlflow/issues/2674
MLflow 1.5 was just released today.
It doesn't specifically mention it in the github notes, but I had the same issue, where it affixed \bin/conda, and now it doesn't do that anymore.
If you don't have conda environment then you can execute the following command from your terminal
mlflow run <enter your local directory name> --no-conda -P alpha=0.5
This should solve the issues with the environment variable.
I solved the issue by removing the MLFLOW_CONDA_HOME environment variable alltogether. Make sure you have added the path to the conda executable to your PATH variable.
Here is one possible solutions (the fastest one, in my opinion).
Key points:
The project virtual environment should be created with conda.
Use pip to install MLFlow.
Follow the steps for Windows:
Install miniconda (in my case, version 3)
Set conda bat file (installation path + condabin dir + conda.bat) in PATH
Create your project without virtual environment (in my case, I set in PyCharm conda instead of venv and it did not create any virtual environment, just added some external libraries), at least not in the project directory.
Create conda virtual environment manually in the project directory. In your project directory, execute conda create -n venv and follow the instructions (I used default for all the questions there).
Open a terminal and activate conda virtual environment. If you use PyCharm, you will be positioned properly, otherwise just prompt yourself in the project directory. Execute conda activate venv where venv is my virtual environment created at point 4.
Execute python -m pip install mlflow
If you want to test it, you can try one of the tests from MLFlow. E.g., you can use mlflow run https://github.com/mlflow/mlflow-example.git -P alpha=5.0
In my case, it worked.
If you're using mlflow.pyfunc.spark_udf and get an error saying Could not find Conda executable conda then try to define the environment variable MLFLOW_CONDA_HOME in spark-env.sh as Spark doesn't recognize variables defined elsewhere. Also make sure to use the absolute path for the Conda executable.
I faced this issue within a kubernetes deployment with miniconda3 as the base image. Fixed this by setting the MLFLOW_CONDA_HOME env variable to "/opt/conda/"

Conda - unable to completely delete environment

I am using Windows 10 (all commands run as administrator). I created an environment called myenv. Then I used
conda env remove -n myenv
Now, if I try
conda info --envs
I only see the base environment. However, if I try
conda activate myenv
I'm still able to activate it! I think because under the folder envs, there is still a folder with the name myenv there which doesn't get deleted.
How do I delete the environment for good?
Command-line options can only go so far, unless you get very specific; perhaps the simplest approach is to delete things manually:
Locate Anaconda folder; I'll use "D:\Anaconda\"
In envs, delete environment of interest: "D:\Anaconda\envs\myenv"
Are you done? Not quite; even while in myenv, conda will still sometimes install packages to the base environment, in "D:\Anaconda\pkgs\"; thus, to clean traces of myenv,
Delete packages installed to myenv that ended up in "D:\Anaconda\pkgs\"
(If above don't suffice) Anaconda Navigator -> Environments -> myenv -> Remove
(If above don't suffice) Likely corrupted Anaconda; make note of installed packages, completely uninstall Anaconda, reinstall.
Note: step 3 is redundant for the goal of simply removing myenv, but it's recommended to minimize future package conflicts.
In addition to the first command in the question posted, I had to complete one additional step to completely remove the environment. I had to go to the folder where the environment was stored (e.g. C:\Users*username*.conda\envs\ on a windows machine) and remove the folder with the same name as the environment I deleted. After this second step, I was able to reuse the environment name without any errors.

Why my conda environment inherits all base packages?

I am trying to create a "clean" Python virtual environment using conda:
conda create -n somename python=3.7 --no-default-packages
But somehow I have access to all the packages installed in base inside this environment. pip list returns all the Python packages, and I can import those packages in Python.
However, the actual environment's "site-packages" folder does not contain those extra Python modules as the base folder does.
So what should I do to create an independent/separate virtual environment? I am using Windows10.
I had PYTHONPATH/HOME explicitly specified in path, after deleting now it works good.
It sounds silly, but make sure that you are actually activating the new environment. Also make sure to check that which python and which pip refer to the new environment, I've had problems before where tmux makes conda activations silently fail.
I would also check your PYTHONPATH variable
echo $PYTHONPATH
just in case you inherit dist-packages (check your ~/.profile and ~/.bashrc)

Add existing virtualenv to virtualenvwrapper

Before I install virtualenvwrapper, I have some virtualenv installed in different location. Is there any way to collect them in virtualenvwrapper so that appear in workon command?
Yes, that should be possible.
Virtualenvwrapper allows you to custom define where your created environments will be stored:
export WORKON_HOME=/path/to/your/envs
If you point this to the location of your virtual environments from virtualenv, it should work.
You should add this line to your .bashrc or .zshrc or whichever else shell you're using.
The problem with this is that you wont be able to activate any environments that are not in that folder.
In that case it will probably work to just copy the whole virtualenv into where your virtualenvwrapper environments are created.
You can find out where that is like this:
mkvirtualenv test
workon test
which python
# Will print path to virtual python interpreter:
/path/to/virtualenvs/test/bin/python
Copy the desired environments so they are in the same folder
as the just created test environment. Here, this folder would be
/path/to/virtualenvs/. I'll call it $VENVS from now on.
After copying it should be something like /path/to/virtualenvs/my-other_env1, /path/to/virtualenvs/my-other_env2.
Assuming you created my-other_env1 and 2 before with virtualenv with default settings, copying my-other-evn1 can be done like:
cp ~/.virtualenvs/my-other-env1 $VENVS/
You can delete the test environment afterwards using
rmvirtualenv test
(Of course, if you already know what that folder is, it's not necessary to create the test environment.)

Categories