We Keep Coding

conda why create new environment for install

I was suggested to conda create a new environment for installing tensorflow
First question, in general:
Why do environment exist in conda or in Python ? (Why) is it preferable to install a new library in a new environment ?
Here, in practice:
After install conda shell says $conda activate test will activate the test environment. Does it mean i can't access the lib in Spyder unless i activate test in conda shell ? Do i need to restart python shell to see the lib ? I can't access the lib (no module named tensorflow) and I assume it has to do with python not finding the path.

After install conda shell says $conda activate test will activate the
test environment. Does it mean i can't access the lib in Spyder unless
i activate test in conda shell ? Do i need to restart python shell to
see the lib ? I can't access the lib (no module named tensorflow) and
I assume it has to do with python not finding the path.
Have you installed TF within the environment?
I haven't used Spyder in a while, but what usually happens is that you can start a program (like Spyder or Jupyter) from an environment if you have installed the application within it and the environment is active. (Some editors/IDE like VS Code lets you choose the environment for a specific project, once it is able to discover all the environments.)
And, also usually, though perhaps not always, you will not need to restart the shell to import a library, after installing it. It's best to refer to the specific library's installation instructions for details like this.

Virtual Environment is used to manage Python packages for different projects. Using virtual environment allows you to avoid installing Python packages globally which could break system tools or other projects. You can install virtual environment using pip.
For example, say you have two projects, and each requires a different version of Tensorflow. This is a real problem for Python since it can’t differentiate between versions in the “site-packages” directory. So both say V1.1 and V2.1 would reside in the same directory with the same name.
This also allows easy clean up, once you are done with the project just delete the virtual environment.
Checkout more, https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/

MLFlow Projects can't find conda executable

I am following the tutorial on MLFlow website. I was able to run the train.py and mlflow ui worked fine. Packaging the project tries to use env variable MLFLOW_CONDA_HOME but can't find conda.
I have tried setting the variable to the path of anaconda3/condabin but it doesn't seem to find my executable. This is the error I get:
ERROR mlflow.cli: === Could not find Conda executable at /anaconda3/condabin\bin/conda. Ensure Conda is installed as per the inst
ructions at https://conda.io/docs/user-guide/install/index.html. You can also configure MLflow to look for a specific Conda executable by setting the MLFLOW_CONDA_HOME environment variable
to the path of the Conda executable ===
Adding \bin/conda at the end of my path seems to be the problem, I am not sure why mlflow is doing it. I even tried setting it to my python.exe in my conda env, but no luck. I can't find bin/conda folder in my Anaconda folder anywhere.

I resolved this by running it from Anaconda Prompt. Make sure mlflow is installed in anaconda first as well, nothing else. But the problem then is that it's not well compatible on windows, you would need to split into two steps, activate the conda environment and then run with --no-conda as mentioned here https://github.com/mlflow/mlflow/issues/2674

MLflow 1.5 was just released today.
It doesn't specifically mention it in the github notes, but I had the same issue, where it affixed \bin/conda, and now it doesn't do that anymore.

If you don't have conda environment then you can execute the following command from your terminal
mlflow run <enter your local directory name> --no-conda -P alpha=0.5
This should solve the issues with the environment variable.

I solved the issue by removing the MLFLOW_CONDA_HOME environment variable alltogether. Make sure you have added the path to the conda executable to your PATH variable.

Here is one possible solutions (the fastest one, in my opinion).
Key points:
The project virtual environment should be created with conda.
Use pip to install MLFlow.
Follow the steps for Windows:
Install miniconda (in my case, version 3)
Set conda bat file (installation path + condabin dir + conda.bat) in PATH
Create your project without virtual environment (in my case, I set in PyCharm conda instead of venv and it did not create any virtual environment, just added some external libraries), at least not in the project directory.
Create conda virtual environment manually in the project directory. In your project directory, execute conda create -n venv and follow the instructions (I used default for all the questions there).
Open a terminal and activate conda virtual environment. If you use PyCharm, you will be positioned properly, otherwise just prompt yourself in the project directory. Execute conda activate venv where venv is my virtual environment created at point 4.
Execute python -m pip install mlflow
If you want to test it, you can try one of the tests from MLFlow. E.g., you can use mlflow run https://github.com/mlflow/mlflow-example.git -P alpha=5.0
In my case, it worked.

If you're using mlflow.pyfunc.spark_udf and get an error saying Could not find Conda executable conda then try to define the environment variable MLFLOW_CONDA_HOME in spark-env.sh as Spark doesn't recognize variables defined elsewhere. Also make sure to use the absolute path for the Conda executable.

I faced this issue within a kubernetes deployment with miniconda3 as the base image. Fixed this by setting the MLFLOW_CONDA_HOME env variable to "/opt/conda/"

Conda - unable to completely delete environment

I am using Windows 10 (all commands run as administrator). I created an environment called myenv. Then I used
conda env remove -n myenv
Now, if I try
conda info --envs
I only see the base environment. However, if I try
conda activate myenv
I'm still able to activate it! I think because under the folder envs, there is still a folder with the name myenv there which doesn't get deleted.
How do I delete the environment for good?

Command-line options can only go so far, unless you get very specific; perhaps the simplest approach is to delete things manually:
Locate Anaconda folder; I'll use "D:\Anaconda\"
In envs, delete environment of interest: "D:\Anaconda\envs\myenv"
Are you done? Not quite; even while in myenv, conda will still sometimes install packages to the base environment, in "D:\Anaconda\pkgs\"; thus, to clean traces of myenv,
Delete packages installed to myenv that ended up in "D:\Anaconda\pkgs\"
(If above don't suffice) Anaconda Navigator -> Environments -> myenv -> Remove
(If above don't suffice) Likely corrupted Anaconda; make note of installed packages, completely uninstall Anaconda, reinstall.
Note: step 3 is redundant for the goal of simply removing myenv, but it's recommended to minimize future package conflicts.

In addition to the first command in the question posted, I had to complete one additional step to completely remove the environment. I had to go to the folder where the environment was stored (e.g. C:\Users*username*.conda\envs\ on a windows machine) and remove the folder with the same name as the environment I deleted. After this second step, I was able to reuse the environment name without any errors.

Add existing virtualenv to virtualenvwrapper

Before I install virtualenvwrapper, I have some virtualenv installed in different location. Is there any way to collect them in virtualenvwrapper so that appear in workon command?

Yes, that should be possible.
Virtualenvwrapper allows you to custom define where your created environments will be stored:
export WORKON_HOME=/path/to/your/envs
If you point this to the location of your virtual environments from virtualenv, it should work.
You should add this line to your .bashrc or .zshrc or whichever else shell you're using.
The problem with this is that you wont be able to activate any environments that are not in that folder.
In that case it will probably work to just copy the whole virtualenv into where your virtualenvwrapper environments are created.
You can find out where that is like this:
mkvirtualenv test
workon test
which python
# Will print path to virtual python interpreter:
/path/to/virtualenvs/test/bin/python
Copy the desired environments so they are in the same folder
as the just created test environment. Here, this folder would be
/path/to/virtualenvs/. I'll call it $VENVS from now on.
After copying it should be something like /path/to/virtualenvs/my-other_env1, /path/to/virtualenvs/my-other_env2.
Assuming you created my-other_env1 and 2 before with virtualenv with default settings, copying my-other-evn1 can be done like:
cp ~/.virtualenvs/my-other-env1 $VENVS/
You can delete the test environment afterwards using
rmvirtualenv test
(Of course, if you already know what that folder is, it's not necessary to create the test environment.)

Anaconda: Permanently include external packages (like in PYTHONPATH)

I know how to install packages in Anaconda using conda install and also how to install packages that are on PyPi which is described in the manual.
But how can I permanently include packages/folders into the PYTHONPATH of an Anaconda environment so that code that I am currently working on can be imported and is still available after a reboot?
My current approach is to use sys:
import sys
sys.path.append(r'/path/to/my/package')
which is not really convenient.
Any hints?

I found two answers to my question in the Anaconda forum:
1.) Put the modules into into site-packages, i.e. the directory $HOME/path/to/anaconda/lib/pythonX.X/site-packages which is always on sys.path. This should also work by creating a symbolic link.
2.) Add a .pth file to the directory $HOME/path/to/anaconda/lib/pythonX.X/site-packages. This can be named anything (it just must end with .pth). A .pth file is just a newline-separated listing of the full path-names of directories that will be added to your path on Python startup.
Alternatively, if you only want to link to a particular conda environment then add the .pth file to ~/anaconda3/envs/{NAME_OF_ENVIRONMENT}/lib/pythonX.X/site-packages/
Both work straightforward and I went for the second option as it is more flexible.
*** UPDATE:
3.) Use conda develop i. e. conda-develop /path/to/module/ to add the module which creates a .pth file as described under option 2.).
4.) Create a setup.py in the folder of your package and install it using pip install -e /path/to/package which is the cleanest option from my point of view because you can also see all installations using pip list. Note that the option -e allows to edit the package code. See here for more information.
Thanks anyway!

I'm able to include local modules using the following:
conda-develop /path/to/module/
I hope it helps.

The way I do this, which I believe is the most native to conda, is by creating env_vars.sh files in my environment, as per the official documentation here.
For macOS and Linux users, the steps are as follows:
Go to your environment folder (e.g. /miniconda1/env/env_name). $CONDA_PREFIX is the environemnt variable for your environment path.
cd $CONDA_PREFIX
Create the activate.d and deactivate.d directories.
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
Inside the each respective directory, create one env_vars.sh file. The one in the activate.d directory will set (or export) your environment variables when you conda activate your environment. The file in the deactivate.d directory will serve to unset the environment variables when you conda deactivate your environment.
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh
First edit the $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh to export the desired environment variables.
#!/bin/sh
export VAR_A='some-thing-here'
export VAR_B=/path/to/my/file/
Afterwards, open to edit the $CONDA_PREFIX/etc/conda/deactivate/env_vars.sh, in order to unset the env variables when you conda deactivate like so:
#!/bin/sh
unset VAR_A
unset VAR_B
Again, the source of my description comes straight from the conda docs here.

Just to add to Cord Kaldemeyer's answer above, for the 2nd option. If you only want to link to a particular conda environment then add the .pth file to ~/anaconda3/envs/{NAME_OF_ENVIRONMENT}/lib/pythonX.X/site-packages/