Activating conda environment during gitlab CI - python

My .gitlab-ci.yml file looks like this:
anomalydetector:
image: continuumio/miniconda:4.7.10
stage: build
tags:
- docker
script:
- conda env create -f environment.yml
- conda activate my-env
- pytest tests/.
On Gitlab, this job starts fine, and the logs read
$ conda env create -f environment.yml
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
==> WARNING: A newer version of conda exists. <==
current version: 4.7.10
latest version: 4.7.11
Ok, so I'm using a conda version later than 4.4, so conda activate should work. However, the job fails with the following:
# To activate this environment, use
#
# $ conda activate my-env
#
# To deactivate an active environment, use
#
# $ conda deactivate
$ conda activate my-env
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
I have then tried editing my .gitlab-ci.yml file so that there is a command
conda init bash
but then get the message
==> For changes to take effect, close and re-open your current shell. <==
How can I activate my conda environment in the gitlab CI process?

conda init touches the .bashrc file. To reinitialize the shell you can source it:
- conda create --name myenv
- conda init bash
- source ~/.bashrc # <- !!!
- conda activate myenv
Whether this is better or worse than source activate myenv is a separate discussion, I guess.

Similarly to Tommy's answer, this needs to be done for the Windows Powershell as well. Contrary to bash conda activate myenv does not fail in the powershell. It just has no effect (i.e. the environment is not switched) without calling conda init powershell which makes it even more awkward. Reloading the profile in the powershell is more complicated since there are six of them [1]. I used:
- conda create --name myenv
- conda init powershell
- "if (test-path $PROFILE.CurrentUserAllHosts) { & $PROFILE.CurrentUserAllHosts}"
- conda activate myenv
Why Conda uses the $PROFILE.CurrentUserAllHosts profile has been asked in an issue [2].
references:
[1] https://devblogs.microsoft.com/scripting/understanding-the-six-powershell-profiles/
[2] https://github.com/conda/conda/issues/8608

Another possibility that you might find more succinct and elegant: source directly the code needed for conda to run run with bash. This also has the effect of adding conda to the PATH, if it's not the case.
This is done with
- source <anaconda_root>/etc/profile.d/conda.sh
- conda activate myenv
(Stolen from https://stackoverflow.com/a/60523131/11343)

Somehow all of these answers failed me. I ended up using conda run instead of activating an environment. That allowed me to run pytest without activating the environment
- conda run -n <environment-name> python -m pytest <addl-pytest-args>

Related

Removing Conda environment

I want to remove a certain environment created with conda. How can I achieve that? Let's say I have an active testenv environment. I tried, by following documentation, with:
$ conda env remove
CondaEnvironmentError: cannot remove current environment. deactivate and run conda remove again
I then deactivate it:
$ source deactivate
I try running again the command to remove it and I still get the same error. What is going wrong here?
You probably didn't fully deactivate the Conda environment - remember, the command you need to use with Conda is conda deactivate (for older versions, use source deactivate). So it may be wise to start a new shell and activate the environment in that before you try. Then deactivate it.
You can use the command
conda env remove -n ENV_NAME
to remove the environment with that name. (--name is equivalent to -n)
Note that you can also place environments anywhere you want using -p /path/to/env instead of -n ENV_NAME when both creating and deleting environments, if you choose. They don't have to live in your conda installation.
UPDATE, 30 Jan 2019: From Conda 4.6 onwards the conda activate command becomes the new official way to activate an environment across all platforms. The changes are described in this Anaconda blog post
After making sure your environment is not active, type:
$ conda env remove --name ENVIRONMENT
Official documentation way worked for me:
conda remove --name myenv --all
Or just conda env remove --name myenv.
To verify that the environment was removed, in your terminal window or an Anaconda Prompt, run:
conda info --envs
The environments list that displays should not show the removed environment.
You anaconda3 enviroments folder might list an empty folder of deleted environment in your anaconda3 installation folder, like:
/opt/anaconda3/envs
If you are in base:
(base) HP-Compaq-Elite-8300-CMT:~$
remove env_name by:
conda env remove -n env_name
if you are already in env_name environment :
(env_name) HP-Compaq-Elite-8300-CMT:~$
deactivate then remove by :
conda deactivate
conda env remove -n env_name
In my windows 10 Enterprise edition os this code works fine:
(suppose for environment namely testenv)
conda env remove --name testenv
Environments created with the --prefix or -p flag must be removed with the -p flag (not -n).
For example:
conda remove -p </filepath/myenvironment> --all, in which </filepath/myenvironment> is substituted with a complete or relative path to the environment.
There're 3 ways to achieve this in total. Assuming you have a environment named myenv,
conda env remove --name myenv, -n is shortcut for --name.
conda remove --name myenv --all.
Delete the env folder directly. (Not recommended)
# list environments and their locations
conda env list
# or
# conda info --envs
# delete the folder listed
rm -rf /Users/username/.local/share/conda/envs/myenv
If you wanna delete the environment without a prompt to let you check again. Use -y, shortcut for --yes. (For global use check silent prompt in conda)
conda env remove -n myenv -y
conda remove -n myenv --all -y
References
conda env --help
conda remove --help
First deactivate the environment that you wish to remove.
Then type the following code:
conda env remove -n <your environment name>
To make sure you have deleted it, you can use the following code.
conda info --envs or conda env list
4.If you wan to remove all the dependencies along with the installed packages, you can use:
conda remove -n <environment name> --all
You may try the following: Open anaconda command prompt and type
conda remove --name myenv --all
This will remove the entire environment.
Further reading: docs.conda.io > Manage Environments
To remove complete conda environment :
conda remove --name YOUR_CONDA_ENV_NAME --all
First you have to deactivate your environment before removing it. You can remove conda environment by using the following command
Suppose your environment name is "sample_env" , you can remove this environment by using
source deactivate
conda remove -n sample_env --all
'--all' will be used to remove all the dependencies
My environment name is: test
conda remove -n test --all
Use source deactivate to deactivate the environment before removing it, replace ENV_NAME with the environment you wish to remove:
source deactivate
conda env remove -n ENV_NAME
First I checkout from the environment (tensorflow):
conda deactivate
Then I removed the environment by:
conda remove -n tensorflow --all
The tensorflow is the name of my environment
You can check your env name using this command:
conda env list
First deactivate the environment and come back to the base environment. From the base, you should be able to run the command conda env remove -n <envname>. This will give you the message
Remove all packages in environment
C:\Users\<username>\AppData\Local\Continuum\anaconda3\envs\{envname}:
This worked for me:
conda env remove --name tensorflow
if you are unfamiliar with the command line , you can remove it using the anaconda dashboard
View the environments in Anaconda or miniconda:
conda env list
If you have created an environment using name then use:
conda remove -n envname --all
if you have created an environment using prefix then use:
conda remove -p [path] --all
Change the envname with your environment name and in case of prefix provide the complete path of the environment eg: C:/Users/techv/Desktop/project/env.
--all will remove all the dependencies of the target environment.
I hope this answer will be helpful.
Because you can only deactivate the active environment, so conda deactivate does not need nor accept arguments. The error message is very explicit here.
Just call conda deactivate
https://github.com/conda/conda/issues/7296#issuecomment-389504269
on terminal it's showing
(base) [root#localhost ~]#
simply hit command : conda deactivate
and you are out of conda env , now your prompt will look like
[root#localhost ~]#

I can not activate virtual environment in conda

I create a virtual environment with conda
$ conda create test_env numpy .....
It gets created successfully.
$ conda env list
# conda environments:
#
base * /home/myname/anaconda3
my_project_env /home/myname/anaconda3/envs/my_project_env
test_env /home/myname/anaconda3/envs/test_env
but I can not activate it
$ source activate my_project_env
returns - activate: No such file or directory
The only place I find activate is within the whole anaconda3 is in /common folder
source anaconda3/envs/my_project_env/lib/python3.6/venv/scripts/common/activate my_project_env
When I run it with this path I get VENV_PROMPT "kind of environment" but when I check libraries with pip list it returns a global list of installed libraries instead of the selected few.
$ source anaconda3/envs/my_project_env/lib/python3.6/venv/scripts/common/activate my_project_env
__VENV_PROMPT__myname#box:~$ pip3 list
I met with the same problem. It is because I have changed the system's $PATH variable from anaconda's main bin directory to the environment's bin directory. Actually, the activate's path is under /home/users/anaconda3/bin/. So I just use the following command to make a link between the two bin directories:
ln -s /home/userName/anaconda3/bin/activate /home/userName/anaconda3/envs/envName/bin/activate
ln -s /home/userName/anaconda3/bin/deactivate /home/userName/anaconda3/envs/envName/bin/deactivate
Try to use this command activate your conda environment:
source activate /home/myname/anaconda3/envs/my_project_env
Recommended command to create environment with python version 2.7 :
conda create -n my_project_env python=2.7
Check your conda version
conda -V
Create virtual environment for your project
conda create -n yourenvname python=x.x anaconda
To activate your virtual environment
source activate yourenvname
As others have mentioned, it may be a PATH issue. However, if you're still able to run other conda commands then you may need to either conda update conda or delete conda and reinstall. In my case, I was running miniconda which I believe simply did not contain the activate binary.
Running conda activate instead of source activate solved my issues.

How to transfer Anaconda env installed on one machine to another? [Both with Ubuntu installed]

I have been using Anaconda(4.3.23) on my GuestOS ubuntu 14.04 which is installed on Vmware on HostOS windows 8.1. I have setup an environment in anaconda and have installed many libraries, some of which were very hectic to install (not straight forward pip installs). few libraries had inner dependencies and had to be build together and from their git source.
Problem
I am going to use Cloud based VM (Azure GPU instance) to use GPU. but I don't want to get into the hectic installation again as i don't want to waste money on the time it will take me to install all the packages and libraries again
Is there any way to transfer/copy my existing env (which has everything already installed) to the Cloud VM?
From the very end of this documentation page:
Save packages for future use:
conda list --export > package-list.txt
Reinstall packages from an export file:
conda create -n myenv --file package-list.txt
If conda list --export failes like this ...
Executing conda list --export > package-list.txt creates a file which looks like this:
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: win-64
_tflow_1100_select=0.0.1=gpu
absl-py=0.5.0=py_0
astor=0.7.1=py_0
...
But creating a new environment by executing conda create -n myenv --file package-list.txt gives me this error:
Solving environment: ...working... failed
PackagesNotFoundError: The following packages are not available from current channels:
- markdown==2.6.11=py_0
...
... then try to use conda env export
According to this discussion execute the following command on your source machine:
source activate yourEnvironment
conda env export --no-builds > file.txt
On the target machine execute:
conda env create --file /path/to/file.txt
The file generated by conda env export looks a bit different, but it contains pip packages as well:
name: yourEnvironment
channels:
- conda-forge
- defaults
dependencies:
- absl-py=0.5.0
...
- pip:
- astroid==2.0.4
...
## You can try below approach to move all the package from one machine to other :
## Note : Machine that packages are being moved should be same and python version also should be same
$ pip install conda-pack
# To package an environment:
## Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env
## Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz
## Pack environment located at an explicit path into my_env.tar.gz
$ conda pack -p /explicit/path/to/my_env
# After following above approach, you will end up with a tar.gz file. Now to install package from this zip file follow below approach.
## To install the environment:
## Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env
## Use Python without activating or fixing the prefixes. Most Python
## libraries will work fine, but things that require prefix cleanups
## will fail.
$ ./my_env/bin/python
## Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate
## Run Python from in the environment
(my_env) $ python
## Cleanup prefixes from in the active environment.
## Note that this command can also be run without activating the environment
## as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack
You can probably get away with copying the whole Anaconda installation to your cloud instance.
According to github thread execute the following command on your source machine:
https://github.com/conda/conda/issues/3847
source activate yourEnvironment
conda env export --no-builds > environment.yml
On the target machine execute:
conda env create -f environment.yml
The file generated by conda env export looks a bit different, but it contains pip packages as well:
name: yourEnvironment
channels:
conda-forge
defaults
dependencies:
absl-py=0.5.0
...
pip:
astroid==2.0.4
...
I found the answer from this
you can export your Anaconda environment using:
conda env export > environment.yml
In order to recreate it on another machine using:
conda env create -f environment.yml
You can modify the environment.yml as required because some of the python libraries may be obsolete or due to version conflict in future releases.

How can you "clone" a conda environment into the root environment?

I'd like the root environment of conda to copy all of the packages in another environment. How can this be done?
There are options to copy dependency names/urls/versions to files.
Recommendation
Normally it is safer to work from a new environment rather than changing root. However, consider backing up your existing environments before attempting changes. Verify the desired outcome by testing these commands in a demo environment. To backup your root env for example:
λ conda activate root
λ conda env export > environment_root.yml
λ conda list --explicit > spec_file_root.txt
Options
Option 1 - YAML file
Within the second environment (e.g. myenv), export names+ to a yaml file:
λ activate myenv
λ conda env export > environment.yml
then update the first environment+ (e.g. root) with the yaml file:
λ conda env update --name root --file environment.yml
Option 2 - Cloning an environment
Use the --clone flag to clone environments (see #DevC's post):
λ conda create --name myclone --clone root
This basically creates a direct copy of an environment.
Option 3 - Spec file
Make a spec-file++ to append dependencies from an env (see #Ormetrom):
λ activate myenv
λ conda list --explicit > spec_file.txt
λ conda install --name root --file spec_file.txt
Alternatively, replicate a new environment (recommended):
λ conda create --name myenv2 --file spec_file.txt
See Also
conda env for more details on the env sub-commands.
Anaconada Navigator desktop program for a more graphical experience.
Docs on updated commands. With older conda versions use activate (Windows) and source activate (Linux/Mac OS). Newer versions of conda can use conda activate (this may require some setup with your shell configuration via conda init).
Discussion on keeping conda env
Extras
There appears to be an undocumented conda run option to help execute commands in specific environments.
# New command
λ conda run --name myenv conda list --explicit > spec_file.txt
The latter command is effective at running commands in environments without the activation/deactivation steps. See the equivalent command below:
# Equivalent
λ activate myenv
λ conda list --explicit > spec_file.txt
λ deactivate
Note, this is likely an experimental feature, so this may not be appropriate in production until official adoption into the public API.
+Conda docs have changed since the original post; links updated.
++Spec-files only work with environments created on the same OS. Unlike the first two options, spec-files only capture links to conda dependencies; pip dependencies are not included.
To make a copy of your root environment (named base), you can use following command; worked for me with Anaconda3-5.0.1:
conda create --name <env_name> --clone base
you can list all the packages installed in conda environment with following command
conda list -n <env_name>
When setting up a new environment and I need the packages from the base environment in my new one (which is often the case) I am building in the prompt a identical conda environment by using a spec-file.txt with:
conda list --explicit > spec-file.txt
The spec-file includes the packages of for example the base environment.
Then using the prompt I install the the packages into the new environment:
conda create --name myenv --file spec-file.txt
The packages from base are then available in the new environment.
The whole process is describe in the doc:
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#building-identical-conda-environments
I also ran into the trouble of cloning an environment onto another machine and wanted to provide an answer. The key issue I had was addressing errors when the current environment contains development packages which cannot be obtained directly from conda install or pip install. For these cases I highly recommend conda-pack (see this answer):
pip install conda-pack
or,
conda install conda-pack
then back up the environment, to use the current environment just omit the my_env name,
# Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env
# Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz
# Pack environment located at an explicit path into my_env.tar.gz
$ conda pack -p /explicit/path/to/my_env
and restoring,
# Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env
# Use Python without activating or fixing the prefixes. Most Python
# libraries will work fine, but things that require prefix cleanups
# will fail.
$ ./my_env/bin/python
# Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate
# Run Python from in the environment
(my_env) $ python
# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack

How to activate an Anaconda environment

I'm on Windows 8, using Anaconda 1.7.5 64bit.
I created a new Anaconda environment with
conda create -p ./test python=2.7 pip
from C:\Pr\TEMP\venv\.
This worked well (there is a folder with a new python distribution). conda tells me to type
activate C:\PR\TEMP\venv\test
to activate the environment, however this returns:
No environment named "C:\PR\temp\venv\test" exists in C:\PR\Anaconda\envs
How can I activate the environment? What am I doing wrong?
If this happens you would need to set the PATH for your environment (so that it gets the right Python from the environment and Scripts\ on Windows).
Imagine you have created an environment called py33 by using:
conda create -n py33 python=3.3 anaconda
Here the folders are created by default in Anaconda\envs, so you need to set the PATH as:
set PATH=C:\Anaconda\envs\py33\Scripts;C:\Anaconda\envs\py33;%PATH%
Now it should work in the command window:
activate py33
The line above is the Windows equivalent to the code that normally appears in the tutorials for Mac and Linux:
$ source activate py33
More info:
https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/8T8i11gO39U
Does `anaconda` create a separate PYTHONPATH variable for each new environment?
Use cmd instead of Powershell!
I spent 2 hours before I switched to cmd and then it worked!
create Environment:
conda create -n your_environment_name
see list of conda environments:
conda env list
activate your environment:
conda activate your_environment_name
That's all folks
Note that the command for activating an environment has changed in Conda version 4.4. The recommended way of activating an environment is now conda activate myenv instead of source activate myenv. To enable the new syntax, you should modify your .bashrc file. The line that currently reads something like
export PATH="<path_to_your_conda_install>/bin:$PATH"
Should be changed to
. <path_to_your_conda_install>/etc/profile.d/conda.sh
This only adds the conda command to the path, but does not yet activate the base environment (which was previously called root). To do also that, add another line
conda activate base
after the first command. See all the details in Anaconda's blog post from December 2017. (I think that this page is currently missing a newline between the two lines, it says .../conda.shconda activate base).
(This answer is valid for Linux, but it might be relevant for Windows and Mac as well)
All the former answers seem to be outdated.
conda activate was introduced in conda 4.4 and 4.6.
conda activate: The logic and mechanisms underlying environment activation have been reworked. With conda 4.4, conda activate and conda deactivate are now the preferred commands for activating and deactivating environments. You’ll find they are much more snappy than the source activate and source deactivate commands from previous conda versions. The conda activate command also has advantages of (1) being universal across all OSes, shells, and platforms, and (2) not having path collisions with scripts from other packages like python virtualenv’s activate script.
Examples
conda create -n venv-name python=3.6
conda activate -n venv-name
conda deactivate
These new sub-commands are available in "Aanconda Prompt" and "Anaconda Powershell Prompt" automatically. To use conda activate in every shell (normal cmd.exe and powershell), check expose conda command in every shell on Windows.
References
How to Get Ready for the Release of conda 4.4
Conda 4.6 Release
As you can see from the error message the paths, that you specified, are wrong. Try it like this:
activate ..\..\temp\venv\test
However, when I needed to install Anaconda, I downloaded it from here and installed it to the default paths (C:\Anaconda), than I put this path to the environment variables, so now Anacondas interpreter is used as default. If you are using PyCharm, for example, you can specify the interpreter there directly.
I've tried to activate env from Jenkins job (in bash) with
conda activate base and it failed, so after many tries, this one worked for me (CentOS 7) :
source /opt/anaconda2/bin/activate base
Below is how it worked for me
C:\Windows\system32>set CONDA_ENVS_PATH=d:\your\location
C:\Windows\system32>conda info
Shows new environment path
C:\Windows\system32>conda create -n YourNewEnvironment --clone=root
Clones default root environment
C:\Windows\system32>activate YourNewEnvironment
Deactivating environment "d:\YourDefaultAnaconda3"...
Activating environment "d:\your\location\YourNewEnvironment"...
[YourNewEnvironment] C:\Windows\system32>conda info -e
conda environments:
#
YourNewEnvironment
* d:\your\location\YourNewEnvironment
root d:\YourDefaultAnaconda3
Though #Simba had a good answer at the time, a lot has changed in the conda env since 4.6. Conda activate (env-name) overthrew source activate (env-name) for good but not without it own challenges. conda activate oftentimes forces your environment to base and makes you see something like this:
and throwing loads of error back at you. This can also be because auto_activate_base is set to True.
You can check this by using the following command
conda config --set auto_activate_base False
source ~/.bashrc
And to reactivate use this
conda config --set auto_activate_base True
source ~/.bashrc
let's assume your environment name is 'demo' and you are using anaconda and want to create a virtual environment:
(if you want python3)
conda create -n demo python=3
(if you want python2)
conda create -n demo python=2
After running above command you have to activate the environment by bellow command:
source activate demo
For me, using Anaconda Prompt instead of cmd or PowerShell is the key.
In Anaconda Prompt, all I need to do is activate XXX
I was having the same, a fix seems to have been made in the source.
One special case: If you are trying to put the activation command in a unix script, you might run into a problem because when you run the bash script, a new (linux) shell environment is created, and then destroyed when you exit that script. Think of this as running bash, then source activate... / conda activate..., then running exit to exit that shell... The result is you end up without an activated shell... More details in How to execute script in the current shell on Linux?:
TL;DR: for linux, to activate in a bash script and leave active:
Add the line #!/bin/bash as the first line of the script (This is anyways always a good practice, it specifies this is a bash script)
Type the command source shell_script.sh or . shell_script.sh
Note: . in bash is equivalent to source in bash.
I'm using conda with Windows 10.
Anaconda was installed in:
C:\User\Admin\Anaconda3
After installation I've added this folders to system PATH:
C:\User\Admin\Anaconda3
C:\User\Admin\Anaconda3\Scripts
Then I ran "Anaconda promt" from windows start menu. From this anaconda promt I created my environment with
conda create --name my_env
And after that, I'm able to activate my conda environment from regular CMD by just typing:
activate my_env
For me running this works in VS Code:
C:/ProgramData/Anaconda3/Scripts/activate
Or just creating a new Terminal.
Window:
conda activate environment_name
Mac: conda activate environment_name

Categories