Python - Activate conda env through shell script - python
I am hoping to run a simple shell script to ease the management around some conda environments. Activating conda environments via conda activate in a linux os works fine in the shell but is problematic within a shell script. Could someone point me into the right direction as to why this is happening?
Example to repeat the issue:
# default conda env
$ conda info | egrep "conda version|active environment"
active environment : base
conda version : 4.6.9
# activate new env to prove that it works
$ conda activate scratch
$ conda info | egrep "conda version|active environment"
active environment : scratch
conda version : 4.6.9
# revert back to my original conda env
$ conda activate base
$ cat shell_script.sh
#!/bin/bash
conda activate scratch
# run shell script - this will produce an error even though it succeeded above
$ ./shell_script.sh
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
I use 'source command' to run the shell script, it works:
source shell_script.sh
The error message is rather helpful - it's telling you that conda is not properly set up from within the subshell that your script is running in. To be able to use conda within a script, you will need to (as the error message says) run conda init bash (or whatever your shell is) first. The behaviour of conda and how it's set up depends on your conda version, but the reason for the version 4.4+ behaviour is that conda is dependent on certain environment variables that are normally set up by the conda shell itself. Most importantly, this changelog entry explains why your conda activate and deactivate commands no longer behave as you expect in versions 4.4 and above.
For more discussion of this, see the official conda issue on GitHub.
Edit: Some more research tells me that the conda init function mentioned in the error message is actually a new v4.6.0 feature that allows a quick environment setup so that you can use conda activate instead of the old source activate. However, the reason why this works is that it adds/changes several environment variables of your current shell and also makes changes to your RC file (e.g.: .bashrc), and RC file changes are never picked up in the current shell - only in newly created shells. (Unless of course you source .bashrc again). In fact, conda init --help says as much:
IMPORTANT: After running conda init, most shells will need to be closed and restarted for changes to take effect
However, you've clearly already run conda init, because you are able to use conda activate interactively. In fact, if you open up your .bashrc, you should be able to see a few lines added by conda teaching your shell where to look for conda commands. The problem with your script, though, lies in the fact that the .bashrc is not sourced by the subshell that runs shell scripts (see this answer for more info). This means that even though your non-login interactive shell sees the conda commands, your non-interactive script subshells won't - no matter how many times you call conda init.
This leads to a conjecture (I don't have conda on Linux myself, so I can't test it) that by running your script like so:
bash -i shell_script.sh
you should see conda activate work correctly. Why? -i is a bash flag that tells the shell you're starting to run in interactive mode, which means it will automatically source your .bashrc. This should be enough to enable you to use conda within your script as if you were using it normally.
Quick solution for bash: prepend the following init script into your Bash scripts.
eval "$(command conda 'shell.bash' 'hook' 2> /dev/null)"
Done.
For other shells, check the init conf of your shell, copy the following content within the shell conf and prepend it into your scripts.
# >>> conda initialize >>>
...
# <<< conda initialize <<<
You can also use
conda init --all --dry-run --verbose
to get the init script you need in your scripts.
Explanation
This is related with the introduction of conda init in conda 4.6.
Quote from conda 4.6 release log
Conda 4.4 allowed “conda activate envname”. The problem was that setting up your shell to use this new feature was not always straightforward. Conda 4.6 adds extensive initialization support so that more shells than ever before can use the new “conda activate” command. For more information, read the output from “conda init –help”
After conda init is introduced in conda 4.6, conda only expose command
conda into the PATH but not all the binaries from "base". And environment switch is unified by conda activate env-name and conda deactivate on all platforms.
But to make these new commands work, you have to do an additional initialization with conda init.
The problem is that your script file is run in a sub-shell, and conda is not initialized in this sub-shell.
References
Conda 4.6 Release
Unix shell initialization
Shell startup scripts
Using conda activate or source activate in shell scripts does not always work and can throw errors like this. An easy work around it to place source ~/miniconda3/etc/profile.d/conda.sh above any conda activate command in the script:
source ~/miniconda3/etc/profile.d/conda.sh # Or path to where your conda is
conda activate some-conda-environment
This is the solution that has worked for me and will also work if sharing scripts. This also gets around having to use conda init as on some clusters I have worked with the system is initialised but conda activate still won't work in a shell script.
if you want to use the shell script to run the other python file in the other conda env, just run the other file via the following command.
os.system('conda run -n <env_name> python <path_to_other_script>')
What is the problem with simply doing something like this in your shell:
source /opt/conda/etc/profile.d/conda.sh
(The conda init is still marked as Experimental, and thus not sure if it is a good idea to use it yet).
I also had the exact same error when trying to activate conda env from C++ or Python file. I solved it by bypassing the conda activate statement and using the absolute path of the specific conda env.
For me, I set up an environment called "testenv" using conda.
I searched all python environments using
whereis python | grep 'miniconda'
It returned a list of python environments. Then I ran my_python_file.py using the following command.
~/miniconda3/envs/testenv/bin/python3.8 my_python_file.py
You can do the same thing on windows too but looking up for python and conda python environments is a bit different.
This answer from Github worked for me (I'm using Ubuntu so it's not for Windows only):
eval "$(conda shell.bash hook)"
conda activate my_env
I just followed a similar solution like the one from hong-xu
So to run a shell command that calls the script with arguments and using a specific conda environment:
from a jupyter cell, goes like this :
p1 = <some-value>
run = f"conda run -n {<env-name>} python {<script-name>.py} \
--parameter_1={p1}"
!{run}
I didn't find any of the above scripts useful. These are fine if you want to run conda in non-interactive mode, but I'd like to run it in interactive mode. If I run:
conda activate my_environment
in a bash script it just runs in the script.
I found that creating an alias in .bashrc is all that is required to change directory to a particular project I'm working on, and set up the correct conda environment for me. So I included in .bashrc:
alias my_environment="cd ~/subdirectory/my_project && conda activate my_environment"
and then:
source ~/.bashrc
Then I can just type at the command line:
my_environment
to change to the correct project and correct environment everytime I want to work on a different project.
This answer is similar to #Lamma answer. This worked for me ->
(1) I defined several variables; the conda activate function, environments directory and environment
conda_activate=~/anaconda3/bin/activate
conda_envs_dir=~/anaconda3/envs
conda_env=<env name>
(2) I source conda activate with the environment
source ${conda_activate} ${conda_envs_dir}/${conda_env}
(3) then you can run your python script
python <path to script.py>
This bypasses the conda init requirement. my .bashrc already was initialized and sourcing the .bashrc file didn't work for me. #Lamma's answer worked for me as well as the above code.
The problem is that when you run the bash script, a new (linux) shell environment is created that was not initialized properly. If your intention is to activate a conda environment, and then run python through the script, you can properly initialize the created shell
environment as discussed in the accepted solution.
If however you want to have the conda environment active after you finish this script, then this will not work because the conda environment has changed on the new shell and you exit that shell when you finish the script. Think of this as running bash, then conda activate... then running exit to exit that bash... More details in How to execute script in the current shell on Linux?:
TL;DR:
Add the line #!/bin/bash as the first line of the script
Type the command source shell_script.sh or . shell_script.sh
Note: . in bash is equivalent to source in bash.
$ conda activate scratch
or
$ source activate scratch
#open terminal or CMD as administrator
$ cd <path Anaconda3 install>\Scripts
$ activate
$ cd ..
$ conda activate scratch
Related
Run python script from Task Manager using specific environment
I'm trying to run a python script through Task Manager from a specific virtual environment. However, "activate.bat" only seems to exist in the root folder: C:\Users\user\anaconda3\Scripts but not in C:\Users\user\anaconda3\envs\env_name\Scripts I've tried using the following batch file as the Task Manager Program/Script but doesn't work (nothing happens). Any ideas on how to call a specific env here? Thanks! #echo off cd C:\Users\*user*\Documents\folderName\Scripts call C:\Users\*user*\anaconda3\Scripts\activate.bat env_name python test.py
Generally, conda run should be preferred for programmatic execution within an environment. The conda activate command is intended for interactive use in a shell. That is, something like #echo off cd C:\Users\*user*\Documents\folderName\Scripts conda run -n env_name python test.py I'm not on Windows, but you may need conda.exe or a full path to the Conda executable instead of simply conda.
How to set a "SHELL" to use Conda activate
I have been given a guide to install tensorflow, keras and sciann, but my python fails and cannot google why. I have an old 2010 MacBook Pro (masOS 10.12.6) and I use Python 3.8.5 via Spyder. This is what do: First, I create an environment ”ml” with python: conda create -n ml python=3.8.5 -y This works I have ckecked the folder on my computer. Then I try to activate this conda activate ml And this is where I get error: CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. To initialize your shell, run $ conda init <SHELL_NAME> Currently supported shells are: - bash - fish - tcsh - xonsh - zsh - powershell See 'conda init --help' for more information and options. IMPORTANT: You may need to close and restart your shell after running 'conda init'. Note: you may need to restart the kernel to use updated packages. I also tried condo init ml without any luck. I have really tried to google but all the solution I find won't help. Can you help me manage around? I don't know what a shell is and what shell I should use.
In your terminal, enter the following command: ps -p $$ to figure out what shell you have. A shell basically allows you access to the operating system of your computer. Then, based on what it says, do conda init. For example, if it says bash, then do: conda init bash conda init bash is basically telling your computer that you can launch the conda program from your shell. In this case, your shell is "bash", which is one of the many different shells.
Silence Conda Shell Warning
When I activate a Conda environment in a Bash script like: #!/bin/bash conda activate ./myenv which python python do_stuff.py The activate line prints out the long warning: CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. To initialize your shell, run $ conda init <SHELL_NAME> Currently supported shells are: - bash - fish - tcsh - xonsh - zsh - powershell See 'conda init --help' for more information and options. IMPORTANT: You may need to close and restart your shell after running 'conda init'. How do I silence this warning? If I run the conda init bash command it suggests, that causes Conda to activate my environment by default for all Bash shells, which is definitely not what I want. That's the default configuration in Conda and I explicitly turned that off after it broke functionality everywhere across my system. I want Conda to function similar to virtualenv, where I have a folder containing the environment, and activate it only when needed for a specific application. Under no circumstances do I want a Conda environment activated for every instance of Bash run anywhere on my system. I'm not sure why it's giving me a CommandNotFoundError, because otherwise Conda populates my Bash shell with the correct paths to Python and other environment-specific resources, and everything else works correctly.
Prepend the following code in your script. eval "$(command conda 'shell.bash' 'hook' 2> /dev/null)" This initialize the conda in the script only.
Once you initialize conda it will auto activate the base environment on every new shell session you open, by default. If you would like to turn this behavior off, just run once: conda config --set auto_activate_base false
Stuck when setting up to use anaconda with VS Code and Integrated Git terminal
I want to learn Data Science and so have used some really popular Python modules likes Pandas, Matplotlib, Numpy, etc. So I clean installed Anaconda and am now using it as my default Python interpreter and also using Conda for installing packages and making virtual environments. I use VS Code as my daily text editor. But I have run into some issues when using the integrated Git terminal in VS Code with the Anaconda Python interpreter.
There are a couple of issues that I am facing. One of the first issues that I see is when I am using CMD to run Python. If I type and enter python in cmd, the Python interpreter provided by anaconda comes up. But I also get a warning:
Warning:
This Python interpreter is in a conda environment, but the environment has not been activated. Libraries may fail to load. To activate this environment please see https://conda.io/activation
I didn't expect to get this output. Anyway, there's another problem in VS code. But first I would like to mention that I have checked "Add to PATH" when installing Anaconda so no issues there. Now, when I open a new Terminal in VS Code, automatically C:/Users/User/Anaconda3/Scripts/activate is run and then conda activate base is run. But when conda activate base is run, automatically, as mentioned, I get a CommandNotFoundError. It states Your shell has not been properly configured to use 'conda activate'.
If using 'conda activate' from a batch script, change your
invocation to 'CALL conda.bat activate'
And then I am told to initialize my shell, so I did conda init bash but still no luck. And this brings me to talk about .bash_profile. I think it has to do something with this bash profile. Anyway, this is what is in my bash profile
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
eval "$('/C/Users/User/Anaconda3/Scripts/conda.exe' 'shell.bash' 'hook')"
# <<< conda initialize <<<
Just a summary of the problem:
Unexpected warning in CMD when running Anaconda Python interpreter
Automatically run Anaconda Scripts and conda activate base when opening new Terminal in VS Code
Conda init bash not helping
P.S I have tried using conda activate [env_name] in CMD and also in Git Bash and they work without any issues. In other words, Anaconda and Conda work perfectly outside of VS Code terminal.
I have figured out the answer myself and would like to share it here. First of all at the time of writing the question I was using Git Bash as my Terminal in VS Code (am still using it). So the issue was that when I ran the command conda init bash in Git Bash or the VS Code Terminal, Conda just basically put the command used for activating Conda environments in the .bash_profile since it is sourced during logging into Bash. But the integrated Terminal in VS Code is a subshell of a Git Bash session. That is why .bash_profile is NOT sourced in VS Code since .bash_profile is only sourced during the main Bash session. The .bashrc file is the file that is sourced when creating a Terminal session in VS Code. So what you actually need to do is take the code that is put into .bash_profile by conda init bash and paste it into your .bashrc file and make the .bash_profile source that .bashrc file automatically.
So, to sum up, using conda init bash will put the conda command in the .bash_profile and it is usually sourced by Git Bash, but VS Code Git Bash terminal will use .bashrc.
So you can just cut and paste the code from .bash_profile to .bashrc (as already mentioned) or if you want, just simply follow this: put this code in your .bash_profile:
if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi
And in your .bashrc, put this code:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
eval "$('{path_to_your_conda.exe}' 'shell.bash' 'hook')"
# <<< conda initialize <<<
# You can get conda.exe path by using `which conda` in Git Bash
Now, when you open up a Git Bash session in VS Code Terminal, you can use conda activate env_name to activate any environments you have.
Everything is now supposed to work as expected in VS Code terminal but I would like to further elaborate about something. If you want, you can skip the conda init bash process (NOT recommended, just read on for additional knowledge but please follow the above steps only). This is a feature that was introduced in conda 4.4.0. Till then the way of activating conda environments was by using source activate but that command was NOT cross-platform, meaning that the command could not be used in OSes like Windows.
So they made this change by introducing commands like: conda activate env_name so that conda environments become much easier to use despite the OS platform.
conda activate also has other advantages. This is directly from their release docs:
conda activate: The logic and mechanisms underlying environment activation have been reworked. With conda 4.4, conda activate and conda deactivate are now the preferred commands for activating and deactivating environments. You'll find they are much more snappy than the source activate and source deactivate commands from previous conda versions. The conda activate command also has advantages of (1) being universal across all OSes, shells, and platforms, and (2) not having path collisions with scripts from other packages like Python virtualenv's activate script.
I used this question as a reference. Check it out to learn more.
Having said that, using source activate env_name will still work if you are using Git Bash, even in VS Code Git Bash terminal. source activate env_name requires no prior set up or config. But it is highly recommended that you only use conda init to set everything up and then use conda activate env_name.
[NOTE]: Locating and modifying the said .bashrc and .bash_profile on Windows is usually not as simple as it is on Linux. But can be done fairly easily like this:
It goes without saying but, you should have Git Bash installed. Having Git Bash installed should, as far as I know, automatically create .bashrc or .bash_profile or maybe both. These files are called "dotfiles" (since they start with a dot) and these are by default hidden on most OSes and definitely on Windows. If they were auto-created by Git Bash on your system, it is most likely that they are placed in your home directory. Home directory on Windows is C:\Users\<you>\. With that said, follow this:
Open Git Bash and go to your home directory with: cd. Just type this and you will be in your home directory
Enter this command: ls -a and you will see all your files, even hidden ones. Look for .bash_profile and .bashrc. Both should be present. If they are, you are ready to follow the above instructions. But if one is not there or if both are missing create them using: touch .bashrc && touch .bash_profile. You should now see these files when you again type: ls -a
That's it. Now that you have your .bashrc and .bash_profile, you can follow the above instructions. Also, to access these two files quicker, open them like this with VS Code: code ~/.bashrc or code ~/.bash_profile. Now, modify these two files as per the instructions.
In the question, I have also talked about VS Code activating Conda environments automatically. There's no issue with VS Code doing that since this is the default behavior.
I misinterpreted that as something that's an issue. But if anyone was looking to stop VS Code from automatically doing that, I would recommend trying to set this in the user settings:
"python.terminal.activateEnvironment": false
EDIT: A better solution than using source activate to get conda activate commands to work in the git bash terminal in VS Code: Run conda init in the Git Bash Terminal in VS Code Type in bash -l in VS Code's Git Bash terminal to launch your configured shell as a login shell You should now be able to run conda activate commands per normal! More info: bash -l runs your ~/.profile/~/.bash_profile/~/.zprofile scripts where the conda executable is actually referenced (but in which Git Bash as a integrated terminal does not run by default and refers to). Hence, git bash does not know where to search for conda when running conda activate commands and per Arafat's explanation above, running conda init changes the git bash PATHs in this .bash_profile file, but is ineffectual as the git bash terminal in VS Code doesn't actually refer to this file! Further info in VS Code's official docs. Supplementing the explanation of the accepted answer, I've posted a solution that worked for me here that might possibly help others (changing user settings did not solve the issue for me). Link could also point to other working solutions if the below or accepted answer above doesn't work. NOTE: Please read Arafat's answer before attempting the source activate method below to understand why it's not normally recommended. That said leaving it up as it still solves the problem. Here's what worked for me using the Git Bash terminal in VS Code on windows in succinct steps: source activate env-name - You should see your line appended by the (base) tag now. After calling on source activate, I've found following conda activate commands to work: i.e. conda activate env2-name What didn't work for Git Bash (as a VS Code terminal) for me: activate env-name and conda activate env-name.
A year later I am still running into this issue. The following is a streamlined and updated approach based on Arafat's answer. Install Git Bash Configure Git Bash to be used in VSC (see How do I use Bash on Windows from the Visual Studio Code integrated terminal?) Open the git bash Terminal from VSC If conda activate is run successfully, skip the rest run conda init bash Check for the exiting bash dot files: ls -al ~/.bash* Likely only one of '.bashrc' and '.bash_profile' exist Check the existing dot file for conda initialization code e.g. cat ~/.bash_profile This included in my case '>>> conda initialize >>> ...' code (But, and this is the source of the problem, it is not executed when the terminal is opened. To check which of the files is executed simple add 'echo hello-X' to each of them.) To fix the problem, we must create the missing dot file and make it execute the OTHER previously existing one: tee -a ~/.bashrc << END if [ -f ~/.bash_profile ]; then source ~/.bash_profile fi END Reopen the Terminal in VSC
How to properly activate an Anaconda environment when not using bash?
I have installed an additional Anaconda environment, running python 3.5, so that now I have: # conda environments: # python3.5 /Users/user/anaconda/envs/python3.5 root * /Users/user/anaconda The Anaconda documentation says that I need to do source activate python3.5 to activate the new environment. What it does not mention is that activate only works in bash. I'm using tcsh, so that I currently have to switch to bash, issue the command above, and switch back to tcsh every time I open a new shell. How can I ease this process?
If you really want to stay inside this shell, you need to replicate the logic that the activate script does for bash in your shell. I have found a gist that might work: https://gist.github.com/mikecharles/f09486e884a0b41e1e8f