I would like to set permanently a conda environment in my docker image in order that the functions of the conda package could be used by the script given as argument to the entrypoint.
This is the dockerfile that I created.
FROM continuumio/anaconda3
RUN conda create -n myenv
RUN echo "source activate myenv" > ~/.bashrc
ENV PATH:="/opt/conda/envs/myenv/bin:$PATH"
SHELL ["/bin/bash", "-c"]
ENTRYPOINT ["python3"]
It seems that the ~/.bashrc file is not sourced when I run the docker container. Am I doing something wrong?
Thank you
As a work around either use 'SHELL ["/bin/bash", "-i", "--login", "-c"]'
-or-
edit the .bashrc file in the image to not exit if not in interactive mode by changing "*) return;;" to read "*) ;;"
Using the first option bash will complain about job control and ttys, but the error can be ignored.
cause of the issue:
the .bashrc file contains the following command:
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
which causes bash to stop sourcing the file if not in interactive mode. (the -i flag)
Unfortunately, I haven't found a way for the conda stanza to be inserted into .bash_profile or .profile automatically instead of (or in addition to) .bashrc, as there doesn't seem to be an option to override or add to the list of what files conda init examines for modification.
Related
I'm on Ubuntu 20.04, and I am having issues setting up the crontab correctly using pyenv + pipenv together. Simply adding pipenv run python script.py to the cronjob does not work; I think it may be due to:
the environment required by pyenv
the non-interactive shell of cronjob
UPDATED BIG QUESTION
How do I use /home/jennings/.pyenv/shims/pipenv correctly in crontab??
I've checked $? for pipenv -v in run.sh scheduled in crontab, and it fails.
https://github.com/pyenv/pyenv#advanced-configuration
00. (for reference) Bash startup
pyenv requires these entries in these startup files for interactive/login shells I don't understand how to translate this to a non-interactive cronjob call. How do I set up my BASH_ENV to emmulate these environents below?
https://stackoverflow.com/a/9954208/9335288
# ~/.profile
eval "$(pyenv init --path)"
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init --path)"
fi
# ~/.bashrc:
export PYENV_ROOT="$HOME/.pyenv"
I. Crontab
I am trying to effectively write a .profile for the cronjob... but if you a better solution, please let me know.
# CRONTAB
SHELL=/bin/bash
BASH_ENV="home/jennings/.custom_bash_env
# BASH_ENV="home/jennings/.profile"
# BASH_ENV="home/jennings/.bashrc"
* * * * * cd $PROJECT_DIR; ./run.sh
# BASH_ENV:
Should I point to .profile or .bashrc instead?*
# PYENV
#eval "$(pyenv init --path)"
#if command -v pyenv 1>/dev/null 2>&1; then
# eval "$(pyenv init --path)"
#fi
# ENV VARIABLES
PYENV_ROOT="/home/jennings/.pyenv"
PYTHONPATH=$SOMEPLACE
PATH=all:of:them:paths
II. Project Folder
# run.sh:
#!/usr/bin/bash
# PYENV
eval "$(pyenv init --path)"
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init --path)"
fi
# actual pipenv command I'm trying to run
pipenv run python main.py
# main.py:
import this
# Does some python and logging
What I've tried and know
Pointing directly at the .profile and .bashrc doesn't work
run.sh will run okay; it's thepipenv step that fails
Scheduling other non-pipenv commands work fine
pyenv init block
Placed in the BASH_ENV file, the cronjob doesn't run at all
Placed in the run.sh, the cronjob now runs, but the pipenv run still fails
pipenv related
I've tried pipenv shell; which python to use that one in the crontab enjoy -- no cigar
Cron has a limited environment. You have to declare the full path to the pipenv executable. Instead of using pipenv run python script.py you should use /usr/local/bin/pipenv run python script.py.
In your case the full path can be different which pipenv will give you the proper path.
This is explained in this post how to run pipenv in cronjob in ubuntu?
I have a project set up in Pycharm, with an existing conda environment. My scripts work when run from within the console.
I would like to be able to run python -m path_to_my_script/script.py from any location, but I need conda activated. Conda recommends I do conda init but I'm worried it may change settings someplace and break things.
What does conda init do?
Strategy for Answering
Exactly what the conda init command does and its consequences are shell-specific. Instead of trying to cover all cases, let's walk through a case, noting along the way that one can replicate this analysis by substituting their shell of interest.
Case Study: conda init zsh
Let's look at zsh as the shell. This is a common shell (default for macOS 10.15+) and very close to bash. Plus, I don't already have it configured.
Probing the Command: Dry Run
Many Conda commands include some form of dry run functionality via a --dry-run, -d flag, which - combined with verbosity flags - enables seeing what this would do without doing them. For the init command, dry run alone will only tell us what files it would modify:
$ conda init -d zsh
no change /Users/mfansler/miniconda3/condabin/conda
no change /Users/mfansler/miniconda3/bin/conda
no change /Users/mfansler/miniconda3/bin/conda-env
no change /Users/mfansler/miniconda3/bin/activate
no change /Users/mfansler/miniconda3/bin/deactivate
no change /Users/mfansler/miniconda3/etc/profile.d/conda.sh
no change /Users/mfansler/miniconda3/etc/fish/conf.d/conda.fish
no change /Users/mfansler/miniconda3/shell/condabin/Conda.psm1
no change /Users/mfansler/miniconda3/shell/condabin/conda-hook.ps1
no change /Users/mfansler/miniconda3/lib/python3.7/site-packages/xontrib/conda.xsh
no change /Users/mfansler/miniconda3/etc/profile.d/conda.csh
modified /Users/mfansler/.zshrc
==> For changes to take effect, close and re-open your current shell. <==
Here we can see that it plans to target the user-level resources file for zsh, /Users/mfansler/.zshrc, but it doesn't tell us how it will modified it. Also, OMG! the UX here is awful, because it in no way reflects the fact that I used the -d flag. But don't worry: as long as the -d flag is there it won't actually change things.
Patch Preview
To see what exactly it will do, add a single verbosity flag (-v) to the command. This will give everything from the previous output, but will now shows us the diff it will use to patch (update) the .zshrc file.
$ conda init -dv zsh
/Users/mfansler/.zshrc
---
+++
## -0,0 +1,16 ##
+
+# >>> conda initialize >>>
+# !! Contents within this block are managed by 'conda init' !!
+__conda_setup="$('/Users/mfansler/miniconda3/bin/conda' 'shell.zsh' 'hook' 2> /dev/null)"
+if [ $? -eq 0 ]; then
+ eval "$__conda_setup"
+else
+ if [ -f "/Users/mfansler/miniconda3/etc/profile.d/conda.sh" ]; then
+ . "/Users/mfansler/miniconda3/etc/profile.d/conda.sh"
+ else
+ export PATH="/Users/mfansler/miniconda3/bin:$PATH"
+ fi
+fi
+unset __conda_setup
+# <<< conda initialize <<<
+
# ...the rest is exactly as above
That is, the plan of action is to add these 16 lines to the .zshrc file. In this case, I don't have an existing .zshrc file, so it plans to add it at line 1. If the file had already existed, it would append these lines.
Interpreting the Shell Code
Let's overview this code, before focusing on the details. Essentially, this is a redundant sequence of attempts to set up some shell functionality. They are ordered from most to least functional.
What Conda Hopes To Do
The code
__conda_setup="$('/Users/mfansler/miniconda3/bin/conda' 'shell.zsh' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
gets something from conda itself, storing the result to a string, and then evaluates that string if the command had a clean exit ($? -eq 0). The neat engineering here is that the subprocess (technically python -m conda) passes back a result that can be run within this current process (zsh), allowing it to define shell functions.
I'll dig deeper into what is going on here in a second.
Fallback 1: Hardcoded Shell Functions
If that strange internal command fails, the devs included a hardcoded version of some essential shell functions (specifically conda activate). This is in:
miniconda3/etc/profile.d/conda.sh
and they simply check the file exists and source it. Let's hit that last option, then we'll swing back to look at the functionality.
Fallback 2: The Last Resort
The absolute last resort is to literally violate the standing recommendation since Conda v4.4, which is to simply put the base environment's bin directory on PATH. In this case, there is no conda activate functionality; this only ensures that Conda is on your PATH.
Details: Shell Functionality
Coming back to the intended case, we can inspect exactly what it would evaluate by simply getting that string result:
$ conda shell.zsh hook
__add_sys_prefix_to_path() {
# In dev-mode CONDA_EXE is python.exe and on Windows
# it is in a different relative location to condabin.
if [ -n "${_CE_CONDA}" ] && [ -n "${WINDIR+x}" ]; then
SYSP=$(\dirname "${CONDA_EXE}")
else
SYSP=$(\dirname "${CONDA_EXE}")
SYSP=$(\dirname "${SYSP}")
fi
if [ -n "${WINDIR+x}" ]; then
PATH="${SYSP}/bin:${PATH}"
PATH="${SYSP}/Scripts:${PATH}"
PATH="${SYSP}/Library/bin:${PATH}"
PATH="${SYSP}/Library/usr/bin:${PATH}"
PATH="${SYSP}/Library/mingw-w64/bin:${PATH}"
PATH="${SYSP}:${PATH}"
else
PATH="${SYSP}/bin:${PATH}"
fi
\export PATH
}
__conda_exe() (
__add_sys_prefix_to_path
"$CONDA_EXE" $_CE_M $_CE_CONDA "$#"
)
__conda_hashr() {
if [ -n "${ZSH_VERSION:+x}" ]; then
\rehash
elif [ -n "${POSH_VERSION:+x}" ]; then
: # pass
else
\hash -r
fi
}
__conda_activate() {
if [ -n "${CONDA_PS1_BACKUP:+x}" ]; then
# Handle transition from shell activated with conda <= 4.3 to a subsequent activation
# after conda updated to >= 4.4. See issue #6173.
PS1="$CONDA_PS1_BACKUP"
\unset CONDA_PS1_BACKUP
fi
\local ask_conda
ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix "$#")" || \return
\eval "$ask_conda"
__conda_hashr
}
__conda_reactivate() {
\local ask_conda
ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix reactivate)" || \return
\eval "$ask_conda"
__conda_hashr
}
conda() {
\local cmd="${1-__missing__}"
case "$cmd" in
activate|deactivate)
__conda_activate "$#"
;;
install|update|upgrade|remove|uninstall)
__conda_exe "$#" || \return
__conda_reactivate
;;
*)
__conda_exe "$#"
;;
esac
}
if [ -z "${CONDA_SHLVL+x}" ]; then
\export CONDA_SHLVL=0
# In dev-mode CONDA_EXE is python.exe and on Windows
# it is in a different relative location to condabin.
if [ -n "${_CE_CONDA:+x}" ] && [ -n "${WINDIR+x}" ]; then
PATH="$(\dirname "$CONDA_EXE")/condabin${PATH:+":${PATH}"}"
else
PATH="$(\dirname "$(\dirname "$CONDA_EXE")")/condabin${PATH:+":${PATH}"}"
fi
\export PATH
# We're not allowing PS1 to be unbound. It must at least be set.
# However, we're not exporting it, which can cause problems when starting a second shell
# via a first shell (i.e. starting zsh from bash).
if [ -z "${PS1+x}" ]; then
PS1=
fi
fi
conda activate base
I'm not going to walk through all this, but the main part is that instead of directly putting bin on PATH, it defines a shell function called conda and this serves as a wrapper for the condabin/conda entrypoint. This also defines a new functionality conda activate, which uses a shell function, __conda_activate(), behind the scenes. At the final step, it then activates the base environment.
Why do it this way?
This is engineered like this in order to be responsive to the configuration settings. Configuration options like auto_activate_base and change_ps1 affect how Conda manipulates the shell, and so that changes what functionality Conda includes in its shell functions.
Does Conda "Pollute the Environment"?
Not really. The main behavioral things like auto-activation and prompt modification can be disabled through configuration settings, so that conda init ultimately just adds the conda activate function to the shell, enabling clean switching between environments without ever having to manually manipulate PATH.
Your caution may be warranted. The conda init command adds Anaconda to the path on Linux or Mac (not recommended on Windows). Anaconda FAQ
You also need to prep for conda init by running source <path to conda>/bin/activate first.
You can run your script in your desired conda environment by specifying the python environment in the shebang statement in the first line of your code.
You can get that value by activating your desired environment and executing which python
(base) -> conda activate py39
(py39) -> which python
/home/user/anaconda3/envs/py39/bin/python
The shebang would be:
#!/home/user/anaconda3/env/py39/bin/python
On Mac and Linux, once you add the shebang you can chmod the file to be executable (e.g. chmod 700 myscript.py) and run from the command line directly. (I'm not a Windows user, so ymmv.)
(base) -> <path-to-script>/myscript.py
(Which now runs in shebang virtual environment instead of base.)
I have an environment.yml in my applications folder
I have this in my dockerfile:
RUN conda env create
RUN source activate myenvfromymlfile
When I run the container though the env is not activated. If I do conda env list Is see /opt/conda is activated:
root#9c7181cf86aa:/app# conda env list
# conda environments:
#
myenvfromymlfile /opt/conda/envs/myenvfromymlfile
root * /opt/conda
If I attach to the container I can manually run source activate myenvfromymlfile and it works, but why doesn't that work in the RUN directive??
In examples, I see this often in dockerfiles that require conda:
CMD [ "source activate your-environment && exec python application.py" ]
Can someone explain why it is necessary to use && to make it a single command? And why running "source activate" in a RUN directive does not work? I want to have my dockerfile look like this:
RUN conda env create
RUN source activate myenvfromymlfile
ENTRYPOINT ["python"]
CMD ["application.py"]
Consider the below Dockerfile
RUN conda env create
RUN source activate myenvfromymlfile
ENTRYPOINT ["python"]
CMD ["application.py"]
Statement #1 conda env create. Create the environment and changes files on the disk.
Statement #2 source activate myenvfromymlfile. Loads some stuff in the bash sessions. No disk changes done here
Statement #3 and #4 specifies what happens when you run the container
ENTRYPOINT ["python"]
CMD ["application.py"]
So now when you run the container. Anything that you did in step#2 is not there, because a shell was launched to run step #2, when it completed the shell was closed. Now when you run the image a new shell is started and it is brand new shell with now no knowledge that in past inside your dockerfile you ran source activate myenvfromymlfile
Now you want to run this application.py in the environment you created. The default shell of docker is sh -c. So when you set CMD as below
CMD [ "source activate your-environment && exec python application.py" ]
The final command executed at start of container becomes
sh -c "source activate your-environment && exec python application.py"
Which activates the environment in current shell and then runs your program.
I want to use a makefile to build my project's environment using a makefile and anaconda/miniconda, so I should be able to clone the repo and simply run make myproject
myproject: build
build:
#printf "\nBuilding Python Environment\n"
#conda env create --quiet --force --file environment.yml
#source /home/vagrant/miniconda/bin/activate myproject
If I try this, however, I get the following error
make: source: Command not found
make: *** [source] Error 127
I have searched for a solution, but [this question/answer(How to source a script in a Makefile?) suggests that I cannot use source from within a makefile.
This answer, however, proposes a solution (and received several upvotes) but this doesn't work for me either
( \
source /home/vagrant/miniconda/bin/activate myproject; \
)
/bin/sh: 2: source: not found
make: *** [source] Error 127
I also tried moving the source activate step to a separate bash script, and executing that script from the makefile. That doesn't work, and I assume for the a similar reason, i.e. I am running source from within a shell.
I should add that if I run source activate myproject from the terminal, it works correctly.
I had a similar problem; I wanted to create, or update, a conda environment from a Makefile to be sure my own scripts could use the python from that conda environment.
By default make uses sh to execute commands, and sh doesn't know source (also see this SO answer). I simply set the SHELL to bash and ended up with (relevant part only):
SHELL=/bin/bash
CONDAROOT = /my/path/to/miniconda2
.
.
install: sometarget
source $(CONDAROOT)/bin/activate && conda env create -p conda -f environment.yml && source deactivate
Hope it helps
You should use this, it's functional for me at moment.
report.ipynb : merged.ipynb
( bash -c "source ${HOME}/anaconda3/bin/activate py27; which -a python; \
jupyter nbconvert \
--to notebook \
--ExecutePreprocessor.kernel_name=python2 \
--ExecutePreprocessor.timeout=3000 \
--execute merged.ipynb \
--output=$< $<" )
I had the same problem. Essentially the only solution is stated by 9000. I have a setup shell script inside which I setup the conda environment (source activate python2), then I call the make command. I experimented with setting up the environment from inside Makefile and no success.
I have this line in my makefile:
installpy :
./setuppython2.sh && python setup.py install
The error messages is:
make
./setuppython2.sh && python setup.py install
running install
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/test-easy-install-29183.write-test'
Essentially, I was able to set up my conda environment to use my local conda that I have write access. But this is not picked up by the make process. I don't understand why the environment set up in my shell script using 'source' is not visible in the make process; the source command is supposed to change the current shell. I just want to share this so that other people don't wast time trying to do this. I know autotoools has a way of working with python. But the make program is probably limited in this respect.
My current solution is a shell script:
cat py2make.sh
#!/bin/sh
# the prefix should be change to the target
# of installation or pwd of the build system
PREFIX=/some/path
CONDA_HOME=$PREFIX/anaconda3
PATH=$CONDA_HOME/bin:$PATH
unset PYTHONPATH
export PREFIX CONDA_HOME PATH
source activate python2
make
This seems to work well for me.
There were a solution for similar situation but it does not seems to work for me:
My modified Makefile segment:
installpy :
( source activate python2; python setup.py install )
Error message after invoking make:
make
( source activate python2; python setup.py install )
/bin/sh: line 0: source: activate: file not found
make: *** [installpy] Error 1
Not sure where am I wrong. If anyone has a better solution please share it.
How do you define a custom prompt to use when activating a Python virtual environment?
I have a bash script for activating a virtualenv I use when calling specific Fabric commands. I want the shell prompt to say something like "(fab)" so I can easily distinguish it from other shells I have open. Following this example, I've tried:
#!/bin/bash
script_dir=`dirname $0`
cd $script_dir
/bin/bash -c ". .env/bin/activate; PS1='(fab) '; exec /bin/bash -i"
but there's no change to the prompt. What am I doing wrong?
The prompt is set in the virtualenv's activate script (located in the bin folder under the virtualenv). If you only want to change the prompt some times, you could set an environment variable before calling activate (make sure to clear it in the corresponding deactivate file). If you simply want the prompt to be different all the time, you can do that right in activate at the line that looks like
set "PROMPT=(virtualenvname) %PROMPT%"
If you're using virtualenvwrapper, you could do all of this in the postactivate and postdeactivate scripts as well.
I couldn't find any way to do this via a script executed as a child process. Calling a separate bash process seems to forget any previously set PS1. However, it turned out to be trivial if I just sourced the script:
#!/bin/bash
script_dir=`dirname $0`
cd $script_dir
. .env/bin/activate
PS1="(fab) "
It appears the
exec /bin/bash -i
is resetting the PS1 variable. When I run
export PS1="foo "; bash
it resets it too. Curiously, when I look into the bash sources (shell.c and variables.c) it appears to use
set_if_not ("PS1", primary_prompt);
to init it. But I'm not exactly sure what happens between this and main(). Giving up.
I tried on cygwin and on linux (RedHat CentOS) as well. I found solution for both.
CYGWIN
After some investigation I found that the problem is that PS1 is set by /etc/bash.bashrc which overrides the PS1 env.var. So You need to disable to run this file using:
/bin/bash -c ". .env/bin/activate; PS1='(fab) ' exec /bin/bash -i --norc"
or
/bin/bash -c ". .env/bin/activate; export PS1='(fab) '; exec /bin/bash -i --norc"
LINUX
It works much simpler:
/bin/bash -c ". .env/bin/activate; PS1='(fab) ' exec /bin/bash -i"
or
/bin/bash -c ". .env/bin/activate; export PS1='(fab) '; exec /bin/bash -i"
If the script You are calling does not export the variables (and I suppose it does not) and the set variables does not appears in the environment then You could try something like this:
/bin/bash -c "PS1='(fab) ' exec /bin/bash --rcfile .env/bin/activate; "
I hope I could help!