I have an environment.yml in my applications folder
I have this in my dockerfile:
RUN conda env create
RUN source activate myenvfromymlfile
When I run the container though the env is not activated. If I do conda env list Is see /opt/conda is activated:
root#9c7181cf86aa:/app# conda env list
# conda environments:
#
myenvfromymlfile /opt/conda/envs/myenvfromymlfile
root * /opt/conda
If I attach to the container I can manually run source activate myenvfromymlfile and it works, but why doesn't that work in the RUN directive??
In examples, I see this often in dockerfiles that require conda:
CMD [ "source activate your-environment && exec python application.py" ]
Can someone explain why it is necessary to use && to make it a single command? And why running "source activate" in a RUN directive does not work? I want to have my dockerfile look like this:
RUN conda env create
RUN source activate myenvfromymlfile
ENTRYPOINT ["python"]
CMD ["application.py"]
Consider the below Dockerfile
RUN conda env create
RUN source activate myenvfromymlfile
ENTRYPOINT ["python"]
CMD ["application.py"]
Statement #1 conda env create. Create the environment and changes files on the disk.
Statement #2 source activate myenvfromymlfile. Loads some stuff in the bash sessions. No disk changes done here
Statement #3 and #4 specifies what happens when you run the container
ENTRYPOINT ["python"]
CMD ["application.py"]
So now when you run the container. Anything that you did in step#2 is not there, because a shell was launched to run step #2, when it completed the shell was closed. Now when you run the image a new shell is started and it is brand new shell with now no knowledge that in past inside your dockerfile you ran source activate myenvfromymlfile
Now you want to run this application.py in the environment you created. The default shell of docker is sh -c. So when you set CMD as below
CMD [ "source activate your-environment && exec python application.py" ]
The final command executed at start of container becomes
sh -c "source activate your-environment && exec python application.py"
Which activates the environment in current shell and then runs your program.
Related
I'm on Ubuntu 20.04, and I am having issues setting up the crontab correctly using pyenv + pipenv together. Simply adding pipenv run python script.py to the cronjob does not work; I think it may be due to:
the environment required by pyenv
the non-interactive shell of cronjob
UPDATED BIG QUESTION
How do I use /home/jennings/.pyenv/shims/pipenv correctly in crontab??
I've checked $? for pipenv -v in run.sh scheduled in crontab, and it fails.
https://github.com/pyenv/pyenv#advanced-configuration
00. (for reference) Bash startup
pyenv requires these entries in these startup files for interactive/login shells I don't understand how to translate this to a non-interactive cronjob call. How do I set up my BASH_ENV to emmulate these environents below?
https://stackoverflow.com/a/9954208/9335288
# ~/.profile
eval "$(pyenv init --path)"
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init --path)"
fi
# ~/.bashrc:
export PYENV_ROOT="$HOME/.pyenv"
I. Crontab
I am trying to effectively write a .profile for the cronjob... but if you a better solution, please let me know.
# CRONTAB
SHELL=/bin/bash
BASH_ENV="home/jennings/.custom_bash_env
# BASH_ENV="home/jennings/.profile"
# BASH_ENV="home/jennings/.bashrc"
* * * * * cd $PROJECT_DIR; ./run.sh
# BASH_ENV:
Should I point to .profile or .bashrc instead?*
# PYENV
#eval "$(pyenv init --path)"
#if command -v pyenv 1>/dev/null 2>&1; then
# eval "$(pyenv init --path)"
#fi
# ENV VARIABLES
PYENV_ROOT="/home/jennings/.pyenv"
PYTHONPATH=$SOMEPLACE
PATH=all:of:them:paths
II. Project Folder
# run.sh:
#!/usr/bin/bash
# PYENV
eval "$(pyenv init --path)"
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init --path)"
fi
# actual pipenv command I'm trying to run
pipenv run python main.py
# main.py:
import this
# Does some python and logging
What I've tried and know
Pointing directly at the .profile and .bashrc doesn't work
run.sh will run okay; it's thepipenv step that fails
Scheduling other non-pipenv commands work fine
pyenv init block
Placed in the BASH_ENV file, the cronjob doesn't run at all
Placed in the run.sh, the cronjob now runs, but the pipenv run still fails
pipenv related
I've tried pipenv shell; which python to use that one in the crontab enjoy -- no cigar
Cron has a limited environment. You have to declare the full path to the pipenv executable. Instead of using pipenv run python script.py you should use /usr/local/bin/pipenv run python script.py.
In your case the full path can be different which pipenv will give you the proper path.
This is explained in this post how to run pipenv in cronjob in ubuntu?
I have the following Makefile:
PYTHON = python
.DEFAULT_GOAL = help
help:
#echo ------------------------------Makefile for Flask app------------------------------
#echo USAGE:
#echo make dependencies Install all project dependencies
#echo make docker Run Docker
#echo make env Set environment variables
#echo make run Run Flask app
#echo make test Run tests for app
#echo ----------------------------------------------------------------------------------
dependencies:
#pip install -r requirements.txt
#pip install -r dev-requirements.txt
docker:
docker compose up
env:
#set CS_HOST_PORT=5000
#set CS_HOST_IP=127.0.0.1
#set DATABASE_URL=postgresql://lv-python-mc:575#127.0.0.1:5482/Realty_DB
#set REDIS_IP=127.0.0.1
#set REDIS_PORT=6379
run:
${PYTHON} app.py test:
#${PYTHON} -m pytest
The set command doesn't work and the environment variables aren't set, what may be the problem?
You can certainly set environment variables that will be in effect for programs make will invoke. But make cannot set environment variables for shells that invoke make. So if your makefile runs a program then you can set an environment variable in your makefile that will be visible in that program.
This has nothing to do with make, by the way. This is a limitation (or feature, depending on your perspective) of the operating system. Try this experiment:
Open a terminal.
Run set FOO=bar
Run echo %FOO%. See that it prints bar.
From that same terminal start a new shell by running cmd.exe
Now here run set FOO=nobar
Run echo %FOO%. See that it prints nobar.
Now exit the new shell by running exit
Now run echo %FOO%
You'll see that instead of nobar, it still prints bar. That's because the OS does not allow a child program to modify the environment of its parent program.
So, there's nothing make can do about this.
For context, this problem relates to a docker image that will be run using azure batch.
Here is the Dockerfile, in full:
FROM continuumio/miniconda3
ADD . /pipegen
ADD environment.yml /tmp/environment.yml
RUN conda env create -f /tmp/environment.yml
RUN echo "conda activate $(head -1 /tmp/environment.yml | cut -d' ' -f2)" >> ~/.bashrc
ENV PATH /opt/conda/envs/$(head -1 /tmp/environment.yml | cut -d' ' -f2)/bin:$PATH
ENV CONDA_DEFAULT_ENV $(head -1 /tmp/environment.yml | cut -d' ' -f2)
ADD classify.py /classify.py
RUN rm -rf /pipegen
pipgen is the local module (where the Dockerfile is located) that is installed using the environment.yml file. Here is the environment.yml file in full:
name: pointcloudz
channels:
- conda-forge
- defaults
dependencies:
- python=3.7
- python-pdal
- entwine
- matplotlib
- geopandas
- notebook
- azure-storage-blob==1.4.0
- pip:
- /pipegen
- azure-batch==6.0.0
For clarity, the directory structure looks like this:
Dockerfile
pipegen
\__ __init__.py
\__ pipegen.py
\__ utils.py
classify.py
batch_containers.py
environment.yml
setup.py
The Dockerfile establishes the environment created using the environment.yml file as the default (conda) python environment when the container is run. Therefore, I can run the container interactively as follows:
docker run -it pdalcontainers.azurecr.io/pdalcontainers/pdal-pipelines
and, from inside the container, execute the classify.py script with some command line arguments, as follows:
python classify.py in.las out.las --defaults
and the script is executed as expected. However, when I run the following command, attempting to execute the very same script from "outside" the container,
docker run -it pdalcontainers.azurecr.io/pdalcontainers/pdal-pipelines python classify.py in.las out.las --defualts
I get the following error:
File "classify.py", line 2, in <module>
from pipegen.pipegen import build_pipeline, write_las
ModuleNotFoundError: No module named 'pipegen'
Just to be clear, the classify.py script imports pipegen, the local module which is now installed in the conda environment created in the Dockerfile. I need to be able to execute the script using the docker run command above due to constraints in how Azure batch runs jobs. I've tried multiple fixes but am now pretty stuck. Any wisdom would be greatly appreciated!
The problem you are facing is because you added the conda activate to the .bashrc script which is only activated for login shells. When you run the container interactively, that is what you are getting. However, when you just try to invoke the python script directly, you do not get a login shell so your conda environment is not activated.
One think you could do is not use the conda activate and instead run the script with conda run. To simplify the command-line, add this entrypoint to your Dockerfile:
ENTRYPOINT ["conda", "run", "-n", "$CONDA_DEFAULT_ENV", "python", "classify.py"]
Using this in the entrypoint also allows the caller to pass command-line arguments via docker run.
From the Dockerfile reference
Command line arguments to docker run will be appended after all elements in an exec form ENTRYPOINT
For a more detailed explanation, see Activating a Conda environment in your Dockerfile
FROM python:3
WORKDIR /Users/vaibmish/Documents/new/graph-report
RUN pip install graphreport==1.2.1
CMD [ cd /Users/vaibmish/Documents/new/graph-report/graphreport_metrics ]
CMD [ graphreport ]
THIS IS PART OF DCOKERFIILE
i wish to remove cd volumes from tha file and have a command like -v there so that whoever runs that can give his or her own volume path in same
The line
CMD [ cd /Users/vaibmish/Documents/new/graph-report/graphreport_metrics ]
is wrong. You achieve the same with WORKDIR:
WORKDIR /Users/vaibmish/Documents/new/graph-report/graphreport_metrics
WORKDIR creates the path if it doesn't exist and then changes the current directory to that path (same as mkdir -p /path/new && cd /path/new)
You can also declare the path as a volume and instruct who runs the container to provide their own path (docker run -v host_path:container_path ...)
VOLUME /Users/vaibmish/Documents/new/graph-report
A final note: It looks like these paths are from the host. Remember that the paths in the Dockerfile are not host paths. They are paths inside the container.
Typical practice here is to pick some fixed path inside the Docker container. It should be a different path from where your application is installed; it does not need to match any particular host path at all.
FROM python:3
RUN pip3 install graphreport==1.2.1
WORKDIR /data
CMD ["graphreport"]
docker build -t me/graphreport:1.2.1 .
docker run --rm \
-v /Users/vaibmish/Documents/new/graph-report:/data \
me/graphreport:1.2.1
(Remember that only the last CMD has an effect, and if it's not a well-formed JSON array, Docker will interpret it as a shell command. What you show in the question would run the test(1) command and not the program you're installing.)
If you're trying to install a single package from PyPI and just run it on local files, a Python virtual environment will be much easier to set up than anything based on Docker, and will essentially work as you expect:
python3 -m venv graphreport
. graphreport/bin/activate
pip3 install graphreport==1.2.1
cd /Users/vaibmish/Documents/new/graph-report
graphreport
deactivate # switch back to system Python/pip
All of the installed Python code is inside the graphreport virtual environment directory, and if you don't need this application again, you can just delete the directory tree.
I would like to set permanently a conda environment in my docker image in order that the functions of the conda package could be used by the script given as argument to the entrypoint.
This is the dockerfile that I created.
FROM continuumio/anaconda3
RUN conda create -n myenv
RUN echo "source activate myenv" > ~/.bashrc
ENV PATH:="/opt/conda/envs/myenv/bin:$PATH"
SHELL ["/bin/bash", "-c"]
ENTRYPOINT ["python3"]
It seems that the ~/.bashrc file is not sourced when I run the docker container. Am I doing something wrong?
Thank you
As a work around either use 'SHELL ["/bin/bash", "-i", "--login", "-c"]'
-or-
edit the .bashrc file in the image to not exit if not in interactive mode by changing "*) return;;" to read "*) ;;"
Using the first option bash will complain about job control and ttys, but the error can be ignored.
cause of the issue:
the .bashrc file contains the following command:
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
which causes bash to stop sourcing the file if not in interactive mode. (the -i flag)
Unfortunately, I haven't found a way for the conda stanza to be inserted into .bash_profile or .profile automatically instead of (or in addition to) .bashrc, as there doesn't seem to be an option to override or add to the list of what files conda init examines for modification.