How does one iteratively develop their app using Docker? I have only just started using it and my workflow is very slow, so I'm pretty sure I'm using it wrong.
I'm following along with a python machine learning course on Youtube, and so I am using Docker to work with python 3. I know I can use virtualenv or a VM, but I want to learn Docker as well so bear with me.
My root directory looks like so:
Dockerfile main.py*
My docker file:
FROM python
COPY . /src
RUN pip install quandl
RUN pip install pandas
CMD ["python", "/src/main.py"]
And the Python file:
#!/usr/bin/env python
import pandas as pd
import quandl
print("Hello world from main.py")
df = quandl.get("WIKI/GOOGL")
print("getting data frame for WIKI/GOOGL")
print(df.head())
My workflow has been:
Learn something new from the tutorial
Update python file
Build the docker image: docker build -t myapp .
Run the app: docker run my app python /src/main.py
Questions:
How can I speed this all up? For every change I want to try, I end up rebuilding. This causes pip to get dependencies each time which takes way too long.
Instead of editing a python file and running it, how might a get an interactive shell from the python version running in the container?
If I wanted my program to write out a file, how could I get this file back to my local system from the container after the program has finished?
Thanks for the help!
Edit:
I should add, this was the tutorial I was following in general to run some python code in Docker: https://www.civisanalytics.com/blog/using-docker-to-run-python/
Speeding up the rebuild process
The simplest thing you can do is reorder your Dockerfile.
FROM python
RUN pip install quandl
RUN pip install pandas
COPY . /src
CMD ["python", "/src/main.py"]
The reason this helps is that Docker will re-use the cached build for commands it has already run. Now when you rebuild after modifying your source code, it will re-use the build results for the pip commands, as they do not need to be run again. It will only run the COPY step.
Getting a python shell
You can exec a shell in the running container and run your python command.
docker exec -it <container-id> bash
python <...>
Or, you can run a container with just a shell, and skip running your app entirely (then, run it however you want).
docker run -it <image> bash
python <...>
Writing outside the container
Mount an external directory into the container. Then write to the mounted path.
docker run -v /local/path:/path <.. rest of command ..>
Then when you write in the container to /path/file, the file will show up outside the container at /local/path/file.
Related
I've been experimenting with using pack to build a docker image. I'm quite happy with using a Dockerfile if that is the better approach.
I have a single python script that uses click to parse arguments. In a Dockerfile, I'd specify as the ENTRYPOINT something like ["python", "myscript.py"] and then I could run:
docker run MYCONTAINER myarg1 myarg2
and replace my script with the docker run.
I am attempting to get the same functionality using buildpacks and a Procfile. Here is what I have, roughly, in the profile:
web: python myscript.py
However, when I try running the resulting successfully built image using docker run (eg., docker run MYCONTAINER myarg1 myarg2), I get errors like:
python myscript.py: python myscript.py: command not found
Running the same docker run without the arguments works as expected and prints the help for my command-line script and exits.
What am I missing? My google-foo has failed me...
My laptop (Macbook) pre-installed an old version of Python (2.7).
I have a couple of different python scripts task1.py and task2.py that require Python 3.7 and pip install some_handy_python_package
Several online sources say updating the system-wide version of Python on a Macbook could break some (unspecified) legacy apps.
Seems like a perfect use-case for Docker, to run some local scripts with a custom Python setup, but I do not find any online examples for this simple use case:
Laptop hosts folder mystuff has two scripts task1.py and task2.py (plus a Dockerfile and docker-compose.yml file)
Create a docker image with python 3.7 and whatever required packages, eg pip install some_handy_python_package
Can run any local-hosted python scripts from inside the docker container
perhaps something like docker run -it --rm some-container-name THEN at a bash prompt 'inside' docker run the script(s) via python task1.py
or perhaps something like docker-compose run --rm console python task1.py
I assume the Dockerfile starts off something like this:
FROM python:3.7
RUN pip install some_handy_python_package
but I'm not sure what to add to either the Dockerfile or a docker-compose.yml file so I can either a) run in Docker a bash prompt that lets me run python task1.py, or b) lets me define a 'console' service that can invoke python task1.py from the command line
In case it helps someone else, here's a kind of basic example how to run some local-folder python scripts inside Dockerized python environment. (A better example would setup a volume share within the Dockerfile.)
cd sc2
pwd # /Users/thisisme/sc2` -- you use this path later, when run docker, to set a volume share
Create Dockerfile
# Dockerfile
FROM python:3.7
RUN pip install some_package
Build the container, named rp in this example:
docker build -t rp .
In the local folder, create some python scripts, for example: task1.py
# task1.py
from some_package import SomePackage
# do some stuff
In the local folder, run the script in the container by creating a app share point:
docker run --rm -v YOUR_PATH_TO_FOLDER:/app rp python /app/task1.py
Specifically:
docker run --rm -v /Users/thisisme/sc2:/app rp python /app/task1.py
And sometimes it is handy to run the python interpreter in the container while developing code:
docker run -it --rm rp1
>>> 2 + 2
4
>>>
I am trying to deploy a Python Webservice (with flask) that uses CNTK in a Docker Container. I use an Ubuntu-Base-Image from Microsoft that is supposed to contain all the neccessary and correct programs and libraries to run CNTK.
The Script works locally (on Windows) and also when I run the container and start a bash from cmd-line with
docker exec -it <container_id> bash
and start the script from "within the container".
An important addition is that the python script uses two precompiled modules that are *.pyd files for windows and *.so files for Linux. So for the docker image I replaced the former for the latter for running the script from within the container.
The problems start when I start the script with a CMD in the Dockerfile. The creation of the image shows no problems. But when I start the container with
docker run -p 1234:80 app
I get the following error:
...
ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory
It seems like the library is missing. But (I repeat) when I run the script from within a bash running in the container (which should only have the containers libraries as far as I can see), everything works fine. I even can look up the library with
ldd $(which python)
And the file is definitely in the folder. So the question is why python can't find its dependency when running the docker container.
It even gets weirder when I try to give the path to the library explicitely by writing it in the environment variable:
ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/root/anaconda3/pkgs/python-3.5.2-0/lib/"
Then the library it seems the library is found but it is not accepted as correct:
ImportError: dynamic module does not define init function (initcython_bbox)
"cython_bbox" is the name of one of the *.pyd / *.so file/library that is to be imported. This is apparantly a typical error for these kinds of filetypes. But I don't have any experience with them.
I am also not at the point (in my personal development) to be able to compile my own files from foreign source or create the docker image itself on my own. I rely on the parts I got from Microsoft. But I would be open to suggestions.
I also already tried to install the library anew inside my Dockerfile after importing the base image with
RUN apt-get install -y libpython3.5
But it provoked the same error as when I put the path in the environment variable.
I am really eager to know what goes wrong here. Why does everything run smoothly "inside the container" but not with Autostart at Initialization of a Container with CMD?
For additional info I add the Dockerfile:
# Use an official Python runtime as a parent image
FROM microsoft/cntk:2.5.1-cpu-python3.5
# Set the working
directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
ADD . /app
RUN apt-get update && apt-get install -y python-pip RUN pip install
--upgrade pip
# Install any needed packages specified in requirements.txt
RUN pip install --trusted-host pypi.python.org -r requirements.txt
# Make port 80 available to the world outside this container
EXPOSE 80
# Run app.py when the container launches
CMD ["python", "test.py"]
The rest of the project is a pretty straightforward flask-webapp that runs without problems when I comment out all import of the actual CNTK-project. It is the CNTK Object Detection with Faster-RCNN by the way, as it can be found in the cntk-git-repository.
EDIT:
I found out what the actual problem is, yet I still have no way to solve it. The thing is that when I start bash with "docker exec" it runs a script at startup that activates an anaconda environment with python3.5 and all the neat libraries. But when CMD just starts python this is done by the standard Bourne shell "sh" which (as I tried out) runs with python2.7.
So I need a way either to start my container with bash (including its autostart scripts) or somehow activate the environment on startup in another way.
I looked up the script and it basically checks if bash is the current shell, sets some environment variables and activates the environment.
if [ -z "$BASH_VERSION" ]; then
echo Error: only Bash is supported.
elif [ "$(basename "$0" 2> /dev/null)" == "activate-cntk" ]; then
echo Error: this script is meant to be sourced. Run 'source activate-cntk'
else
export PATH="/cntk/cntk/bin:$PATH"
export LD_LIBRARY_PATH="/cntk/cntk/lib:/cntk/cntk/dependencies/lib:$LD_LIBRARY_PATH"
source "/root/anaconda3/bin/activate" "/root/anaconda3/envs/cntk-py35"
cat <<MESSAGE
************************************************************
CNTK is activated.
Please checkout tutorials and examples here:
/cntk/Tutorials
/cntk/Examples
To deactivate the environment run
source /root/anaconda3/bin/deactivate
************************************************************
MESSAGE
fi
I tried some dozens of things like linking sh to bash
RUN ln -fs /bin/bash /bin/sh
or using bash as ENTRYPOINT.
I have found a workaround that works for now.
First I manually link python to python3 in my environment:
RUN ln -fs /root/anaconda3/envs/cntk-py35/bin/python3.5 /usr/bin/python
Then I add the environment libraries to the Library-Path:
ENV LD_LIBRARY_PATH "/cntk/cntk/lib:/cntk/cntk/dependencies/lib:$LD_LIBRARY_PATH"
And to be sure I add all important folders to PATH:
ENV PATH "/cntk/cntk/bin:$PATH"
ENV PATH "/root/anaconda3/envs/cntk-py35/bin:$PATH"
I then have to install my python packages again:
RUN pip install flask
And can finally just start my script with:
CMD ["python", "app.py"]
I have also found this GIT Repository doing pretty much the same thing I did. And they also need to start their environment. Realizing that I need to learn how to write better Dockerfiles. I think this is the correct way to do it, i.e. using a shell script as ENTRYPOINT
ENTRYPOINT ["/app/run.sh"]
which activates the environment, installs additional python packages (this could be a problem) and starting the actual app.
#!/bin/bash
source /root/anaconda3/bin/activate root
pip install easydict
pip install azure-ml-api-sdk==0.1.0a9
pip install sanic
python3 /app/app.py
I have a python code for which I want to create a docker image. Now as per my understanding we need a Dockerfile and our python code code.py. Inside a Dockerfile we need to write:
FROM python:3
ADD code.py /
RUN pip3 install imapclient
CMD [ "python", "./code.py" ]
My first question is about this Dockerfile. First we have mentioned FROM python:3 because we want to use python3. Next we have added our code. In RUN we can write a dependency of our code. So for example if our code need python package imapclient we can mention it here so that it will be installed before docker file is build. But what if our code do not have any requirements.? Is this line RUN important. Can we exclude it when we don't need it.?
So now let's say we have finally created our docker image python-hello-world by using command docker build -t python-hello-world .. I can see it using command docker images -a. Now when I do docker ps, it is not listed there because the container is not running. Now to start it, I'll have to do docker run python-hello-world. This will start the code. But I want it to be running always in the background like a Linux service. How to do that.?
Is this line RUN important? Can we exclude it when we don't need it?
Yes if your code doesn't need the packages then you can exclude it.
But I want it to be running always in the background like a Linux service. How to do that?
If you want to run it as background then use below command.
docker run -d --restart=always python-hello-world
This will start container in background and will start automatically when system reboots.
I have a shell script that runs a Python program to pre-process some data and then runs an R program that does some long running task on it. I'm learning to use Docker and I've been running
FROM r-base:latest
FROM python
COPY . /usr/local/src/scripts
WORKDIR /usr/local/src/scripts
CMD ["./myscript.sh"]
To my understanding it does the following:
Get R
Get Python
Copy my current directory to /usr/local/src/scripts
Change directory to /usr/local/src/scripts
Run ./myscript.sh
Inside myscript.sh I use the R CMD ... syntax for running my R script. However, when this docker image is run I get the following error:
./myscript.sh: line 8: R: command not found
This suggests that the script, when run inside the container, is not recognizing the R program. I can confirm that ./myscript.sh works locally but I cannot expose any proprietary code.
The FROM command sets the base image that your Dockerfile builds on. You should only have one of these. After that, if you need additional tools that aren't in the base image, run commands that use platform-dependent package managers, such as
RUN apt-get update && apt-get install -y \
package-foo \
package-bar
This would be for a Debian-based image, such as Ubuntu.