I have a python code for which I want to create a docker image. Now as per my understanding we need a Dockerfile and our python code code.py. Inside a Dockerfile we need to write:
FROM python:3
ADD code.py /
RUN pip3 install imapclient
CMD [ "python", "./code.py" ]
My first question is about this Dockerfile. First we have mentioned FROM python:3 because we want to use python3. Next we have added our code. In RUN we can write a dependency of our code. So for example if our code need python package imapclient we can mention it here so that it will be installed before docker file is build. But what if our code do not have any requirements.? Is this line RUN important. Can we exclude it when we don't need it.?
So now let's say we have finally created our docker image python-hello-world by using command docker build -t python-hello-world .. I can see it using command docker images -a. Now when I do docker ps, it is not listed there because the container is not running. Now to start it, I'll have to do docker run python-hello-world. This will start the code. But I want it to be running always in the background like a Linux service. How to do that.?
Is this line RUN important? Can we exclude it when we don't need it?
Yes if your code doesn't need the packages then you can exclude it.
But I want it to be running always in the background like a Linux service. How to do that?
If you want to run it as background then use below command.
docker run -d --restart=always python-hello-world
This will start container in background and will start automatically when system reboots.
Related
First I am very new to docker, so apologies if this doesn't make sense.
This is my situation:
I have a data science/machine learning project in a python script (written in a single .py file). I want to containerize this application.
I would need to create a Dockerfile to do that. But since this is a machine learning project, there are a lot of packages that I need to pip install. So I found this Docker image from https://hub.docker.com/r/continuumio/miniconda3, which has miniconda installed, which has the packages that I need.
I pulled this image.
And now I don't know what to do with it. How do I continue from here. So far, my Dockerfile is empty. How can I use this image as my starting point and perhaps install more modules if needed and then finally, how to containerize my python script based on this modified image?
Many thanks.
You can specify a base image using the FROM command.
Example:
FROM continuumio/miniconda3:latest
You'll want to use the RUN command to install dependencies (assuming you need any more,) the COPY command to get your main.py file into the container, and CMD can be used to set a default command to run when the container starts up. Documentation for each of those can be found here.
I've been experimenting with using pack to build a docker image. I'm quite happy with using a Dockerfile if that is the better approach.
I have a single python script that uses click to parse arguments. In a Dockerfile, I'd specify as the ENTRYPOINT something like ["python", "myscript.py"] and then I could run:
docker run MYCONTAINER myarg1 myarg2
and replace my script with the docker run.
I am attempting to get the same functionality using buildpacks and a Procfile. Here is what I have, roughly, in the profile:
web: python myscript.py
However, when I try running the resulting successfully built image using docker run (eg., docker run MYCONTAINER myarg1 myarg2), I get errors like:
python myscript.py: python myscript.py: command not found
Running the same docker run without the arguments works as expected and prints the help for my command-line script and exits.
What am I missing? My google-foo has failed me...
My Docker knowledge is very poor, I have Docker installed only because I would use freqtrade, so I followed this simple HOWTO
https://www.freqtrade.io/en/stable/docker_quickstart/
Now , all freqtrade commands run using docker , for example this
D:\ft_userdata\user_data>docker-compose run --rm freqtrade backtesting --config user_data/cryptofrog.config.json --datadir user_data/data/binance --export trades --stake-amount 70 --strategy CryptoFrog -i 5m
Well , I started to have problems when I would had try this strategy
https://github.com/froggleston/cryptofrog-strategies
for freqtrade . This strategy requires Python module finta .
I understood that the Python module finta should be installed in my Docker container
and NOT in my Windows system (it was easy "pip install finta" from console!).
Even if I tried to find a solution over stackoverflow and google, I do no understand how to do this step (install finta python module in freqtrade container).
after several hours I am really lost.
Someone can explain me in easy steps how to do this ?
Freqtrade mount point is
D:\ft_userdata\user_data
You can get bash from your container with this command:
docker-compose exec freqtrade bash
and then:
pip install finta
OR
run only one command:
docker-compose exec freqtrade pip install finta
If the above solutions didn't work, You can run docker ps command and get container id of your container. Then
docker exec -it CONTAINER_ID bash
pip install finta
You need to make your own docker image that has finta installed. Luckily you can build on top of the standard freqtrade docker image.
First make a Dockerfile with these two lines in it
FROM freqtradeorg/freqtrade:stable
RUN pip install finta
Then build the image (calling the new image myfreqtrade) by running the command
docker build -t myfreqtrade .
Finally change the docker-compose.yml file to run your image by changing the line
image: freqtradeorg/freqtrade:stable
to
image: myfreqtrade
And that should be that.
The way to get our Python code running in a container is to pack it as a Docker image and then run a container based on it.
To generate a Docker image we need to create a Dockerfile that contains instructions needed to build the image. The Dockerfile is then processed by the Docker builder which generates the Docker image. Then, with a simple docker run command, we create and run a container with the Python service.
An example of a Dockerfile containing instructions for assembling a Docker image for Python service installing finta is the following
# set base image (host OS)
FROM python:3.8
# install dependencies
RUN pip install finta
# command to run on container start
CMD [ "python", "-V" ]
For each instruction or command from the Dockerfile, the Docker builder generates an image layer and stacks it upon the previous ones. Therefore, the Docker image resulting from the process is simply a read-only stack of different layers.
docker build -t myimage .
Then, we can check the image is in the local image store:
docker images
Please refer to the freqtrade DockerFile https://github.com/freqtrade/freqtrade/blob/develop/Dockerfile
How does one iteratively develop their app using Docker? I have only just started using it and my workflow is very slow, so I'm pretty sure I'm using it wrong.
I'm following along with a python machine learning course on Youtube, and so I am using Docker to work with python 3. I know I can use virtualenv or a VM, but I want to learn Docker as well so bear with me.
My root directory looks like so:
Dockerfile main.py*
My docker file:
FROM python
COPY . /src
RUN pip install quandl
RUN pip install pandas
CMD ["python", "/src/main.py"]
And the Python file:
#!/usr/bin/env python
import pandas as pd
import quandl
print("Hello world from main.py")
df = quandl.get("WIKI/GOOGL")
print("getting data frame for WIKI/GOOGL")
print(df.head())
My workflow has been:
Learn something new from the tutorial
Update python file
Build the docker image: docker build -t myapp .
Run the app: docker run my app python /src/main.py
Questions:
How can I speed this all up? For every change I want to try, I end up rebuilding. This causes pip to get dependencies each time which takes way too long.
Instead of editing a python file and running it, how might a get an interactive shell from the python version running in the container?
If I wanted my program to write out a file, how could I get this file back to my local system from the container after the program has finished?
Thanks for the help!
Edit:
I should add, this was the tutorial I was following in general to run some python code in Docker: https://www.civisanalytics.com/blog/using-docker-to-run-python/
Speeding up the rebuild process
The simplest thing you can do is reorder your Dockerfile.
FROM python
RUN pip install quandl
RUN pip install pandas
COPY . /src
CMD ["python", "/src/main.py"]
The reason this helps is that Docker will re-use the cached build for commands it has already run. Now when you rebuild after modifying your source code, it will re-use the build results for the pip commands, as they do not need to be run again. It will only run the COPY step.
Getting a python shell
You can exec a shell in the running container and run your python command.
docker exec -it <container-id> bash
python <...>
Or, you can run a container with just a shell, and skip running your app entirely (then, run it however you want).
docker run -it <image> bash
python <...>
Writing outside the container
Mount an external directory into the container. Then write to the mounted path.
docker run -v /local/path:/path <.. rest of command ..>
Then when you write in the container to /path/file, the file will show up outside the container at /local/path/file.
So, I'm in a bit of a particular situation and i'm trying to find a clean solution.
Currently we've got 18 different repos, all with python deployment utilities copy and pasted 18 times with venv... to me this is disgusting.
I'd like to bake those utilities into some kind of "tools" docker image, and just execute them wherever i need, instead of having to have each folder install all the dependencies 18 times.
/devtools/venv
/user-service/code
/data-service/code
/proxy-service/code
/admin-service/code
Ultimately I'd like to CD into user-service, and run a command similar to docker run tools version_update.py -- and have the docker image mount user-service's code and run the script against it.
How would I do this, and is there a better way i'm not seeing?
Why use docker?
I would recommend placing your scripts into a "tools" directory alongside your services (or wherever you see fit), then you can cd into one of your service directories and run python ../tools/version_update.py.
It would depend on your docker image, but here is the basic concept.
In your docker image, lets say we have a /code directory where we will mount the source code that we want to do work on, and a /tools directory with all of our scripts.
We can then mount what ever directory we want into the /code directory in the docker image and run what ever script we want. The working directory inside of the container would be set to /code and the path would also have /tools in it. So using your example, the docker run commands would look like this.
docker run -v /user-service/code:/code tools version_update.py
This would run the tools docker image, mount the local /user-service/code directory to /code directory in the container, and then run the version_update.py script on that code. and then exit.
The same image can be used for all the other projects as well, just change the mount point. (assuming they all have the same structure)
docker run -v /data-service/code:/code tools version_update.py
docker run -v /proxy-service/code:/code tools version_update.py
docker run -v /admin-service/code:/code tools version_update.py
And if you want to run a different tool, just change the command that you pass in.
docker run -v /user-service/code:/code tools other_command.py