I work on my mac and have a python-flask application running inside a container. I am using Docker for mac.
Purpose: I want my app to reload automatically everytime I make a change in code. I want to access and make changes to the code from my IDE(atom) in mac.
My Dockerfile creates a virtualenv(/app/venv) when I build the image.
WORKDIR /app
ADD ./myapp /app
RUN virtualenv venv
RUN venv/bin/activate && pip install requirements.lock
When I run the container, I mount the code volume so that I can access and make changes to code from local IDE.
volumes:
- ./myapp:/app
Problem: The problem with this approach is my venv folder which got created in image build disappears because of the volume mount i made.
What is the best practice around it?
Related
I am trying to deploy a python Webapp which is basically an interface for an excel file. The application is based on python.
I've build and ran the container on my local machine and everything is smooth. But when I try to deploy on Azure Web Apps I face an issue which says that my file is missing.
The Dockerfile looks like this
FROM python:3.8-slim-buster AS base
WORKDIR /home/mydashboard
RUN python -m pip install --upgrade pip
RUN pip3 install pipenv
COPY Pipfile /home/mydashboard/Pipfile
COPY Pipfile.lock /home/mydashboard/Pipfile.lock
COPY dataset.csv /home/mydashboard/dataset.csv
COPY src/ /home/mydashboard/src
RUN pipenv install --system
WORKDIR /home/mydashboard/src
EXPOSE 8000
CMD gunicorn -b :8000 dashboard_app.main:server
Weirdly enough when this runs on the Azure App Service I receive a message which was that "dataset.csv" does not exist.
I printed the files inside of /home/dashboard/ it seems that it was an empty folder!!
Who does this happen??
It runs perfectly on my personal computer but it seems that it just run on Azure.
Based on:
Your error message is dataset.csv does not exist
It works on your local machine
Fails in Azure
Probably dataset.csv exists in the copy from location on your local machine, but is missing in Azure.
Trying to host a python http server and works fine.
FROM python:latest
COPY index.html /
CMD python3 -m http.server
But when trying with python virtualenv, facing issues.
FROM python:3
COPY index.html .
RUN pip install virtualenv
RUN virtualenv --python="python3" .virtualenv
RUN .virtualenv/bin/pip install boto3
RUN python3 -m http.server &
CMD ["/bin/bash"]
Please help.
I just want to point up that using virtualenv within docker container might be redundant.
With docker, you are encapsulating your one specific application along with its dependencies (libraries, frameworks, boto3 in your case), as opposed to your local machine where you might have several applications being developed, each with different dependencies.
Thus, I would recommend considering again the necessity of virtualenv within docker.
Second, running the command:
RUN python3 -m http.server &
in the background is also bad practice here. You want to run it with the CMD command in the foreground, so it will run as the first process (PID 1). Then it will receive all docker signals, and start automatically with the container start:
CMD ["virtualenv/bin/python3", "-m", "http.server"]
I am trying to create a Docker container to deploy onto a Raspberry Pi4. The container is to run a Python application. I'm building the container on a Windows 10 PC with the following Dockerfile.
FROM python:3.6-stretch
RUN mkdir /app
WORKDIR /app
ADD . /app/
RUN pip install -r requirements.txt
EXPOSE 5000
CMD ["python", "/app/main.py"]
Once I've built the container on the Windows PC, I'm pushing it to Docker Hub, then using docker run on the Raspberry Pi to pull the image and run it.
I understand that the architectures are different but I was under the impression the the python:3.6-stretch image pulled from Docker Hub was Multi-Arch and so would do the job. However when I try and run the image on the Raspberry Pi it fails with an 'exec format error' suggesting that the image is incorrect for the architecture.
What do I need to change in order to build the container on Windows and run it on Raspberry Pi?
You have to specify the platform. Either in your Dockerfile, or from the command line.
FROM --platform=linux/arm/v7 python:3.6-stretch
You might need to use BuildKit or enable experimental features for your Docker daemon if you want to be able to set the platfrom from the command line:
DOCKER_BUILDKIT=1 docker build --platform=linux/arm/v7 .
You need to have Qemu and docker/binfmt installed to be able to build ARM images on x86_64. The installation process is explained here: https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/
On Linux, you have to install this yourself. From what I've gathered, it's included with Docker for Windows.
I need to use a docker image which has compiled versions of certain programs that are very hard to compile from scratch.
I need to run a program in that environment.
I installed docker and pulled the image (john/custom:py2).
But how do I run a program (A Python program that works in the docker environment) using the environment provided to me by docker, sending in the local folder as input and producing the output back to my local system.
So far all resources tutorials show me how to work inside docker and not the problem I want.
Thanks for our help.
The technical issue is:
docker run -it -v /tmp:/home/ubuntu/myfolder/ john/custom:py2
This sends me to root. But I see none of the folders or files of myfolder in the shell.
i.e ls command gives empty results
How can I run a program inside this docker environment that works on the input of the folder and writes in the same folder.
It sounds like you've reversed the order on the volume syntax. The first half is the host location or volume source, while the second half is the target directory inside the container where the volume is mounted. Try:
docker run -it -v /home/ubuntu/myfolder/:/tmp john/custom:py2
To mount myfolder into the container's /tmp directory.
I tried a little variation using the ubuntu container and it works for me.
$ docker pull ubuntu
$ docker run -it -v /tmp:/home/ubuntu/myfolder ubuntu:latest
$ ls /home/ubuntu/myfolder
Try this and see whether or not it works for you.
I would also try mounting other directories besides /tmp to a directory in the docker container. For example:
$ mkdir /home/john/foo
$ docker run -it -v /home/john/foo:/home/ubuntu/foo ubuntu:latest
/tmp is a little special and I don't know if it's a good idea to mount that directory inside docker.
After installing Docker and googling for hours now, I can't figure out how to place data in a Docker, it seems to become more complex by the minute.
What I did; installed Docker and ran the image that I want to use (kaggle/python). I also read several tutorials about managing and sharing data in Docker containers, but no success so far...
What I want: for now, I simply want to be able to download GitHub repositories+other data to a Docker container. Where and how do I need to store these files? I prefer using GUI or even my GitHub GUI, but simple commands would also be fine I suppose.. Is it also possible to place data or access data from a Docker that is currently not active?
Note that I also assume you are using linux containers. This works in all platforms, but on windows you need to tell your docker process that that you are dealing with linux containers. (It's a dropdown in the tray)
It takes a bit of work to understand docker and the only way to understand it is to get your hands dirty. I recommend starting with making an image of an existing project. Make a Dockerfile and play with docker build . etc.
To cover the docker basics (fast version) first.
In order to run something in docker we first need to build and image
An image is a collection of files
You can add files to an image by making a Dockerfile
Using the FROM keyword on the first line you extend and image
by adding new files to it creating a new image
When staring a container we need to tell what image it should use
and all the files in the image is copied into the containers storage
The simplest way to get files inside a container:
Crate your own image using a Dockerfile and copy in the files
Map a directory on your computer/server into the container
You can also use docker cp, to copy files from and two a container,
but that's not very practical in the long run.
(docker-compose automates a lot of these things for you, but you should probably also play around with the docker command to understand how things work. A compose file is basically a format that stores arguments to the docker command so you don't have to write commands that are multiple lines long)
A "simple" way to configure multiple projects in docker in local development.
In your project directory, add a docker-dev folder (or whatever you want to call it) that contains an environment file and a compose file. The compose file is responsible for telling docker how it should run your projects. You can of course make a compose file for each project, but this way you can run them easily together.
projects/
docker-dev/
.env
docker-compose.yml
project_a/
Dockerfile
# .. all your project files
project_b/
Dockerfile
# .. all your project files
The values in .env is sent as variables to the compose file. We simply add the full path to the project directory for now.
PROJECT_ROOT=/path/to/your/project/dir
The compose file will describe each of your project as a "service". We are using compose version 2 here.
version: '2'
services:
project_a:
# Assuming this is a Django project and we override command
build: ${PROJECT_ROOT}/project_a
command: python manage.py runserver 0.0.0.0:8000
volumes:
# Map the local source inside the container
- ${PROJECT_ROOT}/project_a:/srv/project_a/
ports:
# Map port 8000 in the container to your computer at port 8000
- "8000:8000"
project_a:
# Assuming this is a Django project and we override command
build: ${PROJECT_ROOT}/project_b
volumes:
# Map the local source inside the container
- ${PROJECT_ROOT}/project_b:/srv/project_b/
This will tell docker how to build and run the two projects. We are also mapping the source on your computer into the container so you can work on the project locally and see instant updates in the container.
Now we need to create a Dockerfile for each out our projects, or docker will not know how to build the image for the project.
Example of a Dockerfile:
FROM python:3.6
COPY requirements.txt /requirements.txt
RUN pip install requirements.txt
# Copy the project into the image
# We don't need that now because we are mapping it from the host
# COPY . /srv/project_a
# If we need to expose a network port, make sure we specify that
EXPOSE 8000
# Set the current working directory
WORKDIR /srv/project_a
# Assuming we run django here
CMD python manage.py runserver 0.0.0.0:8000
Now we enter the docker-dev directory and try things out. Try to build a single project at a time.
docker-compose build project_a
docker-compose build project_b
To start the project in background mode.
docker-compose up -d project_a
Jumping inside a running container
docker-compose exec project_a bash
Just run the container in the forground:
docker-compose run project_a
There is a lot of ground to cover, but hopefully this can be useful.
In my case I run a ton of web servers of different kinds. This gets really frustrating if you don't set up a proxy in docker so you can reach each container using a virtual host. You can for example use jwilder-nginx (https://hub.docker.com/r/jwilder/nginx-proxy/) to solve this in a super-easy way. You can edit your own host file and make fake name entires for each container (just add a .dev suffix so you don't override real dns names)
The jwilder-nginx container will automagically send you to a specific container based on a virtualhost name you decide. Then you no longer need to map ports to your local computer except for the nginx container that maps to port 80.
For others who prefer using GUI, I ended up using portainer.
After installing portainer (which is done by using one simple command), you can open the UI by browsing to where it is running, in my case:
http://127.0.1.1:9000
There you can create a container. First specify a name and an image, then scroll down to 'Advanced container options' > Volumes > map additional volume. Click the 'Bind' button, specify a path in the container (e.g. '/home') and the path on your host, and you're done!
Add files to this host directory while your container is not running, then start the container and you will see your files in there. The other way around, accessing in files created by the container, is also possible while the container is not running.
Note: I'm not sure whether this is the correct way of doing things. I will, however, edit this post as soon as I encounter any problems.
After pulling the image, you can use code like this in the shell:
docker run --rm -it -p 8888:8888 -v d:/Kaggles:/d kaggle/python
Run jupyter notebook inside the container
jupyter notebook --ip=0.0.0.0 --no-browser
This mounts the local directory onto the container having access to it.
Then, go to the browser and hit https://localhost:8888, and when I open a new kernel it's with Python 3.5/ I don't recall doing anything special when pulling the image or setting up Docker.
You can find more information from here.
You can also try using datmo in order to easily setup environment and track machine learning projects to make experiments reproducible. You can run datmo task command as follows for setting up jupyter notebook,
datmo task run 'jupyter notebook' --port 8888
It sets up your project and files inside the environment to keep track of your progress.