We Keep Coding

How to create files to avoid permission issues in docker?

I am new using docker and i have a task to create two containers: one for a django project and another for a mysql server.
I made it using Dockerfile and docker-compose.yml.
The project it's working with "docker-compose up", but i have permission problems when i try to create new files inside the container with "python manage.py startapp app_name". Basically i can't modify the files inside my ide(pycharm ubuntu). For what i can understand the user in the docker is root, so i need to have root permission to change those files but i don't want to modify them as root or change the permission of those files each time i make them.
Here is my Dockerfile:
FROM python:3.9.5
ENV PYTHONUNBUFFERED 1
WORKDIR /app
COPY requirements.txt /app/requirements.txt
RUN pip install -r requirements.txt
COPY . /app
CMD python manage.py runserver 0.0.0.0:8000
docker-compose.yml
version: "3.9"
services:
backend:
build:
context: .
dockerfile: Dockerfile
ports:
- 8000:8000
volumes:
- .:/app
depends_on:
- mysqldb
mysqldb:
image: mysql:5.7.34
restart: always
environment:
MYSQL_DATABASE: mydb
MYSQL_USER: django
MYSQL_PASSWORD: djangopass
MYSQL_ROOT_PASSWORD: mysecretpassword
volumes:
- .dbdata:/var/lib/mysql
ports:
- 3306:3306

Better not to use root user. the best way its to create a new user it give it permission as needed. copy the code to the user home directory and run the image not as root.
RUN useradd --create-home myuser
USER myuser
RUN mkdir /home/myuser/code
WORKDIR /home/myuser/code
COPY . .

Django on Docker is starting up but browser gives empty response

For a simple app with Django, Python3, Docker on mac
Dockerfile
FROM python:3
ENV PYTHONUNBUFFERED=1
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN python3 -m pip install -r requirements.txt
CMD python3 manage.py runserver
COPY . /code/
docker-compose.yml
version: "3.9"
services:
# DB
db:
image: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: '****'
MYSQL_USER: '****'
MYSQL_PASSWORD: '****'
MYSQL_DATABASE: 'mydb'
ports:
- "3307:3306"
expose:
# Opens port 3306 on the container
- '3307'
volumes:
- $HOME/proj/sql/mydbdata.sql:/mydbdata.sql
# Web app
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
Also, what I wanted is to execute the SQL the first time the image is created,
after that database should be mounted.
volumes:
- $HOME/proj/sql/mydbdata.sql:/mydbdata.sql
Looks like the Docker is starting but from my browser, I get this response
localhost didn’t send any data.
ERR_EMPTY_RESPONSE
what is that I am missing. Please help

Looks like your django project is running already when you create image. Since you use command option docker-compose.yml file, you don't need CMD command in Dockerfile in this case.
I would rewrite Dockerfile and docker-compose.yml as follows:
FROM python:3
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN python3 -m pip install -r requirements.txt
COPY . /code/
version: "3.9"
services:
db:
image: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: '****'
MYSQL_USER: '****'
MYSQL_PASSWORD: '****'
MYSQL_DATABASE: 'mydb'
ports:
- "3307:3306" # make sure django project connects to 3306 port
volumes:
- $HOME/proj/sql:/docker-entrypoint-initdb.d
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
A few things to point out.
When you run docker-compose up, you will probably see an error, because your django project will already be running even before db is initialised.
That's natural. So you need customized command or shell program to force django project to wait to try to connect db.
In my case I would use a custom command.
version: "3.9"
services:
db:
image: mysql:8
env_file:
- .env
command:
- --default-authentication-plugin=mysql_native_password
restart: always
ports:
- "3308:3306"
web:
build: .
command: >
sh -c "python manage.py wait_for_db &&
python manage.py makemigrations &&
python manage.py migrate &&
python manage.py runserver 0.0.0.0:8000"
volumes:
- .:/code
ports:
- "8001:8000"
depends_on:
- db
env_file:
- .env
Next, wait_for_db.py. This file is what I created in myapp/management/commands/wait_for_db.py. With this you postpone db connection until db is ready. This SO post has helped me a lot.
See Writing custom django-admin command for detail.
import time
from django.db import connection
from django.db.utils import OperationalError
from django.core.management.base import BaseCommand
class Command(BaseCommand):
"""Wait to connect to db until db is initialised"""
def handle(self, *args, **options):
start = time.time()
self.stdout.write('Waiting for database...')
while True:
try:
connection.ensure_connection()
break
except OperationalError:
time.sleep(1)
end = time.time()
self.stdout.write(self.style.SUCCESS(f'Database available! Time taken: {end-start:.4f} second(s)'))
Looks like you want to populate your database with sql file when your db container starts running. Mysql docker hub says
Initializing a fresh instance
When a container is started for the first time, a new database with the specified name will be created and initialized with the provided configuration variables. Furthermore, it will execute files with extensions .sh, .sql and .sql.gz that are found in /docker-entrypoint-initdb.d. Files will be executed in alphabetical order. You can easily populate your mysql services by mounting a SQL dump into that directory and provide custom images with contributed data. SQL files will be imported by default to the database specified by the MYSQL_DATABASE variable.
So your .sql file should be located in /docker-entrypoint-initdb.d in your mysql container. See this post for more.
Last but not least, your db is lost when you run docker-compose down, since you don't have volumes other than sql file. It that's not what you want, you might want to consider the following
version: "3.9"
services:
db:
...
volumes:
- data:/var/lib/mysql
...
volumes:
data:

docker compose up generate (2005, "Unknown MySQL server host 'db' (-2)") error

Hi all i'm trying to dockerize a django application with a connexion to the database. when i run docker compose up i get this error when the dockerfile is making migrations django.db.utils.OperationalError: (2005, "Unknown MySQL server host 'db' (-2)")
ERROR: Service 'web' failed to build: The command '/bin/sh -c python manage.py makemigrations' returned a non-zero code: 1
here is my Dockerfile
FROM python:3.6
ENV PYTHONUNBUFFERED 1
RUN mkdir /SFP_ingestion
WORKDIR /SFP_ingestion
COPY . /SFP_ingestion
RUN pip install -r requirements.txt
RUN python generatemodel.py
RUN python generateapp.py
RUN python manage.py makemigrations
RUN python manage.py migrate
RUN python manage.py migrate easyaudit
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
and here is my docker-compose.yml
services:
db:
image: mysql
restart: always
command: --default-authentication-plugin=mysql_native_password --mysqlx=0
environment:
- MYSQL_HOST=localhost
- MYSQL_PORT=3306 # cannot change this port to other number
- MYSQL_DATABASE=sfp # name you want for the database
- MYSQL_USER=root # change to whatever username you want
- MYSQL_PASSWORD=password #change to the password you want for user
- MYSQL_ROOT_PASSWORD=password #change to good root password
ports:
- "3306:3306"
expose:
- "3306"
volumes:
- "./db:/var/lib/mysql"
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/SFP_ingestion
restart: always
ports:
- "8000:8000"
depends_on:
- db

You have to specify network for your containers.
https://docs.docker.com/compose/networking/

Compose up container exited with code 0 and logs it with empty

I need to containerize a Django Web project with docker. I divided the project into dashboard, api-server and database. When I type docker-compose up, it print api-server exited with code 0 and api-server container Exited (0), and I type docker logs api-server, it return empty, but other container normal. I don't know how to check problem.
api-server directory structure is as follows
api-server
server/
Dockerfile
requirements.txt
start.sh
...
...
Some compose yml content is as follows
dashboard:
image: nginx:latest
container_name: nginx-dashboard
volumes:
- /nginx/nginx/default:/etc/nginx/conf.d/default.conf:ro
- /nginx/dist:/var/www/html:ro
ports:
- "80:80"
depends_on:
- api-server
api-server:
build: /api-server
container_name: api-server
volumes:
- /api-server:/webapps
ports:
- "8000:8000"
depends_on:
- db
db:
image: postgres
container_name: Postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=postgres
ports:
- "5432:5432"
Some Dockerfile content of api-server is as follows
FROM python:3.6
ENV PYTHONUNBUFFERED 1
RUN mkdir /webapps
WORKDIR /webapps
RUN apt-get clean && apt-get update && apt-get upgrade -y && apt-get install -y python3-pip libpq-dev apt-utils
COPY ./requirements.txt /webapps/
RUN pip3 install -r /webapps/requirements.txt
COPY . /webapps/
CMD ["bash","-c","./start.sh"]
start.sh is as follows
#!/usr/bin/env bash
cd server/
python manage.py runserver 0.0.0.0:8000
type docker-compose up result as follows
root#VM:/home/test/Documents/ComposeTest# docker-compose up
Creating network "composetest_default" with the default driver
Creating Postgres ... done
Creating api-server ... done
Creating dashboard ... done
Attaching to Postgres, api-server, dashboard
Postgres | The files belonging to this database system will be owned by user "postgres".
Postgres | This user must also own the server process.
...
...
api-server exited with code 0
api-server exited with code 0
docker logs api-server is empty
I would very appreciate it if you guys can tell me how to check this problems, It is better to provide a solution.

You are already copying api-server to Dockerfile during build time which should work fine, but in Docker compose it all override all the pip packages and code.
volumes:
- /api-server:/webapps
Remove the volume from your Docker compose and it should work.
Second thing set permission to the bash script.
COPY . /webapps/
RUN chmod +x ./start.sh
Third thing, you do need to run python using bash as there is no thing in the bash that CMD can not perform so why not as a CMD?
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

How to initialize a postgres container with data using .dump instead of .sql?

i've seen in the docs and many solution to initialize a postgres container with a database using an init.sql as mentioned in this question:
COPY init.sql /docker-entrypoint-initdb.d/10-init.sql
The problem is that my database data in .sql is to large(8GB). This makes the dump process to long and the file to heavy. I was trying to make something similar to this approach generating a .dump file with (540mb).
I have 2 containers to run my django project. A Dockerfile for my python environment and postgres image. My docker files are shown in the end of this question. So far i've managed to run the dump in my container with these steps:
I add my .dump file to my container in docker-compose
build my containers
Go inside my postgres container and execute a pg_restore command
to dump my database.
Then go to my python/django container to execute a migrate command.
And finally run server.
This works but its not automated. How can i automate this problem in my docker files so i don't need to execute these commands manually?
Bellow are my files and the commands i required to run my .dump file on postgres container:
generate and add ./project-dump/latest.dump to my project
docker-compose up -d web db #build containers
docker-compose exec db bash # go in my postgres container
pg_restore --verbose --clean --no-acl --no-owner -h {pg_container_name} -U postgres -d database_dev
exit # exit postgres container
docker-compose exec web bash # go in my python/django project container
./manage.py migrate # migrate database
./manage.py runserver 0.0.0.0:3000 #run server
Dockerfile:
FROM python:3.7.3-stretch
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/
COPY init.sql /docker-entrypoint-initdb.d/10-init.sql
RUN python manage.py migrate
EXPOSE 3000
docker-compose.yaml
version: '3'
services:
db:
image: postgres
environment:
POSTGRES_USER: postgres
POSTGRES_DB: database_dev
POSTGRES_PASSWORD: secret
ports:
- "5432:5432"
volumes:
- ./project-dump/:/var/www/html
web:
build: .
command: python3 manage.py runserver 0.0.0.0:3000
volumes:
- .:/django-docker
ports:
- "3000:3000"
depends_on:
- db
init.sql
CREATE DATABASE database_dev;
GRANT ALL PRIVILEGES ON DATABASE database_dev TO postgres;