Celery Module Tasks are Cached so edits in dev environment do not apply - python

I have a problem which makes local development with Celery very difficult.
If I edit my local files and restart docker containers none of the CODE changes are applied. Not talking about task result caching here... just the actual function execution.
I have to prune everything for them to be applied.
Does anyone have any solution for this?
supervisord.conf
[supervisord]
nodaemon=true
[program:celeryworker]
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
command=celery -A worker.scheduler.schedule.celery_app worker -l info
[program:celerybeat]
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
command=celery -A worker.scheduler.schedule.celery_app beat -l info
worker.Dockerfile
FROM python:3.10
WORKDIR /usr/src/app
# install supervisord
RUN apt-get update && apt-get install -y supervisor
RUN apt-get install -y python3-pymysql
# copy requirements and install (so that changes to files do not mean rebuild cannot be cached)
RUN mkdir worker
COPY worker/requirements.txt /usr/src/app/worker
COPY worker/supervisord.conf /usr/src/app
RUN pip install -r /usr/src/app/worker/requirements.txt
# copy all files into the container
COPY ./worker /usr/src/app/worker
COPY ./db /usr/src/app/db
# needs to be set else Celery gives an error (because docker runs commands inside container as root)
ENV C_FORCE_ROOT=1
# run supervisord
CMD ["/usr/bin/supervisord"]
compose subset
celery_worker:
build:
context: server
dockerfile: worker.Dockerfile
volumes:
- ./worker:/tmp/worker
environment:
- CELERY_BROKER_URL=amqp://guest:guest#rabbitmq:5672//
- CELERY_RESULT_BACKEND=redis://#redis:6000/0
depends_on:
- db
- rabbitmq
- redis

Related

Django-crontab doesn't execute tasks on Docker container

I've been working on this all week, and I don't seem to understand what I'm missing. The problem is simple, I've a container running a platform on Django, and I need to create a Cronjob for an smaller task, I created a test ask that executes every minute and just print a log just to test, but it is not working, while it installs cron, add the cronjobs, start the cron service, and I can see them in the crontab, they are just never triggered.
When I first started, I had the Cron running in the same instance, but after reading this Question I found that I had to separate it in 2 instances, since apparently having Django running was afecting the cron service, so following that, this is how I have my files:
docker-compose.yml
version: '3'
services:
auth:
build:
context: ./
dockerfile: ./devops/Dockerfile
args:
[Bunch of parameters]
container_name: auth
volumes:
- ./project:/app
ports:
- 8000:8000
environment:
[Bunch of parameters]
command: python manage.py runserver 0.0.0.0:8000
cron:
build:
context: ./
dockerfile: ./devops/Dockerfile
args:
[Bunch of parameters]
container_name: cron
volumes:
- ./project:/app
environment:
[Bunch of parameters]
command: cron -f
Dockerfile
FROM python:3.8
ENV PYTHONUNBUFFERED 1
WORKDIR /app
COPY ./devops/requirements.txt .
COPY ./project .
# COPY ./.env .
RUN apt-get update
RUN apt-get -y install cron
RUN cp ./.env . || echo "file not found"
RUN pip install -r requirements.txt
#Set permission to entrypoint.sh
RUN chmod +x entrypoint.sh
# start web server
ENTRYPOINT ["./entrypoint.sh"]
CMD ["gunicorn", "-b", "0.0.0.0:8000", "project.wsgi:application", "--workers=5"]
entrypoint.sh
#!/bin/sh
# Set up scheduled jobs, if this is the cron container.
if [ "$1" = cron ]; then
service cron start
python ./manage.py crontab add
service cron stop
fi
# Run whatever command we got passed.
exec "$#"
settings.py
CRONJOBS = [
('*/1 * * * *', 'apps.coupons.cron.test'),
]
cron.py
import logging
logger = logging.getLogger(__name__)
def test():
logger.warning('Hello World')
logger.debug('Hello World')
logger.info('Hello World')
logger.error('Hello World')
logger.critical('Hello World')
print("Hello World")
return "Finished"
Here you can see that the cron were added, and that cron is running, and that executing the cronjob manually works.
Still, doesn't matter how long I wait, it doesn't seem like the Cronjob runs automatically every minute (I check this by using the file.log that I setted in the settings.py logger's options). What am I doing wrong or what may be missing to make the cron work?

Running Django's collectstatic in Dockerfile produces empty directory

I'm trying to run Django from a Docker container on Heroku, but to make that work, I need to run python manage.py collectstatic during my build phase. To achieve that, I wrote the following Dockerfile:
# Set up image
FROM python:3.10
WORKDIR /usr/src/app
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
# Install poetry and identify Python dependencies
RUN pip install poetry
COPY pyproject.toml /usr/src/app/
# Install Python dependencies
RUN set -x \
&& apt update -y \
&& apt install -y \
libpq-dev \
gcc \
&& poetry config virtualenvs.create false \
&& poetry install --no-ansi
# Copy source into image
COPY . /usr/src/app/
# Collect static files
RUN python -m manage collectstatic -v 3 --no-input
And here's the docker-compose.yml file I used to run the image:
services:
db:
image: postgres
env_file:
- .env.docker.db
volumes:
- db:/var/lib/postgresql/data
networks:
- backend
ports:
- "5433:5432"
web:
build: .
restart: always
env_file:
- .env.docker.web
ports:
- "8001:$PORT"
volumes:
- .:/usr/src/app
depends_on:
- db
networks:
- backend
command: gunicorn --bind 0.0.0.0:$PORT myapp.wsgi
volumes:
db:
networks:
backend:
driver: bridge
The Dockerfile builds just fine, and I can even see that collectstatic is running and collecting the appropriate files during the build. However, when the build is finished, the only evidence that collectstatic ran is an empty directory called staticfiles. If I run collectstatic again inside of my container, collectstatic works just fine, but since Heroku doesn't persist files created after the build stage, they disappear when my app restarts.
I found a few SO answers discussing how to get collectstatic to run inside a Dockerfile, but that's not my problem; my problem is that it does run, but the collected files don't show up in the container. Anyone have a clue what's going on?
UPDATE: This answer did the trick. My docker-compose.yml was overriding the changes made by collectstatic with this line:
volumes:
- .:/usr/src/app
If, like me, you want to keep the bind mount for ease of local development (so that you don't need to re-build each time), you can edit the command for the web service as follows:
command: bash -c "python -m manage collectstatic && gunicorn --bind 0.0.0.0:$PORT myapp.wsgi"
Note that the image would have run just fine as-is had I pushed it to Heroku (since Heroku doesn't use the docker-compose.yml file), so this was just a problem affecting containers I created on my local machine.
You are overriding the content of /usr/src/app in your container when you added the
volumes:
- .:/usr/src/app
to your docker compose file.
Remove it since you already copied everything during the build.

How to stop a docker database container

Trying to run the following docker compose file
version: '3'
services:
database:
image: postgres
container_name: pg_container
environment:
POSTGRES_USER: partman
POSTGRES_PASSWORD: partman
POSTGRES_DB: partman
app:
build: .
container_name: partman_container
links:
- database
environment:
- DB_NAME=partman
- DB_USER=partman
- DB_PASSWORD=partman
- DB_HOST=database
- DB_PORT=5432
- SECRET_KEY='=321t+92_)#%_4b+f-&0ym(fs2p5-0-_nz5mhb_cak9zlo!bv#'
depends_on:
- database
expose:
- "8000"
- "8020"
ports:
- "127.0.0.1:8020:8020"
volumes:
pgdata: {}
when running docker-compose up-build with the following docker file
# Dockerfile
# FROM directive instructing base image to build upon
FROM python:3.7-buster
RUN apt-get update && apt-get install nginx vim -y --no-install-recommends
COPY nginx.default /etc/nginx/sites-available/default
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log
RUN mkdir .pip_cache \
mkdir -p /opt/app \
mkdir -p /opt/app/pip_cache \
mkdir -p /opt/app/py-partman
COPY start-server.sh /opt/app/
COPY requirements.txt start-server.sh /opt/app/
COPY .pip_cache /opt/app/pip_cache/
COPY partman /opt/app/py-partman/
WORKDIR /opt/app
RUN pip install -r requirements.txt --cache-dir /opt/app/pip_cache
RUN chown -R www-data:www-data /opt/app
RUN /bin/bash -c 'ls -la; chmod +x /opt/app/start-server.sh; ls -la'
EXPOSE 8020
STOPSIGNAL SIGTERM
CMD ["/opt/app/start-server.sh"]
/opt/app/start-server.sh :
#!/usr/bin/env bash
# start-server.sh
ls
pwd
cd py-partman
ls
pwd
python manage.py createsuperuser --no-input
python manage.py makemigrations
python manage.py migrate
python manage.py initialize_entities
the database image keeps on running, i want to stop it because otherwise the jenkins job will keep on waiting for the image to terminate.
Any good ideas / better ideas how to do so ?
Maybe with -> docker stop <"container id or container name">
Use -f to force it, if it can't be stopped.
Try it.
Docker Compose is generally oriented around long-running server-type processes, and where database containers can frequently take 30-60 seconds to start up, it's usually beneficial to not repeat them. (In fact, the artifacts you show look a little odd for not including a python manage.py runserver command.)
It looks like there is a docker-compose up option for what you're looking for
docker-compose up --build --abort-on-container-exit
If you wanted to do this more manually, and especially if your app container's normal behavior is to actually start a server, you can docker-compose run the initialization command. This will start up the container and its dependencies, but it also expects its command to return, and then you can clean up yourself.
docker-compose build
docker-compose run app /opt/app/initialize-only.sh
docker-compose down -v

docker-compose up --build, get stuck while installing the pip package in alpine container

Installing the package in alpine get stuck
it stuck at
(6/12) Installing ncurses-terminfo (6.1_p20190105-r0) OR
(10/12) Installing python2 (2.7.16-r1)
Sometimes it works properly.
Command: sudo docker-compose build
Tried proxy but didn't worked
# Docker Upstart and SysVinit configuration file
#
# THIS FILE DOES NOT APPLY TO SYSTEMD
#
# Please see the documentation for "systemd drop-ins":
# https://docs.docker.com/engine/admin/systemd/
#
# Customize location of Docker binary (especially for development testing).
#DOCKERD="/usr/local/bin/dockerd"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
# If you need Docker to use an HTTP proxy, it can also be specified here.
export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export DOCKER_TMPDIR="/mnt/bigdrive/docker-tmp"
Also tried by increating the MTU
docker-compose.yml
version: '3.7'
services:
admin-api:
container_name: admin-api
build:
context: .
dockerfile: Dockerfile
environment:
- HOME=/home
- NODE_ENV=dev
- DB_1=mongodb://mongo:27017/DB_1
- DB_2=mongodb://mongo:27017/DB_2
volumes:
- '.:/app'
- '/app/node_modules'
- '$HOME/.aws:/home/.aws'
ports:
- '4004:4004'
networks:
- backend
links:
- mongo
mongo:
container_name: mongo
image: mongo:4.2.0-bionic
ports:
- "27018:27017"
networks:
- backend
networks:
backend:
driver: bridge
driver_opts:
com.docker.network.driver.mtu: 1500
Dockerfile
# base image
FROM node:8.16.1-alpine
# set working directory
WORKDIR /app
# add `/app/node_modules/.bin` to $PATH
ENV PATH /app/node_modules/.bin:$PATH
# install and cache app dependencies
COPY package.json /app/package.json
RUN apk add --update-cache py-pip && \
pip install awscli && \
apk --purge -v del py-pip && \
rm -rf /var/cache/apk/*
RUN npm install --silent
RUN npm install -g nodemon
# start app
CMD nodemon
EXPOSE 4004
My work is dependent on AWS and it requires AWS credentials, I installed the AWS using pip and mounted the /home/.aws (local) with /home/.aws container but when I am creating or building the container it gets stuck and doesn't show any error. While building the container, I also checked the network monitor, it shows receiving packets 0 bytes/s
Tried --verbose but it didn't get any useful information

how to run server using docker container?

Django server is running well in localhost. however, When I try to run server on the docker container, it doesn't find the manage.py file when using docker-compose file and even I run the container manually and run the server, it doesn't appear in browser. how can I solve this problem?
So I wrote all the code testing on my local server and using the dockerfile, I built the image of my project.
and I tried to run server on the docker container, suddenly this doesn't run.
what's worse, if I use docker-compose to run the server, it doesn't find the manage.py file though I already checked with 'docker run -it $image_name sh'
here is the code of my project
I am new to docker and new to programming.
hope you can give me a help. thanks!
file structure
current directory
└─example
└─db.sqlite3
└─docker-compose.yml
└─Dockerfile
└─manage.py
└─Pipfile
└─Pipfile.lock
Docker file
# Base image - Python version
FROM python:3.6-alpine
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Set work directory
WORKDIR /code
# Copy Pipfile
COPY Pipfile /code
COPY Pipfile.lock /code
# Install dependencies
RUN pip install pipenv
RUN pipenv install --system
# Copy files
COPY . /code/
docker-compose.yml
# docker-compose.yml
version: '3.3'
services:
web:
build: .
command: python /code/manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- 8000:8000
expected result : running server in web browser like in chrome
actual result :
when using docker-compose :
ERROR like this in the prompt : web_1 | python: can't open file '/code/manage.py': [Errno 2] No such file or directory
when running the container manually with 'docker run -it $image_name sh' and 'python manage.py runserver' on the shell :
server is running but, doesn't connect to web browser. (doesn't show up in browser like chrome'
Yo have done same thing in many ways. You have copy source files using a COPY command and then you have mounted a host volume in your docker-compose.yml file. In first place you don't need a volume because volume mounts are to persisting data generated by and used by Docker containers.
Following simplified Dockerfile and docker-compose file would fix the problem.
# Base image - Python version
FROM python:3.6-alpine
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Copy files
COPY . /code/
# Set work directory
WORKDIR /code
# Install dependencies
RUN pip install pipenv
RUN pipenv install --system
docker-compose.yml -:
# docker-compose.yml
version: '3.3'
services:
web:
build: .
command: python ./manage.py runserver 0.0.0.0:8000
ports:
- 8000:8000

Categories