I was wondering what the best approach is to achieve this
docker-compose.yml:
version: '3.8'
services:
web:
build: ./services/website
volumes:
- ./services/website/src:/usr/src/app/src
- ./data/cntautomation_logs:/var/log/cntautomation_logs
expose:
- 5010
env_file:
- .env
environment:
- DEPLOYMENT_TYPE=production
depends_on:
- db
- rabbit_mongodb
entrypoint: ./entrypoint.sh
db:
image: postgres
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data/
- ./data/db_logs:/var/log/postgresql
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=password
- POSTGRES_DB=database.db
nginx:
build: ./services/nginx
volumes:
- ./data/nginx_logs:/var/log/nginx
ports:
- "80:80"
depends_on:
- web
rabbit:
hostname: rabbit
image: "rabbitmq:3-management"
volumes:
- ./data/rabbitmq_logs:/var/log/rabbitmq
env_file:
- .env
ports:
- "15672:15672"
- "5672:5672"
rabbit_mongodb:
image: mongo:latest
volumes:
- ./data/rabbit_mongodb_logs:/var/log/mongodb
command:
- '--logpath'
- '/var/log/mongodb/rabbit_mongodb_log.log'
ports:
- "27017:27017"
worker:
build: ./services/worker
user: nobody
entrypoint: ./entrypoint.sh
volumes:
- ./services/worker/app:/usr/src/app/app
- ./data/worker_logs:/var/log/worker
env_file:
- .env
depends_on:
- rabbit
- web
- rabbit_mongodb
volumes:
postgres_data:
I have a docker compose file like such. I was wondering what the best method is to write to my postgres db in the celery worker container.
The common solution I see for this is creating a make_celery function that passes in the application context, but I don't see how this would work when celery and flask are split into separate containers. One solution I thought about that I am not sure would even work, would be to pass the application context as a pickled object when ever a task is called in a flask route, then execute the task with the application context.
eg.
task.py:
#task(name="task.add")
def add(app, x, y):
with app.app_context():
# Write to db
return x + y
views.py:
#app.route('/add')
def add():
celery.send_task('task.add', args=[app, 1, 2])
I see three problems with the approach though:
I don't know how I would write to the db with a celery beat scheduled task.
Would I create a replica of flask db models in the worker container and instantiate a new sqlalchemy instance? Or maybe pass the model that needs writing as a pickled object just like the app context?
I have not seen anyone do it, so it probably is not even an acceptable approach
I don't really like this solution at all, but im not sure what the standard approach would be
Related
I am running a dockerized django app using the following dockerfile:
services:
web:
build:
context: .
dockerfile: Dockerfile.prod
command: gunicorn PriceOptimization.wsgi:application --bind 0.0.0.0:8000
volumes:
- static_volume:/home/app/web/staticfiles
networks:
- dbnet
ports:
- "8000:8000"
environment:
aws_access_key_id: ${aws_access_key_id}
redis:
restart: always
image: redis:latest
networks:
- dbnet
ports:
- "6379:6379"
celery:
restart: always
build:
context: .
command: celery -A PriceOptimization worker -l info
volumes:
- ./PriceOptimization:/PriceOptimization
depends_on:
- web
- redis
networks:
- dbnet
environment:
access_key_id: ${access_key_id}
nginx:
build: ./nginx
ports:
- "80:80"
volumes:
- static_volume:/home/app/web/staticfiles
depends_on:
- web
networks:
- dbnet
database:
image: "postgres" # use latest official postgres version
restart: unless-stopped
env_file:
- ./database.env # configure postgres
networks:
- dbnet
ports:
- "5432:5432"
volumes:
- database-data:/var/lib/postgresql/data/ # persist data even if container shuts down
volumes:
database-data:
static_volume:
media_volume:
I have added celery.py to my app, and I am building / running the docker container as follows:
docker-compose -f $HOME/PriceOpt/PriceOptimization/docker-compose.prod.yml up -d --build
Running the application in my development environment lets me check at the command line that the celery app is correctly connected, etc. Is there a way that I can test to see if my celery app is initiated properly at the end of the build process?
I've been up and down StackOverflow and Google, but I can't seem to come close to an answer.
tl;dr How do I register a dockerized Celery worker in a dockerized Flower dashboard? How do I point the worker to the Flower dashboard so that the dashboard "knows" about it?
I have 2 FastAPI apps, both deployed with docker-compose.yml files. The first app's compose file looks like this:
version: '3.8'
services:
web:
build:
context: .
dockerfile: ./compose/local/fastapi/Dockerfile
image: app_web
# '/start' is the shell script used to run the service
command: /start
volumes:
- .:/app
ports:
- 8010:8000
env_file:
- .env/.dev-sample
depends_on:
- redis
redis:
image: redis:6-alpine
celery_worker:
build:
context: .
dockerfile: ./compose/local/fastapi/Dockerfile
image: app_celery_worker
command: /start-celeryworker
volumes:
- .:/app
env_file:
- .env/.dev-sample
depends_on:
- redis
flower:
build:
context: .
dockerfile: ./compose/local/fastapi/Dockerfile
image: app_celery_flower
command: /start-flower
volumes:
- .:/app
env_file:
- .env/.dev-sample
ports:
- 5557:5555
depends_on:
- redis
So this app is responsible for creating the Celery Flower dashboard.
The second app's compose file looks like:
version: '3.8'
services:
web:
build:
context: .
dockerfile: ./compose/local/fastapi/Dockerfile
image: app_two_web
# '/start' is the shell script used to run the service
command: /start
volumes:
- .:/app
ports:
- 8011:8000
env_file:
- .env/.dev-sample
depends_on:
- redis
redis:
image: redis:6-alpine
celery_worker:
build:
context: .
dockerfile: ./compose/local/fastapi/Dockerfile
image: app_two_celery_worker
command: /start-celeryworker
volumes:
- .:/app
env_file:
- .env/.dev-sample
depends_on:
- redis
I can't get this second app's worker to register in the Celery Flower dashboard running on port 5557. Everything works fine, and I can even launch a second Flower dashboard with the second app if on a different port, but I can't seem to connect the second worker to the first app's Flower dashboard.
This is what main.py looks like, for both apps.
from project import create_app
app = create_app()
celery = app.celery_app
def celery_worker():
from watchgod import run_process
import subprocess
def run_worker():
subprocess.call(
["celery", "-A", "main.celery", "worker", "-l", "info"]
)
run_process("./project", run_worker)
if __name__ == "__main__":
celery_worker()
Thanks for any ideas that I can throw at this.
First enable event monitoring by putting "-E" in your worker container "command:"
Second, specify environment variable C_FORCE_ROOT in every worker services in your docker-compose configuration.
My Flask app (webapp) has two directories inside (uploads and images). I want my second container (rq-worker) to have access to them, so it can take something from uploads, and save back to images. How I can organize this inside my docker-compose.yml?
version: '3.5'
services:
web:
build: ./webapp
image: webapp
container_name: webapp
ports:
- "5000:5000"
depends_on:
- redis-server
- mongodb
redis-server:
image: redis:alpine
container_name: redis-server
ports:
- 6379:6379
mongodb:
image: mongo:4.2-bionic
container_name: mongodb
ports:
- "27017:27017"
rq-worker:
image: jaredv/rq-docker:0.0.2
container_name: rq-worker
command: rq worker -u redis://redis-server:6379 high normal low
deploy:
replicas: 1
depends_on:
- redis-server
dashboard:
build: ./dashboard
image: dashboard
container_name: dashboard
ports:
- "9181:9181"
command: rq-dashboard -H redis-server
depends_on:
- redis-server
You'll need to specify a volume like this:
volumes:
data-volume:
And then attach it to your services, e.g. for web:
web:
build: ./webapp
image: webapp
container_name: webapp
ports:
- "5000:5000"
depends_on:
- redis-server
- mongodb
volumes:
- data-volume:/my/mnt/point
The documentation has more info, also how to configure a volume driver, e.g. if you want to have a volume on NFS. Furthermore this lists available volume plugins which enable Docker volumes to persist across multiple Docker hosts.
I have a two services in docker compose 1. an application and 2. db service (can be Mysql or Postgres). Depending upon environment variables set in compose file for db services, I need create DATABASE_URI for sqlalchemy engine. How do I access these ENV in app docker container?
I am trying to access env set inside docker-compose file and not Dockerfile.
Below is how my Docker-compose file looks
version: "3.7"
services:
myapp:
image: ${TAG:-myapp}
build:
context: .
ports:
- "5000:5000"
docker_postgres:
image: "postgres:latest"
environment:
- POSTGRES_PASSWORD=postgres
ports:
- "5432:5432"
You need to set environment variables for your postgres database if you want to build your own database_uri.
environment:
POSTGRES_DB: dev
POSTGRES_USER: username
POSTGRES_PASSWORD: pw
#jonrsharpe You mean to say I can do something like this?
services:
myapp:
image: ${TAG:-myapp}
build:
context: .
environment:
- DB_USER=postgres
- DB_PASSWORD=postgres
- DB_HOST=docker_postgres
ports:
- "5000:5000"
docker_postgres:
image: "postgres:latest"
environment:
- POSTGRES_PASSWORD=postgres
- POSTGRES_USER=postgres
ports:
- "5432:5432"```
what i am trying to do - run airflow in docker with celery
my issue - my celery workers are in containers and i dont know how to scale them
my docker-comopose file:
version: '2'
services:
mysql:
image: mysql:latest
restart: always
ports:
- "3306:3306"
environment:
- MYSQL_RANDOM_ROOT_PASSWORD=true
- MYSQL_USER=airflow
- MYSQL_PASSWORD=airflow
- MYSQL_DATABASE=airflow
volumes:
- mysql:/var/lib/mysql
rabbitmq:
image: rabbitmq:3-management
restart: always
ports:
- "15672:15672"
- "5672:5672"
- "15671:15671"
environment:
- RABBITMQ_DEFAULT_USER=airflow
- RABBITMQ_DEFAULT_PASS=airflow
- RABBITMQ_DEFAULT_VHOST=airflow
volumes:
- rabbitmq:/var/lib/rabbitmq
webserver:
image: airflow:ver5
restart: always
volumes:
- ~/airflow/dags:/usr/local/airflow/dags
- /opt/scripts:/opt/scripts
environment:
- AIRFLOW_HOME=/usr/local/airflow
ports:
- "8080:8080"
links:
- mysql:mysql
- rabbitmq:rabbitmq
- worker:worker
- scheduler:scheduler
depends_on:
- mysql
- rabbitmq
- worker
- schedulerv
command: webserver
env_file: ./airflow.env
scheduler:
image: airflow:ver5
restart: always
volumes:
- ~/airflow/dags:/usr/local/airflow/dags
- /opt/scripts:/opt/scripts
environment:
- AIRFLOW_HOME=/usr/local/airflow
links:
- mysql:mysql
- rabbitmq:rabbitmq
depends_on:
- mysql
- rabbitmq
command: scheduler
env_file: ./airflow.env
worker:
image: airflow:ver5
restart: always
volumes:
- ~/airflow/dags:/usr/local/airflow/dags
- /opt/scripts:/opt/scripts
environment:
- AIRFLOW_HOME=/usr/local/airflow
ports:
- "8793:8793"
links:
- mysql:mysql
- rabbitmq:rabbitmq
depends_on:
- mysql
- rabbitmq
command: worker
env_file: ./airflow.env
So i run the docker-compose command using the above file and it starts an instance of worker on port 8793 on localhost as i am mapping from docker port to localhost. Now what i want to do is scale the number of workers i have and to do that i use the following command:
docker-compose -f docker-compose.yml scale worker=5
but that gives out an error as an instance of worker is already running on 8793. Is there a way to dynamically allocate port to new instances of worker containers as i scale up?
You could allow your worker nodes to expose the worker port to the host machine on a random port number:
worker:
image: airflow:ver5
restart: always
volumes:
- ~/airflow/dags:/usr/local/airflow/dags
- /opt/scripts:/opt/scripts
environment:
- AIRFLOW_HOME=/usr/local/airflow
ports:
- "8793"
links:
- mysql:mysql
- rabbitmq:rabbitmq
depends_on:
- mysql
- rabbitmq
command: worker
env_file: ./airflow.env
Setting port: to - 80 will expose port 80, in the container, to a random port on the host.
Because Docker Compose uses networks, you can actually omit this publish step altogether, and it would work. So simply remove ports: from the worker