I have a dockerized setup running a Django app within which I use Celery tasks. Celery uses Redis as the broker.
Versioning:
Docker version 17.09.0-ce, build afdb6d4
docker-compose version 1.15.0, build e12f3b9
Django==1.9.6
django-celery-beat==1.0.1
celery==4.1.0
celery[redis]
redis==2.10.5
Problem:
My celery workers appear to be unable to connect to the redis container located at localhost:6379. I am able to telnet into the redis server on the specified port. I am able to verify redis-server is running on the container.
When I manually connect to the Celery docker instance and attempt to create a worker using the command celery -A backend worker -l info I get the notice:
[2017-11-13 18:07:50,937: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 99 connecting to localhost:6379. Cannot assign requested address..
Trying again in 4.00 seconds...
Notes:
I am able to telnet in to the redis container on port 6379. On the redis container, redis-server is running.
Is there anything else that I'm missing? I've gone pretty far down the rabbit hole, but feel like I'm missing something really simple.
DOCKER CONFIG FILES:
docker-compose.common.yml here
docker-compose.dev.yml here
When you use docker-compose, you aren't going to be using localhost for inter-container communication, you would be using the compose-assigned hostname of the container. In this case, the hostname of your redis container is redis. The top level elements under services: are your default host names.
So for celery to connect to redis, you should try redis://redis:6379/0. Since the protocol and the service name are the same, I'll elaborate a little more: if you named your redis service "butter-pecan-redis" in your docker-compose, you would instead use redis://butter-pecan-redis:6379/0.
Also, docker-compose.dev.yml doesn't appear to have celery and redis on a common network, which might cause them not to be able to see each other. I believe they need to share at least one network in common to be able to resolve their respective host names.
Networking in docker-compose has an example in the first handful of paragraphs, with a docker-compose.yml to look at.
You may need to add the link and depends_on sections to your docker compose file, and then reference the containers by their hostname.
Updated docker-compose.yml:
version: '2.1'
services:
db:
image: postgres
memcached:
image: memcached
redis:
image: redis
ports:
- '6379:6379'
backend-base:
build:
context: .
dockerfile: backend/Dockerfile-base
image: "/backend:base"
backend:
build:
context: .
dockerfile: backend/Dockerfile
image: "/backend:${ENV:-local}"
command: ./wait-for-it.sh db:5432 -- gunicorn backend.wsgi:application -b 0.0.0.0:8000 -k gevent -w 3
ports:
- 8000
links:
- db
- redis
- memcached
depends_on:
- db
- redis
- memcached
celery:
image: "/backend:${ENV:-local}"
command: ./wait-for-it.sh db:5432 -- celery worker -E -B --loglevel=INFO --concurrency=1
environment:
C_FORCE_ROOT: "yes"
links:
- db
- redis
- memcached
depends_on:
- db
- redis
- memcached
frontend-base:
build:
context: .
dockerfile: frontend/Dockerfile-base
args:
NPM_REGISTRY: http://.view.build
PACKAGE_INSTALLER: yarn
image: "/frontend:base"
links:
- db
- redis
- memcached
depends_on:
- db
- redis
- memcached
frontend:
build:
context: .
dockerfile: frontend/Dockerfile
image: "/frontend:${ENV:-local}"
command: 'bash -c ''gulp'''
working_dir: /app/user
environment:
PORT: 3000
links:
- db
- redis
- memcached
depends_on:
- db
- redis
- memcached
Then configure the urls to redis, postgres, memcached, etc. with:
redis://redis:6379/0
postgres://user:pass#db:5432/database
The issue for me was that all of the containers, including celery had a network argument specified. If this is the case the redis container must also have the same argument otherwise you will get this error. See below, the fix was adding 'networks':
redis:
image: redis:alpine
ports:
- '6379:6379'
networks:
- server
Related
I have a docker-compose file for a Django application.
Below is the structure of my docker-compose.yml
version: '3.8'
volumes:
pypi-server:
services:
backend:
command: "bash ./install-ppr_an_run_dphi.sh"
build:
context: ./backend
dockerfile: ./Dockerfile
volumes:
- ./backend:/usr/src/app
expose:
- 8000:8000
depends_on:
- db
pypi-server:
image: pypiserver/pypiserver:latest
ports:
- 8080:8080
volumes:
- type: volume
source: pypi-server
target: /data/packages
command: -P . -a . /data/packages
restart: always
db:
image: mysql:8
ports:
- 3306:3306
volumes:
- ~/apps/mysql:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=gary
- MYSQL_PASSWORD=tempgary
- MYSQL_USER=gary_user
- MYSQL_DATABASE=gary_db
nginx:
build: ./nginx
ports:
- 80:80
depends_on:
- backend
Django app is dependent on a couple of private packages hosted on the private-pypi-server without which the app won't run.
I created a separate dockerfile for django-backend alone which install packages of requirements.txt and the packages from private-pypi-server. But the dockerfile of django-backend service is running even before the private pypi server is running.
If I move the installation of private packages to docker-compose.yml command code under django-backend service in , then it works fine. Here the issue is that, if the backend is running and I want to run some commands in django-backend(./manage.py migrat) then it says that the private packages are not installed.
Im not sure how to proceed with this, it would be really helpful If i can get all these services running at once by just running the command docker-compose up --build -d
Created a separate docker-compose for pypi-server, which will be up and running even before I build/start other services.
Have you tried adding the pipy service to depends_on of the backend app?
backend:
command: "bash ./install-ppr_an_run_dphi.sh"
build:
context: ./backend
dockerfile: ./Dockerfile
volumes:
- ./backend:/usr/src/app
expose:
- 8000:8000
depends_on:
- db
- pypi-server
Your docker-compose file begs a few questions though.
Why to install custom packages to the backend service at a run time? I can see so many problems which might arise from this such as latency during service restarts, possibly different environments between runs of the same version of the backend service, any problems with the installation would come up during the deployment bring it down, etc. Installation should be done during the build of the docker image. Could you provide your Dockerfile maybe?
Is there any reason why the pypi server has to share docker-compose with the application? I'd suggest having it in a separate deployment especially if it is to be shared among other projects.
Is the pypi server supposed to be used for anything else than a source of the custom packages for the backend service? If not then I'd consider getting rid of it / using it for the builds only.
Is there any good reason why you want to have all the ports exposed? This creates a significant attack surface. E.g. an attacker could bypass the reverse proxy and talk directly to the backend service using port 8000 or they'd be able to connect to the db on the port 3306. Nb docker-compose creates subnetworks among the containers so they can access each other's ports even if those ports are not forwarded to the host machine.
Consider using docker secrets to store db credentials.
I have a docker image using the compose.yml file as below
version: "3.9"
services:
flask:
build:
context: consumelogs/
dockerfile: Dockerfile.web
env_file:
- ./consumelogs/.env
ports:
- "5000:5000"
redis:
image: "redis:alpine"
ports:
- "6379:6379"
worker:
build:
context: consumelogs/
dockerfile: Dockerfile.worker
env_file:
- ./consumelogs/.env
depends_on:
- redis
And my SQL Server's (which is a Docker image) command is
docker run -d --name sql_server -e ‘ACCEPT_EULA=Y’ -e ‘SA_PASSWORD=123456’ --net slackbot-net -p 1433:1433 mcr.microsoft.com/mssql/server:2019-latest
where the default userName = sa and my bridge network = slackbot-net. My python app docker is started using the command
docker-compose up --build --force-recreate
I know that if I use localhost as ServerName, it points to docker itself, but I've tried my machine's IP address, 0.0.0.0, 127.0.0.1 and even the IP address from the query
SELECT
CONNECTIONPROPERTY('net_transport') AS net_transport,
CONNECTIONPROPERTY('protocol_type') AS protocol_type,
CONNECTIONPROPERTY('auth_scheme') AS auth_scheme,
CONNECTIONPROPERTY('local_net_address') AS local_net_address,
CONNECTIONPROPERTY('local_tcp_port') AS local_tcp_port,
CONNECTIONPROPERTY('client_net_address') AS client_net_address
and I still get Cannot assign requested address. Is there something obvious that I'm missing?
Azure Data Studio is able to connect to the SQL Server from docker using localhost and username/password combo, just fine and so does my python tool using pytds, but just not from within docker.
Any help is greatly appreciated!
I had my Django web app running on the Azure App Services using a single docker container instances. However, I plan to add one more container to run the celery service.
Before going to try the compose with celery and Django web app, I first tried using their docker-compose option to run the Django web app before including the compose with celery service.
Following is my docker-compose configuration for Azure App Service
version: '3.3'
services:
web:
image: azureecr.azurecr.io/image_name:15102020155932
command: gunicorn DjangoProj.wsgi:application --workers=4 --bind 0.0.0.0:8000 --log-level=DEBUG
ports:
- 8000:8000
However, the only thing that I see in my App Service logs is:
2020-10-16T07:02:31.653Z INFO - Stopping site MYSITE because it failed during startup.
2020-10-16T13:26:20.047Z INFO - Stopping site MYSITE because it failed during startup.
2020-10-16T14:51:07.482Z INFO - Stopping site MYSITE because it failed during startup.
2020-10-16T16:40:49.109Z INFO - Stopping site MYSITE because it failed during startup.
2020-10-16T16:43:05.980Z INFO - Stopping site MYSITE because it failed during startup.
I tried the combination of celery and Django app using docker-compose on my LOCAL environment and it seems to be working as expected.
Following is the docker-compose file that I am using to run it on local:
version: '3'
services:
web:
image: azureecr.azurecr.io/image_name:15102020155932
build: .
command: gunicorn DjangoProj.wsgi:application --workers=4 --bind 0.0.0.0:8000 --log-level=DEBUG
ports:
- 8000:8000
env_file:
- .env.file
celery:
image: azureecr.azurecr.io/image_name:15102020155932
build: .
command: celery -A DjangoProj worker -l DEBUG
depends_on:
- web
restart: on-failure
env_file:
- .env.file
What am I missing?
I have checked multiple SO questions but they are all left without an answer.
I can provide more details if required.
P.S. there's an option to run both Django and Celery in the same container and call it a day, but I am looking for a cleaner and scalable solution.
You have to change port because Azure does not support multi container app on port 8000.
Exemple of Configuration-file.yaml
version: '3.3'
services:
api:
image: containerdpt.azurecr.io/xxxxxxx
command: python manage.py runserver 0.0.0.0:8080
ports:
- "8080:8080"
Is there any chance you can time the startup of your site? My first concern with this is it's not starting up within 230 seconds or an external dependency such as the celery container is not ready within 230 seconds.
To see if this is the issue, can you try raising the startup time?
Set the WEBSITES_CONTAINER_START_TIME_LIMIT App Setting to the value you want.
Default Value = 230 Sec.
Max Value= 1800 Sec
I am dockerizing a python script, and I run it as CMD ['python', 'script.py'], in the Dockerfile. When I up the container using docker-compose.yml, it runs,
But when I docker exec and go inside the container and do a ps -aux, I see the %CPU is 100%, because of this the purpose of the service is not met.
If I do the same process, i.e, by doing a docker exec and run the script python script.py manually in the container, It works good and I can see only a 5% of the CPU is utilized, as well as the service works and gives the expected result.
Service wrote in docker-compose:
consumer:
restart: always
image: consumer:latest
build: ./consumer
ports:
- "8283:8283"
depends_on:
- redis
environment:
- REDIS_HOST = redis
redis:
image: redis
command: redis-server
volumes:
- ./redis_data:/data
ports:
- "6379:6379"
restart: unless-stopped
It is a consumer application, which consumes the message from the producer and writes into a Redis server.
Can someone advice why such behavior is observed.
I'm trying to build a Flask app that has Kafka as an interface. I used a Python connector, kafka-python and a Docker image for Kafka, spotify/kafkaproxy .
Below is the docker-compose file.
version: '3.3'
services:
kafka:
image: spotify/kafkaproxy
container_name: kafka_dev
ports:
- '9092:9092'
- '2181:2181'
environment:
- ADVERTISED_HOST=0.0.0.0
- ADVERTISED_PORT=9092
- CONSUMER_THREADS=1
- TOPICS=PROFILE_CREATED,IMG_RATED
- ZK_CONNECT=kafka7zookeeper:2181/root/path
flaskapp:
build: ./flask-app
container_name: flask_dev
ports:
- '9000:5000'
volumes:
- ./flask-app:/app
depends_on:
- kafka
Below is the Python snippet I used to connect to kafka. Here, I used the Kafka container's alias kafka to connect, as Docker would take care of mapping the alias to it's IP address.
from kafka import KafkaConsumer, KafkaProducer
TOPICS = ['PROFILE_CREATED', 'IMG_RATED']
BOOTSTRAP_SERVERS = ['kafka:9092']
consumer = KafkaConsumer(TOPICS, bootstrap_servers=BOOTSTRAP_SERVERS)
I got NoBrokersAvailable error. From this, I could understand that the Flask app could not find the Kafka server.
Traceback (most recent call last):
File "./app.py", line 11, in <module>
consumer = KafkaConsumer("PROFILE_CREATED", bootstrap_servers=BOOTSTRAP_SERVERS)
File "/usr/local/lib/python3.6/site-packages/kafka/consumer/group.py", line 340, in __init__
self._client = KafkaClient(metrics=self._metrics, **self.config)
File "/usr/local/lib/python3.6/site-packages/kafka/client_async.py", line 219, in __init__
self.config['api_version'] = self.check_version(timeout=check_timeout)
File "/usr/local/lib/python3.6/site-packages/kafka/client_async.py", line 819, in check_version
raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
Other Observations:
I was able to run ping kafka from the Flask container and get packets from the Kafka container.
When I run the Flask app locally, trying to connect to the Kafka container by setting BOOTSTRAP_SERVERS = ['localhost:9092'], it works fine.
UPDATE
As mentioned by cricket_007, given that you are using the docker-compose provided below, you should use kafka:29092 to connect to Kafka from another container. So your code would look like this:
from kafka import KafkaConsumer, KafkaProducer
TOPICS = ['PROFILE_CREATED', 'IMG_RATED']
BOOTSTRAP_SERVERS = ['kafka:29092']
consumer = KafkaConsumer(TOPICS, bootstrap_servers=BOOTSTRAP_SERVERS)
END UPDATE
I would recommend you use the Kafka images from Confluent Inc, they have all sorts of example setups using docker-compose that are ready to use and they are always updating them.
Try this out:
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
flaskapp:
build: ./flask-app
container_name: flask_dev
ports:
- '9000:5000'
volumes:
- ./flask-app:/app
I used this docker-compose.yml and added your service on top
Please note that:
The config used here exposes port 9092 for external connections to the broker i.e. those from outside the docker network. This could be from the host machine running docker, or maybe further afield if you've got a more complicated setup. If the latter is true, you will need to change the value 'localhost' in KAFKA_ADVERTISED_LISTENERS to one that is resolvable to the docker host from those remote clients
Make sure you check out the other examples, may be useful for you especially when moving to production environments: https://github.com/confluentinc/cp-docker-images/tree/5.0.1-post/examples
Also worth checking:
It seems that you need to specify the api_version to avoid this error. For more details check here.
Version 1.3.5 of this library (which is latest on pypy) only lists certain API versions 0.8.0 to 0.10.1. So unless you explicitly specify api_version to be (0, 10, 1) the client library's attempt to discover the version will cause a NoBrokersAvailable error.
producer = KafkaProducer(
bootstrap_servers=URL,
client_id=CLIENT_ID,
value_serializer=JsonSerializer.serialize,
api_version=(0, 10, 1)
)
This should work, interestingly enough setting the api_version is accidentally fixing the issue according to this:
When you set api_version the client will not attempt to probe brokers for version information. So it is the probe operation that is failing. One large difference between the version probe connections and the general connections is that the former only attempts to connect on a single interface per connection (per broker), where as the latter -- general operation -- will cycle through all interfaces continually until a connection succeeds. #1411 fixes this by switching the version probe logic to attempt a connection on all found interfaces.
The actual issue is described here
I managed to get this up-and-running using a network named stream_net between all services.
# for local development
version: "3.7"
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
networks:
- stream_net
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
networks:
- stream_net
flaskapp:
build: ./flask-app
container_name: flask_dev
ports:
- "9000:5000"
volumes:
- ./flask-app:/app
networks:
- stream_net
depends_on:
- kafka
networks:
stream_net:
connection from outside the containers on localhost:9092
connection within the network on kafka:29092
of course it is strange to put all containers that are already running within a network within a network. But in this way the containers can be named by their actual name. Maybe someone can explain exactly how this works, or it helps someone else to understand the core of the problem and to solve it properly.