FastAPI app:
import fastapi as _fastapi
from celery import Celery
from celery.result import AsyncResult
app = _fastapi.FastAPI()
celery_app = Celery(
"worker",
broker_url="amqp://guest:guest#rabbit:5672//",
result_backend="rpc://",
)
celery_app.conf.task_routes = {"celery_worker.test_celery": "test-queue"}
celery_app.conf.update(task_track_started=True)
#app.get("/{word}")
async def root(word: str):
task = celery_app.send_task("celery_worker.test_celery", args=[word])
return {"message": "Word received", "id": f"{task}"}
#app.get("/api/result/{task_id}")
async def result(task_id: str):
task = AsyncResult(task_id)
# Task Not Ready
if not task.ready():
return {"status": task.status}
# Task done: return the value
task_result= task.get()
result = task_result.get("result")
return {"task_id": str(task_id),
"status": task_result.get("status"),
"result": result,
}
Dockerfile:
FROM python:3.10-slim
WORKDIR /app
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
COPY ./requirements.txt .
RUN pip install --upgrade pip && pip install -r requirements.txt --no-cache-dir
COPY . .
docker-compose.yml:
version: '3.8'
services:
ylab:
container_name: ylab
build:
context: .
command: "uvicorn main:app --reload --host 0.0.0.0"
ports:
- "8000:8000"
networks:
- api_network
rabbit:
container_name: rabbit
image: rabbitmq:3.10.7-management
ports:
- "15672:15672"
- "5672:5672"
networks:
- api_network
celery_worker:
container_name: celery_worker
build:
context: .
command: celery -A main.celery_app worker --loglevel=INFO
networks:
- api_network
networks:
api_network:
name: api_network
The root() function works well. I can send messages, return a task id, and see all the messages in the RabbitMQ queue, but the result() function for any task id returns task.ready() == False
Can anyone tell me what is the error in this code?
Services info:
RabbitMQ 3.10.7
Celery:
celery#415bde516932 v5.2.3 (dawn-chorus)
Linux-5.10.0-18-amd64-x86_64-with-glibc2.31 2023-02-05 12:02:49
app: worker:0x7f3679306c20
transport: amqp://guest:**#rabbit:5672//
results: rpc://
concurrency: 8 (prefork)
task events: OFF (enable -E to monitor tasks in this worker)
[queues]
.> celery exchange=celery(direct) key=celery
According to the documentation for task_track_started:
If True the task will report its status as ‘started’ when the task is
executed by a worker.
But in your code, you don't seem to have anything consuming the tasks that you're placing on the queue. They will stay in PENDING state forever.
I started by writing your code to use automatic task routing, using <func>.delay to call a task rather than the lower-level send_task method:
import time
import fastapi as _fastapi
from celery import Celery
from celery.result import AsyncResult
app = _fastapi.FastAPI()
celery_app = Celery(
"worker",
broker_url="amqp://guest:guest#rabbit:5672//",
result_backend="rpc://",
)
celery_app.conf.update(task_track_started=True)
#celery_app.task
def test_celery(word):
time.sleep(10)
return word.upper()
#app.get("/{word}")
async def root(word: str):
task = test_celery.delay(word)
return {"message": "Word received", "id": f"{task}"}
#app.get("/api/result/{task_id}")
async def result(task_id: str):
task = AsyncResult(task_id)
# Task Not Ready
if not task.ready():
return {"status": task.status}
# Task done: return the value
task_result= task.get()
return {"task_id": str(task_id),
"result": task_result,
}
When running the above code, a connection to /foo results in:
{"message":"Word received","id":"34bfe48d-6ab3-4dec-ad7d-aa567315a609"}
A subsequent call to /api/result/34bfe48d-6ab3-4dec-ad7d-aa567315a609 yields:
{"status":"STARTED"}
And if we wait for 10 seconds, the same request results in:
{"task_id":"34bfe48d-6ab3-4dec-ad7d-aa567315a609","result":"FOO"}
We've demonstrated that things work correctly when using automatic task routing. So why isn't your original code working? There are three problems:
You don't have anything watch test-queue.
You're delivering tasks into test-queue, but your Celery worker is watching the default celery queue. You need to use the -Q argument to have it watch test-queue instead:
celery_worker:
container_name: celery_worker
build:
context: .
command: celery -A main.celery_app worker --loglevel=INFO -Q test-queue
networks:
- api_network
You don't have any tasks defined.
If you add the -Q test-queue argument from the previous step and restart the environment, attempts to connect to /foo will result in the following traceback in your Celery worker:
celery_worker | [2023-02-05 14:12:40,864: ERROR/MainProcess] Received unregistered task of type 'celery_worker.test_celery'.
celery_worker | The message has been ignored and discarded.
[...]
celery_worker | Traceback (most recent call last):
celery_worker | File "/usr/local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 591, in on_task_received
celery_worker | strategy = strategies[type_]
celery_worker | KeyError: 'celery_worker.test_celery'
We can fix that by registering the appropriate task with Celery:
#celery_app.task(name="celery_worker.test_celery")
def test_celery(word):
time.sleep(10)
return word.upper()
With the previous two changes, your code will successfully submit the task to Celery and Celery will pass it to the test_celery function. However, calls to /api/result/<id> will fail with:
File "/app/./main.py", line 39, in result
result = task_result.get("result")
AttributeError: 'str' object has no attribute 'get'
You need to to modiofy your result function so that it looks more like:
#app.get("/api/result/{task_id}")
async def result(task_id: str):
task = AsyncResult(task_id)
# Task Not Ready
if not task.ready():
return {"status": task.status}
# Task done: return the value
task_result = task.get()
return {
"task_id": str(task_id),
"result": task_result,
}
With these three changes, your original code works as intended. The complete modified code looks like:
import time
import fastapi
from celery import Celery
from celery.result import AsyncResult
app = fastapi.FastAPI()
celery_app = Celery(
"worker",
broker_url="amqp://guest:guest#rabbit:5672//",
result_backend="rpc://",
)
celery_app.conf.task_routes = {"celery_worker.test_celery": "test-queue"}
celery_app.conf.update(task_track_started=True)
#celery_app.task(name="celery_worker.test_celery")
def test_celery(word):
time.sleep(10)
return word.upper()
#app.get("/{word}")
async def root(word: str):
task = celery_app.send_task("celery_worker.test_celery", args=[word])
return {"message": "Word received", "id": f"{task}"}
#app.get("/api/result/{task_id}")
async def result(task_id: str):
task = AsyncResult(task_id)
# Task Not Ready
if not task.ready():
return {"status": task.status}
# Task done: return the value
task_result = task.get()
return {
"task_id": str(task_id),
"result": task_result,
}
Related
I'm new to learning celery and was following tutorials and setup my celery setup with docker
I'm having issue with sending and executing celery task.
So have 4 docker container one for rabbitmq server, celery producer server and 2 worker.
Celery tasks file:
"""
CELERY MAIN FILE
"""
from celery import Celery
from time import sleep
celery_obj = Celery()
celery_obj.config_from_object('celery_config') #config file we created in same folder
#celery_obj.task
def add(num1,num2):
print("executing add function")
sleep(5)
return num1 + num2
My celery config file for Producer:
"""
CELERY CONFIGURATION FILE
"""
from kombu import Exchange, Queue
broker_url = "pyamqp://rabbitmq_user:123#172.17.0.2/res_opt_rabbitmq_vhost"
result_backend = 'rpc://'
#celery_result_backend = ""
celery_imports = ('res_opt_code.tasks')
task_queues = (
Queue('worker_A_kombu_queue',Exchange('celery',type='direct'),routing_key='worker_A_rabbitmq_queue'),
Queue('worker_B_kombu_queue',Exchange('celery',type='direct'),routing_key='worker_B_rabbitmq_queue')
)
Config file for worker_A:
"""
CELERY CONFIGURATION FILE
"""
from kombu import Exchange, Queue
broker_url = "pyamqp://rabbitmq_user:123#172.17.0.2/res_opt_rabbitmq_vhost"
result_backend = 'rpc://'
#celery_result_backend = ""
celery_imports = ('worker_code.tasks')
task_queues = (
Queue('worker_A_kombu_queue',Exchange('celery',type='direct'),routing_key='worker_A_rabbitmq_queue'),
Queue('worker_B_kombu_queue',Exchange('celery',type='direct'),routing_key='worker_B_rabbitmq_queue')
)
Command for starting celery on producer:
celery -A tasks worker --loglevel=DEBUG -f log_file.txt
command for starting celery on worker:
celery -A tasks worker -n celery_worker_A -Q worker_A_kombu_queue --loglevel=DEBUG
Function call from producer:
from tasks import add
add.apply_async([4,4],routing_key='worker_A_rabbitmq_queue')
#also tried local executing the function but not logs of functions it's in pending
add.delay(4,4)
could you guyz please help me what I'm doing wrong here
In Logs I'm able to see worker_A connected but no logs for function
Tried further troubleshooting and changed the argument in apply_async from routing key to queue and it working with the queue argument
was following this tutorial:
https://www.youtube.com/watch?v=TM1a3m65zaA
old:
add.apply_async([4,4],routing_key='worker_A_rabbitmq_queue')
new:
add.apply_async([4,4],queue='worker_A_rabbitmq_queue')
I'm following one of the various tutorials out on the internet and set up a Flask/RabbitMQ/Celery app using Docker/Docker Compose. The containers all appear to run successfully but when I hit the endpoint, the app stalls. The task appears to be stuck in PENDING and never actually completes. There are no errors in the Docker output, so I'm really confused why this isn't working. The only output I see when I hit my endpoint is this:
rabbit_1 | 2021-05-13 01:38:07.942 [info] <0.760.0> accepting AMQP connection <0.760.0> (172.19.0.4:45414 -> 172.19.0.2:5672)
rabbit_1 | 2021-05-13 01:38:07.943 [info] <0.760.0> connection <0.760.0> (172.19.0.4:45414 -> 172.19.0.2:5672): user 'rabbitmq' authenticated and granted access to vhost '/'
rabbit_1 | 2021-05-13 01:38:07.952 [info] <0.776.0> accepting AMQP connection <0.776.0> (172.19.0.4:45416 -> 172.19.0.2:5672)
rabbit_1 | 2021-05-13 01:38:07.953 [info] <0.776.0> connection <0.776.0> (172.19.0.4:45416 -> 172.19.0.2:5672): user 'rabbitmq' authenticated and granted access to vhost '/'
I'm really not sure what I am doing wrong as the documentation hasn't been much help.
Dockerfile
FROM python:3
COPY ./requirements.txt /app/requirements.txt
WORKDIR /app
RUN pip install -r requirements.txt
COPY . /app
ENTRYPOINT [ "python" ]
CMD ["app.py","--host=0.0.0.0"]
Flask app.py
from workerA import add_nums
from flask import (
Flask,
request,
jsonify,
)
app = Flask(__name__)
#app.route("/add")
def add():
first_num = request.args.get('f')
second_num = request.args.get('s')
result = add_nums.delay(first_num, second_num)
return jsonify({'result': result.get()}), 200
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0')
Celery workerA.py
from celery import Celery
# Celery configuration
CELERY_BROKER_URL = 'amqp://rabbitmq:rabbitmq#rabbit:5672/'
CELERY_RESULT_BACKEND = 'rpc://'
# Initialize Celery
celery = Celery('workerA', broker=CELERY_BROKER_URL, backend=CELERY_RESULT_BACKEND)
#celery.task()
def add_nums(a, b):
return a + b
docker-compose.yml
version: "3"
services:
web:
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- "5000:5000"
depends_on:
- rabbit
volumes:
- .:/app
rabbit:
hostname: rabbit
image: rabbitmq:management
environment:
- RABBITMQ_DEFAULT_USER=rabbitmq
- RABBITMQ_DEFAULT_PASS=rabbitmq
ports:
- "5673:5672"
- "15672:15672"
worker_1:
build:
context: .
hostname: worker_1
entrypoint: celery
command: -A workerA worker --loglevel=info -Q workerA
volumes:
- .:/app
links:
- rabbit
depends_on:
- rabbit
Alright, after much research I determined that the issue was the queue name for the task. Celery was using the default name for the queue and it was causing some problems. I adjusted my decorated like so:
#celery.task(queue='workerA')
def add_nums(a, b):
return a + b
And now it works!
I'm trying a FastAPI based API with celery, redis, and rabitMQ as the background tasks.
when doing docker-compose up, the redis, rabbit, and flower parts work, I'm able to access the flower dashboard.
but it then gets stuck in the celery part.
the error:
rabbitmq_1 | 2020-09-08 06:32:38.552 [info] <0.716.0> connection <0.716.0> (172.22.0.6:49290 -> 172.22.0.2:5672): user 'user' authenticated and granted access to vhost '/'
celery-flower_1 | [W 200908 06:32:41 control:44] 'stats' inspect method failed
celery-flower_1 | [W 200908 06:32:41 control:44] 'active_queues' inspect method failed
celery-flower_1 | [W 200908 06:32:41 control:44] 'registered' inspect method failed
celery-flower_1 | [W 200908 06:32:41 control:44] 'scheduled' inspect method failed
celery-flower_1 | [W 200908 06:32:41 control:44] 'active' inspect method failed
celery-flower_1 | [W 200908 06:32:41 control:44] 'reserved' inspect method failed
celery-flower_1 | [W 200908 06:32:41 control:44] 'revoked' inspect method failed
celery-flower_1 | [W 200908 06:32:41 control:44] 'conf' inspect method failed
My docker-compose file:
version: "3.7"
services:
rabbitmq:
image: "bitnami/rabbitmq:3.7"
ports:
- "4000:4000"
- "5672:5672"
volumes:
- "rabbitmq_data:/bitnami"
redis:
image: "bitnami/redis:5.0.4"
environment:
- REDIS_PASSWORD=password123
ports:
- "5000:5000"
volumes:
- "redis_data:/bitnami/redis/data"
celery-flower:
image: gregsi/latest-celery-flower-docker:latest
environment:
- AMQP_USERNAME=user
- AMQP_PASSWORD=bitnami
- AMQP_ADMIN_USERNAME=user
- AMQP_ADMIN_PASSWORD=bitnami
- AMQP_HOST=rabbitmq
- AMQP_PORT=5672
- AMQP_ADMIN_HOST=rabbitmq
- AMQP_ADMIN_PORT=15672
- FLOWER_BASIC_AUTH=user:test
ports:
- "5555:5555"
depends_on:
- rabbitmq
- redis
fastapi:
build: .
ports:
- "8000:8000"
depends_on:
- rabbitmq
- redis
volumes:
- "./:/app"
command: "poetry run uvicorn app/app/main:app --bind 0.0.0.0:8000"
worker:
build: .
depends_on:
- rabbitmq
- redis
volumes:
- "./:/app"
command: "poetry run celery worker -A app.app.worker.celery_worker -l info -Q test-queue -c 1"
volumes:
rabbitmq_data:
driver: local
redis_data:
driver: local
My celery app:
celery_app = Celery(
"worker",
backend="redis://:password123#redis:6379/0",
broker="amqp://user:bitnami#rabbitmq:5672//"
)
celery_app.conf.task_routes = {
"app.app.worker.celery_worker.compute_stock_indicators": "stocks-queue"
}
celery_app.conf.update(task_track_started=True)
celery worker:
#celery_app.task(acks_late=True)
def compute_stock_indicators(stocks: list, background_task):
stocks_with_indicators = {}
for stock in stocks:
current_task.update_state(state=Actions.STARTED,
meta={f"starting to fetch {stock}'s indicators"})
stock_indicators = fetch_stock_indicators(stock) # Fetch the stock most recent indicators
current_task.update_state(state=Actions.FINISHED,
meta={f"{stock}'s indicators fetched"})
stocks_with_indicators.update({stock: stock_indicators})
current_task.update_state(state=Actions.PROGRESS,
meta={f"predicting {stocks}s..."})
The Fast API function:
log = logging.getLogger(__name__)
rabbit = RabbitMQHandler(host='localhost', port=5672, level="DEBUG")
log.addHandler(rabbit)
def celery_on_message(body):
"""
Logs the initiation of the function
"""
log.warning(body)
def background_on_message(task):
"""
logs the function when it is added to queue
"""
log.warning(task.get(on_message=celery_on_message, propagate=False))
app = FastAPI(debug=True)
#app.post("/")
async def initiator(stocks: FrozenSet, background_task: BackgroundTasks, ):
"""
:param stocks: stocks to be analyzed
:type stocks: set
:param background_task: initiate the tasks queue
:type background_task: starlette.background.BackgroundTasks
"""
log.warning(msg=f'beginning analysis on: {stocks}')
task_name = "app.app.worker.celery_worker.compute_stock_indicators"
task = celery_app.send_task(task_name, args=[stocks, background_task])
background_task.add_task(background_on_message, task)
return {"message": "Stocks indicators successfully calculated,stocks sent to prediction"}
On the docker-compose, on the worker section, the command reads:
command: "poetry run celery worker -A app.app.worker.celery_worker -l info -Q test-queue -c 1"
So essentially you are asking the worker to "watch" a queue named test-queue.
But on the celery_app, on the following section:
celery_app.conf.task_routes = {
"app.app.worker.celery_worker.compute_stock_indicators": "stocks-queue"
}
you are defining a queue named stocks-queue.
Either change the docker-compose's or the celery_app's queue name to match the other.
if you use Docker Toolbox on windows , so you should add port 5555 to VM virtualBOX network:
frist run following command on cmd:
docker-machine stop default
then open VM virtualBOX , go to Settings >Networks > advanced>port forwarding >add a row with port 5555 and leave name field
click OK and on cmd, run following command:
docker-machine start default
I am attempting to get a result backend working on my local machine for a project I'm working on but I am running into an issue.
Currently I am trying to create a queue system in order for my lab to create cases. This is to prevent duplicate sequence numbers from being used. I am already using Celery for our printing so I figured I would create a new Celery queue and use that to handle the case. The front-end also needs to get the results of the case creations to display the case number that was created.
http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#rabbitmq
I was following the above tutorial on getting my Celery configured. Below is the source:
celeryconfig.py:
from kombu import Queue
CELERY_DEFAULT_QUEUE = 'celery'
CELERY_DEFAULT_EXCHANGE = 'celery'
CELERY_DEFAULT_EXCHANGE_TYPE = 'direct'
CELERY_RESULT_BACKEND = 'rpc://'
CELERY_RESULT_PERSISTENT = False
CELERY_QUEUES = (
Queue('celery', routing_key="celery"),
Queue('case_creation', routing_key='create.#')
)
CELERY_ROUTES = {
'case.tasks.create_case': {
'queue': 'case_creation',
'routing_key': 'create.1'
},
'print.tasks.connect_and_serve': {
'queue': 'celery',
'routing_key': 'celery'
}
}
celery.py:
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings.local')
app = Celery('proj', broker='amqp://guest#localhost//')
app.config_from_object('proj.celeryconfig')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
tasks.py:
import celery
from django.db import IntegrityError
from case.case_create import CaseCreate
#celery.task(bind=True)
def create_case(self, data, user, ip):
try:
acc = CaseCreate(data, user, ip)
return acc.begin()
except IntegrityError as e:
self.retry(exc=e, countdown=2)
Here is my view that calls the above task:
#require_authentication()
#requires_api_signature()
#csrf_exempt
#require_http_methods(['POST'])
def api_create_case(request):
result = create_case.delay(json.loads(request.body.decode('utf-8')), request.user, get_ip_address(request))
print(str(result)) # Prints the Task ID
print(str(result.get(timeout=1))) # Throws error
return HttpResponse(json.dumps({'result': str(result)}), status=200)
I start my celery queue with the following command:
celery -A proj worker -Q case_creation -n case_worker -c 1
When I run the celery worker I do see results show up under config:
-------------- celery#case_worker v3.1.16 (Cipater)
---- **** -----
--- * *** * -- Windows-8-6.2.9200
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: proj:0x32a2990
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: rpc://
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> case_creation exchange=celery(direct) key=create.#
When I run the program and submit a new case this is the error message that I get:
No result backend configured. Please see the documentation for more information.
I have attempted every single thing I can find online. Is there anyone out there that can point me in the right direction? I'm so very close and so very tired of looking at this code.
If you want to keep your result, try this Keeping Results
app = Celery('proj', backend='amqp', broker='amqp://guest#localhost//')
EDIT
Make sure the client is configured with the right backend.
If for some reason the client is configured to use a different backend than the worker, you will not be able to receive the result, so make sure the backend is correct by inspecting it:
Try this to see the output:
>>> result = task.delay(…)
>>> print(result.backend)
other solutions will be instead of
app = Celery('proj',
backend='amqp',
broker='amqp://',
include=['proj.tasks'])
Try:
app = Celery('proj',
broker='amqp://',
include=['proj.tasks'])
app.conf.update(
CELERY_RESULT_BACKEND='amqp'
)
I am starting celery via supervisord, see the entry below.
[program:celery]
user = foobar
autostart = true
autorestart = true
directory = /opt/src/slicephone/cloud
command = /opt/virtenvs/django_slice/bin/celery beat --app=cloud -l DEBUG -s /home/foobar/run/celerybeat-schedule --pidfile=/home/foobar/run/celerybeat.pid
priority = 100
stdout_logfile_backups = 0
stderr_logfile_backups = 0
stdout_logfile_maxbytes = 10MB
stderr_logfile_maxbytes = 10MB
stdout_logfile = /opt/logs/celery.stdout.log
stderr_logfile = /opt/logs/celery.stderr.log
pip freeze | grep celery
celery==3.1.0
But any usage of:
#celery.task
def test_rabbit_running():
import logging
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
logger.setLevel(logging.DEBUG)
logger.info("foobar")
doesn't show up in the logs. Instead I get entries like the following.
celery.stdout.log
celery beat v3.1.0 (Cipater) is starting.
__ - ... __ - _
Configuration ->
. broker -> redis://localhost:6379//
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> /home/foobar/run/celerybeat-schedule
. logfile -> [stderr]#%DEBUG
. maxinterval -> now (0s)
celery.stderr.log
[2013-11-12 05:42:39,539: DEBUG/MainProcess] beat: Waking up in 2.00 seconds.
INFO Scheduler: Sending due task test_rabbit_running (retail.tasks.test_rabbit_running)
[2013-11-12 05:42:41,547: INFO/MainProcess] Scheduler: Sending due task test_rabbit_running (retail.tasks.test_rabbit_running)
DEBUG retail.tasks.test_rabbit_running sent. id->34268340-6ffd-44d0-8e61-475a83ab3481
[2013-11-12 05:42:41,550: DEBUG/MainProcess] retail.tasks.test_rabbit_running sent. id->34268340-6ffd-44d0-8e61-475a83ab3481
DEBUG beat: Waking up in 6.00 seconds.
What do I have to do to make my logging calls appear in the log files?
It doesn't log anything because it doesn't execute any tasks (and it's ok).
See also Celerybeat not executing periodic tasks
I'd try to put the call to log inside a task as the name of the util function implies get_task_logger, or just start with a simple print, or have your own log set up as suggested in Django Celery Logging Best Practice (best way to go IMO)