We Keep Coding

Celery configuration in settings.py file

Can anyone explain about these lines in celery RabbitMQ in Django. Which time it will be use ?
I ran 2 tasks(addition operation and endpoint in django) in celery RabbitMq without these lines successfully. So Please explain when it will be used in settings.py and celery rabbitmq
CELERY_BROKER_URL = 'amqp://localhost'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
__init__.py :
from .celery import app as celery_app
__all__ = ('celery_app',)
Thanks in advance

The reason why your tasks are still running even without those explicit settings is because Celery has default values for them as written in its documentation.
https://docs.celeryproject.org/en/stable/userguide/configuration.html
To visualize this, here is a run where we wouldn't set the broker_url.
$ cat > tasks.py
from celery import Celery
app = Celery('my_app')
$ celery --app=tasks worker --loglevel=INFO
...
- ** ---------- [config]
- ** ---------- .> app: my_app:0x7f5a09295160
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 5 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
...
As you can see, even if we didn't set the broker explicitly, it defaulted to transport: amqp://guest:**#localhost:5672// which is the default for RabbitMQ as stated in the docs:
broker_url
Default: "amqp://"
The transport part is the broker implementation to use, and the
default is amqp, (uses librabbitmq if installed or falls back to
pyamqp).
Here is a run where we would explicitly set the broker_url. To see the difference, let's say our RabbitMQ broker listens at port 666 of localhost 127.0.0.1 with a different password.
$ cat > tasks.py
from celery import Celery
app = Celery('my_app')
app.conf.broker_url = "amqp://guest:a-more-secure-password#127.0.0.1:666"
$ celery --app=tasks worker --loglevel=INFO
...
- ** ---------- [config]
- ** ---------- .> app: my_app:0x7fb02579f160
- ** ---------- .> transport: amqp://guest:**#127.0.0.1:666//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 5 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
...
Now, the broker was set to our configured value transport: amqp://guest:**#127.0.0.1:666//
You need to change those settings if the value would be different from the default ones. For further details about each configurable setting, please refer to the docs.
One particular use case of overriding the default value is as seen above in the example for broker_url, where we need to explicitly set it to use the RabbitMQ running in amqp://guest:a-more-secure-password#127.0.0.1:666 instead of the supposed-to-be default value of amqp://guest:guest#127.0.0.1:5672 which would have resulted to error consumer: Cannot connect to amqp://guest:**#127.0.0.1:5672//: [Errno 104] Connection reset by peer. Trying again in 2.00 seconds... (1/100) if we didn't set it.
Other references:
Default RabbitMQ user guest:guest
Default RabbitMQ port 5672

Celery not connecting to Redis

I'm working in a Django project and i'm trying to integrate celery into my project but my app doesn't seem to be able to connect to my redis-server that I'm running locally.
This is what it shows me when I run celery -A project worker.
---- **** -----
--- * *** * -- Linux-4.15.0-58-generic-x86_64-with-Ubuntu-18.04-bionic 2020-03-18 19:06:29
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: proyecto_is2:0x7f7c98351518
- ** ---------- .> transport: redis://h:**#ec2-34-202-114-79.compute-1.amazonaws.com:17729//
- ** ---------- .> results: redis://127.0.0.1:6379/0
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[2020-03-18 19:31:24,595: ERROR/MainProcess] consumer: Cannot connect to
redis://h:**#ec2-34-202-114-79.compute-1.amazonaws.com:17729//: Error 110 connecting to
ec2-34-202-114-79.compute-1.amazonaws.com:17729. Connection timed out..
I noticed that the url in transport is not my localhost and I don't know where that got set up.
I'm using celery 4.3.0, redis 3.2.1.
In my celery.py I have
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proyecto_is2.settings.dev_settings')
app = Celery('proyecto_is2')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
In my settings i have
... other configs
CELERY_BROKER_URL = 'redis://127.0.0.1:6379'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'

This is definitely a config issue. Something either in your settings.py, celery.py, or environment is overriding your broker to point to redis://34.202.114.79.

Celery Worker don't execute cassandra queries

I'm using
celery == 4.1.0 (latentcall)
[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
Python 2.7.14
I'm trying to execute Cassandra Query in Celery worker function. But Celery worker received task but not execute Query.
tasks.py
from cassandra.cluster import Cluster
from celery import Celery
app = Celery('<workername>', backend="rpc://", broker='redis://localhost:6379/0')
dbSession = Cluster().connect()
#app.tasks()
def get_data():
query = "SELECT * FROM customers"
CustomerObj = dbSession.execute(dbSession.prepare(query))
return CustomerObj
get_data.delay()
I start worker using :
$ celery worker -A <worker_name> -l INFO -c 1
-------------- celery#ubuntu v4.1.0 (latentcall)
---- **** -----
--- * *** * -- Linux-4.13.0-21-generic-x86_64-with-Ubuntu-17.10-artful 2018-04-20 14:31:41
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: Woker:0x7fa4a0e6f310
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: rpc://
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. Worker.get_data
[2018-04-20 14:31:41,271: INFO/MainProcess] Connected to redis://localhost:6379/0
[2018-04-20 14:31:41,285: INFO/MainProcess] mingle: searching for neighbors
[2018-04-20 14:31:42,315: INFO/MainProcess] mingle: all alone
.............
[2018-04-20 14:31:42,332: INFO/MainProcess] celery#ubuntu ready.
[2018-04-20 14:31:43,823: INFO/MainProcess] Received task: <worker_name>.get_data[8de91fdf-1388-4d5c-bb22-8cb00c1c065e]
Worker process is just stopped there.It will not execute that SELECT query and give any data.
Anyone suggest me How can I run this code to execute Cassandra Queries.

I think that you can't define dbSession globally.
Celery task can run in different workers, so the connection can't be global.
I can suggest two options:
Create the session within the task. It should work. The pros is that you'll create new session per each task. Maybe lazy (#LazyProperty) should help here.
You can create the connection in the worker level: try to create your session when worker start, maybe with worker_init signal (ref). The problem here is that you can have concurrency level > 1 (depends how you start the worker) - and than you need pool of sessions to serve more than one celery task at a time (handle more than one Cassandra session at a time).
By the way you should use the global keyword in python. If you are running one instance it may fix too.
Here is a related question that might help you: Celery Worker Database Connection Pooling
Good luck!

Since celery doesn't use the application's connection instance. Initiate a new connection at celery initiation. the below snippet is as per Cassandra documentation for celery
from celery import Celery
from celery.signals import worker_process_init, beat_init
from cassandra.cqlengine import connection
from cassandra.cqlengine.connection import (
cluster as cql_cluster, session as cql_session)
def cassandra_init(**kwargs):
""" Initialize a clean Cassandra connection. """
if cql_cluster is not None:
cql_cluster.shutdown()
if cql_session is not None:
cql_session.shutdown()
connection.setup()
# Initialize worker context for both standard and periodic tasks.
worker_process_init.connect(cassandra_init)
beat_init.connect(cassandra_init)
app = Celery()
This worked for me

Setting up a result backend (rpc) with Celery in Django

I am attempting to get a result backend working on my local machine for a project I'm working on but I am running into an issue.
Currently I am trying to create a queue system in order for my lab to create cases. This is to prevent duplicate sequence numbers from being used. I am already using Celery for our printing so I figured I would create a new Celery queue and use that to handle the case. The front-end also needs to get the results of the case creations to display the case number that was created.
http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#rabbitmq
I was following the above tutorial on getting my Celery configured. Below is the source:
celeryconfig.py:
from kombu import Queue
CELERY_DEFAULT_QUEUE = 'celery'
CELERY_DEFAULT_EXCHANGE = 'celery'
CELERY_DEFAULT_EXCHANGE_TYPE = 'direct'
CELERY_RESULT_BACKEND = 'rpc://'
CELERY_RESULT_PERSISTENT = False
CELERY_QUEUES = (
Queue('celery', routing_key="celery"),
Queue('case_creation', routing_key='create.#')
)
CELERY_ROUTES = {
'case.tasks.create_case': {
'queue': 'case_creation',
'routing_key': 'create.1'
},
'print.tasks.connect_and_serve': {
'queue': 'celery',
'routing_key': 'celery'
}
}
celery.py:
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings.local')
app = Celery('proj', broker='amqp://guest#localhost//')
app.config_from_object('proj.celeryconfig')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
tasks.py:
import celery
from django.db import IntegrityError
from case.case_create import CaseCreate
#celery.task(bind=True)
def create_case(self, data, user, ip):
try:
acc = CaseCreate(data, user, ip)
return acc.begin()
except IntegrityError as e:
self.retry(exc=e, countdown=2)
Here is my view that calls the above task:
#require_authentication()
#requires_api_signature()
#csrf_exempt
#require_http_methods(['POST'])
def api_create_case(request):
result = create_case.delay(json.loads(request.body.decode('utf-8')), request.user, get_ip_address(request))
print(str(result)) # Prints the Task ID
print(str(result.get(timeout=1))) # Throws error
return HttpResponse(json.dumps({'result': str(result)}), status=200)
I start my celery queue with the following command:
celery -A proj worker -Q case_creation -n case_worker -c 1
When I run the celery worker I do see results show up under config:
-------------- celery#case_worker v3.1.16 (Cipater)
---- **** -----
--- * *** * -- Windows-8-6.2.9200
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: proj:0x32a2990
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: rpc://
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> case_creation exchange=celery(direct) key=create.#
When I run the program and submit a new case this is the error message that I get:
No result backend configured. Please see the documentation for more information.
I have attempted every single thing I can find online. Is there anyone out there that can point me in the right direction? I'm so very close and so very tired of looking at this code.

If you want to keep your result, try this Keeping Results
app = Celery('proj', backend='amqp', broker='amqp://guest#localhost//')
EDIT
Make sure the client is configured with the right backend.
If for some reason the client is configured to use a different backend than the worker, you will not be able to receive the result, so make sure the backend is correct by inspecting it:
Try this to see the output:
>>> result = task.delay(…)
>>> print(result.backend)
other solutions will be instead of
app = Celery('proj',
backend='amqp',
broker='amqp://',
include=['proj.tasks'])
Try:
app = Celery('proj',
broker='amqp://',
include=['proj.tasks'])
app.conf.update(
CELERY_RESULT_BACKEND='amqp'
)

Celery AsyncResult is always PENDING

I'm working on a demo and the code is simple:
# The Config
class Config:
BROKER_URL = 'redis://127.0.0.1:6379/0'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
CELERY_ACCEPT_CONTENT = ['application/json']
# The Task
#celery_app.task()
def add(x, y):
return x + y
To start the worker:
$ celery -A appl.task.celery_app worker --loglevel=info -broker=redis://localhost:6379/0
-------------- celery#ALBERTATMP v3.1.13 (Cipater)
---- **** -----
--- * *** * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.6
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: celery_test:0x293ffd0
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: disabled
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
To schedule task:
>>> from appl.task import add
>>> r = add.delay(1, 2)
>>> r.id
'c41d4e22-ccea-408f-b48f-52e3ddd6bd66'
>>> r.task_id
'c41d4e22-ccea-408f-b48f-52e3ddd6bd66'
>>> r.status
'PENDING'
>>> r.backend
<celery.backends.redis.RedisBackend object at 0x1f35b10>
Then the worker will execute the task:
[2014-07-29 17:54:37,356: INFO/MainProcess] Received task: appl.task.add[beeef023-c582-42e1-baf7-9e19d9de32a0]
[2014-07-29 17:54:37,358: INFO/MainProcess] Task appl.task.add[beeef023-c582-42e1-baf7-9e19d9de32a0] succeeded in 0.00108124599865s: 3
But the result remains PENDING:
>>> res = add.AsyncResult(r.id)
>>> res.status
'PENDING'
I've tried the official FAQ. But it did not help.
>>> celery_app.conf['CELERY_IGNORE_RESULT']
False
What did I do wrong? Thanks!

Its been a while, but am leaving this more for others who come along with a similar issue:
In your screenshot, you see that the results are disabled
When you instantiate your celery instance, make sure that you have the right config inputs
from celery import Celery,Task
# here im using an AMQP broker with a memcached backend to store the results
celery = Celery('task1',broker='amqp://guest:guest#127.0.0.1:5672//',backend='cache+memcached://127.0.0.1:11211/')
For some reason, i always have trouble getting the celery instance parametered through the config file and hence explicitly passed in the broker and backend during instantiation as shown above
Now you'll see the results rightly configured to be memcached (in my instance - should be redis in yours). Also make sure that your task is picked up in the list of tasks (task1.add)
If you still cant get it to work, while starting celery try using the debug option as below
celery worker -A task1.celery -l debug
see if something is going wrong in the information it spews out
In my case, it fixed your error and result was set to success and i was able to recover 3 on r.get()

Try to change your broker to something else (like rabbitmq) and check the status again.
Make sure your redis server is up and accessible for celery.
redis-cli
keys *
and you should see some keys related to celery, if not it means there is a issue in your broker

This works for me:
from celery.result import AsyncResult
celery_task_result = AsyncResult(task_id)
task_state = celery_task_result.state
and task_state get all kinds of status: 'FAILURE', 'SUCCESS', 'PENDING', etc.