We Keep Coding

Celery configuration in settings.py file

Can anyone explain about these lines in celery RabbitMQ in Django. Which time it will be use ?
I ran 2 tasks(addition operation and endpoint in django) in celery RabbitMq without these lines successfully. So Please explain when it will be used in settings.py and celery rabbitmq
CELERY_BROKER_URL = 'amqp://localhost'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
__init__.py :
from .celery import app as celery_app
__all__ = ('celery_app',)
Thanks in advance

The reason why your tasks are still running even without those explicit settings is because Celery has default values for them as written in its documentation.
https://docs.celeryproject.org/en/stable/userguide/configuration.html
To visualize this, here is a run where we wouldn't set the broker_url.
$ cat > tasks.py
from celery import Celery
app = Celery('my_app')
$ celery --app=tasks worker --loglevel=INFO
...
- ** ---------- [config]
- ** ---------- .> app: my_app:0x7f5a09295160
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 5 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
...
As you can see, even if we didn't set the broker explicitly, it defaulted to transport: amqp://guest:**#localhost:5672// which is the default for RabbitMQ as stated in the docs:
broker_url
Default: "amqp://"
The transport part is the broker implementation to use, and the
default is amqp, (uses librabbitmq if installed or falls back to
pyamqp).
Here is a run where we would explicitly set the broker_url. To see the difference, let's say our RabbitMQ broker listens at port 666 of localhost 127.0.0.1 with a different password.
$ cat > tasks.py
from celery import Celery
app = Celery('my_app')
app.conf.broker_url = "amqp://guest:a-more-secure-password#127.0.0.1:666"
$ celery --app=tasks worker --loglevel=INFO
...
- ** ---------- [config]
- ** ---------- .> app: my_app:0x7fb02579f160
- ** ---------- .> transport: amqp://guest:**#127.0.0.1:666//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 5 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
...
Now, the broker was set to our configured value transport: amqp://guest:**#127.0.0.1:666//
You need to change those settings if the value would be different from the default ones. For further details about each configurable setting, please refer to the docs.
One particular use case of overriding the default value is as seen above in the example for broker_url, where we need to explicitly set it to use the RabbitMQ running in amqp://guest:a-more-secure-password#127.0.0.1:666 instead of the supposed-to-be default value of amqp://guest:guest#127.0.0.1:5672 which would have resulted to error consumer: Cannot connect to amqp://guest:**#127.0.0.1:5672//: [Errno 104] Connection reset by peer. Trying again in 2.00 seconds... (1/100) if we didn't set it.
Other references:
Default RabbitMQ user guest:guest
Default RabbitMQ port 5672

Setting up a result backend (rpc) with Celery in Django

I am attempting to get a result backend working on my local machine for a project I'm working on but I am running into an issue.
Currently I am trying to create a queue system in order for my lab to create cases. This is to prevent duplicate sequence numbers from being used. I am already using Celery for our printing so I figured I would create a new Celery queue and use that to handle the case. The front-end also needs to get the results of the case creations to display the case number that was created.
http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#rabbitmq
I was following the above tutorial on getting my Celery configured. Below is the source:
celeryconfig.py:
from kombu import Queue
CELERY_DEFAULT_QUEUE = 'celery'
CELERY_DEFAULT_EXCHANGE = 'celery'
CELERY_DEFAULT_EXCHANGE_TYPE = 'direct'
CELERY_RESULT_BACKEND = 'rpc://'
CELERY_RESULT_PERSISTENT = False
CELERY_QUEUES = (
Queue('celery', routing_key="celery"),
Queue('case_creation', routing_key='create.#')
)
CELERY_ROUTES = {
'case.tasks.create_case': {
'queue': 'case_creation',
'routing_key': 'create.1'
},
'print.tasks.connect_and_serve': {
'queue': 'celery',
'routing_key': 'celery'
}
}
celery.py:
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings.local')
app = Celery('proj', broker='amqp://guest#localhost//')
app.config_from_object('proj.celeryconfig')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
tasks.py:
import celery
from django.db import IntegrityError
from case.case_create import CaseCreate
#celery.task(bind=True)
def create_case(self, data, user, ip):
try:
acc = CaseCreate(data, user, ip)
return acc.begin()
except IntegrityError as e:
self.retry(exc=e, countdown=2)
Here is my view that calls the above task:
#require_authentication()
#requires_api_signature()
#csrf_exempt
#require_http_methods(['POST'])
def api_create_case(request):
result = create_case.delay(json.loads(request.body.decode('utf-8')), request.user, get_ip_address(request))
print(str(result)) # Prints the Task ID
print(str(result.get(timeout=1))) # Throws error
return HttpResponse(json.dumps({'result': str(result)}), status=200)
I start my celery queue with the following command:
celery -A proj worker -Q case_creation -n case_worker -c 1
When I run the celery worker I do see results show up under config:
-------------- celery#case_worker v3.1.16 (Cipater)
---- **** -----
--- * *** * -- Windows-8-6.2.9200
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: proj:0x32a2990
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: rpc://
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> case_creation exchange=celery(direct) key=create.#
When I run the program and submit a new case this is the error message that I get:
No result backend configured. Please see the documentation for more information.
I have attempted every single thing I can find online. Is there anyone out there that can point me in the right direction? I'm so very close and so very tired of looking at this code.

If you want to keep your result, try this Keeping Results
app = Celery('proj', backend='amqp', broker='amqp://guest#localhost//')
EDIT
Make sure the client is configured with the right backend.
If for some reason the client is configured to use a different backend than the worker, you will not be able to receive the result, so make sure the backend is correct by inspecting it:
Try this to see the output:
>>> result = task.delay(…)
>>> print(result.backend)
other solutions will be instead of
app = Celery('proj',
backend='amqp',
broker='amqp://',
include=['proj.tasks'])
Try:
app = Celery('proj',
broker='amqp://',
include=['proj.tasks'])
app.conf.update(
CELERY_RESULT_BACKEND='amqp'
)

Celery periodic task doesn't start

I am using celery in my django project to run periodic tasks. Following the standard tutorial from celery's website, here is my project structure
project
|_ settings.py
|_ __init__.py
|_ celery_app.py (instead of celery.py)
|_ app
|_ tasks.py
Relevant parts in settings.py looks like this -
CELERY_RESULT_BACKEND = "amqp"
CELERY_IMPORTS = ["app.tasks"]
CELERY_ALWAYS_EAGER = True
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_DB_REUSE_MAX = 1
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'America/New York'
CELERYBEAT_SCHEDULE = {
'test_task': {
'task': 'tasks.test_task',
'schedule': timedelta(seconds=5),
'args': (),
},
}
celery_app.py looks like this -
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app = Celery('project')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
tasks.py looks like this -
from __future__ import absolute_import
from celery.utils.log import get_task_logger
from celery import task
from celery_app import app
logger = get_task_logger(__name__)
#app.task
def test_task():
for i in range(0, 4):
logger.debug("celery test task")
When I run the celery worker, I can see my task being discovered -
$ python manage.py celery -A project worker --loglevel=DEBUG --app=celery_app:app
-------------- celery#-MBP.home v3.1.19 (Cipater)
---- **** -----
--- * *** * -- Darwin-15.4.0-x86_64-i386-64bit
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: project:0x10d5b9a10
- ** ---------- .> transport: amqp://sgcelery1:**#localhost:5672/sgceleryhost
- ** ---------- .> results: djcelery.backends.database:DatabaseBackend
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. app.tasks.test_task
[2016-04-24 23:54:55,452: INFO/MainProcess] Connected to amqp://sgcelery1:**#127.0.0.1:5672/sgceleryhost
[2016-04-24 23:54:55,471: INFO/MainProcess] mingle: searching for neighbors
[2016-04-24 23:54:56,481: INFO/MainProcess] mingle: all alone
[2016-04-24 23:54:56,497: WARNING/MainProcess] celery#-MBP.home ready.
And when I run beat, it shows the task being picked up from settings.py, but it never actually runs.
$ python manage.py celery -A project beat --app=celery_app:app --loglevel=DEBUG
[2016-04-24 23:55:04,059: DEBUG/MainProcess] Current schedule:
<ModelEntry: test_task tasks.test_task(*[], **{}) {4}>
What am I missing here ?

Try setting CELERY_ALWAYS_EAGER = False.
Setting CELERY_ALWAYS_EAGER = True makes the tasks run synchronously without celery being used. By switching it to False will make celery beat pick it up and run it periodically. The False switch is used in Development mode when you don't want celery to process tasks.
Checkout the docs here

try to use command:
celery -A project worker -B -E -Q beat --concurrency=1 -l INFO
this link about celery keys and options

Celery AsyncResult is always PENDING

I'm working on a demo and the code is simple:
# The Config
class Config:
BROKER_URL = 'redis://127.0.0.1:6379/0'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
CELERY_ACCEPT_CONTENT = ['application/json']
# The Task
#celery_app.task()
def add(x, y):
return x + y
To start the worker:
$ celery -A appl.task.celery_app worker --loglevel=info -broker=redis://localhost:6379/0
-------------- celery#ALBERTATMP v3.1.13 (Cipater)
---- **** -----
--- * *** * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.6
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: celery_test:0x293ffd0
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: disabled
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
To schedule task:
>>> from appl.task import add
>>> r = add.delay(1, 2)
>>> r.id
'c41d4e22-ccea-408f-b48f-52e3ddd6bd66'
>>> r.task_id
'c41d4e22-ccea-408f-b48f-52e3ddd6bd66'
>>> r.status
'PENDING'
>>> r.backend
<celery.backends.redis.RedisBackend object at 0x1f35b10>
Then the worker will execute the task:
[2014-07-29 17:54:37,356: INFO/MainProcess] Received task: appl.task.add[beeef023-c582-42e1-baf7-9e19d9de32a0]
[2014-07-29 17:54:37,358: INFO/MainProcess] Task appl.task.add[beeef023-c582-42e1-baf7-9e19d9de32a0] succeeded in 0.00108124599865s: 3
But the result remains PENDING:
>>> res = add.AsyncResult(r.id)
>>> res.status
'PENDING'
I've tried the official FAQ. But it did not help.
>>> celery_app.conf['CELERY_IGNORE_RESULT']
False
What did I do wrong? Thanks!

Its been a while, but am leaving this more for others who come along with a similar issue:
In your screenshot, you see that the results are disabled
When you instantiate your celery instance, make sure that you have the right config inputs
from celery import Celery,Task
# here im using an AMQP broker with a memcached backend to store the results
celery = Celery('task1',broker='amqp://guest:guest#127.0.0.1:5672//',backend='cache+memcached://127.0.0.1:11211/')
For some reason, i always have trouble getting the celery instance parametered through the config file and hence explicitly passed in the broker and backend during instantiation as shown above
Now you'll see the results rightly configured to be memcached (in my instance - should be redis in yours). Also make sure that your task is picked up in the list of tasks (task1.add)
If you still cant get it to work, while starting celery try using the debug option as below
celery worker -A task1.celery -l debug
see if something is going wrong in the information it spews out
In my case, it fixed your error and result was set to success and i was able to recover 3 on r.get()

Try to change your broker to something else (like rabbitmq) and check the status again.
Make sure your redis server is up and accessible for celery.
redis-cli
keys *
and you should see some keys related to celery, if not it means there is a issue in your broker

This works for me:
from celery.result import AsyncResult
celery_task_result = AsyncResult(task_id)
task_state = celery_task_result.state
and task_state get all kinds of status: 'FAILURE', 'SUCCESS', 'PENDING', etc.

Celery in Amazon Server

I have two nodes (two computers are already connected) in the server and every node has 4 cores, how can I cluster these two computers using celery + RabbitMQ, when I start celeryd, it shows that:
-------------- celery#ip-10-77-137-41 v3.0.19 (Chiastic Slide)
---- **** -----
--- * *** * -- Linux-3.2.0-36-virtual-x86_64-with-Ubuntu-12.04-precise
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> broker: amqp://celeryuser#localhost:5672/celeryvhost
- ** ---------- .> app: default:0x29ecf90 (.default.Loader)
- ** ---------- .> concurrency: 2 (processes)
- *** --- * --- .> events: OFF (enable -E to monitor this worker)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery: exchange:celery(direct) binding:celery
Can I just set concurrency as 8 to using these two nodes?

On each server launch a celery worker with concurrency level 4
celery worker --concurrency=4