We Keep Coding

Why celery beat doesn't schedule periodic tasks?

I've followed the celery doc https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html to create an app and periodic tasks as below:
$ tree demo/
demo/
├── config.py
├── __init__.py
└── tasks.py
$ cat demo/__init__.py
# -*- coding: utf-8 -*-
from celery import Celery
app = Celery('demo')
app.config_from_object('demo.config')
$ cat demo/config.py
# -*- coding: utf-8 -*-
BROKER_URL = "redis://127.0.0.1:6379"
CELERY_TIMEZONE='UTC'
CELERY_IMPORTS = [
"demo.tasks",
]
$ cat demo/tasks.py
from demo import app
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(3.0, say.s(), name='say hello every 3s')
#app.task
def say():
print("Hello!")
And then run celery beat as below:
$ celery beat -A demo -l info --max-interval 10
celery beat v4.3.0 (rhubarb) is starting.
__ - ... __ - _
LocalTime -> 2019-12-12 16:26:41
Configuration ->
. broker -> redis://127.0.0.1:6379//
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]#%INFO
. maxinterval -> 10.00 seconds (10.0s)
[2019-12-12 16:26:41,234: INFO/MainProcess] beat: Starting...
Wait for a while, there isn't any task scheduled.
However if I change to set CELERYBEAT_SCHEDULE in config, it can work well. I do that by changing demo/config.py and demo/tasks.py as below:
$ cat demo/config.py
# -*- coding: utf-8 -*-
from datetime import timedelta
BROKER_URL = "redis://127.0.0.1:6379"
CELERY_TIMEZONE='UTC'
CELERY_IMPORTS = [
"demo.tasks",
]
CELERYBEAT_SCHEDULE = {
'say hello every 10 seconds': {
'task': 'demo.tasks.say',
'schedule': timedelta(seconds=3),
},
}
$ cat demo/tasks.py
from demo import app
#app.task
def say():
print("Hello!")
Then run celery beat with the same command as before, the periodic tasks can be scheduled every 3 seconds as expected.
What's wrong with my previous setup?

FYI, I figured out another solution, changing the decorator #app.on_after_configure.connect to #app.on_after_finalize.connect can make it work. Though I don't know the exact reason at this moment.

Celery 4.x does not use the upper-case config variables - see the New lowercase settings section of the Celery documentation. Scheduler-specific configuration arguments are listed here. So, try to modify your code to have beat_schedule = { instead of CELERYBEAT_SCHEDULE = {. The link you say you followed also uses lower-case names...

You must specify the scheduler by --scheduler key according this userguide. For example you could try something like this code below:
celery beat -A demo -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler
P.S. I don't sure that --max-interval not interfere with your schedule period. Think is better to remove this option. IMHO.

Django Celery periodic task example

I need a minimum example to do periodic task (run some function after every 5 minutes, or run something at 12:00:00 etc.).
In my myapp/tasks.py, I have,
from celery.task.schedules import crontab
from celery.decorators import periodic_task
from celery import task
#periodic_task(run_every=(crontab(hour="*", minute=1)), name="run_every_1_minutes", ignore_result=True)
def return_5():
return 5
#task
def test():
return "test"
When I run celery workers it does show the tasks (given below) but does not return any values (in either terminal or flower).
[tasks]
. mathematica.core.tasks.test
. run_every_1_minutes
Please provide a minimum example or hints to achieve the desired results.
Background:
I have a config/celery.py which contains the following:
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
app = Celery('config')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
And in my config/__init__.py, I have
from .celery import app as celery_app
__all__ = ['celery_app']
I added a function something like below in myapp/tasks.py
from celery import task
#task
def test():
return "test"
When I run test.delay() from shell, it runs successfully and also shows the task information in flower

To run periodic task you should run celery beat also. You can run it with this command:
celery -A proj beat
Or if you are using one worker:
celery -A proj worker -B

Periodic task is not working properly in django celery

Hello friends I need your help in periodic tasks for django application. I am trying to do periodic task using celery but it is not properly working. I have a simple task, in which there is only a print statement. Celery is working only for 1st time, and I also tried 'celery beat', but haven't got result
my "task.py" is
from __future__ import absolute_import
from myapp.celery import app
from celery.schedules import crontab
from celery.task import periodic_task
from celery.registry import tasks
#periodic_task(run_every=(crontab(minute='*/1')), name="some_task")
def every_minute(a,b):
print("This is running after one minute",a+b)
return "task done"
tasks.register(every_minute)
and "view.py"
from django.http import HttpResponse
from django.views.generic import View
from .tasks import *
from .models import *
from datetime import datetime, timedelta
class CeleryTest(View):
def get(self,request):
send_date = datetime.now() + timedelta(seconds=200)
task=every_minute.apply_async([5,6],etc=send_date)
while not task.ready():
print "calling............task is not ready"
return HttpResponse("hi get ur task")
I just added this schedule in earlier celery setting."setting.py"
CELERYBEAT_SCHEDULE = {
'every_minute': {
'task': 'every_minute.add',
'schedule': crontab(minute='*/1'),
'args': (5, 6),
},
}
Thank you friends for your time.

use your Schedule like this....
CELERYBEAT_SCHEDULE = {
'every_minute': {
'task': 'every_minute',
},
}
and run this command for celery
python manage.py celeryd -BE -l info
now my periodic tasks running fine.

Celery task group not being executed in background and results in exception

My Celery task isn't executing in the background in my Django 1.7/Python3 project.
# settings.py
BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULTBACKEND = BROKER_URL
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_ALWAYS_EAGER = False
I have celery.py in my root app module as such:
from __future__ import absolute_import
import os
import django
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'my_app.settings')
django.setup()
app = Celery('my_app')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
and load the app in __init__.py in the root module:
from __future__ import absolute_import
from .celery import app as celery_app
My task is set up as a shared task in a tasks.py file in my app module:
from __future__ import absolute_import
from celery import shared_task
#shared_task
def update_statistics(profile, category):
# more code
and I call the task as a group:
. . .
job = group([update_statistics(f.profile, category)
for f in forecasts])
job.apply_async()
However, I'm not seeing any status updates in my task queue, which I am starting via:
$ celery -A my_app worker -l info
The task is being executed, just not in the background. If I add a print statement to the task code, I will see the output in my Django development server console instead of the Celery queue.
After the task runs in the foreground, I'm greeted with this exception:
'NoneType' object has no attribute 'app'
Here's the full traceback if you're interested: https://gist.github.com/alsoicode/0263d251e3744227ba46

You're calling the tasks directly in your list comprehension when you create the group, so they're executed then and there. You need to use the .subtask() method (or its shortcut, .s()) to create the subtasks without calling them:
job = group([update_statistics.s(f.profile, category) for f in forecasts])

Celery task routing doesn't work when using current_app.send_task

I have an issue with Celery queue routing when using current_app.send_task
I have two workers (each one for each queue)
python manage.py celery worker -E -Q priority --concurrency=8 --loglevel=DEBUG
python manage.py celery worker -Q low --concurrency=8 -E -B --loglevel=DEBUG
I have two queues defined in celeryconfig.py file:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.core.exceptions import ImproperlyConfigured
from celery import Celery
from django.conf import settings
try:
app = Celery('proj', broker=getattr(settings, 'BROKER_URL', 'redis://'))
except ImproperlyConfigured:
app = Celery('proj', broker='redis://')
app.conf.update(
CELERY_TASK_SERIALIZER='json',
CELERY_ACCEPT_CONTENT=['json'],
CELERY_RESULT_SERIALIZER='json',
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
CELERY_DEFAULT_EXCHANGE='tasks',
CELERY_DEFAULT_EXCHANGE_TYPE='topic',
CELERY_DEFAULT_ROUTING_KEY='task.priority',
CELERY_QUEUES=(
Queue('priority',routing_key='priority.#'),
Queue('low', routing_key='low.#'),
),
CELERY_DEFAULT_EXCHANGE='priority',
CELERY_IMPORTS=('mymodule.tasks',)
CELERY_ENABLE_UTC = True
CELERY_TIMEZONE = 'UTC'
if __name__ == '__main__':
app.start()
In the definition of tasks, we use decorator to explicit the queue:
#task(name='mymodule.mytask', routing_key='low.mytask', queue='low')
def mytask():
# does something
pass
This task is run indeed in the low queue when this task is run using:
from mymodule.tasks import mytask
mytask.delay()
But it's not the case when it's run using: (it's run in the default queue: "priority")
from celery import current_app
current_app.send_task('mymodule.mytask')
I wonder why this later way doesn't route the task to the "low" queue!
p.s: I use redis.

send_task is a low-level method. It sends directly to the broker the task signature without going through your task decorator.
With this method, you can even send a task without loading the task code/module.
To solve your problem, you can fetch the routing_key/queue from configuration directly:
route = celery.amqp.routes[0].route_for_task("mymodule.mytask")
Out[10]: {'queue': 'low', 'routing_key': 'low.mytask'}
celery.send_task("myodule.mytask", queue=route['queue'], routing_key=route['routing_key']`