I am using Celery 4.3.0. I am trying to update the schedule of celery beat every 5 seconds based on a schedule in a json file, so that when I’m manually editing, adding or deleting scheduled tasks in that json file, the changes are picked up by the celery beat scheduler without restarting it.
What I tried is creating a task that update this schedule by updating app.conf['CELERYBEAT_SCHEDULE']. The task successfully runs every 5 seconds but celery beat doesn’t update to the new schedule, even though I set beat_max_loop_interval to 1 sec.
tasks.py
from celery import Celery
app = Celery("tasks", backend='redis://', broker='redis://')
app.config_from_object('celeryconfig')
#app.task
def hello_world():
return "Hello World!"
#app.task
def update_schedule():
with open("path_to_scheduler.json", "r") as f:
app.conf['CELERYBEAT_SCHEDULE'] = json.load(f)
celeryconfig.py
beat_max_loop_interval = 1 # Maximum number of seconds beat can sleep between checking the schedule
beat_schedule = {
"greet-every-10-seconds": {
"task": "tasks.hello_world",
"schedule": 10.0
},
'update_schedule': {
'task': 'tasks.update_schedule',
'schedule': 5.0
},
}
schedule.json
{
"greet-every-2-seconds": {
"task": "tasks.hello_world",
"schedule": 2.0
},
"update_schedule": {
"task": "tasks.update_schedule",
"schedule": 5.0
}
}
Any help would be appreciated. If you know a better way to reload the beat schedule from a file I’m also keen to hear it.
Related
I have the following celery beat schedule
CELERY_BEAT_SCHEDULE = {
'task-number-one': {
'task': 'frontend.tasks2.tasks.test_task', #this doesn't work throws following error
'schedule': 1.0
},
}
and the following error comes when I start celery beat and worker
[2018-08-20 19:26:31,606: ERROR/MainProcess] Received unregistered task of type 'frontend.tasks2.tasks.test_task'.
The message has been ignored and discarded.
Did you remember to import the module containing this task?
Or maybe you're using relative imports?
I have the following project structure
frontend
|-tasks2
| |-tasks.py
| |-test_task()
|-tasks.py
|-test_task()
But if I change the beat schedule to following it starts working, both test_task() functions are same.
CELERY_BEAT_SCHEDULE = {
'task-number-one': {
'task': 'frontend.tasks.test_task', # this does work properly
'schedule': 1.0
},
}
Where have I gone wrong?
I want to monitor the status of a celery task.
for example:
I have a celery task that read that reads a table every minute, when the task is executed at minute 2 I do not want to read what I already read the first task
app.conf.beat_schedule = {
'add-every-1-minute': {
'task': 'read_data_base',
'schedule': 1.0
}
}
I'm not good at english, so if you cannot understand my sentence, give me any comment.
I use celery for periodic task on django.
CELERYBEAT_SCHEDULE = {
'send_sms_one_pm': {
'task': 'tasks.send_one_pm',
'schedule': crontab(minute=0, hour=13),
},
'send_sms_ten_am': {
'task': 'tasks.send_ten_am',
'schedule': crontab(minute=0, hour=10),
},
'night_proposal_noti': {
'task': 'tasks.night_proposal_noti',
'schedule': crontab(minute=0, hour=10)
},
}
This is my celery schedule and i use redis for celery queue.
Problem is, when the biggest task is start, other task is on hold.
biggest task will be processed for 10hours, and, other tasks are start after 10 hours.
My task looks like
#app.task(name='tasks.send_one_pm')
def send_one_pm():
I found, celery give me task.apply_asnyc(), but couldn't find periodic tasks can working on asnyc.
So, i want to know celery's periodic task can work as asnyc task. my celery worker are 8 workers.
Did you assign CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler' in your settings as well?
If you want one task starts to run after another one, you should use apply_asnyc() with link kwargs, it looks like this:
res=[signature(your_task.name, args=(...), options=kwargs, immutable=True),..]
task.apply_async((args), link=res)
I've been working on getting manual routing set up with Celery, but can't seem to get specific tasks into specific queues. Here's what I've got going on so far pretty much:
CELERY_QUEUES = {
"default": {
"binding_key": "default"},
"medium": {
"binding_key": "medium"},
"heavy": {
"binding_key": "heavy"},
}
with the routes defined like
CELERY_ROUTES = ({ "tasks.some_heavy_task": {
"queue": "heavy",
"routing_key": "tasks.heavy"
}}, )
and the daemons started like
celeryd -l INFO -c 3 -Q heavy
The "some_heavy_task"'s never get run though. When I remove the routing and just have a default queue I can get them to run. What am I doing wrong here, any suggestions?
I created special celeryconfig file for each tasks, all tasks stored in special queue.
Here is example:
CELERY_IMPORTS = ('cleaner_on_celery.tasks',)
CELERYBEAT_SCHEDULE = {
'cleaner': {
"task": "cleaner_on_celery.tasks.cleaner",
"schedule": timedelta(seconds=CLEANER_TIMEOUT),
},
}
CELERY_QUEUES = {
"cleaner": {"exchange": "cleaner", "binding_key": "cleaner"}
}
CELERY_DEFAULT_QUEUE = "cleaner"
from celeryconfig import *
You can see in the bottom: I import common celeryconfig module. In this case you can start few celeryd instances. Also I recommend to use it with supervisord, after creating supervisord.conf file for each task you can easy manage them as:
supervisorctl start cleaner
supervisorctl stop cleaner
How do I schedule a task with celery that runs on 1st of every month?
Since Celery 3.0 the crontab schedule now supports day_of_month
and month_of_year arguments: http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#crontab-schedules
You can do this using Crontab schedules and you cand define this either:
in your django settings.py:
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
'my_periodic_task': {
'task': 'my_app.tasks.my_periodic_task',
'schedule': crontab(0, 0, day_of_month='1'), # Execute on the first day of every month.
},
}
in celery.py config:
from celery import Celery
from celery.schedules import crontab
app = Celery('app_name')
app.conf.beat_schedule = {
'my_periodic_task': {
'task': 'my_app.tasks.my_periodic_task',
'schedule': crontab(0, 0, day_of_month='1'), # Execute on the first day of every month.
},
}