I'm not good at english, so if you cannot understand my sentence, give me any comment.
I use celery for periodic task on django.
CELERYBEAT_SCHEDULE = {
'send_sms_one_pm': {
'task': 'tasks.send_one_pm',
'schedule': crontab(minute=0, hour=13),
},
'send_sms_ten_am': {
'task': 'tasks.send_ten_am',
'schedule': crontab(minute=0, hour=10),
},
'night_proposal_noti': {
'task': 'tasks.night_proposal_noti',
'schedule': crontab(minute=0, hour=10)
},
}
This is my celery schedule and i use redis for celery queue.
Problem is, when the biggest task is start, other task is on hold.
biggest task will be processed for 10hours, and, other tasks are start after 10 hours.
My task looks like
#app.task(name='tasks.send_one_pm')
def send_one_pm():
I found, celery give me task.apply_asnyc(), but couldn't find periodic tasks can working on asnyc.
So, i want to know celery's periodic task can work as asnyc task. my celery worker are 8 workers.
Did you assign CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler' in your settings as well?
If you want one task starts to run after another one, you should use apply_asnyc() with link kwargs, it looks like this:
res=[signature(your_task.name, args=(...), options=kwargs, immutable=True),..]
task.apply_async((args), link=res)
Related
I am using Celery 4.3.0. I am trying to update the schedule of celery beat every 5 seconds based on a schedule in a json file, so that when I’m manually editing, adding or deleting scheduled tasks in that json file, the changes are picked up by the celery beat scheduler without restarting it.
What I tried is creating a task that update this schedule by updating app.conf['CELERYBEAT_SCHEDULE']. The task successfully runs every 5 seconds but celery beat doesn’t update to the new schedule, even though I set beat_max_loop_interval to 1 sec.
tasks.py
from celery import Celery
app = Celery("tasks", backend='redis://', broker='redis://')
app.config_from_object('celeryconfig')
#app.task
def hello_world():
return "Hello World!"
#app.task
def update_schedule():
with open("path_to_scheduler.json", "r") as f:
app.conf['CELERYBEAT_SCHEDULE'] = json.load(f)
celeryconfig.py
beat_max_loop_interval = 1 # Maximum number of seconds beat can sleep between checking the schedule
beat_schedule = {
"greet-every-10-seconds": {
"task": "tasks.hello_world",
"schedule": 10.0
},
'update_schedule': {
'task': 'tasks.update_schedule',
'schedule': 5.0
},
}
schedule.json
{
"greet-every-2-seconds": {
"task": "tasks.hello_world",
"schedule": 2.0
},
"update_schedule": {
"task": "tasks.update_schedule",
"schedule": 5.0
}
}
Any help would be appreciated. If you know a better way to reload the beat schedule from a file I’m also keen to hear it.
I have the following celery beat schedule
CELERY_BEAT_SCHEDULE = {
'task-number-one': {
'task': 'frontend.tasks2.tasks.test_task', #this doesn't work throws following error
'schedule': 1.0
},
}
and the following error comes when I start celery beat and worker
[2018-08-20 19:26:31,606: ERROR/MainProcess] Received unregistered task of type 'frontend.tasks2.tasks.test_task'.
The message has been ignored and discarded.
Did you remember to import the module containing this task?
Or maybe you're using relative imports?
I have the following project structure
frontend
|-tasks2
| |-tasks.py
| |-test_task()
|-tasks.py
|-test_task()
But if I change the beat schedule to following it starts working, both test_task() functions are same.
CELERY_BEAT_SCHEDULE = {
'task-number-one': {
'task': 'frontend.tasks.test_task', # this does work properly
'schedule': 1.0
},
}
Where have I gone wrong?
I want to monitor the status of a celery task.
for example:
I have a celery task that read that reads a table every minute, when the task is executed at minute 2 I do not want to read what I already read the first task
app.conf.beat_schedule = {
'add-every-1-minute': {
'task': 'read_data_base',
'schedule': 1.0
}
}
I don't know exactly why, but I am getting duplicated tasks. I thing this may be related with time change of the last weekend (The clock was delayed for an hour in the system).
The first task should not be executed, since I say explicitly hour=2. Any idea why this happens?
[2017-11-01 01:00:00,001: INFO/Beat] Scheduler: Sending due task every-first-day_month (app.users.views.websites_down)
[2017-11-01 02:00:00,007: INFO/Beat] Scheduler: Sending due task every-first-day_month (app.users.views.websites_down)
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
'every-first-day_month': {
'task': 'app.users.views.websites_down',
'schedule': crontab(hour=2, minute=0, day_of_month=1),
}
}
CELERY_TIMEZONE = "Europe/Lisbon"
I run celery:
celery multi start --app=myapp fast_worker
slow_worker
-Q:fast_worker fast-queue
-Q:slow_worker slow-queue
-c:fast_worker 1 -c:slow_worker 1
--logfile=%n.log --pidfile=%n.pid
And celerybeat:
celery beat -A myapp
Task:
#task.periodic_task(run_every=timedelta(seconds=5), ignore_result=True)
def test_log_task_queue():
import time
time.sleep(10)
print "test_log_task_queue"
Routing:
CELERY_ROUTES = {
'myapp.tasks.test_log_task_queue': {
'queue': 'slow-queue',
'routing_key': 'slow-queue',
},
}
I use rabbitMQ. When I open rabbitMQ admin panel, I see that my tasks are in slow-queue, but when I open logs I see task output for both workers. Why do both workers execute my tasks, even when task not in worker queue?
It looks like celery multi creates something like shared queues. To fix this problem, I added -X option:
celery multi start --app=myapp fast_worker
slow_worker
-Q:fast_worker fast-queue
-Q:slow_worker slow-queue
-X:fast_worker slow-queue
-X:slow_worker fast-queue
-c:fast_worker 1 -c:slow_worker 1
--logfile=%n.log --pidfile=%n.pid