how to monitor the status of Celery task - python

I want to monitor the status of a celery task.
for example:
I have a celery task that read that reads a table every minute, when the task is executed at minute 2 I do not want to read what I already read the first task
app.conf.beat_schedule = {
'add-every-1-minute': {
'task': 'read_data_base',
'schedule': 1.0
}
}

Related

How to dynamically change the schedule of celery beat?

I am using Celery 4.3.0. I am trying to update the schedule of celery beat every 5 seconds based on a schedule in a json file, so that when I’m manually editing, adding or deleting scheduled tasks in that json file, the changes are picked up by the celery beat scheduler without restarting it.
What I tried is creating a task that update this schedule by updating app.conf['CELERYBEAT_SCHEDULE']. The task successfully runs every 5 seconds but celery beat doesn’t update to the new schedule, even though I set beat_max_loop_interval to 1 sec.
tasks.py
from celery import Celery
app = Celery("tasks", backend='redis://', broker='redis://')
app.config_from_object('celeryconfig')
#app.task
def hello_world():
return "Hello World!"
#app.task
def update_schedule():
with open("path_to_scheduler.json", "r") as f:
app.conf['CELERYBEAT_SCHEDULE'] = json.load(f)
celeryconfig.py
beat_max_loop_interval = 1 # Maximum number of seconds beat can sleep between checking the schedule
beat_schedule = {
"greet-every-10-seconds": {
"task": "tasks.hello_world",
"schedule": 10.0
},
'update_schedule': {
'task': 'tasks.update_schedule',
'schedule': 5.0
},
}
schedule.json
{
"greet-every-2-seconds": {
"task": "tasks.hello_world",
"schedule": 2.0
},
"update_schedule": {
"task": "tasks.update_schedule",
"schedule": 5.0
}
}
Any help would be appreciated. If you know a better way to reload the beat schedule from a file I’m also keen to hear it.

Calling a task via celery beat which is inside a folder throws Error

I have the following celery beat schedule
CELERY_BEAT_SCHEDULE = {
'task-number-one': {
'task': 'frontend.tasks2.tasks.test_task', #this doesn't work throws following error
'schedule': 1.0
},
}
and the following error comes when I start celery beat and worker
[2018-08-20 19:26:31,606: ERROR/MainProcess] Received unregistered task of type 'frontend.tasks2.tasks.test_task'.
The message has been ignored and discarded.
Did you remember to import the module containing this task?
Or maybe you're using relative imports?
I have the following project structure
frontend
|-tasks2
| |-tasks.py
| |-test_task()
|-tasks.py
|-test_task()
But if I change the beat schedule to following it starts working, both test_task() functions are same.
CELERY_BEAT_SCHEDULE = {
'task-number-one': {
'task': 'frontend.tasks.test_task', # this does work properly
'schedule': 1.0
},
}
Where have I gone wrong?

Duplicated tasks after time change

I don't know exactly why, but I am getting duplicated tasks. I thing this may be related with time change of the last weekend (The clock was delayed for an hour in the system).
The first task should not be executed, since I say explicitly hour=2. Any idea why this happens?
[2017-11-01 01:00:00,001: INFO/Beat] Scheduler: Sending due task every-first-day_month (app.users.views.websites_down)
[2017-11-01 02:00:00,007: INFO/Beat] Scheduler: Sending due task every-first-day_month (app.users.views.websites_down)
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
'every-first-day_month': {
'task': 'app.users.views.websites_down',
'schedule': crontab(hour=2, minute=0, day_of_month=1),
}
}
CELERY_TIMEZONE = "Europe/Lisbon"

celery periodic task as asnyc on django

I'm not good at english, so if you cannot understand my sentence, give me any comment.
I use celery for periodic task on django.
CELERYBEAT_SCHEDULE = {
'send_sms_one_pm': {
'task': 'tasks.send_one_pm',
'schedule': crontab(minute=0, hour=13),
},
'send_sms_ten_am': {
'task': 'tasks.send_ten_am',
'schedule': crontab(minute=0, hour=10),
},
'night_proposal_noti': {
'task': 'tasks.night_proposal_noti',
'schedule': crontab(minute=0, hour=10)
},
}
This is my celery schedule and i use redis for celery queue.
Problem is, when the biggest task is start, other task is on hold.
biggest task will be processed for 10hours, and, other tasks are start after 10 hours.
My task looks like
#app.task(name='tasks.send_one_pm')
def send_one_pm():
I found, celery give me task.apply_asnyc(), but couldn't find periodic tasks can working on asnyc.
So, i want to know celery's periodic task can work as asnyc task. my celery worker are 8 workers.
Did you assign CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler' in your settings as well?
If you want one task starts to run after another one, you should use apply_asnyc() with link kwargs, it looks like this:
res=[signature(your_task.name, args=(...), options=kwargs, immutable=True),..]
task.apply_async((args), link=res)

How do I schedule a task with Celery that runs on 1st of every month?

How do I schedule a task with celery that runs on 1st of every month?
Since Celery 3.0 the crontab schedule now supports day_of_month
and month_of_year arguments: http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#crontab-schedules
You can do this using Crontab schedules and you cand define this either:
in your django settings.py:
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
'my_periodic_task': {
'task': 'my_app.tasks.my_periodic_task',
'schedule': crontab(0, 0, day_of_month='1'), # Execute on the first day of every month.
},
}
in celery.py config:
from celery import Celery
from celery.schedules import crontab
app = Celery('app_name')
app.conf.beat_schedule = {
'my_periodic_task': {
'task': 'my_app.tasks.my_periodic_task',
'schedule': crontab(0, 0, day_of_month='1'), # Execute on the first day of every month.
},
}

Categories