Hello friends I need your help in periodic tasks for django application. I am trying to do periodic task using celery but it is not properly working. I have a simple task, in which there is only a print statement. Celery is working only for 1st time, and I also tried 'celery beat', but haven't got result
my "task.py" is
from __future__ import absolute_import
from myapp.celery import app
from celery.schedules import crontab
from celery.task import periodic_task
from celery.registry import tasks
#periodic_task(run_every=(crontab(minute='*/1')), name="some_task")
def every_minute(a,b):
print("This is running after one minute",a+b)
return "task done"
tasks.register(every_minute)
and "view.py"
from django.http import HttpResponse
from django.views.generic import View
from .tasks import *
from .models import *
from datetime import datetime, timedelta
class CeleryTest(View):
def get(self,request):
send_date = datetime.now() + timedelta(seconds=200)
task=every_minute.apply_async([5,6],etc=send_date)
while not task.ready():
print "calling............task is not ready"
return HttpResponse("hi get ur task")
I just added this schedule in earlier celery setting."setting.py"
CELERYBEAT_SCHEDULE = {
'every_minute': {
'task': 'every_minute.add',
'schedule': crontab(minute='*/1'),
'args': (5, 6),
},
}
Thank you friends for your time.
use your Schedule like this....
CELERYBEAT_SCHEDULE = {
'every_minute': {
'task': 'every_minute',
},
}
and run this command for celery
python manage.py celeryd -BE -l info
now my periodic tasks running fine.
Related
this function i was using under management/commands and it was working but i'd like to use it to update feeds periodically. like every 15 mins or so.
celery.py
from celery import Celery
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")
app = Celery('mysite',
broker='redis://',
backend='rpc://',
include=['mysite.tasks'],
timezone='UTC')
if __name__ == '__main__':
app.start()
tasks.py
from .celery import app
from news.models import Feed, Article
import feedparser
from datetime import datetime, timezone
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(60.0, handle.s(), name='update feeds every 60 seconds')
#app.task
def handle(self, *args, **kwargs):
feeds = Feed.objects.all()
for feed in feeds:
new_list = feedparser.parse(feed.url)
for entry in new_list.entries:
start = datetime.now(timezone.utc)
end = datetime(*(entry.published_parsed[0:6])).replace(tzinfo=timezone.utc)
if (start - end).days < 2 and not Article.objects.filter(url=entry.link).exists():
article = Article()
article.title = entry.title
article.url = entry.link
article.description = entry.description
dateString = end.strftime('%Y-%m-%d %H:%M:%S %z')
article.publication_date = dateString
article.save()
else:
pass
i run celery -A mysite worker -l INFO and there is
[tasks]
. mysite.tasks.handle
i tried also celery -A mysite beat
no errors but i don't see any effect on my site
i also tried to run both commands in two terminals.
in settings directory create file for celery
celery_conf.py
import os
from celery import Celery
from datetime import timedelta
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<path to settings like=> config.settings>')
celery_app = Celery('<settings direcory name like=>config>')
celery_app.autodiscover_tasks()
celery_app.conf.broker_url = 'amqp://'
celery_app.conf.result_backend = 'rpc://'
in your apps,Wherever you need celery,create tasks.py file
Create your own function(in tasks.py file) and set shared_task decorator to function
tasks.py
from celery import shared_task
#shared_task
def send_email():
.....
Celery using shared_task to identify tasks
*** Don't forget to use delay in task calling***
I am using the below packages.
celery==5.1.2
Django==3.1
I have 2 periodic celery tasks, in which I want the first task to run every 15 mins and the second to run every 20 mins. But the problem is that the first task is running on time, while the second is running on random timing.
Although I'm getting a message on console on time for both tasks:
Scheduler: Sending due task <task_name> (<task_name>)
Please find the following files,
celery.py
from celery import Celery, Task
app = Celery('settings')
...
class PeriodicTask(Task):
#classmethod
def on_bound(cls, app):
app.conf.beat_schedule[cls.name] = {
"schedule": cls.run_every,
"task": cls.name,
"args": cls.args if hasattr(cls, "args") else (),
"kwargs": cls.kwargs if hasattr(cls, "kwargs") else {},
"options": cls.options if hasattr(cls, "options") else {}
}
tasks.py
from celery.schedules import crontab
from settings.celery import app, PeriodicTask
...
#app.task(
base=PeriodicTask,
run_every=crontab(minute='*/15'),
name='task1',
options={'queue': 'queue_name'}
)
def task1():
logger.info("task1 called")
#app.task(
base=PeriodicTask,
run_every=crontab(minute='*/20'),
name='task2'
)
def task2():
logger.info("task2 called")
Please help me to find the bug here. Thanks!
I need a minimum example to do periodic task (run some function after every 5 minutes, or run something at 12:00:00 etc.).
In my myapp/tasks.py, I have,
from celery.task.schedules import crontab
from celery.decorators import periodic_task
from celery import task
#periodic_task(run_every=(crontab(hour="*", minute=1)), name="run_every_1_minutes", ignore_result=True)
def return_5():
return 5
#task
def test():
return "test"
When I run celery workers it does show the tasks (given below) but does not return any values (in either terminal or flower).
[tasks]
. mathematica.core.tasks.test
. run_every_1_minutes
Please provide a minimum example or hints to achieve the desired results.
Background:
I have a config/celery.py which contains the following:
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
app = Celery('config')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
And in my config/__init__.py, I have
from .celery import app as celery_app
__all__ = ['celery_app']
I added a function something like below in myapp/tasks.py
from celery import task
#task
def test():
return "test"
When I run test.delay() from shell, it runs successfully and also shows the task information in flower
To run periodic task you should run celery beat also. You can run it with this command:
celery -A proj beat
Or if you are using one worker:
celery -A proj worker -B
I am using celery and trying to run the corntab. Below is my celery.py
from __future__ import absolute_import
from celery.schedules import crontab
from celery import Celery
app = Celery('Celery_1',
broker='amqp://test:test#localhost//',
include=['Celery_1.tasks'])
# Optional configuration, see the application user guide.
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=3600,
CELERYBEAT_SCHEDULE = {
'T1': {
'task': 'Celery_1.tasks.add',
'schedule': crontab(minute='*/1'),
'args': (4, 5)
}
},
CELERY_IMPORTS = ('Celery_1.tasks', )
)
if __name__ == '__main__':
app.start()
And my tasks.py
from __future__ import absolute_import
from Celery_1.celery import app
#app.task(name='Celery_1.add')
def add(x, y):
return x + y
when i schedule by celery beat
but it is not running task for every minute. Can any one please help me ?
You should run it this way (while still in the Celery_1 directory):
echo $null >> __init__.py #to make your directory a python project
cd ..
celery -A Celery_1 beat
Celery_1 is the name of your app.
I have an issue with Celery queue routing when using current_app.send_task
I have two workers (each one for each queue)
python manage.py celery worker -E -Q priority --concurrency=8 --loglevel=DEBUG
python manage.py celery worker -Q low --concurrency=8 -E -B --loglevel=DEBUG
I have two queues defined in celeryconfig.py file:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.core.exceptions import ImproperlyConfigured
from celery import Celery
from django.conf import settings
try:
app = Celery('proj', broker=getattr(settings, 'BROKER_URL', 'redis://'))
except ImproperlyConfigured:
app = Celery('proj', broker='redis://')
app.conf.update(
CELERY_TASK_SERIALIZER='json',
CELERY_ACCEPT_CONTENT=['json'],
CELERY_RESULT_SERIALIZER='json',
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
CELERY_DEFAULT_EXCHANGE='tasks',
CELERY_DEFAULT_EXCHANGE_TYPE='topic',
CELERY_DEFAULT_ROUTING_KEY='task.priority',
CELERY_QUEUES=(
Queue('priority',routing_key='priority.#'),
Queue('low', routing_key='low.#'),
),
CELERY_DEFAULT_EXCHANGE='priority',
CELERY_IMPORTS=('mymodule.tasks',)
CELERY_ENABLE_UTC = True
CELERY_TIMEZONE = 'UTC'
if __name__ == '__main__':
app.start()
In the definition of tasks, we use decorator to explicit the queue:
#task(name='mymodule.mytask', routing_key='low.mytask', queue='low')
def mytask():
# does something
pass
This task is run indeed in the low queue when this task is run using:
from mymodule.tasks import mytask
mytask.delay()
But it's not the case when it's run using: (it's run in the default queue: "priority")
from celery import current_app
current_app.send_task('mymodule.mytask')
I wonder why this later way doesn't route the task to the "low" queue!
p.s: I use redis.
send_task is a low-level method. It sends directly to the broker the task signature without going through your task decorator.
With this method, you can even send a task without loading the task code/module.
To solve your problem, you can fetch the routing_key/queue from configuration directly:
route = celery.amqp.routes[0].route_for_task("mymodule.mytask")
Out[10]: {'queue': 'low', 'routing_key': 'low.mytask'}
celery.send_task("myodule.mytask", queue=route['queue'], routing_key=route['routing_key']`