I've been working on getting manual routing set up with Celery, but can't seem to get specific tasks into specific queues. Here's what I've got going on so far pretty much:
CELERY_QUEUES = {
"default": {
"binding_key": "default"},
"medium": {
"binding_key": "medium"},
"heavy": {
"binding_key": "heavy"},
}
with the routes defined like
CELERY_ROUTES = ({ "tasks.some_heavy_task": {
"queue": "heavy",
"routing_key": "tasks.heavy"
}}, )
and the daemons started like
celeryd -l INFO -c 3 -Q heavy
The "some_heavy_task"'s never get run though. When I remove the routing and just have a default queue I can get them to run. What am I doing wrong here, any suggestions?
I created special celeryconfig file for each tasks, all tasks stored in special queue.
Here is example:
CELERY_IMPORTS = ('cleaner_on_celery.tasks',)
CELERYBEAT_SCHEDULE = {
'cleaner': {
"task": "cleaner_on_celery.tasks.cleaner",
"schedule": timedelta(seconds=CLEANER_TIMEOUT),
},
}
CELERY_QUEUES = {
"cleaner": {"exchange": "cleaner", "binding_key": "cleaner"}
}
CELERY_DEFAULT_QUEUE = "cleaner"
from celeryconfig import *
You can see in the bottom: I import common celeryconfig module. In this case you can start few celeryd instances. Also I recommend to use it with supervisord, after creating supervisord.conf file for each task you can easy manage them as:
supervisorctl start cleaner
supervisorctl stop cleaner
Related
I have taken a free trial for Redis and it gave me an endpoint with a password. I haven't done anything with Redis or celery before so I really don't have any idea how it works. From the Docs of Celery everyone connects to the local host but how can I connect to this endpoint?
CELERY_BROKER_URL='redis://localhost:6379',
CELERY_RESULT_BACKEND='redis://localhost:6379'
What should I replace this with? Where should I give the password?
My endpoint looks something like this: redis-18394.c252.######.cloud.redislabs.com:18394, Should I add the password at the end of this after a / ?
According to celery's documentation, the format is
redis://:password#hostname:port/db_number
By default, redis has 16 databases so you can use any number from 0-15 for db_number. Use a different db number for broker and result backend.
https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/redis.html#configuration
You can use channel_redis for this
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": ["password#your_ip"],
},
},
}
I am using Celery 4.3.0. I am trying to update the schedule of celery beat every 5 seconds based on a schedule in a json file, so that when I’m manually editing, adding or deleting scheduled tasks in that json file, the changes are picked up by the celery beat scheduler without restarting it.
What I tried is creating a task that update this schedule by updating app.conf['CELERYBEAT_SCHEDULE']. The task successfully runs every 5 seconds but celery beat doesn’t update to the new schedule, even though I set beat_max_loop_interval to 1 sec.
tasks.py
from celery import Celery
app = Celery("tasks", backend='redis://', broker='redis://')
app.config_from_object('celeryconfig')
#app.task
def hello_world():
return "Hello World!"
#app.task
def update_schedule():
with open("path_to_scheduler.json", "r") as f:
app.conf['CELERYBEAT_SCHEDULE'] = json.load(f)
celeryconfig.py
beat_max_loop_interval = 1 # Maximum number of seconds beat can sleep between checking the schedule
beat_schedule = {
"greet-every-10-seconds": {
"task": "tasks.hello_world",
"schedule": 10.0
},
'update_schedule': {
'task': 'tasks.update_schedule',
'schedule': 5.0
},
}
schedule.json
{
"greet-every-2-seconds": {
"task": "tasks.hello_world",
"schedule": 2.0
},
"update_schedule": {
"task": "tasks.update_schedule",
"schedule": 5.0
}
}
Any help would be appreciated. If you know a better way to reload the beat schedule from a file I’m also keen to hear it.
I have been following this article - https://blog.mangoforbreakfast.com/2017/02/13/django-channels-on-aws-elastic-beanstalk-using-an-alb/
to get my django-channels app working on aws..but only non-websockets request are getting handled.
my channel layer setting is :
CHANNEL_LAYERS = {
"default": {
"BACKEND": "asgi_redis.RedisChannelLayer",
"CONFIG": {
"hosts": [os.environ.get('REDIS_URL', 'redis://localhost:6379')],
},
"ROUTING": "malang.routing.channel_routing",
},
}
I have two target group as mentioned in the article. One forwarding path / to port 80 and /ws/* to 5000.
My supervisord.conf is -
[program:Daphne]
environment=PATH="/opt/python/run/venv/bin"
command=/opt/python/run/venv/bin/daphne -b 0.0.0.0 -p 5000
malang.asgi:channel_layer
directory=/opt/python/current/app
autostart=true
autorestart=true
redirect_stderr=true
user=root
stdout_logfile=/tmp/daphne.out.log
[program:Worker]
environment=PATH="/opt/python/run/venv/bin"
command= /opt/python/run/venv/bin/python manage.py runworker
directory=/opt/python/current/app
process_name=%(program_name)s_%(process_num)02d
numprocs=4
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/workers.out.log
When I check the result of supervisorctl status on aws logs it shows them running fine. But still I get 404 response for ws.
Please help and let me know if you want some more info..
Does the project run locally? If not, the issue is with the software. If so, the issue is with your deployment. I would check the security group/firewall/ELB configuration to ensure the correct ports are accessible.
It makes no sense to run a Redis backend locally on each instance, provided the fact that you actually deployed it, which you don't given your info.
Redis is a cache system that allow data sharing through different instances. Closer to a DB on architectural point of view that a simple daemon thread.
You should use a external Redis Cache instead and refer to it on you Django conf.
CHANNEL_LAYERS = {
"default": {
"BACKEND": "asgi_redis.RedisChannelLayer",
"ROUTING": "<YOUR_APP>.routing.application",
"CONFIG": {
"hosts": ["redis://"+REDIS_URL+":6379"],
},
},
}
See AWS ElasticCache service for that.
I'm not good at english, so if you cannot understand my sentence, give me any comment.
I use celery for periodic task on django.
CELERYBEAT_SCHEDULE = {
'send_sms_one_pm': {
'task': 'tasks.send_one_pm',
'schedule': crontab(minute=0, hour=13),
},
'send_sms_ten_am': {
'task': 'tasks.send_ten_am',
'schedule': crontab(minute=0, hour=10),
},
'night_proposal_noti': {
'task': 'tasks.night_proposal_noti',
'schedule': crontab(minute=0, hour=10)
},
}
This is my celery schedule and i use redis for celery queue.
Problem is, when the biggest task is start, other task is on hold.
biggest task will be processed for 10hours, and, other tasks are start after 10 hours.
My task looks like
#app.task(name='tasks.send_one_pm')
def send_one_pm():
I found, celery give me task.apply_asnyc(), but couldn't find periodic tasks can working on asnyc.
So, i want to know celery's periodic task can work as asnyc task. my celery worker are 8 workers.
Did you assign CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler' in your settings as well?
If you want one task starts to run after another one, you should use apply_asnyc() with link kwargs, it looks like this:
res=[signature(your_task.name, args=(...), options=kwargs, immutable=True),..]
task.apply_async((args), link=res)
I am a bit confused on how to configure Django+Celery.
I have followed what is reported in this guide.
Here it is the bunch of configuration I have to write:
BROKER_URL = 'amqp://...'
queue_arguments = {'x-max-length': 1}
CELERY_QUEUES = (
Queue('queue1', routing_key='queue1', queue_arguments=queue_arguments),
Queue('queue2', routing_key='queue2', queue_arguments=queue_arguments))
from datetime import timedelta
CELERYBEAT_SCHEDULE = {
'task1': {
'task': 'MyProject.tasks.this_is_task_1',
'schedule': timedelta(seconds=1)
},
'task2': {
'task': 'MyProject.tasks.this_is_task_2',
'schedule': timedelta(seconds=1)
}
}
CELERY_ROUTES = {
'MyProject.tasks.this_is_task_1': {
'queue': 'queue1',
'routing_key': 'queue1',
},
'MyProject.tasks.this_is_task_2': {
'queue': 'queue2',
'routing_key': 'queue2',
}
}
app = Celery('MyProject')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'MyProject.settings')
app = Celery('MyProject')
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=30,
CELERY_IGNORE_RESULT=True,
)
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
Based on 1, I should put:
The tasks in MyProject/tasks.py
The 'app' variable creation and initialization in MyProject/celery.py
The configuration variables in MyProject/settings.py
If I do so, i receive the following:
Couldn't apply scheduled task check_block_height: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'x-max-length'for queue 'queue2' in vhost '...': received the value '1' of type 'signedint' but current is none
[2015-11-04 00:30:12,899: DEBUG/MainProcess] beat: Waking up now.
It behaves as if the queue has already been created, but without any option.
If I keep just CELERYBEAT_SCHEDULE and CELERY_ROUTES in settings.py, everything seems to be working. The truth is that the queue configuration is ignored, that is, the CELERY_QUEUE configuration is not used.
Thanks
I solved the problem.
The issue was related to the fact that I already had created the queues, which were not limited.
By deleting them and restarting, everything worked perfectly by putting:
1. all the configuration in settings.py
2. all the Celery creation in celery.py