We Keep Coding

Multiple Python-Celery Scripts conflict and doesn't execute

I have two different python scripts in different directories that has celery schedulers.
Script 1:
import requests
from celery import Celery
from celery.schedules import crontab
import subprocess
celery = Celery()
celery.conf.enable_utc = False
#celery.task()
def proxy():
response = requests.get(url="XYZ")
proxies = response.text
paid_proxies = open("paid_proxies.txt", "w+")
paid_proxies.write(proxies.strip())
paid_proxies.close()
celery.conf.beat_schedule = {
"proxy-api": {
"task": "scheduler1.proxy",
"schedule": crontab(minute="*/5")
}
}
Commands that I use for executing it:
celery beat -A scheduler1.celery
celery worker -A scheduler1.celery
Script 2:
from celery import Celery
from celery.schedules import crontab
import subprocess
celery = Celery()
celery.conf.enable_utc = False
#celery.task()
def daily():
subprocess.run(["python3", "cross_validation.py"])
celery.conf.beat_schedule = {
"daily-scraper": {
"task": "scheduler2.daily",
"schedule": crontab(day_of_week="*", hour=15, minute=23)
}
}
Commands that I use for executing it:
celery beat -A scheduler2.celery
celery worker -A scheduler2.celery
The issues is when I execute Script 1, it works perfectly. But when I try to execute Script 2, I get this error as Scheduler2 tries to execute tasks of scheduler1:
[2019-09-14 15:10:00,127: ERROR/MainProcess] Received unregistered task of type 'scheduler1.proxy'.
The message has been ignored and discarded.
Did you remember to import the module containing this task?
Or maybe you're using relative imports?
Please see
http://docs.celeryq.org/en/latest/internals/protocol.html
for more information.
The full contents of the message body was:
'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (77b)
Traceback (most recent call last):
File "/home/PycharmProjects/data_scraping/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 559, in on_task_received
strategy = strategies[type_]
KeyError: 'scheduler1.proxy'
I tried referring multiple answers but nothing worked.

The problem that you are seeing is that celery is using the same "broker" in both project 1 and project 2. In order to use two different celery projects simultaneously, all you have to do is give them different brokers. You can specify a broker using the broker_url setting.
We typically use redis as a broker, so it is very simple to put one project on redis db 0 and the other project on redis db 1. That said, there is a lot of thinking that normally goes into which broker to use, and deciding on a broker is outside the scope of this particular question.

Reload celery beat config

I'm using celery and celery-beat without Django and I have a task which needs to modify celery-beat schedule when run.
Now I have the following code (module called celery_tasks):
# __init__.py
from .celery import app as celery_app
__all__ = ['celery_app']
#celery.py
from celery import Celery
import config
celery_config = config.get_celery_config()
app = Celery(
__name__,
include=[
'celery_tasks.tasks',
],
)
app.conf.update(celery_config)
# tasks.py
from celery_tasks import celery_app
from celery import shared_task
#shared_task
def start_game():
celery_app.conf.beat_schedule = {
'process_round': {
'task': 'celery_tasks.tasks.process_round',
'schedule': 5,
},
}
I start celery with the following command:
celery worker -A celery_tasks -E -l info --beat
start_game executes and exists normally, but beat process_round task never runs.
How can I force-reload beat schedule (restarting all workers doesn't seem as a good idea)?

the problem with normal celery backend when you start the celerybeat process. it will create a config file and write all tasks and schedules in to that file so it cannot change dynamically
you can use the package
celerybeat-sqlalchemy-scheduler so you can edit schedule on DB itself so that celerybeat will pickup the new schedule from DB itself
also there is another package celery-redbeat which using redis-server as backend
you can refer this this also

Using schedule config also seems bad idea. What if initially process_round task will be active and check if game is not started task just do nothing.

My custom django-admin command won't call celery task

I am trying to write a custom django-admin command that executes a celery task, however the task doesn't execute and django just hangs when I try.
from django.core.management.base import BaseCommand
from myapp.tasks import my_celery_task
class Command(BaseCommand):
def handle(self, *args, **options):
print "starting task"
my_celery_task.delay()
print "task has been sent"
The output I receive when calling the command is:
starting task
I never reach the "task has been sent" line. It just hangs. I'm not sure why the task isn't running. Celery tasks are called perfectly when called by a view.

This seems to work for me with celery 5.1.2, but not if it is a shared_task
====== tasks.py
from celery import current_app
from celery.schedules import crontab
app = current_app._get_current_object()
#app.task
def my_celery_task():
# Do something interesting
return 0 # Return the result of this operation
====== my_command.py
from accounts.tasks import my_celery_task
...
def handle(self, *args, **options):
self.stdout.write("Queing request")
result = my_celery_task.delay()
self.stdout.write("Waiting for task to end")
updated = result.wait(timeout=120.0)
self.stdout.write(f"Returned {updated}")
And, if I put the task in the celery.py file I ended up with a deadlock which was a PITA to debug

The issue was actually with RabbitMQ on Mac after upgrading to High Sierra.

Setting up periodic tasks in Celery (celerybeat) dynamically using add_periodic_task

I'm using Celery 4.0.1 with Django 1.10 and I have troubles scheduling tasks (running a task works fine). Here is the celery configuration:
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')
app = Celery('myapp')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.conf.BROKER_URL = 'amqp://{}:{}#{}'.format(settings.AMQP_USER, settings.AMQP_PASSWORD, settings.AMQP_HOST)
app.conf.CELERY_DEFAULT_EXCHANGE = 'myapp.celery'
app.conf.CELERY_DEFAULT_QUEUE = 'myapp.celery_default'
app.conf.CELERY_TASK_SERIALIZER = 'json'
app.conf.CELERY_ACCEPT_CONTENT = ['json']
app.conf.CELERY_IGNORE_RESULT = True
app.conf.CELERY_DISABLE_RATE_LIMITS = True
app.conf.BROKER_POOL_LIMIT = 2
app.conf.CELERY_QUEUES = (
Queue('myapp.celery_default'),
Queue('myapp.queue1'),
Queue('myapp.queue2'),
Queue('myapp.queue3'),
)
Then in tasks.py I have:
#app.task(queue='myapp.queue1')
def my_task(some_id):
print("Doing something with", some_id)
In views.py I want to schedule this task:
def my_view(request, id):
app.add_periodic_task(10, my_task.s(id))
Then I execute the commands:
sudo systemctl start rabbitmq.service
celery -A myapp.celery_app beat -l debug
celery worker -A myapp.celery_app
But the task is never scheduled. I don't see anything in the logs. The task is working because if in my view I do:
def my_view(request, id):
my_task.delay(id)
The task is executed.
If in my configuration file if I schedule the task manually, like this it works:
app.conf.CELERYBEAT_SCHEDULE = {
'add-every-30-seconds': {
'task': 'tasks.my_task',
'schedule': 10.0,
'args': (66,)
},
}
I just can't schedule the task dynamically. Any idea?

EDIT: (13/01/2018)
The latest release 4.1.0 have addressed the subject in this ticket #3958 and has been merged
Actually you can't not define periodic task at the view level, because the beat schedule setting will be loaded first and can not be rescheduled at runtime:
The add_periodic_task() function will add the entry to the beat_schedule setting behind the scenes, and the same setting can also can be used to set up periodic tasks manually:
app.conf.CELERYBEAT_SCHEDULE = {
'add-every-30-seconds': {
'task': 'tasks.my_task',
'schedule': 10.0,
'args': (66,)
},
}
which means if you want to use add_periodic_task() it should be wrapped within an on_after_configure handler at the celery app level and any modification on runtime will not take effect:
app = Celery()
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(10, my_task.s(66))
As mentioned in the doc the the regular celerybeat simply keep track of task execution:
The default scheduler is the celery.beat.PersistentScheduler, that simply keeps track of the last run times in a local shelve database file.
In order to be able to dynamically manage periodic tasks and reschedule celerybeat at runtime:
There’s also the django-celery-beat extension that stores the schedule in the Django database, and presents a convenient admin interface to manage periodic tasks at runtime.
The tasks will be persisted in django database and the scheduler could be updated in task model at the db level. Whenever you update a periodic task a counter in this tasks table will be incremented, and tells the celery beat service to reload the schedule from the database.
A possible solution for you could be as follow:
from django_celery_beat.models import PeriodicTask, IntervalSchedule
schedule= IntervalSchedule.objects.create(every=10, period=IntervalSchedule.SECONDS)
task = PeriodicTask.objects.create(interval=schedule, name='any name', task='tasks.my_task', args=json.dumps([66]))
views.py
def update_task_view(request, id)
task = PeriodicTask.objects.get(name="task name") # if we suppose names are unique
task.args=json.dumps([id])
task.save()

Python, Celery, Flask, "working outside of application context"

I am trying to schedule tasks using Celery and Python for a Flask app. I basically want to run a function in another directory every x amount of time and make it a celery task. I import the function test_check and I try and put it under a celery task called testcheck(), however, I get the error:
working outside of application context
How can I fix this? Here is my setup:
from app import app
from celery import Celery
from datetime import timedelta
from app.mod_check.views import test_check
celery = Celery(__name__,
broker='amqp://guest:#localhost/',
backend='amqp://guest:#localhost/'
)
celery.config_from_object(__name__)
#celery.task
def add(x, y):
print "celery working!"
return x + y
#celery.task
def testcheck():
test_check()
CELERYBEAT_SCHEDULE = {
'add-every-30-seconds': {
'task': 'tasks2.testcheck',
'schedule': timedelta(seconds=5),
#'args': (16, 16)
},
}
CELERY_TIMEZONE = 'Europe/London'

Whatever test_check is, it does something that needs a request context. Since Celery tasks are not part of the HTTP request/response cycle, you need to set up a request context manually.
with app.test_request_context():
test_check()