I want to explicitly revoke a task from celery. This is how I'm currently doing:-
from celery.task.control import revoke
revoke(task_id, terminate=True)
where task_id is string(have also tried converting it into UUID uuid.UUID(task_id).hex).
After the above procedure, when I start celery again celery worker -A proj it still consumes the same message and starts processing it. Why?
When viewed via flower, the message is still there in the broker section. how do I delete the message so that it cant be consumed again?
How does revoke works?
When calling the revoke method the task doesn't get deleted from the queue immediately, all it does is tell celery(not your broker!) to save the task_id in a in-memory set(look here if you like reading source code like me).
When the task gets to the top of the queue, Celery will check if is it in the revoked set, if it does, it won't execute it.
It works this way to prevent O(n) search for each revoke call, where checking if the task_id is in the in-memory set is just O(1)
Why after restarting celery, your revoked tasks executed?
Understanding how things works, you now know that the set is just a normal python set, that being saved in-memory - that means when you restart, you lose this set, but the task is(of course) persistence and when the tasks turn comes, it will be executed as normal.
What can you do?
You will need to have a persistence set, this is done by initial your worker like this:
celery worker -A proj --statedb=/var/run/celery/worker.state
This will save the set on the filesystem.
References:
Celery source code of the in-memory set
Revoke doc
Persistent revokes docs
Related
I have been trying to revoke some Celery tasks by using
app.control.revoke(task_id)
As I am currently using gevent pool, I do understand that you would not be able to revoke a task that is currently executing as seen here.
However, it seems like I can't revoke tasks that are queued as well (base on what I can see from rabbitmq, these tasks are unacked). Do correct me if I am wrong, these queued tasks have yet to be executed and it should have nothing to do with the execution pool. How revoke works is that workers will have a copy of that task_id that is supposed to be revoked, save it (persistent revoke), and when the task reaches the worker, the worker will skip it.
I am using docker to coordinate everything and have included --statedb=/celery/celery.state . This file exists in the directory of the docker worker container so that should be fine.
I am using Celery with a RabbitMQ server. I have a publisher, which could potentially be terminated by a SIGKILL and since this signal cannot be watched, I cannot revoke the tasks. What would be a common approach to revoke the tasks where the publisher is not alive anymore?
I experimented with an interval on the worker side, but the publisher is obviously not registered as a worker, so I don't know how I can detect a timeout
There's nothing built-in to celery to monitor the producer / publisher status -- only the worker / consumer status. There are other alternatives that you can consider, for example by using a redis expiring key that has to be updated periodically by the publisher that can serve as a proxy for whether a publisher is alive. And then in the task checking to see if the flag for a publisher still exists within redis, and if it doesn't the task returns doing nothing.
I am pretty sure what you want is not possible with Celery, so I suggest you to shift your logic around and redesign everything to be part of a Celery workflow (or several Celery canvases depends on the actual use-case). My experience with Celery is that you can build literally any workflow you can imagine with those Celery primitives and/or custom Celery signatures.
Another solution, which works in my case, is to add the next task only if the current processed ones are finished. In this case the queue doesn't fill up.
I have different celery queues and at a certain point I want workers to stop consuming from my queues
celery_app.control.cancel_consumer(consumer_queue)
In a while I want to be able to resume consumers and I do that with the next command
celery.control.add_consumer(
consumer_queue,
routing_key=consumer_queue,
destination=['worker-name'],
)
At this point I expect worker-name will be fetching tasks from consumer_queue which my custom router redirects by a routing_key. But instead I have this output from a celery inspect
celery.control.inspect().active_queues()
{'celery#worker-name': []}
Some details
Celery: celery==3.1.23
Kombu: kombu==3.0.35
billiard: billiard==3.3.0.23
Note: adding consumer via celery flower (flower==0.8.4) works even that the command is the same.
What am I doing wrong and how to reenable consuming in a proper way?
Ok, it was a premature question with a plain solution: I have provided wrong name for the worker, instead of setting worker-name I should provide celery#worker-name identifier.
For debug purposes it's also useful to set reply=True argument
response = celery.control.add_consumer(
consumer_queue,
routing_key=consumer_queue,
destination=['celery#{}'.format(consumer)],
reply=True,
)
print(response)
and you'll see whether operation was successful or not
[{u'celery#worker-name': {u'ok': u'add consumer consumer-queue'}}]
I'm running Django, Celery and RabbitMQ. What I'm trying to achieve is to ensure, that tasks related to one user are executed in order (specifically, one at the time, I don't want task concurrency per user)
whenever new task is added for user, it should depend on the most recently added task. Additional functionality might include not adding task to queue, if task of this type is queued for this user and has not yet started.
I've done some research and:
I couldn't find a way to link newly created task with already queued one in Celery itself, chains seem to be only able to link new tasks.
I think that both functionalities are possible to implement with custom RabbitMQ message handler, though it might be hard to code after all.
I've also read about celery-tasktree and this might be an easiest way to ensure execution order, but how do I link new task with already "applied_async" task_tree or queue? Is there any way that I could implement that additional no-duplicate functionality using this package?
Edit: There is this also this "lock" example in celery cookbook and as the concept is fine, I can't see a possible way to make it work as intended in my case - simply if I can't acquire lock for user, task would have to be retried, but this means pushing it to the end of queue.
What would be the best course of action here?
If you configure the celery workers so that they can only execute one task at a time (see worker_concurrency setting), then you could enforce the concurrency that you need on a per user basis. Using a method like
NUMBER_OF_CELERY_WORKERS = 10
def get_task_queue_for_user(user):
return "user_queue_{}".format(user.id % NUMBER_OF_CELERY_WORKERS)
to get the task queue based on the user id, every task will be assigned to the same queue for each user. The workers would need to be configured to only consume tasks from a single task queue.
It would play out like this:
User 49 triggers a task
The task is sent to user_queue_9
When the one and only celery worker that is listening to user_queue_9 is ready to consume a new task, the task is executed
This is a hacky answer though, because
requiring just a single celery worker for each queue is a brittle system -- if the celery worker stops, the whole queue stops
the workers are running inefficiently
I have a reminder type app that schedules tasks in celery using the "eta" argument. If the parameters in the reminder object changes (e.g. time of reminder), then I revoke the task previously sent and queue a new task.
I was wondering if there's any good way of keeping track of revoked tasks across celeryd restarts. I'd like to have the ability to scale celeryd processes up/down on the fly, and it seems that any celeryd processes started after the revoke command was sent will still execute that task.
One way of doing it is to keep a list of revoked task ids, but this method will result in the list growing arbitrarily. Pruning this list requires guarantees that the task is no longer in the RabbitMQ queue, which doesn't seem to be possible.
I've also tried using a shared --statedb file for each of the celeryd workers, but it seems that the statedb file is only updated on termination of the workers and thus not suitable for what I would like to accomplish.
Thanks in advance!
Interesting problem, I think it should be easy to solve using broadcast commands.
If when a new worker starts up it requests all the other workers to dump its revoked
tasks to the new worker. Adding two new remote control commands,
you can easily add new commands by using #Panel.register,
Module control.py:
from celery.worker import state
from celery.worker.control import Panel
#Panel.register
def bulk_revoke(panel, ids):
state.revoked.update(ids)
#Panel.register
def broadcast_revokes(panel, destination):
panel.app.control.broadcast("bulk_revoke", arguments={
"ids": list(state.revoked)},
destination=destination)
Add it to CELERY_IMPORTS:
CELERY_IMPORTS = ("control", )
The only missing problem now is to connect it so that the new worker
triggers broadcast_revokes at startup. I guess you could use the worker_ready
signal for this:
from celery import current_app as celery
from celery.signals import worker_ready
def request_revokes_at_startup(sender=None, **kwargs):
celery.control.broadcast("broadcast_revokes",
destination=sender.hostname)
I had to do something similar in my project and used celerycam with django-admin-monitor. The monitor takes a snapshot of tasks and saves them in the database periodically. And there is a nice user interface to browse and check the status of all tasks. And you can even use it even if your project is not Django based.
I implemented something similar to this some time ago, and the solution I came up with was very similar to yours.
The way I solved this problem was to have the worker fetch the Task object from the database when the job ran (by passing it the primary key, as the documentation recommends). In your case, before the reminder is sent the worker should perform a check to ensure that the task is "ready" to be run. If not, it should simply return without doing any work (assuming that the ETA has changed and another worker will pick up the new job).