Have following chunked tasks
update_matching_product.chunks(update_matching_products, 10).apply_async(priority=5)
When inside update_matching_product raise self.retry() the chunked task never retry. Instead of that shows Task can be retried
If we open celery code we see:
if request.called_directly:
# raises orig stack if PyErr_Occurred,
# and augments with exc' if that argument is defined.
raise_with_context(exc or Retry('Task can be retried', None))
So question - why chanked tasks called_directly, and how to retry failed chunked tasks?
PS: Celery is not in task_always_eager mode
Related
I have a Python RQ job that downloads a resource from a webserver.
In case of a non-responding webserver, can the download-job reschedule itself and retry the download after a certain interval?
Several transformation-jobs depend on the download-job via
job_queue.enqueue(transformation_task, depends_on=download_job)
If the download-job could reschedule itself, are the dependent jobs kept along, and would finally execute, once the download-job finishes?
I asked the question on the python-rq GitHub project and the functionality is now included in version 1.5.0 of RQ.
RQ lets you now easily retry failed jobs. To configure retries, use RQ’s Retry object that accepts max and interval arguments.
Dependent jobs are kept in the deferred job registry until the job they depend upon succeeds and are executed only then.
For example:
from redis import Redis
from rq import Queue, Retry
from somewhere import randomly_failing_task, dependent_task
job_queue = Queue(connection=Redis())
randomly_failing_job = job_queue.enqueue(randomly_failing_task, retry=Retry(max=3))
dependent_job = job_queue.enqueue(dependent_task, depends_on=randomly_failing_job)
And the sample tasks:
from random import choice
def randomly_failing_task():
print('I am a task, I will fail 50% of the times :/')
success = choice([True, False])
if success:
print('I succeed :)')
else:
print('I failed :(')
raise Exception('randomly_failing_task failed!')
def dependent_task():
print('I depend upon the randomly_failing_task.')
print('I am only executed, once the randomly_failing_task succeeded.’)
I have a client celery application issuing task for a worker (using Redis), it's working ok. Both client and worker applications uses the same config :
app = Celery('clientApp', broker='redis://redis:6379/0',backend='redis://redis:6379/0')
# Listen to queue2
app = Celery('workerApp', broker='redis://redis:6379/0',backend='redis://redis:6379/0')
# Listen to queue1
Now I want to execute handler on success or error, so I used something like this :
task = Signature('mytask', queue='queue1')
task.apply_async(
link=Signature("handle_success", queue='queue2'),
link_error=Signature("handle_error", queue='queue2'))
This call handle_success correctly on success but do not call handle_error when mytask raise an Exception. Can you see any reason why ? The goal here would be that the client to execute handle_error on task failed by the worker (like it does execute handle_sucess when the worker task complete sucessfully).
celery.exceptions.NotRegistered: 'handle_error'
I have no error or info messages when celery applications starts, backend is the same url for both apps and handle_success / handle_error correctly shows up in registered tasks for the client.
Resolved by using the hack listed in this article.
If the link_error argument is a single task, it will get executed by the worker directly (unlike link), one way to force the worker to send the task to the client is to use a chain.
from .tasks import error_callback
app.send_task("system_b.foo", link_error=(error_callback.si() | error_callback.si())
Or use a dummy task plus helper functions to make it more clear, see article.
I had in the back of my mind the impression that if a celery worker gets a task , and it is retried - it remains in the worker's memory (with the eta) - and doesn't return to the queue.
resulting in that if a celery task is retried and the worker is busy working on different tasks , and that task eta arrives- it has to wait until it finishes processing the other tasks.
I tried looking in the documentation for something that is aligned with what I remembered , but I can't find anything.
what I did to try and check it is create two tasks.
#app.task(bind=True, name='task_that_holds_worker', rate_limit='4/m',
default_retry_delay=5 * 60,
max_retries=int(60 * 60 * 24 * 1 / (60 * 5)))
def task_that_holds_worker(self, *args, **kwargs):
import time
time.sleep(50000)
#app.task(bind=True, name='retried_task', rate_limit='2/m',
default_retry_delay=10 * 60,
max_retries=int(60 * 60 * 24 * 1 / (60 * 10)))
def retried_task(self, *args, **kwargs):
self.retry()
the simplest tasks , just to check that if a task is busy with other task - the retried task is not processed by another worker.
I then launched one worker - and triggered those two tasks in the following way:
from some_app import tasks
from some_app.celery_app import app
current_app = app.tasks
async_result = tasks.retried_task.delay()
import time
time.sleep(20)
async_result = tasks.task_that_holds_worker.delay()
the worker processed the retried task , and retried it,
and then moved to the task that sleeps.
I then launched another worker and i can see that it is not getting the 'retried' task, only the first worker.
each worker launched was launced with --prefetch-multiplier=1 --concurrency=1
Is there something wrong with the way I reproduced this?
or is this the way a celery retried task behaves?
Thanks in advance!
celery: 4.1.2
Python: 3.6.2
Rabbitmq Image: rabbitmq:3.6.9-management
Seems like this is an issue with tasks with eta. the first available worker counts down until the task eta and doesn't release it back to the queue. (prefetch count is increased and ignored)
https://github.com/celery/celery/issues/2541
There is an error with how you reproduced it. Unless you have a special broker, celery will always requeue a task retry request back to the broker. Workers do not retain any memory of which task they attempted, and there is no data added to the retry request that allows celery to route the task request back to the same worker. There is no guarantee or assurance that the same worker will retry a task that it has seen before. You can confirm this in the code for celery in celery/app.task.py
# get the signature of the task as called
S = self.signature_from_request(
request, args, kwargs,
countdown=countdown, eta=eta, retries=retries,
**options
)
if max_retries is not None and retries > max_retries:
if exc:
# On Py3: will augment any current exception with
# the exc' argument provided (raise exc from orig)
raise_with_context(exc)
raise self.MaxRetriesExceededError(
"Can't retry {0}[{1}] args:{2} kwargs:{3}".format(
self.name, request.id, S.args, S.kwargs))
ret = Retry(exc=exc, when=eta or countdown)
if is_eager:
# if task was executed eagerly using apply(),
# then the retry must also be executed eagerly.
S.apply().get()
if throw:
raise ret
return ret
try:
S.apply_async()
except Exception as exc:
raise Reject(exc, requeue=False)
if throw:
raise ret
return ret
I've bolded the part where you can see how the retry works. Celery gets the tasks request signature (this include the task name, and the arguments to the task, and sets the eta, countdown, and retries). And then celery will simply call apply_async, which under the hood will just queue up a new task request to the broker.
Your sample did not work because celery workers will often pull more than one task request off of the broker, so what likely happened is that the first worker grabbed the task off of the broker before the second worker had come online.
I have the following scripts:
celery_tasks.py
from celery import Celery
app = Celery(broker='amqp://guest:guest#localhost:5672//')
app.conf.task_default_queue = 'test_queue'
#app.task(acks_late=True)
def test(a):
return a
publish.py
from celery_tasks import test
test.delay('abc')
When i run publish.py and start the worker (celery -A celery_tasks worker --loglevel=DEBUG), the 'abc' content is published in the 'test_queue' and is consumed by the worker.
Is there a way for the worker to consume something from a queue that was not posted by Celery? For example, when I put something in the test_queue straight through RabbitMQ, without going through the Celery publisher, and run the Celery worker, it gave me the following warning:
WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: 'abc' (3b)
{content_type:None content_encoding:None
delivery_info:{'exchange': '', 'redelivered': False, 'delivery_tag': 1, 'consumer_tag': 'None2', 'routing_key': 'test_queue'} headers={}}
Is there a way to solve this?
Celery has a specific format and a set of headers that needs to be maintained to comply with it. Therefore you would have to reverse engineer it to make celery-compliant message not produced by celery.
Keep in mind that celery is not really made to send messages across the broker, but to send tasks, which are enhanced messages therefore have extras in the header part of the amqp message
It's a late answer but custom consumers might help you. I'm using this for consuming messages from rabbitmq. Where these messages are being populated from another app with pika.
http://docs.celeryproject.org/en/latest/userguide/extending.html#custom-message-consumers
I'm using Python 2.7 (sigh), celery==3.1.19, librabbitmq==1.6.1, rabbitmq-server-3.5.6-1.noarch, and redis 2.8.24 (from redis-cli info).
I'm attempting to send a message from a celery producer to a celery consumer, and obtain the result back in the producer. There is 1 producer and 1 consumer, but 2 rabbitmq's (as brokers) and 1 redis (for results) in between.
The problem I'm facing is:
In the consumer, I get back get an AsyncResult via async_result =
ZipUp.delay(unique_directory), but async_result.ready() never
returns True (at least for 9 seconds it doesn't) - even for a
consumer task that does essentially nothing but return a string.
I can see, in the rabbitmq management web interface, my message
being received by the rabbitmq exchange, but it doesn't show up in
the corresponding rabbitmq queue. Also, a log message sent by the
very beginning of the ZipUp task doesn't appear to be getting
logged.
Things work if I don't try to get a result back from the AsyncResult! But I'm kinda hoping to get the result of the call - it's useful :).
Below are configuration specifics.
We're setting up Celery as follows for returns:
CELERY_RESULT_BACKEND = 'redis://%s' % _SHARED_WRITE_CACHE_HOST_INTERNAL
CELERY_RESULT = Celery('TEST', broker=CELERY_BROKER)
CELERY_RESULT.conf.update(
BROKER_HEARTBEAT=60,
CELERY_RESULT_BACKEND=CELERY_RESULT_BACKEND,
CELERY_TASK_RESULT_EXPIRES=100,
CELERY_IGNORE_RESULT=False,
CELERY_RESULT_PERSISTENT=False,
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
)
We have another Celery configuration that doesn't expect a return value, and that works - in the same program. It looks like:
CELERY = Celery('TEST', broker=CELERY_BROKER)
CELERY.conf.update(
BROKER_HEARTBEAT=60,
CELERY_RESULT_BACKEND=CELERY_BROKER,
CELERY_TASK_RESULT_EXPIRES=100,
CELERY_STORE_ERRORS_EVEN_IF_IGNORED=False,
CELERY_IGNORE_RESULT=True,
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
)
The celery producer's stub looks like:
#CELERY_RESULT.task(name='ZipUp', exchange='cognition.workflow.ZipUp_%s' % INTERNAL_VERSION)
def ZipUp(directory): # pylint: disable=invalid-name
""" Task stub """
_unused_directory = directory
raise NotImplementedError
It's been mentioned that using queue= instead of exchange= in this stub would be simpler. Can anyone confirm that (I googled but found exactly nothing on the topic)? Apparently you can just use queue= unless you want to use fanout or something fancy like that, since not all celery backends have the concept of an exchange.
Anyway, the celery consumer starts out with:
#task(queue='cognition.workflow.ZipUp_%s' % INTERNAL_VERSION, name='ZipUp')
#StatsInstrument('workflow.ZipUp')
def ZipUp(directory): # pylint: disable=invalid-name
'''
Zip all files in directory, password protected, and return the pathname of the new zip archive.
:param directory Directory to zip
'''
try:
LOGGER.info('zipping up {}'.format(directory))
But "zipping up" doesn't get logged anywhere. I searched every (disk-backed) file on the celery server for that string, and got two hits: /usr/bin/zip, and my celery task's code - and no log messages.
Any suggestions?
Thanks for reading!
It appears that using the following task stub in the producer solved the problem:
#CELERY_RESULT.task(name='ZipUp', queue='cognition.workflow.ZipUp_%s' % INTERNAL_VERSION)
def ZipUp(directory): # pylint: disable=invalid-name
""" Task stub """
_unused_directory = directory
raise NotImplementedError
In short, it's using queue= instead of exchange= .