We Keep Coding

PreconditionFailed with Celery and RabbitMQ

I am experiencing an issue between Celery:5.2.7 and RabbitMQ:3.11 (through Docker). My program is periodically sending tasks to Celery, like below:
from tasks import getElements
def collectElements(elements):
for el in elements:
getElements.apply_async(queue="queue_getelements",kwargs={"elDict": el.__dict__})
collectElements(elements)
time.sleep(600)
while(True):
collectElements(elements)
time.sleep(120)
Strangely, the queue queue_getelements freezes after the first launch of collectElements(elements) (After 5 minutes/600 seconds), and the message appears after 30 minutes (Which is the default consumer_timeout time):
[2022-10-05 02:45:32,706: CRITICAL/MainProcess] Unrecoverable error: PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')
I have tried to change the default consumer_timeout time in the configuration by increasing the timer (Like seen as a solution here and here), but the freezes still happens before the first loop of my program. Celery seems to receive the tasks only the first time in the queue, and freezes afterward. If I relaunch the queue after stopped it, it receives again the tasks that are waiting on RabbitMQ:
celery -A tasks worker --loglevel=info -E -Q queue_getelements -c 4
Does anyone experienced this issue before? Any help would be appreciated, thank you in advance!

Heroku with Django, Celery and CloudAMPQ - timeout error

I am building an online shop, following Chapter 7 in the book "Django 3 by Example." The book was written by Antonio Melé.
Everything works fine in my local machine. It also works well when I deploy it to Heroku.
However, when I try to use Celery and RabbitMQ (CloudAMQP, Little Lemur, on Heroku), the email message the worker was supposed to send to the customer is not sent. The task takes more then 30 seconds and then it crashes:
heroku[router]: at=error code=H12 desc="Request timeout" method=POST
I have created a tasks.py file with the task of sending emails. My settings.py file include the following lines for Celery:
broker_url = os.environ.get('CLOUDAMQP_URL')
broker_pool_limit = 1
broker_heartbeat = None
broker_connection_timeout = 30
result_backend = None
event_queue_expires = 60
worker_prefetch_multiplier = 1
worker_concurrency = 50
This was taken from https://www.cloudamqp.com/docs/celery.html
And my Procfile is as follows,
web: gunicorn shop.wsgi --log-file -
worker: celery worker --app=tasks.app
Am I missing something?
Thanks!

So fairly familiar with heroku, though not your tech stack. So the general approach to deal with heroku timeout is this:
First, determine exactly what is causing the timeout. One or more things are taking a lot of time.
Now you have 3 main options.
Heroku Scheduler (or one of several other similar addons). Very useful if you can run a script of some sort via a terminal command, and 10 minutes/1 hour/24 hour checks to see if the script needs to be run is good enough for you. I typically find this to be the most straightforward solution, but it's not always an acceptable one. Depending on what you are emailing, an email being delayed 5-15 minutes might be acceptable.
Background process worker. Looks like this is what you are trying to do with Celery, but it's not configured right, probably can't help much on that.
Optimize. The reason heroku sets a 30 second timeout is because generally speaking there really isn't a good reason for a user to wait 30 seconds for a response. I'm scratching my head as to why sending an email would take more than 30 seconds, unless you need to send a few hundred of them or the email is very, very large. Alternatively, you might be doing a lot of work before you send the email, though that raises the question fo why not do that work seperately from the send email command. I suspect you should probably look into the why of this before you try to get a background process worker setup.

After several days trying to solve this issue, I contacted the support department at CLOUDAMQP
They helped me figure out that the problem was related to Celery not identifying my BROKER_URL properly.
Then I came across this nice comment by #jainal09 here. There was an extra variable that should be set in settings.py:
CELERY_BROKER_URL = '<broker address given by Heroku config>'
Adding that extra line solved the problem. Now Heroku is able to send the email correctly.

Celery worker stops after being idle for a few hours

I have a Flask app that uses WSGI. For a few tasks I'm planning to use Celery with RabbitMQ. But as the title says, I am facing an issue where the Celery tasks run for a few minutes and then after a long time of inactivity it just dies off.
Celery config:
CELERY_BROKER_URL='amqp://guest:guest#localhost:5672//'
BROKER_HEARTBEAT = 10
BROKER_HEARTBEAT_CHECKRATE = 2.0
BROKER_POOL_LIMIT = None
From this question, I added BROKER_HEARTBEAT and BROKER_HEARTBEAT_CHECKRATE.
I run the worker inside the venv with celery -A acmeapp.celery worker & to run it in the background. And while checking the status, for the first few minutes, it shows that one node is online and gives an OK response. But after a few hours of the app being idle, when I check the Celery status, it shows Error: No nodes replied within time constraint..
I am new to Celery and I don't know what to do now.

Your Celery worker might be trying to reconnect to the app until it reaches the retry limit. If that is the case, setting up this options in your config file will fix that problem.
BROKER_CONNECTION_RETRY = True
BROKER_CONNECTION_MAX_RETRIES = 0
The first line will make it retry whenever it fails, and the second one will disable the retry limit.
If that solution does not suit you enough, you can also try a high timeout (specified in seconds) for your app using this option:
BROKER_CONNECTION_TIMEOUT = 120
Hope it helps!

Should RabbitMQ take this long to setup the connection?

I am trying a basic hello world example with RabbitMQ in Python, and it is taking about 8 seconds to set up a basic blocking connection. This seems excessive, but this is my first experience with RabbitMQ, so my question is: is this normal? Can I reduce this time? Or should I look for another option? Here is my code:
import time
import pika
start = time.time()
connection = pika.BlockingConnection(pika.ConnectionParameters(host="localhost"))
end = time.time()
print "Elapsed time: %s" % (end-start)
channel = connection.channel()
channel.queue_declare(queue="hello")
channel.basic_publish(exchange="",
routing_key="hello",
body="Hello world!")
connection.close()
and my output is Elapsed time: 8.01042914391.
Thanks for the help!
[Edit] I have noticed that every time I run it, it takes almost exactly 8 seconds, to within .2%. I'm not sure if that means anything.

You need to configure channel configurations for inbound channel, outbound chanel, just like thread pool executor. Default values for these threads is 1 which might cause in delay under some load.

Sleep takes 86% of time in rabbitpy consumer?

I found a rather interesting problem today. I have a queue with 2K messages in it.
The consumer: import rabbitpy
with rabbitpy.Connection('amqp://guest:guest#localhost:5672/%2f') as conn:
with conn.channel() as channel:
queue = rabbitpy.Queue(channel, 'example')
for message in queue.consume_messages():
message.ack()
takes 41 seconds to get these messages and ack them. (messages vary from 4kB to 52kB)
The publisher, however, took 15 seconds to publish them.
Upon profiling, I found that there is a call to sleep were we spend 86% of the time. This, to my application, is not acceptable. Could someone help me get rid of this sleep? (I'm ok if the CPU cycles are wasted or whatever till a message arrives.)
Zoomed in screenshot