When I listen my queue with python pika library, I always get StreamLostError and my code crushes.
In my code, I must listen the queue forever without exception and I must get messages 1 by 1.
Here is my code(I simplified it).
def callback(ch, method, properties, body):
ch.basic_ack(delivery_tag = method.delivery_tag)
#doing work here, it gets minimum 5 minutes, sometimes maximum 1 hour
credentials = pika.PlainCredentials(username, password)
parameters = pika.ConnectionParameters(ip, port, '/', credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.basic_qos(prefetch_count=1)
channel.queue_declare(queue=queuename, durable=True)
channel.basic_consume(queue=queuename, on_message_callback=callback, auto_ack=False)
channel.start_consuming()
try to set connection_attempts and retry_delay parameters in the request if you are using pika URLParameters. Look down the below link for more information.In my case I added ?connection_attempts=20&retry_delay=1 after the AMPQ https://pika.readthedocs.io/en/stable/modules/parameters.html#urlparameters
The problem is that your work takes too long and blocks Pika's I/O loop. This causes heartbeats to be missed, and RabbitMQ thinks your connection is dead.
Please see this code for one correct way to do long-running work:
https://github.com/pika/pika/blob/master/examples/basic_consumer_threaded.py
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
I'm using python with pika, and have the following two similar use cases:
Connect to RabbitMQ server A and server B (at different IP addrs with different credentials), listen on exchange A1 on server A; when a message arrives, process it and send to an exchange on server B
Open an HTTP listener and connect to RabbitMQ server B; when a specific HTTP request arrives, process it and send to an exchange on server B
Alas, in both these cases using my usual techniques, by the time I get to sending to server B the connection throws ConnectionClosed or ChannelClosed.
I assume this is the cause: while waiting on the incoming messages, the connection to server B (its "driver") is starved of CPU cycles, and it never gets a chance to service is connection socket, thus it can't respond to heartbeats from server B, thus the servers shuts down the connection.
But I can't noodle out the fix. My current work around is lame: I catch the ConnectionClosed, reopen a connection to server B, and retry sending my message.
But what is the "right" way to do this? I've considered these, but don't really feel I have all the parts to solve this:
Don't just sit forever in server A's basic_consume (my usual pattern), but rather, use a timeout, and when I catch the timeout somehow "service" heartbeats on server B's driver, before returning to a "consume with timeout"... but how do I do that? How do I "let service B's connection driver service its heartbeats"?
I know the socket library's select() call can wait for messages on several sockets and once, then service the socket who has packets waiting. So maybe this is what pika's SelectConnection is for? a) I'm not sure, this is just a hunch. b) Even if right, while I can find examples of how to create this connection, I can't find examples of how to use it to solve my multiconnection case.
Set up the the two server connections in different processes... and use Python interprocess queues to get the processed message from one process to the next. The concept is "two different RabbitMQ connections in two different processes should thus then be able to independently service their heartbeats". Except... I think this has a fatal flaw: the process with "server B" is, instead, going to be "stuck" waiting on the interprocess queue, and the same "starvation" is going to happen.
I've checked StackOverflow and Googled this for an hour last night: I can't for the life of me find a blog post or sample code for this.
Any input? Thanks a million!
I managed to work it out, basing my solution on the documentation and an answer in the pika-python Google group.
First of all, your assumption is correct — the client process that's connected to server B, responsible for publishing, cannot reply to heartbeats if it's already blocking on something else, like waiting a message from server A or blocking on an internal communication queue.
The crux of the solution is that the publisher should run as a separate thread and use BlockingConnection.process_data_events to service heartbeats and such. It looks like that method is supposed to be called in a loop that checks if the publisher still needs to run:
def run(self):
while self.is_running:
# Block at most 1 second before returning and re-checking
self.connection.process_data_events(time_limit=1)
Proof of concept
Since proving the full solution requires having two separate RabbitMQ instances running, I have put together a Git repo with an appropriate docker-compose.yml, the application code and comments to test this solution.
https://github.com/karls/rabbitmq-two-connections
Solution outline
Below is a sketch of the solution, minus imports and such. Some notable things:
Publisher runs as a separate thread
The only "work" that the publisher does is servicing heartbeats and such, via Connection.process_data_events
The publisher registers a callback whenever the consumer wants to publish a message, using Connection.add_callback_threadsafe
The consumer takes the publisher as a constructor argument so it can publish the messages it receives, but it can work via any other mechanism as long as you have a reference to an instance of Publisher
The code is taken from the linked Git repo, which is why certain details are hardcoded, e.g the queue name etc. It will work with any RabbitMQ setup needed (direct-to-queue, topic exchange, fanout, etc).
class Publisher(threading.Thread):
def __init__(
self,
connection_params: ConnectionParameters,
*args,
**kwargs,
):
super().__init__(*args, **kwargs)
self.daemon = True
self.is_running = True
self.name = "Publisher"
self.queue = "downstream_queue"
self.connection = BlockingConnection(connection_params)
self.channel = self.connection.channel()
self.channel.queue_declare(queue=self.queue, auto_delete=True)
self.channel.confirm_delivery()
def run(self):
while self.is_running:
self.connection.process_data_events(time_limit=1)
def _publish(self, message):
logger.info("Calling '_publish'")
self.channel.basic_publish("", self.queue, body=message.encode())
def publish(self, message):
logger.info("Calling 'publish'")
self.connection.add_callback_threadsafe(lambda: self._publish(message))
def stop(self):
logger.info("Stopping...")
self.is_running = False
# Call .process_data_events one more time to block
# and allow the while-loop in .run() to break.
# Otherwise the connection might be closed too early.
#
self.connection.process_data_events(time_limit=1)
if self.connection.is_open:
self.connection.close()
logger.info("Connection closed")
logger.info("Stopped")
class Consumer:
def __init__(
self,
connection_params: ConnectionParameters,
publisher: Optional["Publisher"] = None,
):
self.publisher = publisher
self.queue = "upstream_queue"
self.connection = BlockingConnection(connection_params)
self.channel = self.connection.channel()
self.channel.queue_declare(queue=self.queue, auto_delete=True)
self.channel.basic_qos(prefetch_count=1)
def start(self):
self.channel.basic_consume(
queue=self.queue, on_message_callback=self.on_message
)
try:
self.channel.start_consuming()
except KeyboardInterrupt:
logger.info("Warm shutdown requested...")
except Exception:
traceback.print_exception(*sys.exc_info())
finally:
self.stop()
def on_message(self, _channel: Channel, m, _properties, body):
try:
message = body.decode()
logger.info(f"Got: {message!r}")
if self.publisher:
self.publisher.publish(message)
else:
logger.info(f"No publisher provided, printing message: {message!r}")
self.channel.basic_ack(delivery_tag=m.delivery_tag)
except Exception:
traceback.print_exception(*sys.exc_info())
self.channel.basic_nack(delivery_tag=m.delivery_tag, requeue=False)
def stop(self):
logger.info("Stopping consuming...")
if self.connection.is_open:
logger.info("Closing connection...")
self.connection.close()
if self.publisher:
self.publisher.stop()
logger.info("Stopped")
I am trying to get message from activemq using stomp.py and then doing some processing on it. But there is a case when that processing fails for certain messages and that message is lost.
How can I prevent the deletion of message untill the message is fully processed?
For example in my code when there is new entry in queue the on_message function will be called and the processing starts but if it is interrupted in between the message is lost. How do I stop it?
Here is my code:
conn = stomp.Connection([(host, 61613)])
conn.set_listener('ML', MyListener())
conn.start()
conn.connect('admin', 'admin', wait=True)
conn.subscribe(destination=/queue/someque, id=1, ack='auto')
print "running"
while 1:
print 'waiting'
time.sleep(2.5)
Here is my Listener class:
class MyListener(stomp.ConnectionListener):
def on_message(self, headers, message):
print headers
print message
do_something()
Thanks in advance.
The issue appears to be that you are using the 'auto' ack mode so the message will be acknowledged before delivery to the client by the broker meaning that even if you fail to process it, it's to late as it is already forgotten on the broker side. You'd need to use either 'client' ack or 'client-individual' ack mode as described in the STOMP specification. Using one of the client ack modes you control when a message or messages are actually acknowledged and dropped by the broker.
I'm writing a python client to accept job messages from a RabbitMQ broker and process the jobs, returning the results to another server. My script that sends messages to the RabbitMQ broker starts up fine, but my worker throws the following error when running channel.declare_queue(queue='task_queue')
pika.exceptions.AMQPChannelError: (406, "PRECONDITION_FAILED - parameters for queue 'task_queue' in vhost '/' not equivalent")
Client:
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host=cmdargs.server))
channel = connection.channel()
channel.queue_declare(queue='task_queue')
channel.basic_qos(prefetch_count=1)
channel.basic_consume(ProcJobCallback, queue='task_queue')
channel.start_consuming()
Server method that interacts with RabbitMQ:
def addNewJob(self, newJob):
self.jobList.append(newJob)
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue')
for tile in newJob.TileStatus:
message = "{0},{1},{2}".format(newJob, tile[0], tile[1])
channel.basic_publish(exchange='', routing_key='task_queue', body=message, properties=pika.BasicProperties(delivery_mode = 2, ))
connection.close()
Any help or insight is greatly appreciated.
EDIT: I discovered why I was getting an error with the code listed above. I was specifying delivery_mode=2 when publishing my messages, but when I declared the queue, I forgot to add the Durable=True parameter.
Are you sure you are connecting to the same server (host) on the publisher and consumer side?
connection = pika.BlockingConnection(pika.ConnectionParameters(host=cmdargs.server))
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
if your queue is durable just remove the declaration "channel.queue_declare(queue='task_queue')", that should be enough in your case.
i meet the same problem when I try to make the queue msg persistent with durable=True.
Try to rename the queue name, it works well with my script. Maybe kill the queue new_task, and re-run your script also works.
Simple question, but Google or the Pika open source code did not help. Is there a way to query the current queue size (item counter) in Pika?
I know that this question is a bit old, but here is an example of doing this with pika.
Regarding AMQP and RabbitMQ, if you have already declared the queue, you can re-declare the queue with the passive flag on and keeping all other queue parameters identical. The response to this declaration declare-ok will include the number of messages in the queue.
Here is an example With pika 0.9.5:
import pika
def on_callback(msg):
print msg
params = pika.ConnectionParameters(
host='localhost',
port=5672,
credentials=pika.credentials.PlainCredentials('guest', 'guest'),
)
# Open a connection to RabbitMQ on localhost using all default parameters
connection = pika.BlockingConnection(parameters=params)
# Open the channel
channel = connection.channel()
# Declare the queue
channel.queue_declare(
callback=on_callback,
queue="test",
durable=True,
exclusive=False,
auto_delete=False
)
# ...
# Re-declare the queue with passive flag
res = channel.queue_declare(
callback=on_callback,
queue="test",
durable=True,
exclusive=False,
auto_delete=False,
passive=True
)
print 'Messages in queue %d' % res.method.message_count
This will print the following:
<Method(['frame_type=1', 'channel_number=1', "method=<Queue.DeclareOk(['queue=test', 'message_count=0', 'consumer_count=0'])>"])>
<Method(['frame_type=1', 'channel_number=1', "method=<Queue.DeclareOk(['queue=test', 'message_count=0', 'consumer_count=0'])>"])>
Messages in queue 0
You get the number of messages from the message_count member.
Here is how you can get queue length using pika(Considering you are using default user and password on localhost)
replace q_name by your queue name.
import pika
connection = pika.BlockingConnection()
channel = connection.channel()
q = channel.queue_declare(q_name)
q_len = q.method.message_count
Have you tried PyRabbit? It has a get_queue_depth() method which sounds like what you're looking for.
There are two ways to get the queue size in the AMQP protocol. You can either use Queue.Declare or Basic.Get.
If you are consuming messages as they arrive using Basic.Consume, then you can't get this info unless you disconnect (timeout) and redeclare the queue, or else get one message but don't ack it. In newer versions of AMQP you can actively requeue the message.
As for Pika, I don't know the specifics but Python clients for AMQP have been a thorn in my side. Often you will need to monkeypatch classes in order to get the info you need, or to allow a queue consumer to timeout so that you can do other things at periodic intervals like record stats or find out how many messages are in a queue.
Another way around this is to give up, and use the Pipe class to run sudo rabbitmqctl list_queues -p my_vhost. Then parse the output to find the size of all queues. If you do this you will need to configure /etc/sudoers to not ask for the usual sudo password.
I pray that someone else with more Pika experience answers this by pointing out how you can do all the things that I mentioned, in which case I will download Pika and kick the tires. But if that doesn't happen and you are having difficulty with monkeypatching the Pika code, then have a look at haigha. I found their code to be much more straightforward than other Python AMQP client libraries because they stick closer to the AMQP protocol.
I am late to the party but this is an example getting queue count using pyrabbit or pyrabbit2 from AWS AmazonMQ with HTTPS, should work on RabbitMQ as well:
from pyrabbit2.api import Client
cl = Client('b-xxxxxx.mq.ap-southeast-1.amazonaws.com', 'user', 'password', scheme='https')
if not cl.is_alive():
raise Exception("Failed to connect to rabbitmq")
for i in cl.get_all_vhosts():
print(i['name'])
queues = [q['name'] for q in cl.get_queues('/')]
print(queues)
itemCount = cl.get_queue_depth('/', 'event.stream.my-api')
print(itemCount)
Just posting this in case anyone else comes across this discussion. The answer with the most votes, i.e.:
# Re-declare the queue with passive flag
res = channel.queue_declare(
callback=on_callback,
queue="test",
durable=True,
exclusive=False,
auto_delete=False,
passive=True
)
was very helpful for me, but it comes with a serious caveat. According to the pika documentation, the passive flag is used to "Only check to see if the queue exists." As such, one would imagine you can use the queue_declare function with the passive flag to check if a queue exists in situations where there's a chance that the queue was never declared. From my testing, if you call this function with the passive flag and the queue does not exist, not only does the api throw an exception; it will also cause the broker to disconnect your channel, so even if you catch the exception gracefully, you've lost your connection to the broker. I tested this with 2 different python scripts against a plain vanilla RabbitMQ container running in minikube. I've run this test many times and I get the same behavior every time.
My test code:
import logging
import pika
logging.basicConfig(level="INFO")
logger = logging.getLogger(__name__)
logging.getLogger("pika").setLevel(logging.WARNING)
def on_callback(msg):
logger.info(f"Callback msg: {msg}")
queue_name = "testy"
credentials = pika.PlainCredentials("guest", "guest")
connection = pika.BlockingConnection(
pika.ConnectionParameters(host="localhost", port=5672, credentials=credentials)
)
logger.info("Connection established")
channel = connection.channel()
logger.info("Channel created")
channel.exchange_declare(exchange="svc-exchange", exchange_type="direct", durable=True)
response = channel.queue_declare(
queue=queue_name, durable=True, exclusive=False, auto_delete=False, passive=True
)
logger.info(f"queue_declare response: {response}")
channel.queue_delete(queue=queue_name)
connection.close()
The output:
INFO:__main__:Connection established
INFO:__main__:Channel created
WARNING:pika.channel:Received remote Channel.Close (404): "NOT_FOUND - no queue 'testy' in vhost '/'" on <Channel number=1 OPEN conn=<SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1047e2700> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>>
Traceback (most recent call last):
File "check_queue_len.py", line 29, in <module>
response = channel.queue_declare(
File "/Users/dbailey/dev/asc-service-deployment/venv/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 2521, in queue_declare
self._flush_output(declare_ok_result.is_ready)
File "/Users/dbailey/dev/asc-service-deployment/venv/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 1354, in _flush_output
raise self._closing_reason # pylint: disable=E0702
pika.exceptions.ChannelClosedByBroker: (404, "NOT_FOUND - no queue 'testy' in vhost '/'")
When I set passive to False:
scripts % python check_queue_len.py
INFO:__main__:Connection established
INFO:__main__:Channel created
INFO:__main__:queue_declare response: <METHOD(['channel_number=1', 'frame_type=1', "method=<Queue.DeclareOk(['consumer_count=0', 'message_count=0', 'queue=testy'])>"])>
Please let me know if I'm somehow missing something here.
I have my software running on a bunch of clients around my network. I've been playing around with RabbitMQ as a solution for passing messages between each client.
My test code is this:
#!/usr/bin/python2
import pika
import time
connection = pika.AsyncoreConnection(pika.ConnectionParameters(
'localhost'))
channel = connection.channel()
def callback(ch, method, properties, body):
# send messages back on certain events
if body == '5':
channel.basic_publish(exchange='',
routing_key='test',
body='works')
print body
channel.queue_declare(queue='test')
channel.basic_consume(callback, queue='test', no_ack=True)
for i in range(0, 8):
channel.basic_publish(exchange='',
routing_key='test',
body='{}'.format(i))
time.sleep(0.5)
channel.close()
Picture this as kind of a 'chat program'. Each client will need to constantly listen for messages. At times, the client will need to send messages back to the server.
This code works, but I've ran into an issue. When the code below sends out the message works, it then retreives that again from the RabbitMQ queue. Is there a way to tell have my client, a producer and a consumer, not receive the message it just sent?
I can't see this functionality built into RabbitMQ so I figured I'd send messages in the form of:
body='{"client_id" : 1, "message" : "this is the message"}'
Then I can parse that string and check the client_id. The client can then ignore all messagess not destined to it.
Is there a better way? Should I look for an alternative to RabbitMQ?
You can have as many queue in RabbitMQ. Why not have a queue for messages to the server as well as a queue for each client?