I have a message queue using ActiveMQ. A web request puts messages into the queue with persistency=True. Now, I have 2 consumers that are both connected as separate sessions to this queue. Consumer 1 always acknowledges the message, but consumer 2 never does.
Now, I read this http://activemq.apache.org/how-does-a-queue-compare-to-a-topic.html which states:
A JMS Queue implements load balancer semantics. A single message will
be received by exactly one consumer. If there are no consumers
available at the time the message is sent it will be kept until a
consumer is available that can process the message. If a consumer
receives a message and does not acknowledge it before closing then the
message will be redelivered to another consumer. A queue can have many
consumers with messages load balanced across the available consumers.
What I understand from this is that I would expect all messages to eventually be processed by consumer 1 since it always acknowledges. Since consumer 2 does not acknowledge, the message should then get sent to consumer 1.
But what I am noticing is the following:
1. When I submit a request, I see only every 2nd request coming to consumer 1. The other request does not show up and it stored in ActiveMQ. I suppose it went to consumer 2 who did not acknowledge. So should it come to consumer 1 next?
I just need to ensure that the message gets processed by one consumer only. In my case, this consumer is a machine in country (site) X. Each message needs to be handled in only one country (machine). But all countries (machines) should received the message. If the country id matches in the message, it will acknowledge. So only 1 acknowledgement/message will be sent.
My code to receive/process messages looks like this:
# --------------------------------------------- MODULE IMPORT ---------------------------------------------------------#
import argparse
import json
import logging
import multiprocessing as mp
import sys
import stomp
from tvpv_portal.services.msgbkr import MsgBkr
from utils import util
# --------------------------------------------- DEVELOPMENT CODE ------------------------------------------------------#
log = logging.getLogger(__name__)
class MessageProcessingListener(stomp.ConnectionListener):
"""This class is responsible for processing (consuming) the messages from ActiveMQ."""
def __init__(self, conn, cb):
"""Initialization.
Args:
conn -- Connection object
cb -- Callback function
"""
self._conn = conn
self._cb = cb
def on_error(self, headers, body):
"""When we get an error.
Args:
headers -- Message header
body -- Message body
"""
log.error('Received error=%s', body)
def on_message(self, headers, body):
"""When we receive a message.
Args:
headers -- Message header
body -- Message body
"""
log.info('Received message')
# Deserialize the message.
item = json.loads(body)
import pprint
pprint.pprint(item)
# TODO: check if msg is to be handled by this SITE. If so, acknowledge and queue it. Otherwise, ignore.
# Put message into queue via callback (queue.put) function.
#self._cb(item)
# TODO: we only send acknowledge if we are supposed to process this message.
# Send acknowledgement to ActiveMQ indicating message is consumed.
self._conn.ack(headers['message-id'], headers['subscription'])
def worker(q):
"""Worker to retrieve item from queue and process it.
Args:
q -- Queue
"""
# Run in an infinite loop. Get an item from the queue to process it. We MUST call q.task_done() to indicate
# that item is processed to prevent deadlock.
while True:
try:
item = q.get()
# TODO: We will call external script from here to run on Netbatch in the future.
log.info('Processed message')
finally:
q.task_done()
def flash_mq_rst_handler_main():
"""Main entry to the request handler."""
# Define arguments.
parser = argparse.ArgumentParser(description='Flash message queue request handler script',
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
add_help=False)
opts = parser.add_argument_group('Options')
opts.add_argument('-h', '--help', action='help',
help='Show this help message and exit')
opts.add_argument('--workers', metavar='val', type=int, default=4,
help='Number of worker processes')
opts.add_argument('--log', metavar='file', type=util.get_resolved_abspath, default='flash_mq_rst_handler.log',
help='Log file')
# Parse arguments.
args = parser.parse_args()
# Setup logger.
util.configure_logger(args.log)
log.info('Command line %s', ' '.join(map(str, sys.argv)))
# Create a managed queue to store messages retrieved from message queue.
queue = mp.Manager().JoinableQueue()
# Instantiate consumer message broker + ensure connection.
consumer = MsgBkr(producer=False)
if not consumer.is_connected():
log.critical('Unable to connect to message queue; please debug')
sys.exit(1)
# Register listener with consumer + queue.put as the callback function to trigger when a message is received.
consumer.set_listener('message_processing_listener', MessageProcessingListener, cb=queue.put)
# Run in an infinite loop to wait form messages.
try:
log.info('Create pool with worker=%d to process messages', args.workers)
with mp.Pool(processes=args.workers) as pool:
p = pool.apply_async(worker, (queue,))
p.get()
except KeyboardInterrupt:
pass
# See MsgBkr. It will close the connection during exit() so we don't have to do it.
sys.exit(0)
if __name__ == '__main__':
flash_mq_rst_handler_main()
This got addressed with the JMS bridge: https://activemq.apache.org/components/artemis/documentation/1.1.0/jms-bridge.html
Was able to get IT to configure create N+1 queues. The source (incoming) queue is where all messages are put. Based on some selector in the message (like 'some_key': 'some_value' in the header), the message can be routed to one of the N destination (outgoing) queues. Then, each site can listen to a particular queue for messages. Multiple consumers on the same destination queue will get messages in a round-robin fashion.
Related
This is an update to my previous question. I realized that I should have added some code to explain my issue further. I am currently trying to implement threading to queues being consumed from a rabbitmq exchange. As I am new to rabbitmq and threading, I am finding it difficult to amalgamate both and apply them. I was wondering if anyone could provide any templates I could apply to begin with.
I am coding in visual studio platform, where a simulator is being used to generate data, emulating the producer(a smart device).
The first part of the code assigns a few variables relevant to the project. I have imported the required libraries and a few extra scripts I have made myself. The import inthe second line are scripts that assist with communicating with the smart device.
import pika, sys, os
import SockAlertMessage_pb2, SockDataProcessedMessage_pb2, SockDataRawMessage_pb2, SockDataSessionEndMessage_pb2, SockDataSessionStartMessage_pb2, SockMessage_pb2
import numpy as np
import scipy
import ampd
import python_file_3
import heartpy
import time
import threading
from scipy.signal import detrend
from python_file_3 import filter_signal, get_hrv, get_rmssd, get_std, heart_rate
from ampd import find_peaks_originalode here
The second part of the code
sock_data_session_start_queue = 'sock_data_session_start_queue'
sock_data_session_end_queue= 'sock_data_session_end_queue'
sock_data_raw_queue = 'sock_data_raw_queue'
tx_queue = 'tbd'
# establish connection with rabbitmq server
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
# create rx message queues
channel.queue_declare(queue=sock_data_session_start_queue, durable = True)
channel.queue_declare(queue=sock_data_session_end_queue, durable = True)
# create tx message queue
channel.queue_declare(queue=tx_queue, durable = True)e here
def sock_data_session_start_callback(ch, method, properties, body):
message = SockDataSessionStartMessage_pb2.SockDataSessionStartMessage()
message.ParseFromString(body)
new_dict['1'] = message
print(new_dict)
# todo: create thread w/ state
print(" [x] Start session %r" % message)
# send message
def sock_data_session_end_callback(ch, method, properties, body):
message = SockDataSessionEndMessage_pb2.SockDataSessionEndMessage()
message.ParseFromString(body)
# todo: destroy thread w/ state
print(" [x] End session %r" % message)
def sock_data_raw_callback(ch, method, properties, body):
message = SockDataRawMessage_pb2.SockDataRawMessage()
message.ParseFromString(body)
print(message)
# todo: destroy thread w/ state
print(" [x] Sock data raw %r" % message)
if __name__ == '__main__':
try:
channel.basic_consume(queue=sock_data_session_start_queue, auto_ack=True, on_message_callback=sock_data_session_start_callback)
channel.basic_consume(queue=sock_data_session_end_queue, auto_ack=True, on_message_callback=sock_data_session_end_callback)
channel.basic_consume(queue=sock_data_raw_queue, auto_ack=True, on_message_callback=sock_data_raw_callback)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
except KeyboardInterrupt:
print('Interrupted')
# close connection
connection.close()
try:
sys.exit(0)
except SystemExit:
os._exit(0)
The start and end data callbacks refer to data sessions, where data session connection is acknowledged and started and then later ended. I believe the raw data callback is where I may need to implement my threads, where data will be processed and then sent back to another queue. The challenge is to make each data session a thread, and then process that.
I am trying to fetch messages from a consumer and send it to a queue. For this I am using Stomp.py After going through articles and posts, I wrote below code:
import ssl
import stomp
stompurl = "xxxxxxxx.mq.us-west-2.amazonaws.com"
stompuser = "stomuser"
stomppass = "password"
class MyListener(stomp.ConnectionListener):
msg_list = []
def __init__(self):
self.msg_list = []
def on_error(self, frame):
self.msg_list.append('(ERROR) ' + frame.body)
def on_message(self, frame):
self.msg_list.append(frame.body)
conn = stomp.Connection(host_and_ports=[(stompurl, "61614")], auto_decode=True)
conn.set_ssl(for_hosts=[(stompurl, "61614")], ssl_version=ssl.PROTOCOL_TLS)
lst = MyListener()
listener = conn.set_listener('', lst)
conn.connect(stompuser, stomppass, wait=True)
# conn.send(body='Test message', destination='Test_QUEUE')
conn.subscribe('Test_QUEUE', '102')
print(listener.message_list)
import time; time.sleep(2)
messages = lst.msg_list
# conn.disconnect()
print(messages)
With this code I am able to send messages to Test_QUEUE but I can't fetch all messages from consumer. How can I pull out all messages from a consumer and post to a queue for processing.
I'm not a Python + STOMP expert, but in every other language I've used when you create an asynchronous (i.e. non-blocking) message listener as you have done then you must prevent your application from exiting. You have a time.sleep(2) in there, but is that realistically enough time to fetch all the messages from the queue?
It appears your application will exit after print(messages) which means that if you don't get all the messages during the time.sleep(2) then your application will simply terminate.
I've been looking for resources but I can't seem to find what I need.. I have an Azure function with a Service Bus trigger. From this, I make an HTTP call with one of the values found in the Service Bus message.
An additional requirement for me is to deadletter a message if it the HTTP call fails. But as I understand it, the message is not present in the subscription anymore because it was properly received. Is there a way for me to keep the message in the subscription, and then dispose it once it is successful (transfer to DLQ if not?)
I found this piece of code but I'm not sure how it's sending to the DLQ?
https://github.com/Azure/azure-sdk-for-python/blob/azure-servicebus_7.3.0/sdk/servicebus/azure-servicebus/samples/sync_samples/receive_deadlettered_messages.py
"""
Example to show receiving dead-lettered messages from a Service Bus Queue.
"""
# pylint: disable=C0111
import os
from azure.servicebus import ServiceBusClient, ServiceBusMessage, ServiceBusSubQueue
CONNECTION_STR = os.environ['SERVICE_BUS_CONNECTION_STR']
QUEUE_NAME = os.environ["SERVICE_BUS_QUEUE_NAME"]
servicebus_client = ServiceBusClient.from_connection_string(conn_str=CONNECTION_STR)
with servicebus_client:
sender = servicebus_client.get_queue_sender(queue_name=QUEUE_NAME)
messages = [ServiceBusMessage("Message to be deadlettered") for _ in range(10)]
with sender:
sender.send_messages(messages)
print('dead lettering messages')
receiver = servicebus_client.get_queue_receiver(queue_name=QUEUE_NAME)
with receiver:
received_msgs = receiver.receive_messages(max_message_count=10, max_wait_time=5)
for msg in received_msgs:
print(str(msg))
receiver.dead_letter_message(msg)
print('receiving deadlettered messages')
dlq_receiver = servicebus_client.get_queue_receiver(queue_name=QUEUE_NAME, sub_queue=ServiceBusSubQueue.DEAD_LETTER)
with dlq_receiver:
received_msgs = dlq_receiver.receive_messages(max_message_count=10, max_wait_time=5)
for msg in received_msgs:
print(str(msg))
dlq_receiver.complete_message(msg)
print("Receive is done.")
Here is a code snippet in mine:
async def main(msg: func.ServiceBusMessage):
try:
logging.info('Python ServiceBus queue trigger processed message: %s',
msg.get_body().decode('utf-8'))
await asyncio.gather(wait(), wait())
result = json.dumps({
'message_id': msg.message_id,
'metadata' : msg.metadata
})
msgobj = json.loads(result)
val = msgobj['metadata']['value']
run_pipeline(val, msg)
except Exception as e:
logging.error(f"trigger failed: {e}")
TLDR; How do I keep the message in the subscription and either dispose them (if successful) or send them to the DLQ if not?
The code that you pasted is to Recieve Deadletter Messages from the deadletter queue.
I found some code in the docs. You can use this snippet from their example
from azure.servicebus import ServiceBusClient
import os
connstr = os.environ['SERVICE_BUS_CONNECTION_STR']
queue_name = os.environ['SERVICE_BUS_QUEUE_NAME']
with ServiceBusClient.from_connection_string(connstr) as client:
with client.get_queue_receiver(queue_name) as receiver:
for msg in receiver:
print(str(msg))
receiver.dead_letter_message(msg)
You can look at using this above code in your Exception handler
There're four methods to settle a message after receipt:
Complete:
Declares the message processing to be successfully completed, removing the message from the queue.
receiver.complete_message(msg)
Abandon:
Abandon processing of the message for the time being, returning the message immediately back to the queue to be picked up by another (or the same) receiver.
receiver.abandon_message(msg)
DeadLetter:
Transfer the message from the primary queue into the DQL.
receiver.dead_letter_message(msg)
Defer:
Defer is subtly different from the prior settlement methods. It prevents the message from being directly received from the queue by setting it aside.
receiver.defer_message(msg)
To answer your question "How do I keep the message in the subscription and either dispose them (if successful) or send them to the DLQ if not?":
keep the message in the subscription: use abandon_message
dispose them (if successful): use complete_message
send them to the DLQ: use dead_letter_message
I'm experimenting with creating a combination of the topic exchange mentioned in tutorial #5 and RPC mentioned in tutorial #6, and while it works once, it doesn't work again unless I restart the consumer code.
In the client code, which runs on the machine with the RabbitMQ server, I have register_request() which receives a message (from a higher level made with Flask) and adds it to the exchange based on a routing key, and then waits for a response. The callback reply_queue_callback() adds responses to a dictionary where the keys are the correlation ID.
class QueueManager(object):
def __init__(self):
"""
Initializes an exchange and a reply queue.
"""
self.responses = {}
self.connection = pika.BlockingConnection(pika.ConnectionParameters(host="localhost"))
atexit.register(self.close_connection)
self.channel = self.connection.channel()
self.channel.exchange_declare(exchange=EXCHANGE_NAME, type="topic")
result = self.channel.queue_declare(exclusive=True)
self.reply_queue = result.method.queue
self.channel.basic_consume(self.reply_queue_callback, no_ack=True, queue=self.reply_queue)
def close_connection(self):
"""
Closes the connection to RabbitMQ. Runs upon destruction of the instance.
"""
print "*** Closing queue connection..."
self.connection.close()
def reply_queue_callback(self, ch, method, props, body):
"""
A callback that is executed when there's a new message in the reply queue.
"""
self.responses[props.correlation_id] = literal_eval(body)
def register_request(self, routing_key, message):
"""
Adds a message to the exchange.
"""
corr_id = str(uuid.uuid4())
self.channel.basic_publish(exchange=EXCHANGE_NAME, routing_key=routing_key,
properties=pika.BasicProperties(
reply_to=self.reply_queue,
correlation_id=corr_id),
body=message)
print "*** Sent request with correlation ID", corr_id
return corr_id
def fetch_response(self, corr_id):
"""
A polling function that waits for a message in the reply queue.
"""
print "Waiting for a response..."
while not self.responses.get(corr_id):
self.connection.process_data_events()
return self.responses.pop(corr_id)
In the consumer's code, which runs on a separate machine, receive_requests() is the main function and request_callback() is the callback function for a new message.
def request_callback(ch, method, props, message):
"""
A callback that is executed when a relevant message is found in the exchange.
"""
print "Pulled a request with correlation ID %s" % props.correlation_id
response = produce_response(message)
print "Produced a response, publishing..."
ch.basic_publish(exchange="",
routing_key=props.reply_to,
properties=pika.BasicProperties(correlation_id=props.correlation_id),
body=response)
ch.basic_ack(delivery_tag=method.delivery_tag)
print " [*] Waiting for new messages\n"
def receive_requests():
"""
The main loop. Opens a connection to the RabbitMQ server and consumes messages from the exchange.
"""
connection = pika.BlockingConnection(pika.ConnectionParameters(host=RABBITMQ_IP))
channel = connection.channel()
channel.exchange_declare(exchange=EXCHANGE_NAME, type="topic")
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
for binding_key in BINDING_KEYS:
channel.queue_bind(exchange=EXCHANGE_NAME, queue=queue_name, routing_key=binding_key)
channel.basic_consume(request_callback, queue=queue_name, no_ack=True)
try:
print(" [*] Waiting for messages. To exit press CTRL+C\n")
channel.start_consuming()
except KeyboardInterrupt:
print "Aborting..."
When I produce a message the first time, the consumer handles it and I get a response back, but with the second message it seems that nothing reaches the consumer (the client prints that it added the new message to the exchange, but the consumer doesn't print anything). I assume something's wrong with the consumer, because if after the first message I restart the consumer's code and keep the client running as is, a second message works fine.
Any idea what the problem is? Perhaps I'm missing something in the consumer's callback?
I would like to send a message (directly) from a script and than process it, and send back the results.
So it's like a double publish-subscribe.
I have 2 scripts:
Processer
Client
The Client sends a message directly (simple string) to the Processer, and than the Processer script counts the characters in the string and sends back the results to the client.
This is how I tried to do:
The Processer waits for a message, calculates something and than answers back to the original sender.
#Processer.py:
import pika
import sys
#Sends back the score
#addr: Connection address
#exchName: Exchange name (where to send)
#rKey: Name of the queue for direct messages
#score: The detected score
def SendActualScore(addr, exchName, rKey, score):
#Send the image thru the created channel with the given routing key (queue name)
channel.basic_publish(exchange=exchName, routing_key=rKey, body=score)
print "(*) Sent: " + score
#When we receive something this is called
def CallbackImg(ch, method, properties, body):
print "(*) Received: " + str(body)
score = str(len(body))
#Send back the score
SendActualScore('localhost', 'valami', rKey, score)
#Subscribe connection
#Receive messages thru this
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
#RECEIVE MESSAGES - Subscribe
channel.exchange_declare(exchange='valami', type='direct')
#Define a queue, where we don't need the name
#After we disconnected delete the queue (exclusive flag)
result = channel.queue_declare(exclusive=True)
#We need the name of our temporary queue
queue_name = result.method.queue
rKeys = sys.argv[1:]
for rKey in rKeys:
channel.queue_bind(exchange='valami', queue=queue_name, routing_key = rKey)
channel.basic_consume(CallbackImg, queue=queue_name, no_ack=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
The Client just sends the message and than waits for the answer.
#Client.py:
import pika
import sys
connAddr = 'localhost'
#Establish connection
connection = pika.BlockingConnection(pika.ConnectionParameters(connAddr))
channel = connection.channel()
#Define an exchange channel, we don't need a queue
channel.exchange_declare(exchange='valami', type='direct')
#Send the image thru the created channel
channel.basic_publish(exchange='valami', routing_key='msg', body='Message in the body')
print "[*] Sent"
def Callback(ch, method, properties, body):
print "(*) Received: " + str(body)
result = channel.queue_declare(exclusive=True)
#We need the name of our temporary queue
queue_name = result.method.queue
channel.queue_bind(exchange='valami', queue=queue_name)
channel.basic_consume(Callback, queue=queue_name, no_ack=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
There could be multiple Clients and I don't know how to send back the messages directly to them.
Have you checked the tutorials for RPC in RabbitMQ w/ python and pika? http://www.rabbitmq.com/tutorials/tutorial-six-python.html
The gist of what you need to do in your client, is found in the RPC tutorial, but with a few modifications.
In your client, you will need to create an exclusive queue - the same way you did in your server.
When you send your message from the client, you need to set the reply_to to the name of the client's exclusive queue
from the tutorial:
channel.basic_publish(exchange='',
routing_key='rpc_queue',
properties=pika.BasicProperties(
reply_to = callback_queue,
),
body=request)
On the server, when you receive a message, you need to read the reply_to header from the message and then basic_publish the reply to that queue.
Rather than thinking about "client" and "server", it may be helpful to frame this in terms of "message producer" and "message consumer".
In your scenario, you need both of your processes to be both a publisher and consumer. The "client" will publish the original message and consume the response. The "server" will consume the original message and publish a response.
The only real difference in your code will be the use of the reply_to header on the original message. This is the name of the queue to which you should publish the response.
Hope that helps!
P.S. I cover the core outline of this in my RabbitMQ Patterns eBook - both RPC and request / reply like you are needing. The book talks in principles and patterns, not in specific programming language (though I mostly write node.js and don't really know python).