I have a Python Kafka consumer application where I consume the messages and then call an external webservice synchronously. The webservice takes a minute to process the message and send the response.
Is there a way to consume the message, send the request to the Web service and consume the next message without waiting for the response?
from kafka import KafkaConsumer
from json import loads
consumer = KafkaConsumer(
'spring_test',
bootstrap_servers=['localhost:9092'],
auto_offset_reset='earliest',
enable_auto_commit=True,
group_id='my-group',
value_deserializer=lambda x: loads(x.decode('utf-8')));
This is how I wait for the messages and send an external Web request
def consume_msgs():
for message in consumer:
message = message.value;
send('{}'.format(message))
consume_msgs()
The function send() takes one minute before I get the response. I want to start consuming the next message in the meantime asynchronously but I don't know where to start
def send(pload) :
import requests
r = requests.post('someurl',data = pload)
print(r)
Not sure if this is what you need but could you just spin each call to send out into a thread? Something like this the below. This way the for loop will continue without waiting for send to return. You may have to throttle the number of threads somehow if you are consuming data far quicker than you are processing it.
from threading import Thread
def consume_msgs():
for message in consumer:
message = message.value;
Thread(target=send, args = ('{}'.format(message),)).start()
consume_msgs()
Related
What does the channel.basic_consume' function return?
how i can access to message using variable i want consumed message and show it in browser?
i build django application send message to rabbitmq and consume messsage from it to show message
in browser like chat
import pika, sys
global message
def consume(room,username):
credentials = pika.PlainCredentials('admin', 'admin')
parameters = pika.ConnectionParameters('192.168.1.14',5672,'/', credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.exchange_declare(exchange='topic_exchange', exchange_type='topic')
result = channel.queue_declare('', exclusive=True)
queue_name = result.method.queue
arr= [room,username]
binding_key ='.'.join([str(i) for i in arr])
channel.queue_bind(exchange='topic_exchange', queue=queue_name, routing_key=binding_key)
print(' [*] Waiting for logs. To exit press CTRL+C')
def callback(ch, method, properties, body):
print(" [x] %r:%r" % (method.routing_key, body))
channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
global message
#message =
channel.start_consuming()
return message
This isn't going to work. You are running a process that is consuming the messages and printing them. Some other process (django) is listening for requests from your browser.
Not sure how you hope it will work, but consider these alternatives-
your consumer writes the messages to a file, and django reads that file when it gets a request
django connects to the message bus and reads all waiting messages when it gets a request
your consumer writes the messages to a database
You build a websocket application that can push messages to currently connected browsers when it receives a message
I have a message queue using ActiveMQ. A web request puts messages into the queue with persistency=True. Now, I have 2 consumers that are both connected as separate sessions to this queue. Consumer 1 always acknowledges the message, but consumer 2 never does.
Now, I read this http://activemq.apache.org/how-does-a-queue-compare-to-a-topic.html which states:
A JMS Queue implements load balancer semantics. A single message will
be received by exactly one consumer. If there are no consumers
available at the time the message is sent it will be kept until a
consumer is available that can process the message. If a consumer
receives a message and does not acknowledge it before closing then the
message will be redelivered to another consumer. A queue can have many
consumers with messages load balanced across the available consumers.
What I understand from this is that I would expect all messages to eventually be processed by consumer 1 since it always acknowledges. Since consumer 2 does not acknowledge, the message should then get sent to consumer 1.
But what I am noticing is the following:
1. When I submit a request, I see only every 2nd request coming to consumer 1. The other request does not show up and it stored in ActiveMQ. I suppose it went to consumer 2 who did not acknowledge. So should it come to consumer 1 next?
I just need to ensure that the message gets processed by one consumer only. In my case, this consumer is a machine in country (site) X. Each message needs to be handled in only one country (machine). But all countries (machines) should received the message. If the country id matches in the message, it will acknowledge. So only 1 acknowledgement/message will be sent.
My code to receive/process messages looks like this:
# --------------------------------------------- MODULE IMPORT ---------------------------------------------------------#
import argparse
import json
import logging
import multiprocessing as mp
import sys
import stomp
from tvpv_portal.services.msgbkr import MsgBkr
from utils import util
# --------------------------------------------- DEVELOPMENT CODE ------------------------------------------------------#
log = logging.getLogger(__name__)
class MessageProcessingListener(stomp.ConnectionListener):
"""This class is responsible for processing (consuming) the messages from ActiveMQ."""
def __init__(self, conn, cb):
"""Initialization.
Args:
conn -- Connection object
cb -- Callback function
"""
self._conn = conn
self._cb = cb
def on_error(self, headers, body):
"""When we get an error.
Args:
headers -- Message header
body -- Message body
"""
log.error('Received error=%s', body)
def on_message(self, headers, body):
"""When we receive a message.
Args:
headers -- Message header
body -- Message body
"""
log.info('Received message')
# Deserialize the message.
item = json.loads(body)
import pprint
pprint.pprint(item)
# TODO: check if msg is to be handled by this SITE. If so, acknowledge and queue it. Otherwise, ignore.
# Put message into queue via callback (queue.put) function.
#self._cb(item)
# TODO: we only send acknowledge if we are supposed to process this message.
# Send acknowledgement to ActiveMQ indicating message is consumed.
self._conn.ack(headers['message-id'], headers['subscription'])
def worker(q):
"""Worker to retrieve item from queue and process it.
Args:
q -- Queue
"""
# Run in an infinite loop. Get an item from the queue to process it. We MUST call q.task_done() to indicate
# that item is processed to prevent deadlock.
while True:
try:
item = q.get()
# TODO: We will call external script from here to run on Netbatch in the future.
log.info('Processed message')
finally:
q.task_done()
def flash_mq_rst_handler_main():
"""Main entry to the request handler."""
# Define arguments.
parser = argparse.ArgumentParser(description='Flash message queue request handler script',
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
add_help=False)
opts = parser.add_argument_group('Options')
opts.add_argument('-h', '--help', action='help',
help='Show this help message and exit')
opts.add_argument('--workers', metavar='val', type=int, default=4,
help='Number of worker processes')
opts.add_argument('--log', metavar='file', type=util.get_resolved_abspath, default='flash_mq_rst_handler.log',
help='Log file')
# Parse arguments.
args = parser.parse_args()
# Setup logger.
util.configure_logger(args.log)
log.info('Command line %s', ' '.join(map(str, sys.argv)))
# Create a managed queue to store messages retrieved from message queue.
queue = mp.Manager().JoinableQueue()
# Instantiate consumer message broker + ensure connection.
consumer = MsgBkr(producer=False)
if not consumer.is_connected():
log.critical('Unable to connect to message queue; please debug')
sys.exit(1)
# Register listener with consumer + queue.put as the callback function to trigger when a message is received.
consumer.set_listener('message_processing_listener', MessageProcessingListener, cb=queue.put)
# Run in an infinite loop to wait form messages.
try:
log.info('Create pool with worker=%d to process messages', args.workers)
with mp.Pool(processes=args.workers) as pool:
p = pool.apply_async(worker, (queue,))
p.get()
except KeyboardInterrupt:
pass
# See MsgBkr. It will close the connection during exit() so we don't have to do it.
sys.exit(0)
if __name__ == '__main__':
flash_mq_rst_handler_main()
This got addressed with the JMS bridge: https://activemq.apache.org/components/artemis/documentation/1.1.0/jms-bridge.html
Was able to get IT to configure create N+1 queues. The source (incoming) queue is where all messages are put. Based on some selector in the message (like 'some_key': 'some_value' in the header), the message can be routed to one of the N destination (outgoing) queues. Then, each site can listen to a particular queue for messages. Multiple consumers on the same destination queue will get messages in a round-robin fashion.
I have a subscriber application which pulls from a Google Cloud Pub/Sub asynchronously using the google-cloud-pubsub python library.
I am running into intermittent issues where my callback function doesnt finish running/is interrupted. Unfortunately I dont have any errors, I only know this is the case because it does not finish writing data to an external source.
for example, in the following sample code:
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project, subscription_name)
def callback(message):
print('Received message: {}'.format(message))
message.ack()
# Limit the subscriber to only have ten outstanding messages at a time.
flow_control = pubsub_v1.types.FlowControl(max_messages=10)
subscriber.subscribe(
subscription_path, callback=callback, flow_control=flow_control)
# The subscriber is non-blocking, so we must keep the main thread from
# exiting to allow it to process messages in the background.
print('Listening for messages on {}'.format(subscription_path))
while True:
time.sleep(60)
the code in the callback function may sometimes take a while, and in some cases, I have noticed that it does not finish executing/seems to be disrupted by something.
Could that ever happen? is there a timeout on this function?
I created a zmq_forwarder.py that's run separately and it passes messages from the app to a sockJS connection, and i'm currently working on right now on how a flask app could receive a message from sockJS via zmq. i'm pasting the contents of my zmq_forwarder.py. im new to ZMQ and i dont know why everytime i run it, it uses 100% CPU load.
import zmq
# Prepare our context and sockets
context = zmq.Context()
receiver_from_server = context.socket(zmq.PULL)
receiver_from_server.bind("tcp://*:5561")
forwarder_to_server = context.socket(zmq.PUSH)
forwarder_to_server.bind("tcp://*:5562")
receiver_from_websocket = context.socket(zmq.PULL)
receiver_from_websocket.bind("tcp://*:5563")
forwarder_to_websocket = context.socket(zmq.PUSH)
forwarder_to_websocket.bind("tcp://*:5564")
# Process messages from both sockets
# We prioritize traffic from the server
while True:
# forward messages from the server
while True:
try:
message = receiver_from_server.recv(zmq.DONTWAIT)
except zmq.Again:
break
print "Received from server: ", message
forwarder_to_websocket.send_string(message)
# forward messages from the websocket
while True:
try:
message = receiver_from_websocket.recv(zmq.DONTWAIT)
except zmq.Again:
break
print "Received from websocket: ", message
forwarder_to_server.send_string(message)
as you can see, i've setup 4 sockets. the app connects to port 5561 to push data to zmq, and port 5562 to receive from zmq (although im still figuring out how to actually set it up to listen for messages sent by zmq). on the other hand, sockjs receives data from zmq on port 5564 and sends data to it on port 5563.
i've read the zmq.DONTWAIT makes receiving of message asynchronous and non-blocking so i added it.
is there a way to improve the code so that i dont overload the CPU? the goal is to be able to pass messages between the flask app and the websocket using zmq.
You are polling your two receiver sockets in a tight loop, without any blocking (zmq.DONTWAIT), which will inevitably max out the CPU.
Note that there is some support in ZMQ for polling multiple sockets in a single thread - see this answer. I think you can adjust the timeout in poller.poll(millis) so that your code only uses lots of CPU if there are lots of incoming messages, and idles otherwise.
Your other option is to use the ZMQ event loop to respond to incoming messages asynchronously, using callbacks. See the PyZMQ documentation on this topic, from which the following "echo" example is adapted:
# set up the socket, and a stream wrapped around the socket
s = ctx.socket(zmq.REP)
s.bind('tcp://localhost:12345')
stream = ZMQStream(s)
# Define a callback to handle incoming messages
def echo(msg):
# in this case, just echo the message back again
stream.send_multipart(msg)
# register the callback
stream.on_recv(echo)
# start the ioloop to start waiting for messages
ioloop.IOLoop.instance().start()
I have my software running on a bunch of clients around my network. I've been playing around with RabbitMQ as a solution for passing messages between each client.
My test code is this:
#!/usr/bin/python2
import pika
import time
connection = pika.AsyncoreConnection(pika.ConnectionParameters(
'localhost'))
channel = connection.channel()
def callback(ch, method, properties, body):
# send messages back on certain events
if body == '5':
channel.basic_publish(exchange='',
routing_key='test',
body='works')
print body
channel.queue_declare(queue='test')
channel.basic_consume(callback, queue='test', no_ack=True)
for i in range(0, 8):
channel.basic_publish(exchange='',
routing_key='test',
body='{}'.format(i))
time.sleep(0.5)
channel.close()
Picture this as kind of a 'chat program'. Each client will need to constantly listen for messages. At times, the client will need to send messages back to the server.
This code works, but I've ran into an issue. When the code below sends out the message works, it then retreives that again from the RabbitMQ queue. Is there a way to tell have my client, a producer and a consumer, not receive the message it just sent?
I can't see this functionality built into RabbitMQ so I figured I'd send messages in the form of:
body='{"client_id" : 1, "message" : "this is the message"}'
Then I can parse that string and check the client_id. The client can then ignore all messagess not destined to it.
Is there a better way? Should I look for an alternative to RabbitMQ?
You can have as many queue in RabbitMQ. Why not have a queue for messages to the server as well as a queue for each client?