I have a callback associated with a rabbitmq queue through pika's basic_consume like so.
channel.basic_consume(queue=REQUESTS_QUEUE,
on_message_callback=request_callback, auto_ack=False)
And the request callback function is:
def request_callback(channel, method, properties, body):
try:
readings = json_util.loads(body)
location_updater.update_location(readings)
channel.basic_ack(delivery_tag=method.delivery_tag)
except Exception:
logger.exception('EXCEPTION: ')
Whenever the code inside the except block is executed, this particular callback stops working (i.e. it stops being called when a message is sent to its associated queue). All the other callbacks I have associated with other queues keep working fine. If I comment out the try...except logic the callback keeps working fine for further requests, even after a exception occurs.
I'm still getting used to Python, so it might be something simple. Can anyone help?
I'm assuming the exception comes from a statement before channel.basic_ack, and I'm also assuming you're calling channel.basic_qos to set a prefetch value.
The exception prevents the call to basic_ack, which prevents RabbitMQ from removing the message from the queue. If you have reached the prefetch value, no further messages will be delivered to that client because RabbitMQ assumes your client is still processing them.
You need to decide what to do with that message when an exception happens. I'm assuming that the message can't be considered to be processed, so you should reject (nack) the message (https://www.rabbitmq.com/nack.html). Do this in the except block. This will cause the message to be re-enqueued and re-delivered, potentially to a different consumer.
Closing the channel and/or the connection will also have the same effect. This ensures that clients that crash do not permanently "hold onto" messages.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
I have a class that contains a websocket-client WebSocketApp, that is started in a thread.
The websocket connection recieves messages and sets some class variables accordingly.
One example is the login:
A login message is sent, and after a while the on_message function recieves a successfull login message. When the function catches this message a self.logged_in variable is set to true.
Currently I'm "waiting" for the variable to become true using a busy wait, which is obviously not very good.
while websocket.logged_in:
pass
What I need is something like this
wait(websocket.logged_in=True, timeout=100)
Found a nice solution using synchronized Queues
You just need to call queue.put(var) in your thread where the message arrives.
In the main you call queue.get() which will wait until an element shows up in the queue.
In my case I will probably use multiple queues with just one element for different responses that I'm going to get
I'm looking to implement something similar to the on_message functionality WebSocketApp but using the websockets library, i.e. always read the latest message. However it looks like they only offer recv() in order to read the next message in the queue.
My temporary work around is to run ws.recv() in a while loop, but I'm not sure whether this is the right approach:
async with websockets.connect(self.socket) as ws:
while True:
msg = await asyncio.wait_for(ws.recv(), timeout=3)
self.on_msg(msg)
With this approach I have two concerns:
I'm subscribed to a very high-frequency channel, so I'm concerned recv will fall behind in reading the messages in the queue. (I previously had a 0.1s delay in the while loop and noticed that the message I was getting was not the latest. Even after terminating my internet connection, it kept reading from the message queue before raising a connection closed error)
I'm currently using a timeout to detect if a message hangs for unforeseen reasons, but there are cases where there are simply no messages and it raises a timeout error. I'm not able to differentiate the two using the current method.
What's the best way to always get the latest message, even if there's a queue of unread messages? Also, how would you do error handling to avoid the issue in 2)?
I have a tornado coroutine hander that looks in part like:
class QueryHandler(tornado.web.RequestHandler):
queryQueues = defaultdict(tornado.queues.Queue)
#tornado.gen.coroutine
def get(self, network):
qq = self.queryQueues[network]
query = yield qq.get()
# do some work with with the dequeued query
self.write(response)
On the client side, I use python-requests to long poll it:
fetched = session.get(QueryURL)
I can make a query, the server blocks waiting on the queue until cough up a something to process and finally respond.
This works pretty slick until... the long poll gets shutdown and restarted while the handler is blocking on the queue. When I stop the query on the client side, the handler stays happily blocked. Worse if I restart the query on the client side, I now have a second handler instance blocking on the queue. So when the queue DOES have data show up, the stale handler processes it and replies to the bitbucket, and the restarted query is now blocked indefinitely.
Is there a pattern I can use to avoid this? I had hoped that when the client side closed, the handler would receive some sort of exception indicating that things have gone south. The queue.get() can have a timeout, but what I really want is not a timeout but a sort of "unless I close" exception.
You want a "queue with guaranteed delivery" which is a hard problem in distributed systems. After all, even if "self.write" succeeds, you can't be certain the other end really received the message.
A basic approach would look like this:
each entry in the queue gets an id greater than all previous ids
when the client connects it asks to subscribe to the queue
when a client is disconnected, it reconnects and asks for all entries with ids greater than the last id it saw
when your QueryHandler receives a get with a non-None id, it first serves all entries with ids greater than id, then begins waiting on the queue
when your QueryHandler raises an exception from self.write, ignore it: the client is responsible for retrieving the lost entry
keep all past entries in a list
expire the oldest list entries after some time (hours?)
Consider the following twisted code, using deferLater:
import random
from twisted.internet.task import deferLater
from twisted.internet import reactor
def random_exception(msg='general'):
if random.random() < 0.5:
raise Exception("Random exception with 50%% likelihood occurred in %s!" % msg)
def dolater():
random_exception('dolater')
print "it's later!"
def whoops(failure):
failure.trap(Exception)
print failure
defer = deferLater(reactor, 10, dolater)
defer.addErrback(whoops)
reactor.run()
An exception is raised during the 10 second sleep (namely a KeyboardInterrupt), however, it seems that the whoops method is never called. My assumption is that since I add the errBack after the deferred kicks off, it's never properly registered. Advice appreciated.
EDIT:
Alright, no one likes my use of the signal (not the exception) KeyboardInterrupt to show an error condition outside of the defer. I thought pretty hard about an actual exception that might occur out of the defer callback, but couldn't think of a particularly good one, most everything would be some kind of signal (or developer error), so signal handling is fine for now- but that wasn't really the heart of the question.
As I understand it, twisted's callback/errback system handles errors within the callback structure - e.g. if dolater raises an Exception of some kind. To show this, I have added an exception that could occur during dolater, to show that if the exception occurs in dolater, the errback handles the exception just fine.
My concern was if something went wrong while the reactor was just reacting normally, and the only thing I could get to go wrong was a keyboard interrupt, then I wanted whoops to fire. It appears that if I put other async events into the reactor and raise exceptions from there, then the dolater code wouldn't be affected, and I would have to add errbacks to those other async events. There is no master error handling for an entire twisted program.
So signals it is, until I can find some way to cause the reactor to fail without a signal.
If by KeyboardInterrupt you mean a signal (ctrl-c, SIGINT, etc), then what you need to do is setup a signal handler with your whoops function as the callback.
By following two previous answers from #jean-paul-calderone twisted: catch keyboardinterrupt and shutdown properly and twisted - interrupt callback via KeyboardInterrupt, I tried the following, and I think it matches your need:
def dolater():
print "it's later!"
def whoops(signal, stackframe):
print "I'm here because of signal number " + str(signal)
reactor.stop()
defer = task.deferLater(reactor, 10, dolater)
signal.signal(signal.SIGINT, whoops)
reactor.run()
That will call whoops on a SIGINT. I put a reactor.stop() in the whoops because otherwise the reactor would just keep on running, take that out if you really want it to keep running in the face of a ctrl-c.
Note: I'm not explicitly showing how to fire a err-back in the signal system because (at least to my understanding) that doesn't really map to how defer'ed should be used. I imagine if you found a way to get the defer'ed into the signal handler you could fire its errback but I think thats out of the expected use-case for twisted and may have crazy consequences.
The problem is with the actual exception you're trying to catch, specifically KeyboardInterrupt is not a subclass of Exception, thus can not be catched with it. If you'd just change the line:
failure.trap(Exception)
into:
failure.trap(KeyboardInterrupt)
it surely would catch it. More on Python's exception hierarchy can be found in the official Python docs: https://docs.python.org/2/library/exceptions.html
Twisted is a library for doing many things concurrently. The things are kept as isolated as possible (given that this is still Python, there's still global state, etc).
If you have a TCP server with two clients connect to it and one of them sends you some bad data that triggers a bug in your parser that leads to an exception being raised, that exception isn't going to cause the other client to receive any error. Nor would you want it to, I hope (at least not automatically).
Similarly, if you have a client connected to your server and you start a delayed call with deferLater and the client triggers that bug, you wouldn't want the error to be delivered to the errback on the Deferred returned by deferLater.
The idea here is that separate event sources are generally treated separately (until you write some code that glues them together somehow).
For the ten seconds that are passing between when you call deferLater and when Twisted begins to run the function you passed to deferLater, any errors that happen - including you hitting C-c on your keyboard to make Python raise a KeyboardInterrupt - aren't associated with that delayed call and they won't be delivered to the errback you attach to its Deferred.
Only exceptions raised by your dolater function will cause the errback chain of that Deferred to begin execution.
can both consuming and publishing be done in one Python thread using RabbitMQ channels?
Actually this isn't a problem at all and you can do it quite easily with for example pika the problem is however that you'd have to stop the consuming since it's a blocking loop or do the producing during the consume of a message.
Consuming and producing is a normal usecase, especially in pika since it isn't threadsafe, when for example you'd want to implement some form of filter on the messages, or, perhaps a smart router, which in turn will pass on the messages to another queue.
I don't think you should want to. MQ means asynch processing. Doing both consuming and producing in the same thread defeats the purpose in my opinion.
I'd recommend taking a look at Celery (http://celery.readthedocs.org/en/latest/) to manage worker tasks. With that, you won't need to integrate with RMQ directly as it will handle the the producing and consuming for you.
But, if you do desire to integrate with RMQ directly and manage your own workers, check out Kombu (http://kombu.readthedocs.org/en/latest/) for the integration. There are non-blocking consumers and producers that would permit you to have both in the same event loop.
I think the simple answer to your question is yes. But it depends on what you want to do. My guess is you have a loop that is consuming from your thread on one channel and after some (small or large) processing it decides to send it on to another queue (or exchange) on a different channel then I do not see any problem with that at all. Though it might be preferable to dispatch it to a different thread it is not necessary.
If you give more details about your process then it might help give a more specific answer.
Kombu is a common python library for working with RabbitMQ (Celery uses it under the hood). It is worth pointing out here that the answer to your question for the simplest use of Kombu that I tried is "No - you can't receive and publish on the same consumer callback thread."
Specifically if there are several messages in the queue for a consumer that has registered a callback for that topic and that callback does some processing and publishes the results then the publishing of the result will cause the 2nd message in the queue to hit the callback before it has returned from the publish from 1st message - so you end up with a recursive call to the callback. If you have n message on the queue your call stack will end up n message deep before it unwinds. Obviously that explodes pretty quickly.
One solution (not necessarily the best) is to have the callback just post the message into a simple queue internal to the consumer that could be processed on the main process thread (i.e. off the callback thread)
def process_message(self, body: str, message: Message):
# Queue the message for processing off this thread:
print("Start process_message ----------------")
self.do_process_message(body, message) if self.publish_on_callback else self.queue.put((body, message))#
print("End process_message ------------------")
def do_process_message(self, body: str, message: Message):
# Deserialize and "Process" the message:
print(f"Process message: {body}")
# ... msg processing code...
# Publish a processing output:
processing_output = self.get_processing_output()
print(f"Publishing processing output: {processing_output}")
self.rabbit_msg_transport.publish(Topics.ProcessingOutputs, processing_output)
# Acknowledge the message:
message.ack()
def run_message_loop(self):
while True:
print("Waiting for incoming message")
self.rabbit_connection.drain_events()
while not self.queue.empty():
body, message = self.queue.get(block=False)
self.do_process_message(body, message)
In this snippet above process_message is the callback. If publish_on_callback is True you'll see recursion in the callback n deep for n message on rabbit queue. If publish_on_callback is False it runs correctly without recursion in the callback.
Another approach is to use a second Connection for the Producer Exchange - separate from the Connection used for the Consumer. This also works so that callback from consuming a message and publishing the result completes before the callback is again fired for the next message on queue.