I have a python server that is available through a websocket endpoint.
During serving a connection, it also communicates with some backend services. This communication is asynchronous and may trigger the send() method of the websocket.
When a single client is served, it seems to work ok. However, when multiple clients are served in parallel, some of the routines that handle the connections get stuck occasionally. More precisely, it seem to block in the recv() method.
The actual code is somehow complex and the issue is slightly more complicated than I have described, nevertheless, I provide a minimal skeleton of code that sketch the way in which I use he websockets:
class MinimalConversation(object):
def __init__(self, ws, worker_sck, messages, should_continue_conversation, should_continue_listen):
self.ws = ws
self.messages = messages
self.worker_sck = worker_sck
self.should_continue_conversation = should_continue_conversation
self.should_continue_listen = should_continue_listen
async def run_conversation(self):
serving_future = asyncio.ensure_future(self.serve_connection())
listening_future = asyncio.ensure_future(self.handle_worker())
await asyncio.wait([serving_future, listening_future], return_when=asyncio.ALL_COMPLETED)
async def serve_connection(self):
while self.should_continue_conversation():
await self.ws.recv()
logger.debug("Message received")
self.sleep_randomly(10, 5)
await self.worker_sck.send(b"Dummy")
async def handle_worker(self):
while self.should_continue_listen():
self.sleep_randomly(50, 40)
await self.worker_sck.recv()
await self.ws.send(self.messages.pop())
def sleep_randomly(self, mean, dev):
delta = random.randint(1, dev) / 1000
if random.random() < .5:
delta *= -1
time.sleep(mean / 1000 + delta)
Obviously, in the real code I do not sleep for random intervals and don't use given list of messages but this sketches the way I handle the websockets. In the real setting, some errors may occur that are sent over the websocket too, so parallel sends() may occur in theory but I have never encountered such a situation.
The code is run from a handler function which is passed as a parameter to websockets.serve(), initialize the MinimalConversation object and calls the run_conversation() method.
My questions are:
Is there something fundamentally wrong with such usage of the websockets?
Are concurrent calls of the send() methods dangerous?
Can you suggest some good practices regarding usage of websockets and asyncio?
Thak you.
recv function yields back only when a message is received, and it seems that there are 2 connections awaiting messages from each other, so there might be a situation similar to "deadlock" when they are waiting for each other's messages and can't send anything. Maybe you should try to rethink the overall algorithm to be safer from this.
And, of course, try adding more debug output and see what really happens.
are concurrent calls of the send() methods dangerous?
If by concurrent you mean in the same thread but in independently scheduled coroutines then parallel send is just fine. But be careful with "parallel" recv on the same connection, because order of coroutine scheduling might be far from obvious and it's what decides which call to recv will get a message first.
Can you suggest some good practices regarding usage of websockets and asyncio?
In my experience, the easiest way is to create a dedicated task for incoming connections which will repeatedly call recv on the connection, until connection is closed. You can store the connection somewhere and delete it in finally block, then it can be used from other coroutines to send something.
Related
I have a minimal async python server based on aiohttp.
It is very straightforward, just a websocket endpoint exposed as in
#routes.get('/my_endpoint')
async def my_func(request):
ws = web.WebSocketResponse()
await ws.prepare(request)
return ws
I want to expose as prometheus metrics the request rate (and potentially the error rate).
After performing a brief investigation on the topic, I realised that it seems like there is a distinction between approaching prometheus metrics exposure when it comes to sync vs async apps.
For my case, where I want a simple request count/rate, is there a reason not to just use the plain' old prometheus python client (e.g by simply decorating my_func?)
Would the request count actually fail in such a case?
The following is based on my understanding on asyncio and the way the official prometheus client describes how it exposes metrics.
aiohttp is to be used on top of asyncio. Now, asyncio is running something called an "event loop" which runs inside a single thread (usually the main thread)
You can look at it as an entity that decides to suspend or execute functions that were assigned to run in the loop. In your case my_func.
For prometheus_client to expose your metrics you will probably need to run it in a different thread
Metrics are usually exposed over HTTP, to be read by the Prometheus server. The easiest way to do this is via start_http_server, which will start a HTTP server in a daemon thread on the given port
This is outside "the control of the event loop" which might lead to performance issues and to unexpected behavior as a result. So the request count might not fail, but if for some reason its doing some blocking task (I/O) it will block the main thread as well. If you'd use the async approach and run it as part of the event loop your blocking task can be awaited and give back the control to the main thread.
There are open source projects that support prometheus in async functions such as aioprometheus and prometheus-async.
I've a trouble understand the reason for the below code, hope someone can shed some lights on this. I'm new in async programming.
This is from websockets documentation
#!/usr/bin/env python
import asyncio
import websockets
async def echo(websocket, path):
async for message in websocket:
await websocket.send(message)
asyncio.get_event_loop().run_until_complete(
websockets.serve(echo, 'localhost', 8765))
asyncio.get_event_loop().run_forever()
I have some questions about asyncio design and how I can take advantage of this.
First of all, looking at the last two lines. If I understand correctly, shouldn't run_until_complete shutdown after finished its work? How can the second loop kept it alive with no jobs submitted into the loop?
Secondly, I was trying to build a back_end that can process some data from the front_end using websockets and return the calculation in realtime. There will be two kinds of tasks, one is bit longer that need computation power a bit but will happen one time for a session, and bunch of streaming data processing that need to be send back immediately (90 frames per seconds).
For the bigger tasks, should I just fire up another websocket server that process the longer work, and use the main websocket to consume on it? Or use another process to do the work in a chained async function? And for the smaller tasks, what would go wrong if I do the same as above?
TLDR:
Async programming is hammering my brain.
Maintain a nonblocking session
Which process hard work and light work simultaneously
The light work should have zero latency
The hard work should be as fast as possible but not effecting the light work.
Thanks!!
I am wanting to create a RabbitMQ receiver/consumer in Python and am not sure how to check for messages. I am trying to do this in my own loop, not using the call-backs in pika.
If I understand things, in the Java client I can use getBasic() to check to see if there are any messages available without blocking. I don't mind blocking while getting messages, but I don't want to block until there is a message.
I don't find any clear examples and haven't yet figured out the corresponding call in pika.
If you want to do it synchronously then you will need to look at the pika BlockingConnection
The BlockingConnection creates a layer on top of Pika’s asynchronous
core providng methods that will block until their expected response
has returned. Due to the asynchronous nature of the Basic.Deliver and
Basic.Return calls from RabbitMQ to your application, you are still
required to implement continuation-passing style asynchronous methods
if you’d like to receive messages from RabbitMQ using basic_consume or
if you want to be notified of a delivery failure when using
basic_publish.
More info and an example here
https://pika.readthedocs.org/en/0.9.12/connecting.html#blockingconnection
You can periodically check the queue size using the example of this answer Get Queue Size in Pika (AMQP Python)
Queue processing loop can be done iteratively with the help of process_data_events():
import pika
# A stubborn callback that still wants to be in the code.
def mq_callback(ch, method, properties, body):
print(" Received: %r" % body)
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
queue_state = channel.queue_declare(queue="test")
# Configure a callback.
channel.basic_consume(mq_callback, queue="test")
try:
# My own loop here:
while(True):
# Do other processing
# Process message queue events, returning as soon as possible.
# Issues mq_callback() when applicable.
connection.process_data_events(time_limit=0)
finally:
connection.close()
can both consuming and publishing be done in one Python thread using RabbitMQ channels?
Actually this isn't a problem at all and you can do it quite easily with for example pika the problem is however that you'd have to stop the consuming since it's a blocking loop or do the producing during the consume of a message.
Consuming and producing is a normal usecase, especially in pika since it isn't threadsafe, when for example you'd want to implement some form of filter on the messages, or, perhaps a smart router, which in turn will pass on the messages to another queue.
I don't think you should want to. MQ means asynch processing. Doing both consuming and producing in the same thread defeats the purpose in my opinion.
I'd recommend taking a look at Celery (http://celery.readthedocs.org/en/latest/) to manage worker tasks. With that, you won't need to integrate with RMQ directly as it will handle the the producing and consuming for you.
But, if you do desire to integrate with RMQ directly and manage your own workers, check out Kombu (http://kombu.readthedocs.org/en/latest/) for the integration. There are non-blocking consumers and producers that would permit you to have both in the same event loop.
I think the simple answer to your question is yes. But it depends on what you want to do. My guess is you have a loop that is consuming from your thread on one channel and after some (small or large) processing it decides to send it on to another queue (or exchange) on a different channel then I do not see any problem with that at all. Though it might be preferable to dispatch it to a different thread it is not necessary.
If you give more details about your process then it might help give a more specific answer.
Kombu is a common python library for working with RabbitMQ (Celery uses it under the hood). It is worth pointing out here that the answer to your question for the simplest use of Kombu that I tried is "No - you can't receive and publish on the same consumer callback thread."
Specifically if there are several messages in the queue for a consumer that has registered a callback for that topic and that callback does some processing and publishes the results then the publishing of the result will cause the 2nd message in the queue to hit the callback before it has returned from the publish from 1st message - so you end up with a recursive call to the callback. If you have n message on the queue your call stack will end up n message deep before it unwinds. Obviously that explodes pretty quickly.
One solution (not necessarily the best) is to have the callback just post the message into a simple queue internal to the consumer that could be processed on the main process thread (i.e. off the callback thread)
def process_message(self, body: str, message: Message):
# Queue the message for processing off this thread:
print("Start process_message ----------------")
self.do_process_message(body, message) if self.publish_on_callback else self.queue.put((body, message))#
print("End process_message ------------------")
def do_process_message(self, body: str, message: Message):
# Deserialize and "Process" the message:
print(f"Process message: {body}")
# ... msg processing code...
# Publish a processing output:
processing_output = self.get_processing_output()
print(f"Publishing processing output: {processing_output}")
self.rabbit_msg_transport.publish(Topics.ProcessingOutputs, processing_output)
# Acknowledge the message:
message.ack()
def run_message_loop(self):
while True:
print("Waiting for incoming message")
self.rabbit_connection.drain_events()
while not self.queue.empty():
body, message = self.queue.get(block=False)
self.do_process_message(body, message)
In this snippet above process_message is the callback. If publish_on_callback is True you'll see recursion in the callback n deep for n message on rabbit queue. If publish_on_callback is False it runs correctly without recursion in the callback.
Another approach is to use a second Connection for the Producer Exchange - separate from the Connection used for the Consumer. This also works so that callback from consuming a message and publishing the result completes before the callback is again fired for the next message on queue.
I have a server in twisted, implementing a LineReceiver protocol.
When I call sendLine in response to a client message, it writes the line to the client immediately, as one would expect.
But say the client asks the server to do a lengthy calculation. I want the server to periodically send a progress message to the client. When the server takes initiative and calls sendLine without the client having asked for anything, it seems to wait for the client to send a message to the server before sending anything.
How do I send a message from the server to the client immediately, without having the client explicitly ask for it?
Use deferreds if you perform calculation asynchronously.
Other way if it's some long calculation in separate Thread, started by lets say deferrToThread(), use reactor.callFromThread()
(I assume we don't do heavy calculation in main loop - that's very, very wrong :))
little example:
def some_long_foo(data_array, protocol):
def send_msg(msg, protocol):
# It actually looks petter in classes without pushing protocol here and
# there
protocol.transport.write(msg)
for n, chunk in enumerate(data_array):
do_something_cool(chunk)
if n and (n % 10 == 0):
from twisted.internet import reactor
# here send_msg will be safely executed in main reactor loop
reactor.callFromThread(send_msg, '10 more chunks processed',
protocol)
# Somwhere in lineReceived we start long calculation
def cb(result):
self.transport.write('got result: {}'.format(result))
d = threads.deferToThread(some_long_foo, data_array, self)
d.addCallback(cb)
Thus now we'll notify client about processing every 10 chunks of data, and then finally send him result.
code may be little incorrect, it's just e.g.
docs
UPD:
just for clarification:
missed sendLine part. Generally it doesn't matter, call it insted of transport.write()
You can send a line using sendLine whenever you want and it should arrive immediately, but you may have a problem related to your server blocking.
A call to sendLine is deferred, so if you make a call in the middle of a bunch of processing, it's possible that it's not being actioned for a while, and then when a message is received, the reactor interrupts the processing, receives the message, and gets the queued message sent before going back to processing. You should read some of the other answers here and make sure that your processing isn't blocking up your main thread.
If by "immediately" you mean "when the client connects", try calling sendLine in your LineReceiver subclass's connectionMade.