I have an end-to-end pipeline of an web application like below in Python3.6
Socket(connection from client to server) -> Flask Server -> Kafka Producer ->Kafka Consumer ->NLPService
Now when I get some result back from the NLPService, I need to send it back to the client. I am thinking below steps
NLP service writes the result to a different topic on Kafka producer (done)
Kafka consumer retrieves the result from Kafka broker (done)
Kafka consumer needs to write the result to the flask server
Then flask server will send the result back to the socket
Socket writes to client
I have already done steps 1-2. But stuck at step 3, 4. How do I write from Kafka to the flask server? If I just call a function at my server.py, then logically it seems like I have to create a socket within at function at server.py which will do the job of sending to client through socket. But syntax wise it looks weird. What am I missing?
at consumer.py
#receiving reply
topicReply = 'Reply'
consumerReply = KafkaConsumer(topicReply, value_deserializer=lambda m: json.loads(m.decode('ascii')))
for message in consumerReply:
#send reply back to Server
fromConsumer(message.value)
at server.py
socketio = SocketIO(app)
def fromConsumer(msg):
#socketio.on('reply')
def replyMessage(msg):
send(msg)
The above construct in server.py doesn't make sense to me. Please suggest.
Related
I would like to set-up a server that can subscribe to an external stream over a websocket (ws_ext) and then republish that data (after curating) to internal clients connecting to this server over websockets (ws_int).
My approach so far is to set up a fastapi server that can open websockets (ws_int) with internal clients .
However, I don't understand how to have a listener embedded in this server that can listen to external stream and then publish to these internal clients in a non blocking way.
Can someone point me to a working example that can help?
Here is what i would like to achieve:
p.s: I HAVE BEEN able to make it work by decoupling broadcaster from subscriber using redis pubsub. So, what i have now setup is a client that listens to external stream, curate and pushes it to redis pubsub. then i have a separate broadcaster that listens to redis pubsub and pushes it out to clients after curating on its websockets. I would still love to combine these two without using redis or some such backend.
if you have all clients connected to an async ws located in the broadcaster then the same time push whatever that's coming asynchronously to broadcaster from external website the process should be unblocking supposedly
the update process can have a async stream pipeline to filter results coming from external website for each client in broadcaster
as for example to async client for WebSocket it can go by "with async"
async def hello():
async with websockets.connect(
'ws://localhost:8765', ssl=ssl_context) as websocket:
name = input("What's your name? ")
await websocket.send(name)
print(f"> {name}")
greeting = await websocket.recv()
print(f"< {greeting}")
asyncio.get_event_loop().run_until_complete(hello())
Can I send messages from one client directly to another using python-socketio?
It can be done using socketio as follows:
socket.join('room')
io.sockets.in('room').emit('event_name', data)
Source: socket.io client to client messaging
I can't find any documentation on how this works with python-socketio. Could someone please provide an example?
The following in socketio
io.sockets.in('room').emit('event_name', data)
is the same as this in python-socketio:
io.emit("HelloWorld", some_dict, room="room")
Here is the link to the official doc -> link
The Socket.IO protocol allows bidirectional client-to-server communication, but clients are not connected among themselves so they cannot communicate directly. To implement client-to-client you have to pass through the server and it works as follows:
Client A emits an event to the server, indicating who is Client B, the recipient of the message, and what data it wishes to send to that client.
In the server, the handler for the event from Client A, emits an event addressed to Client B with the data passed by Client A in the first event.
I load messages from kafka topic to database. Loading to database can fail. Also I do not want to lose unsent messages.
App code:
import faust
app = faust.App('App', broker='kafka://localhost:9092')
source_topic = app.topic('source_topic')
failed_channel = app.channel() # channel for unsent messages
#app.agent(source_topic)
async def process(stream):
async for batch in stream.take(100_000, within=60):
# here we have not info about partitions and keys
# to reuse them when resending if sending failed
try:
pass # send to database. can fail
except ConnectionError:
for record in batch:
# sending to channel is faster than sending to topic
await failed_channel.send(value=record)
#app.agent(failed_channel)
async def resend_failed(stream):
async for unsent_msg in stream:
await source_topic.send(value=unsent_msg)
Maybe there is more standart way to handle such situations? Adding app.topic('source_topic', acks=False) works only after restarting app.
I load messages from kafka topic to database
Maybe there is more standart way to handle such situations
Yes - it's called Kafka Connect :-)
The standard pattern is to do any processing on your data and write it [back to] Kafka topics. Then you use the Kafka topic as a source for a Kafka Connect sink connector, in this case the Kafka Connect JDBC Sink connector.
Kafka Connect is part of Apache Kafka, and handles restarts, scaleout, failures, etc etc.
See also Kafka Connect in Action: JDBC Sink
I'm using the gevent-websocket module with Bottle Python framework.
When a client closes the browser, this code
$(window).on('beforeunload', function() { ws.close(); });
helps to close the websocket connection properly.
But if the client's network connection is interrupted, no "close" information can be sent to the server.
Then, often, even 1 minute later, the server still believes the client is connected, and the websocket is still open on the server.
Question: How to detect properly that a websocket is closed because the client is disconnected from network?
Is there a websocket KeepAlive feature available in Python/Bottle/gevent-websocket?
One answer from Web Socket: cannot detect client connection on internet disconnect suggests to use a heartbeat/ping packet every x seconds to tell the server "I'm still alive". The other answer suggests using a setKeepAlive(true). feature. Would this feature be available in gevent-websocket?
Example server code, taken from here:
from bottle import get, template, run
from bottle.ext.websocket import GeventWebSocketServer
from bottle.ext.websocket import websocket
users = set()
#get('/')
def index():
return template('index')
#get('/websocket', apply=[websocket])
def chat(ws):
users.add(ws)
while True:
msg = ws.receive()
if msg is not None:
for u in users:
u.send(msg)
else:
break
users.remove(ws)
run(host='127.0.0.1', port=8080, server=GeventWebSocketServer)
First you need to add a timeout to the receive() method.
with gevent.Timeout(1.0, False):
msg = ws.receive()
Then the loop will not block, if you send even an empty packet and the client doesn't respond, WebsocketError will be thrown and you can close the socket.
I am testing Redis in heroku. I have simple flask app that can create the redis client and kill the client but ...
#app.route('/client-status')
def client_status():
redis.client_setname("first")
redis.client_kill('addr') #10.157.2.68:60097
return "Success"
The question is how to get addr ? I know a way to get name redis.client_getname()...
you should use the command CLIENT LIST http://redis.io/commands/client-list to get all the information about the client connections server, and then retrieve the address field