I would like to set-up a server that can subscribe to an external stream over a websocket (ws_ext) and then republish that data (after curating) to internal clients connecting to this server over websockets (ws_int).
My approach so far is to set up a fastapi server that can open websockets (ws_int) with internal clients .
However, I don't understand how to have a listener embedded in this server that can listen to external stream and then publish to these internal clients in a non blocking way.
Can someone point me to a working example that can help?
Here is what i would like to achieve:
p.s: I HAVE BEEN able to make it work by decoupling broadcaster from subscriber using redis pubsub. So, what i have now setup is a client that listens to external stream, curate and pushes it to redis pubsub. then i have a separate broadcaster that listens to redis pubsub and pushes it out to clients after curating on its websockets. I would still love to combine these two without using redis or some such backend.
if you have all clients connected to an async ws located in the broadcaster then the same time push whatever that's coming asynchronously to broadcaster from external website the process should be unblocking supposedly
the update process can have a async stream pipeline to filter results coming from external website for each client in broadcaster
as for example to async client for WebSocket it can go by "with async"
async def hello():
async with websockets.connect(
'ws://localhost:8765', ssl=ssl_context) as websocket:
name = input("What's your name? ")
await websocket.send(name)
print(f"> {name}")
greeting = await websocket.recv()
print(f"< {greeting}")
asyncio.get_event_loop().run_until_complete(hello())
Related
Can I send messages from one client directly to another using python-socketio?
It can be done using socketio as follows:
socket.join('room')
io.sockets.in('room').emit('event_name', data)
Source: socket.io client to client messaging
I can't find any documentation on how this works with python-socketio. Could someone please provide an example?
The following in socketio
io.sockets.in('room').emit('event_name', data)
is the same as this in python-socketio:
io.emit("HelloWorld", some_dict, room="room")
Here is the link to the official doc -> link
The Socket.IO protocol allows bidirectional client-to-server communication, but clients are not connected among themselves so they cannot communicate directly. To implement client-to-client you have to pass through the server and it works as follows:
Client A emits an event to the server, indicating who is Client B, the recipient of the message, and what data it wishes to send to that client.
In the server, the handler for the event from Client A, emits an event addressed to Client B with the data passed by Client A in the first event.
I load messages from kafka topic to database. Loading to database can fail. Also I do not want to lose unsent messages.
App code:
import faust
app = faust.App('App', broker='kafka://localhost:9092')
source_topic = app.topic('source_topic')
failed_channel = app.channel() # channel for unsent messages
#app.agent(source_topic)
async def process(stream):
async for batch in stream.take(100_000, within=60):
# here we have not info about partitions and keys
# to reuse them when resending if sending failed
try:
pass # send to database. can fail
except ConnectionError:
for record in batch:
# sending to channel is faster than sending to topic
await failed_channel.send(value=record)
#app.agent(failed_channel)
async def resend_failed(stream):
async for unsent_msg in stream:
await source_topic.send(value=unsent_msg)
Maybe there is more standart way to handle such situations? Adding app.topic('source_topic', acks=False) works only after restarting app.
I load messages from kafka topic to database
Maybe there is more standart way to handle such situations
Yes - it's called Kafka Connect :-)
The standard pattern is to do any processing on your data and write it [back to] Kafka topics. Then you use the Kafka topic as a source for a Kafka Connect sink connector, in this case the Kafka Connect JDBC Sink connector.
Kafka Connect is part of Apache Kafka, and handles restarts, scaleout, failures, etc etc.
See also Kafka Connect in Action: JDBC Sink
Update 13. Apr: dart code works consistently
Background:
My TV (Samsung 2019 RU7000) offers a secure websocket connection with which json packets can be send to remote control it. For example
"method":"ms.remote.control",
"params":{ "Cmd": "Click", "DataOfCmd":"KEY_MUTE" }
can be send to mute the TV.
To enable access, a handshake is done. The client connects to wss://ip:8002/api/v2/samsung.remote.control?name=value where the value of name is a base64 encoded string. After a successful connection, a popup window appears on the TV which needs to be accepted. If the user accepts the request, the TV sends a JSON response containing a token:
"data":{
"clients":[ ... ],
"id":"...",
"token":"28852140"
},
"event":"ms.channel.connect"
This token is used for authenticating connections with the same name by attaching &token=value to the URL. Sending commands before the user accepts the popup is ignored.
What works
Command line approaches with wscat and curl are working. The TV shows a pop-up and sends a response:
$ wscat -n -c https://192.168.0.227:8002/api/v2/channels/samsung.remote.control?name=aW9Ccm9rZXI=
Connected (press CTRL+C to quit)
< {"data":{"clients":[...], "id":"...", "token":"57940060"}, "event":"ms.channel.connect"}
dart - dart.io.WebSocket
The following code triggers the popup and gets a response as well.
WebSocket ws = await WebSocket.connect(url,
compression: CompressionOptions.compressionOff);
ws.pingInterval = Duration(seconds: 10000);
ws.listen(print, onError: print, onDone: () {
print("done");
print(ws.closeCode);
print(ws.closeReason);
});
await Future.delayed(Duration(seconds: 30));
ws.close();
Problem
I want to use python to connect to the websocket, authenticate my session and send remote control commands. This does not work for the following implementations.
python - websocket liris
The popup does not appear and there is no response from the TV. The socket does not close.
from websocket import create_connection
from ssl import CERT_NONE
sock = create_connection(url, sslopt={"cert_reqs": CERT_NONE})
print(sock.recv())
python - websockets aaugustin
The pop-up does not appear as well and it seems the connection is closed by the TV.
from websockets.client import connect, WebSocketClientProtocol
import ssl
import asyncio
async def connect():
async with connect(url, ssl=ssl.CERT_NONE) as websocket:
res = await websocket.recv()
print(res)
asyncio.get_event_loop().run_until_complete(connect())
The wireshark log shows that it's sending an HTTP GET asking for a websocket upgrade. The TV responds by closing the connection with FIN and RST.
Observations
Connections to publicly available websocket servers do not share the same issues I am experiencing.
Many popular remote controls available for python have the two websocket libraries as the implemented solution. None of these were working for me.
Comparing the packets sent by the python implementations and the command line tools do not show a noticeable difference to me.
wscat sends an 80 byte long packet to the TV every four seconds. I suppose this is ping/pong.
How can I investigate further?
What about trying Tornado's websocket implementation?
async def samsung_ws():
ws_req = HTTPRequest("wss://ip:8002/api/v2/samsung.remote.control?name=value", validate_cert=False)
ws = await websocket_connect(ws_req)
ws.write_message("hello")
while True:
msg = await ws.read_message()
if not msg:
break
print(msg)
Depending on how the webserver is built, you might need to request the webpage before connecting to the websocket because in your wscat example I noticed you are using https:// instead of wss://.
Can you post a sample of a dart run (with the url used)?
Um.... I have some experience on websockets connection using python websockets.
In article, you indicate that your websockets connections will automatically disconnected from the server.
I think is the mechanism call "ping-pong" in the websockets module cause this problem.
The mechanism default status is true, that means you will send a ping sign to the websocket server periodically, if the server doesn't send pong back to you, module will consider that server have been shutdown.So you need to do is just set the "ping-pong"
status to False.
async def connect():
async with connect(url, ssl=ssl.CERT_NONE,close_timeout = None,ping_interval = None) as websocket:
res = await websocket.recv()
print(res)
asyncio.get_event_loop().run_until_complete(connect())
This is all my personal opinion, you can try this out.
Why not using a higher level Python module like requests?
You could try the following:
import requests
params = (
('name', 'aW9Ccm9rZXI='),
)
response = requests.get('https://192.168.0.227:8002/api/v2/channels/samsung.remote.control', params=params)
I have an end-to-end pipeline of an web application like below in Python3.6
Socket(connection from client to server) -> Flask Server -> Kafka Producer ->Kafka Consumer ->NLPService
Now when I get some result back from the NLPService, I need to send it back to the client. I am thinking below steps
NLP service writes the result to a different topic on Kafka producer (done)
Kafka consumer retrieves the result from Kafka broker (done)
Kafka consumer needs to write the result to the flask server
Then flask server will send the result back to the socket
Socket writes to client
I have already done steps 1-2. But stuck at step 3, 4. How do I write from Kafka to the flask server? If I just call a function at my server.py, then logically it seems like I have to create a socket within at function at server.py which will do the job of sending to client through socket. But syntax wise it looks weird. What am I missing?
at consumer.py
#receiving reply
topicReply = 'Reply'
consumerReply = KafkaConsumer(topicReply, value_deserializer=lambda m: json.loads(m.decode('ascii')))
for message in consumerReply:
#send reply back to Server
fromConsumer(message.value)
at server.py
socketio = SocketIO(app)
def fromConsumer(msg):
#socketio.on('reply')
def replyMessage(msg):
send(msg)
The above construct in server.py doesn't make sense to me. Please suggest.
I have a very simple Python (Flask socket.io) application which works as a server and another app written in AngularJS which is a client.
In order to handle connected and disconnected client I use respectlivy:
#socketio.on('connect')
def on_connect():
print("Client connected")
#socketio.on('disconnect')
def on_disconnect():
print("Client disconnected")
When Client connects to my app I get information about it, in case if client disconnect (for example because of problems with a network) I don't get any information.
What is the proper way to handle the situation in which client disconnects unexpectedly?
There are two types of connections: using long-pooling or WebSocket.
When you use WebSocket clients knows instantly that server was disconnected.
In the case of long-polling, there is need to set ping_interval and pint_timeout parameters (I also find information about heartbeat_interval and heartbeat_timeout but I don't know how they are related to ping_*).
From the server perspective: it doesn't know that client was disconnected and the only way to get that information is to set ping_interval and ping_timeout.