I'm trying to get a var to my consumers.py to send data to the client in real time as a function does API calls and returns that to the browser.
I know channels needs Redis to function, but why? Why can we not just pass a list as it's built to the consumers class or any variable for that matter? From another answer: to store the necessary information required for different instances of consumers to communicate with one another. But what if I only will use one websocket connection, and only one user is allowed to be logged in at a time? This will be locally hosted only and the function is outside of consumers.py that returns the data so subscribing to groups may be where I need these.
Am I missing something or is redis / memurai a must here? I just can't help but to feel there's an easier way.
I ended up using server side events (SSE), specifically django-eventstream, and so far it's worked great as I didn't need the client to interact with the server, for a chat application this would not work.
Eventstream creates an endpoint at /events/ the client can connect to and receive a streaming http response.
Sending data from externalFunc.py:
send_event('test', 'message', {'text': 'Hello World'})
Event listener in HTML page:
var es = new ReconnectingEventSource('/events/');
es.addEventListener('message', function (e) {
console.log(e.data);
var source = new EventSource("/events/")
var para = document.createElement("P");
const obj = JSON.parse(event.data)
para.innerText = obj.text;
document.body.appendChild(para)
}, false);
es.addEventListener('stream-reset', function (e) {
}, false);
Related
I want to send a message to a websocket client when it connects to the server on AWS lambda and API gateway. Currently, I use wscat as a client. Since the response 'connected' is not shown on the wscat console when I connect to the server, I added post_to_connection to send a message 'hello world' to the client. However, it raises GoneException.
An error occurred (GoneException) when calling the PostToConnection
operation
How can I solve this problem and send some message to wscat when connecting to the server?
My python code is below. I use Python 3.8.5.
import os
import boto3
import botocore
dynamodb = boto3.resource('dynamodb')
connections = dynamodb.Table(os.environ['TABLE_NAME'])
def lambda_handler(event, context):
domain_name = event.get('requestContext',{}).get('domainName')
stage = event.get('requestContext',{}).get('stage')
connection_id = event.get('requestContext',{}).get('connectionId')
result = connections.put_item(Item={ 'id': connection_id })
apigw_management = boto3.client('apigatewaymanagementapi',
endpoint_url=F"https://{domain_name}/{stage}")
ret = "hello world";
try:
_ = apigw_management.post_to_connection(ConnectionId=connection_id,
Data=ret)
except botocore.exceptions.ClientError as e:
print(e);
return { 'statusCode': 500,
'body': 'something went wrong' }
return { 'statusCode': 200,
"body": 'connected'};
Self-answer: you cannot post_to_connection to the connection itself in onconnect.
I have found that the GoneException can occur when the client that initiated the websocket has disconnected from the socket and its connectionId can no longer be found. Is there something causing the originating client to disconnect from the socket before it can receive your return message?
My use case is different but I am basically using a DB to check the state of a connection before replying to it, and not using the request context to do that. This error's appearance was reduced by writing connectionIds to DynamoDB on connect, and deleting them from the table upon disconnect events. Messaging now writes to connectionIds in the table instead of the id in the request context. Most messages go through but some errors are still emitted when the client leaves the socket but does not emit a proper disconnect event which leaves orphans in the table. The next step is to enforce item deletes when irregular disconnections occur. Involving a DB may be overkill for your situation, just sharing what helped me make progress on the GoneException error.
We need to post to connection after connecting (i.e. when the routeKey is not $connect)
routeKey = event.get('requestContext', {}).get('routeKey')
print(routeKey) # for debugging
if routeKey != '$connect': # if we have defined multiple route keys we can choose the right one here
apigw_management.post_to_connection(ConnectionId=connection_id, Data=ret)
#nemy's answer is totally true but it doesn't explain the reason. So, I just want to explain...
So, first of all What is GoneException or GoneError 410 ?
A 410 Gone error occurs when a user tries to access an asset which no longer exists on the requested server. In order for a request to return a 410 Gone status, the resource must also have no forwarding address and be considered to be gone permanently.
you can find more details about GoneException in this article.
In here, GoneException has occured; it means that the POST connection we are trying to make, doesn't exist - which fits perfectly in the scenario. Because we still haven't established the connection between Client and Server. The way APIGatewayWebsocketAPIs work is that you request an Endpoint(Route) and that Endpoint will invoke that Lambda Function (In our case it is ConnectionLambdaFunction for $connect Route).
Now, if The Lambda function resolves with statusCode: 200 then and only then the API Gateway will allow the connection to be established. So, basically untill we return statusCode: 200 from our Lambda Function we are not connected and untill then we are totally unknown to server and thats why the Post call that has been made before the return statement itself will throw an error.
I was wondering, if it is possible to connect (as an admin) with a user and to write messages? I already modified the message-table in the database, however, the message does only appear after reloading the whole website - which is not the optimal outcome. I appreciate every hint. Thank you.
You have to use a pub/sub service to get notification on the client side when a message is sent in server side
A good start is pusher.com service, in your server side code, you will run this example code whenever you send a message to your user:
pusher.trigger('my-channel', 'my-event', {
'message': 'hello world'
})
and in your client side, you add a code that listen to the above channel/topic and act accordingly
var channel = pusher.subscribe('my-channel');
channel.bind('my-event', function(data) {
alert('Received my-event with message: ' + data.message);
});
I am planning to build a home automation system where IoT devices communicate with the MQTT broker.The system also involves a Django web server that serves API for ios/android devices. I will describe an example of what I want to implement.
An API call is made from mobile app to Django server to turn ON a device. When such API request is made to Django, it should push 'Turn ON' data to IoT device via MQTT protocol.
Also, IoT devices send some real time data to the MQTT broker. Upon receiving such data I want to send push notifications to a mobile app via apns/fcm.
How can I implement this?. Will Django channels serve the purpose or should I code my Django server to act as an MQTT client and communicate with the MQTT broker?. Or is there any other methods to implement this.
well, i did a little project on paho-MQTT it's a nice experience with google chrome extension MQTTLens.(you should try this if u aren't using this already)
in your case, I think you can use rest-framework of Django for building an API and on the front-end, you can use crispy-form to make ON-OFF signals and this will directly communicate to the views of Django in which you can write the client and subscriber details.
lets focus on An API call is made from mobile app to Django server to turn ON a device. When such API request is made to Django, it should push 'Turn ON' data to IoT device via MQTT protocol.
you can make views which response to the API call from any devices for that you can check django-rest-framework this is the best option that we have.
and now IoT devices send some real time data to the MQTT broker
for this, you can check Google's this artical. MQTT broker can be handed with the Django views easily and this process not very complex if you use the modular structure with Django's DRY concpet.
on the other hand, you can also make different views for just client or for broker it's up to you but i think this approach will take a long time to devlope such application I don't know about mobile development so i can't help you with that :(.
You can handle the task using JavaScript. I have experience in implementing MQTT protocol in Django and Django-REST projects using JavaScript. You should embed a JavaScript code block in your frontend file (in my case, HTML). First, you should call the Paho-MQTT jQuery package in your file.
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.2/mqttws31.min.js"></script>
Then add this block of code.
#parameters
var hostname = "mqtt.eclipse.org"; #There are different brokers. You should enter the broker's hostname.
var port = 80; #The port number can be different based on a TLS or non-TLS connection.
var ClientID = "ClientID_" + parseInt(Math.random()*100);
#Create a client instance
var client = new Paho.MQTT.Client(hostname, Number(port), ClientID);
#Set callback handlers
client.onConnectionLost = onConnectionLost;
client.onMessageArrived = onMessageArrived;
#Connect the client
client.connect(
{onSuccess: onConnect}
);
#Called when client connects
function onConnect() {
#Once a connection has been established, make a subscription and send a message
console.log("onConnect");
client.subscribe("subTopic");
alert("Connected.");
}
#Called when the client loses its connection
function onConnectionLost(responseObject){
if(responseObject.errorCode != 0){
console.log("onConnectionLost:" + responseObject.errorMessage);
}
}
#Called when a message arrives
function onMessageArrived(message) {
console.log("Message arrived: topic=" + message.destinationName + ", message=" + message.payloadString);
if (message.destinationName == "subTopic") {
#Do something
}
By using the code your application will be connected to the broker and listen to one or multiple topics. It means you can get sensors' data realtime. This necessitates to publish sensor's data on your hardware device, say ESP module or Raspberry PI.
It is very likely that you want to send commands from your application to the actuators to turn them on or off. For doing that you need to publish some messages from your application that your hardware will listen to.
Assume you have a toggle switch that you want to publish a message by toggling that.
<label id="switch{{ device.unique_id }}" class="switch">
<input id="state{{ device.unique_id }}" type="checkbox" onclick="publish('{{ device.unique_id }}')">
<span class="slider round"></span>
</label>
The above HTML block should reside into a django for block. Then you should write the publish onclick function to call that on toggling the switch. You can see an example of such a function below.
function publish(x) {
if(!client){
return;
}
var status = document.getElementById(x);
if (status.innerHTML == 'ON'){
status.innerHTML = 'OFF';
var message = new Paho.MQTT.Message("TurnOFF");
message.destinationName = "pubTopic";
client.send(message);
} else {
status.innerHTML = 'ON';
var message = new Paho.MQTT.Message("TurnON");
message.destinationName = "pubTopic";
client.send(message);
}
}
x in the publish function is the id that is embedded in the HTML file. To receive your published messages you should listen to the specific topic(s) on your hardware device.
I am using Django 1.10 as the backend of a warehousing app we built.
I am adding another new feature where a webpage that will be displayed on a giant monitor. This webpage will show nothing except for 1 giant number.
I have a RFID device that once detects a RFID tag, will send a http request to my Django server.
When that happens, I want the number from the RFID tag to be displayed on the webpage mentioned earlier.
I have read briefly about socket.io but I want to as much as possible keep within the Django universe. I have read briefly about Django Channels as well.
My questions are:
should I use Django Channels for this use case?
if so, how do i do that with my use case above?
It really depends on the usage of the information you're displaying, if you don't need that number to be in real-time then you can opt for the regular AJAX poll once every X seconds like mentionned in zwer's comment.
Now if you need that number to be in real-time, then you should go for websockets and django channels, it's really easy to setup a code base that does what you want.
Assuming you installed django channels and configured your settings.
First you need to setup the consumers.py and routing.py logic that manages the websockets (think of those as views.py and urls.py but for websocket logic).
consumers.py
from channels import Group
from channels.auth import channel_session_user_from_http, channel_session_user
#channel_session_user_from_http
def ws_add(message):
# Authenticates the client
# ASGI WebSocket packet-received and send-packet message types
# both have a "text" key for their textual data.
message.reply_channel.send({
"accept": True,
})
Group("rfid-group").add(message.reply_channel)
#channel_session_user
def ws_message(message):
# You can process messages you receive from the socket here
# Apply w/e logic validation
#channel_session_user
def ws_disconnect(message):
Group("rfid-group").discard(message.reply_channel)
routing.py
from channels.routing import route
from .consumers import ws_message, ws_add, ws_disconnect
routing_routing = [
route("websocket.connect", ws_add),
route("websocket.receive", ws_message),
route("websocket.disconnect", ws_disconnect),
]
Now you need to write the front-end websocket logic:
<script>
socket = new WebSocket("ws://" + window.location.host);
socket.onmessage = function(e) {
console.log("Message received");
// Process the received number here
console.log(e.data);
}
</script>
This will establish a websocket connection, subscribes the client to a group called "rfid-group", now any message sent to that Group will be echoed to all subscribers of that Group, this can handle multiple clients.
Now we need the part that will listen to a request from the rfid device, process it, and send the result to the display, this should be a simple view as the RFID device will send regular HTTP information.
from django.http import HttpResponse
from channels import Group
def rfid_processor(request):
'''
Consider authenticating your rfid_num producer to prevent
someone messing with your api
'''
rfid_num = request.GET.get("rfid", "")
if rfid_num:
Group("rfid-group").send({"text": rfid_num})
return HttpResponse(status=200)
return HttpResponse(status=500)
Hook it to a url:
from app.views import rfid_processor
urlpatterns = [
url(r'^rfid/$', rfid_processor),
]
This is all you need to setup a minimal working django channels project that will echo the number received from the RFID device to the display screen(s).
Hope this helps.
Try reading on this website. https://realpython.com/blog/python/django-and-ajax-form-submissions/
I think this might help you. Also, HassenPy has a pretty good answer. Be sure to read that answer.
My Setup:
I have an existing python script that is using Tweepy to access the Twitter Streaming API. I also have a website that shows aggregate real-time information from other sources from various back-ends.
My Ideal Scenario:
I want to publish real-time tweets as well as real-time updates of my other information to my connected users using Socket.IO.
It would be really nice if I could do something as simple as an HTTP POST (from any back-end) to broadcast information to all the connected clients.
My Problem:
The Socket.IO client implementation is super straight forward... i can handle that. But I can't figure out if the functionality I'm asking for already exists... and if not, what would be the best way to make it happen?
[UPDATE]
My Solution: I created a project called Pega.IO that does what I was looking for. Basically, it lets you use Socket.IO (0.8+) as usual, but you can use HTTP POST to send messages to connected users.
It uses the Express web server with a Redis back-end. Theoretically this should be pretty simple to scale -- I will continue contributing to this project going forward.
Pega.IO - github
To install on Ubuntu, you just run this command:
curl http://cloud.github.com/downloads/Gootch/pega.io/install.sh | sh
This will create a Pega.IO server that is listening on port 8888.
Once you are up and running, just:
HTTP POST http://your-server:8888/send
with data that looks like this:
channel=whatever&secretkey=mysecret&message=hello+everyone
That's all there is to it. HTTP POST from any back-end to your Pega.IO server.
The best way I've found for this sort of thing is using a message broker. Personally, I've used RabbitMQ for this, which seems to meet the requirements mentioned in your comment on the other answer (socket.io 0.7 and scalable). If you use RabbitMQ, I'd recommend the amqp module for node, available through npm, and the Pika module for Python.
An example connector for Python using pika. This example accepts a single json-serialized argument:
def amqp_transmit(message):
connection = pika.AsyncoreConnection(pika.ConnectionParameters(host=settings.AMQP_SETTINGS['host'],
port=settings.AMQP_SETTINGS['port'],
credentials=pika.PlainCredentials(settings.AMQP_SETTINGS['username'],
settings.AMQP_SETTINGS['pass'])))
channel = connection.channel()
channel.exchange_declare(exchange=exchange_name, type='fanout')
channel.queue_declare(queue=NODE_CHANNEL, auto_delete=True, durable=False, exclusive=False)
channel.basic_publish(exchange=exchange_name,
routing_key='',
body=message,
properties=pika.BasicProperties(
content_type='application/json'),
)
print ' [%s] Sent %r' %(exchange_name, message)
connection.close()
Very basic connection code on the node end might look like this:
var connection = amqp.createConnection(
{host: amqpHost,
port: amqpPort,
password: amqpPass});
function setupAmqpListeners() {
connection.addListener('ready', amqpReady)
connection.addListener('close', function() {
console.log('Uh oh! AMQP connection failed!');
});
connection.addListener('error', function(e) {throw e});
}
function amqpReady(){
console.log('Amqp Connection Ready');
var q, exc;
q = connection.queue(queueName,
{autoDelete: true, durable: false, exclusive: false},
function(){
console.log('Amqp Connection Established.');
console.log('Attempting to get an exchange named: '+exchangeName);
exc = connection.exchange(exchangeName,
{type: 'fanout', autoDelete: false},
function(exchange) {
console.log('Amqp Exchange Found. ['+exchange.name+']');
q.bind(exc, '#');
console.log('Amqp now totally ready.');
q.subscribe(routeAmqp);
}
);
}
);
}
routeAmqp = function(msg) {
console.log(msg);
doStuff(msg);
}
Edit: The example above uses a fan-out exchange that does not persist messages. Fan-out exchange is likely going to be your best option since scalability is a concern (ie: you are running more than one box running Node that clients can be connected to).
Why not write your Node app so that there are two parts:
The Socket.IO portion, which communicates directly with the clients, and
An HTTP API of some sort, which receives POST requests and then broadcasts appropriate messages with Socket.IO.
In this way, your application becomes a "bridge" between your non-Node apps and your users' browsers. The key here is to use Socket.IO for what it was made for--real time communication to browsers--and rely on other Node technologies for other parts of your application.
[Update]
I'm not on a development environment at the moment, so I can't get you a working example, but some pseudocode would look something like this:
http = require('http');
io = require('socket.io');
server = http.createServer(function(request, response) {
// Parse the HTTP request to get the data you want
io.sockets.emit("data", whatever); // broadcast the data to Socket.IO clients
});
server.listen(8080);
socket_server = io.listen(server);
With this, you'd have a web server on port 8080 that you can use to listen to web requests (you could use a framework such as Express or one of many others to parse the body of the POST request and extract the data you need).