I have web application based on Node.js and python based hardware. I want to keep status of connectivity between web server and hardware. If Hardware gets disconnected from web application then web application should get event or notification so based on that i can send notification to user. I have used mqtt for data communication, but to keep connection status I can't use MQTT coz it is connected with broker. I don't want to increase more load on server.
Which tools/technology/protocol/method should i use to keep connection status that device is offline or online?. That I also want use when user tries to send data to hardware using web application and if device is not connected with server then user should get notification that device is offline based on connection status.
The following code demonstrates the process I was hinting at in the comments.
The LWT feature tells the broker to publish a message marking the client as offline when it fails to respond in 1.5 times the keepalive period. If the client cleanly disconnects then it needs to mark it's self as offline. The client marks it's self as online when it connects to the broker.
All the status messages have the retained bit set so they will always be delivered when a client subscribes to the status topic.
import paho.mqtt.client as mqtt
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("some/application/topic")
# set status message to online
client.publish("status/client1", payload="online", retain=True)
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
if str(msg.payload) == "shutdown":
# update status to offline as this will be a clean dissconect
client.publish("status/client1", payload="offline", retain=True)
client.disconnect()
client = mqtt.Client(client_id="client1")
client.on_connect = on_connect
client.on_message = on_message
client.will_set("status/client1", payload="offline", retain=True)
client.connect("mqtt.eclipse.org", 1883, 60)
# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_forever()
It will be up to the OP to implement the notification about sending messages to offline clients requested at the end of the question.
Related
I would like to set-up a server that can subscribe to an external stream over a websocket (ws_ext) and then republish that data (after curating) to internal clients connecting to this server over websockets (ws_int).
My approach so far is to set up a fastapi server that can open websockets (ws_int) with internal clients .
However, I don't understand how to have a listener embedded in this server that can listen to external stream and then publish to these internal clients in a non blocking way.
Can someone point me to a working example that can help?
Here is what i would like to achieve:
p.s: I HAVE BEEN able to make it work by decoupling broadcaster from subscriber using redis pubsub. So, what i have now setup is a client that listens to external stream, curate and pushes it to redis pubsub. then i have a separate broadcaster that listens to redis pubsub and pushes it out to clients after curating on its websockets. I would still love to combine these two without using redis or some such backend.
if you have all clients connected to an async ws located in the broadcaster then the same time push whatever that's coming asynchronously to broadcaster from external website the process should be unblocking supposedly
the update process can have a async stream pipeline to filter results coming from external website for each client in broadcaster
as for example to async client for WebSocket it can go by "with async"
async def hello():
async with websockets.connect(
'ws://localhost:8765', ssl=ssl_context) as websocket:
name = input("What's your name? ")
await websocket.send(name)
print(f"> {name}")
greeting = await websocket.recv()
print(f"< {greeting}")
asyncio.get_event_loop().run_until_complete(hello())
EDIT: In an initial version of this question I was using the Python paho-mqtt client version paho-mqtt 1.5.1. The error message I received with that version was on_disconnect 1: Out of memory. error code 1. It was recommended to install the newer version (1.6) of the paho-mqtt client and rerun the code. I did that and still encounter the same spontaneous disconnection, yet the newer version has more descriptive error messages, With the 1.6 version I receive the disconnection error code 7, which means that "The connection was lost".
I am simulating an IoT device. As part of this simulation I have created a Messenger class, which, when instantiated, communicates with a virtual device on Google Cloud's IoT core with the MQTT protocol. The client I am using frequently disconnects leading to lost messages. I am hoping to find a way to prevent the client from randomly disconnecting.
The client used for this communication, which is embedded in the Messenger class, is the python paho mqtt client.
The client id is derived from the project_id, cloud_region, registry_id and device_id using the following code
client_id = "projects/{}/locations/{}/registries/{}/devices/{}".format(project_id, cloud_region, registry_id, device_id)
The client communicates with IoT core in three different ways:
It publishes data to the events topic
sub_topic = "events"
mqtt_topic = "/devices/{}/{}".format(self.device_id, sub_topic)
client.publish(mqtt_topic, payload, qos=1)
It subscribes to the virtual device's config topic
mqtt_config_topic = "/devices/{}/config".format(self.device_id)
sub_status = client.subscribe(mqtt_config_topic, qos=1)
It sends state information to the device's state topic at a frequency no greater than once every 12 seconds. (Google Cloud recommends no more than once every 10 seconds so I added a few more seconds for wiggle room).
sub_topic = "state"
mqtt_topic = "/devices/{}/{}".format(self.device_id, sub_topic)
payload = json.dumps(new_state)
client.publish(mqtt_topic, payload, qos=0)
Initially I noticed that not all of the data sent to be published to the events topic was sent. After examining the log file, I determined the issue was caused by the frequent disconnects of the client. The error code given for the client disconnect is 1, which is technically an "out of memory" error, but also occurs when GCP disconnects the client. I think the issue lies with GCP disconnecting the client, but I am not entirely sure
Here is an excerpt of a log file:
DEBUG Controller4.testA [04/Nov/2021:12:54:57.862] new client created
DEBUG Controller4.testA [04/Nov/2021:12:54:57.865] Publishing state to /devices/<DeviceID>/state
DEBUG Controller4.testA [04/Nov/2021:12:54:57.867] Sending PUBLISH (d0, q0, r0, m2), 'b'/devices/<DeviceID>/state'', ... (197 bytes)
DEBUG Controller4.testA [04/Nov/2021:12:54:57.870] Not publishing state: Previous state update 0:00:00.004303 seconds ago
DEBUG Controller4.testA [04/Nov/2021:12:54:57.870] on_publish client: <paho.mqtt.client.Client object at 0x15bf73670> userdata: None mid: 2
DEBUG Controller4.testA [04/Nov/2021:12:54:57.871] Sending PUBACK (Mid: 1)
DEBUG Controller4.testA [04/Nov/2021:12:54:57.947] Received CONNACK (0, 0)
DEBUG Controller4.testA [04/Nov/2021:12:54:57.951] on_connect Connection Accepted.
DEBUG Controller4.testA [04/Nov/2021:12:54:57.976] Received SUBACK
DEBUG Controller4.testA [04/Nov/2021:12:54:57.976] Subscribed: (1,)
DEBUG Controller4.testA [04/Nov/2021:12:54:57.978] Received PUBLISH (d0, q1, r0, m1), '/devices/<DeviceID>/config', ... (20 bytes)
INFO Controller4.testA [04/Nov/2021:12:54:57.979] Received new state message '{"osmo": 6, "pH": 4}' on topic '/devices/<DeviceID>/config' with Qos 1
DEBUG Controller4.testA [04/Nov/2021:12:54:57.980] Not publishing state: Previous state update 0:00:00.114637 seconds ago
DEBUG Controller4.testA [04/Nov/2021:12:54:57.981] Sending PUBACK (Mid: 1)
DEBUG Controller4.testA [04/Nov/2021:12:55:01.958] on_disconnect 7: The connection was lost. error code 7<paho.mqtt.client.Client object at 0x15bf60b80>None
I am currently working on a virtual private cloud, but I am not sure that matters.
Why does the client keep disconnecting? How can I further troubleshoot this problem?
I have a very simple Python (Flask socket.io) application which works as a server and another app written in AngularJS which is a client.
In order to handle connected and disconnected client I use respectlivy:
#socketio.on('connect')
def on_connect():
print("Client connected")
#socketio.on('disconnect')
def on_disconnect():
print("Client disconnected")
When Client connects to my app I get information about it, in case if client disconnect (for example because of problems with a network) I don't get any information.
What is the proper way to handle the situation in which client disconnects unexpectedly?
There are two types of connections: using long-pooling or WebSocket.
When you use WebSocket clients knows instantly that server was disconnected.
In the case of long-polling, there is need to set ping_interval and pint_timeout parameters (I also find information about heartbeat_interval and heartbeat_timeout but I don't know how they are related to ping_*).
From the server perspective: it doesn't know that client was disconnected and the only way to get that information is to set ping_interval and ping_timeout.
I'm using the gevent-websocket module with Bottle Python framework.
When a client closes the browser, this code
$(window).on('beforeunload', function() { ws.close(); });
helps to close the websocket connection properly.
But if the client's network connection is interrupted, no "close" information can be sent to the server.
Then, often, even 1 minute later, the server still believes the client is connected, and the websocket is still open on the server.
Question: How to detect properly that a websocket is closed because the client is disconnected from network?
Is there a websocket KeepAlive feature available in Python/Bottle/gevent-websocket?
One answer from Web Socket: cannot detect client connection on internet disconnect suggests to use a heartbeat/ping packet every x seconds to tell the server "I'm still alive". The other answer suggests using a setKeepAlive(true). feature. Would this feature be available in gevent-websocket?
Example server code, taken from here:
from bottle import get, template, run
from bottle.ext.websocket import GeventWebSocketServer
from bottle.ext.websocket import websocket
users = set()
#get('/')
def index():
return template('index')
#get('/websocket', apply=[websocket])
def chat(ws):
users.add(ws)
while True:
msg = ws.receive()
if msg is not None:
for u in users:
u.send(msg)
else:
break
users.remove(ws)
run(host='127.0.0.1', port=8080, server=GeventWebSocketServer)
First you need to add a timeout to the receive() method.
with gevent.Timeout(1.0, False):
msg = ws.receive()
Then the loop will not block, if you send even an empty packet and the client doesn't respond, WebsocketError will be thrown and you can close the socket.
I have installed the last version of raspbian on my raspberry pi, and I have opened an account AWS IoT on Amazon, then in the IoT web interface I have created a thing, with "RaspberryPi_2" name and create certificate and connect the certificate to the thing, I have followed this guide:
http://blog.getflint.io/blog/get-started-with-aws-iot-and-raspberry-pi
I have then created the script in the guide, to connect and subscribed the raspberry pi, this is my code:
#!/usr/bin/python3
#required libraries
import sys
import ssl
import paho.mqtt.client as mqtt
#called while client tries to establish connection with the server
def on_connect(mqttc, obj, flags, rc):
if rc==0:
print ("Subscriber Connection status code: "+str(rc)+" | Connection status: successful")
elif rc==1:
print ("Subscriber Connection status code: "+str(rc)+" | Connection status: Connection refused")
#called when a topic is successfully subscribed to
def on_subscribe(mqttc, obj, mid, granted_qos):
print("Subscribed: "+str(mid)+" "+str(granted_qos)+"data"+str(obj))
#called when a message is received by a topic
def on_message(mqttc, obj, msg):
print("Received message from topic: "+msg.topic+" | QoS: "+str(msg.qos)+" | Data Received: "+str(msg.payload))
#creating a client with client-id=mqtt-test
mqttc = mqtt.Client(client_id="mqtt-test")
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
mqttc.on_message = on_message
#Configure network encryption and authentication options. Enables SSL/TLS support.
#adding client-side certificates and enabling tlsv1.2 support as required by aws-iot service
mqttc.tls_set("/home/pi/aws_iot/things/raspberryPi_2/certs/aws-iot-rootCA.crt",
certfile="/home/pi/aws_iot/things/raspberryPi_2/certs/0ea2cd7eb6-certificate.pem.crt",
keyfile="/home/pi/aws_iot/things/raspberryPi_2/certs/0ea2cd7eb6-private.pem.key",
tls_version=ssl.PROTOCOL_TLSv1_2,
ciphers=None)
#connecting to aws-account-specific-iot-endpoint
mqttc.connect("A2GF7W5U5A46J1.iot.us-west-2.amazonaws.com", port=8883) #AWS IoT service hostname and portno
#the topic to publish to
mqttc.subscribe("$aws/things/RaspberryPi_2/shadow/update/#", qos=1) #The names of these topics start with $aws/things/thingName/shadow."
#automatically handles reconnecting
mqttc.loop_forever()
but when I do this command:
python3 mqtt_test.py
or this command:
python mqtt_test.py
and press enter, and the cursor flash and doesn't print anything and stay stuck there, someone can help me?
I haven't also understand if the client-id name should be the same of the things name, and the meaning of the subscribe path, for example in a tutorial I have found this:
mqttc.publish("temperature", tempreading, qos=1)
why there isn't the complete path?
or this:
$aws/things/RaspberryPi_2/shadow/update/delta
so I can put everything I want in the path?
thanks
The code is subscribing to a topic but there is no one publishing to it. So, the code also has a on_connect function that would be triggered after a success connection. Is the message "Subscriber Connection status code: ..." being printed? If it is, the message from on_subscribe should also appear. Is it?
If it is not you have a problem before connect to the AWS server. Use netstat command to see where your Raspberry Pi is connected or not and post more debug information in this case.
If the connect and subscribe messages are shown and nothing happens after it, this is normal because you are only subscribing to a topic but not publishing anything.
Regarding topics, think of them as a directory structure. There is no strict rule for topic hierarchy. A "temperature" topic would be temperature topic on the server and a "temperature/livingroom" would be temperature/livingroom, you can subscribe to one, another or both on the same server. The path you choose for your things shall be meaningful to your application. A house, for instance, might be represented as:
house/kitchen/env/temperature
house/kitchen/env/humidity
house/kitchen/lamp/sinklamp
house/kitchen/lamp/mainlap
house/masterbed/env/temperature
house/masterbed/env/humidity
house/masterbed/lamp/readinglampleft
house/masterbed/lamp/readinglampright
house/masterbed/lamp/mainlamp
house/masterbed/lamp/mirrorlamp
And so on.
Let´s say you have a thermostat at master bedroom. It is interested only in temperature but not humidity. It is also interested only in master bedroom temperature. This thermostat shall subscribe to house/masterbed/env/temperature. Opposite to this, a room wide panel that shows state of every thing in the room, would subscribe to house/masterbed/#, meaning "everything after house/masterbed". Read more on wildcards here
The topic you subscribed for: $aws/things/RaspberryPi_2/shadow/update/# means, "every thing after $aws/things/RaspberryPi_2/shadow/update/". Notice that his is a special topic, it starts with $aws, specially, it starts with $ character. In the AWS context this means:
Any topics beginning with $ are considered reserved and are not
supported for publishing and subscribing except when working with the
Thing Shadows service. For more information, see Thing Shadows.
So you need to understand what thing shadow is. This is a AWS specific (and very util) concept. Please read docs on this subject.
Finally, I would you suggest you install a local broker (mosquitto is available on respbian) and try with it before got to AWS. This way you can master mqtt concept without connectivity issues. Later you put AWS to the mix.