My script reads data from MQTT server and writes it to postgres table.
I'm using loop_forever().
The program is supposed to run nonstop.
When the first connection is received everything works fine, but after some time (from minutes to days) on_connect() is called again. The program works (in the meaning that there is no error in connection) but no meassages are received any more.
In order to debug I tried following:
induce disconnect by switching off and on the network connection
shutting off and on the server
calling client.disconnect()
To my surprise first and second thing did nothing - there was no logs about new connection and the running program just kept running after the connection revived.
The third attempt was unsuccesfull, I couldn't make it work.
Other remarks:
I tried using loop_start() instead of loop_forever but was not succesfull with that at all
So basically the questions are:
how to counter-act ?
how to disconnect manually to replicate the problem of calling on_connect (and loosing incoming data)
My code:
import json
import sys
from paho.mqtt import client as mqtt_client
import psycopg2
import logging as log
from datetime import datetime
import certifi
from collections import defaultdict
def connect_mqtt(userdict) -> mqtt_client:
def on_connect(client, userdata, flags, rc):
log.info(f"{datetime.now()}: Trying connect")
if rc == 0:
log.info(f"{datetime.now()}: Connection returned result: " + mqtt_client.connack_string(rc))
else:
log.info("Failed to connect, return code %d\n", rc)
client = mqtt_client.Client(client_id=conf['client_id'], protocol=mqtt_client.MQTTv31, userdata=userdict)
client.tls_set(certifi.where())
client.tls_insecure_set(True)
client.username_pw_set(conf['username'], conf['password'])
client.on_connect = on_connect
client.connect(conf['broker'], conf['port'])
return client
def on_message(client, userdata, msg):
now_ts_in_s = round(datetime.timestamp(datetime.now()))
now_dt_in_s = datetime.fromtimestamp(now_ts_in_s)
try:
value = float(msg.payload.decode())
data = [now_dt_in_s, value]
insert_to_psql(userdata['conn'], data)
except ValueError:
pass
def insert_to_psql(conn, data):
cursor = conn.cursor()
insert_query = "INSERT INTO data (time, value) VALUES (%s, %s) ON CONFLICT " \
"DO " \
"NOTHING;"
cursor.execute(insert_query, data)
conn.commit()
def run():
psql_conn = "postgres://postgres:blablabla"
conn = psycopg2.connect(psql_conn)
userdict = {'collected_data': defaultdict(list), 'conn': conn, 'first_conn': True}
client = connect_mqtt(userdict)
client.subscribe(conf['topic'])
client.on_message = on_message
try:
client.loop_forever()
finally:
client.disconnect()
conn.close()
if __name__ == '__main__':
with open(sys.argv[1]) as f:
conf = json.load(f)
run()
If connect is called, then disconnect is probably called before that. Could be some temporary network issue. You should configure the corresponding callback.
Note, that because of this, it's most important that you subscribe in the on_connect callback and not one time outside that. When paho disconnects and connects again, it won't resubscribe automatically. That's why subscriptions should be made in the on_connect callback.
To you question how to test this. You can run a local MQTT broker, and just shut it down after your app has connected the first time and then start it again.
Apart from that, messages won't be lost if you configure your broker accordingly. MQTT has various QOS settings for that specific purpose.
If you think your app is the problem, and its not some networking issue, you could gain a somewhat more solid setup by deploying your app multiple times and let the replicas subscribe via shared subscription. https://www.hivemq.com/blog/mqtt5-essentials-part7-shared-subscriptions/
Related
I am having an issue with the code below.
The code works perfectly at the beginning. First it says Connected to MQTT Broker! and receives data from it. But after a long time (like 6 hours, or 10 hours etc.) it says again Connected to MQTT Broker! and after that id does not receive any other data.
I am trying to make this program work forever, but i don't know what i have done wrong.
Any ideas?
# python3.6
import random
import mysql.connector
from paho.mqtt import client as mqtt_client
import json
# Code for MQTT Connection
broker = 'YOUR_BROKER'
port = 1883
topic = "YOUR_TOPIC"
# generate client ID with pub prefix randomly
client_id = f'python-mqtt-{random.randint(0, 100)}'
username = "THE_USERNAME"
password = "THE_PASSWORD"
# Function to connect on mqtt
def connect_mqtt() -> mqtt_client:
def on_connect(client, userdata, flags, rc):
if rc == 0:
print("Connected to MQTT Broker!")
else:
print("Failed to connect, return code %d\n", rc)
client = mqtt_client.Client(client_id)
client.username_pw_set(username, password)
client.on_connect = on_connect
client.connect(broker, port)
return client
# function to subscribe from mqtt
def subscribeFunc(client: mqtt_client):
def on_messageFunc(client, userdata, msg):
print(f"Received `{msg.payload.decode()}` from `{msg.topic}` topic")
client.subscribe(topic)
client.on_message = on_messageFunc
def run():
client = connect_mqtt()
subscribeFunc(client)
client.loop_forever()
if __name__ == '__main__':
run()
I tried to find the problem but it seems that nothing changed significantly.
I am expecting this program to receive data without stopping.
Network connections may not be 100% reliable (and servers etc are restarted from time to time) so expecting the connection to remain up forever is unrealistic. The issue with your code is that it connects then subscribes; if the connection is dropped it does not resubscribe. As the connection is clean_session=true and subscription qos=0 (the defaults) the broker will forget about any active subscriptions when the connection drops (so the client will reconnect but not receive any more messages).
The Simple solution is to use the approach shown in the docs and subscribe in the on_connect callback (that way the subscription will be renewed after reconnection):
def on_connect(client, userdata, flags, rc):
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("$SYS/#")
client = mqtt.Client()
client.on_connect = on_connect
client.connect("mqtt.eclipseprojects.io", 1883, 60)
You may also want to consider the advice in this answer (as per the comment from #mnikley) because that way the broker will queue up messages for you while the connection is down (otherwise these will be lost).
I'm writing a program with websockets in python. I've got an example server and client code running and they work well if only one client is connected. If there are multiple clients, data from the server will go randomly to one of the clients.
I would like for:
Server to keep track of the various clients connected
Server to be able to direct messages to a specific client out of multiple(For eg. 5) clients
websockets is the library I'm using.
Python version 3.7.2
Server Code:
import asyncio
import websockets
uri='localhost'
async def response(websocket, path):
msg = input("What do you want to send : ")
print("message:",msg)
await websocket.send(msg)
start_server = websockets.serve(response, uri, 5000)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Client Code:
import asyncio
import websockets
uri="ws://localhost:5000"
async def message():
async with websockets.connect(uri) as socket:
print(await socket.recv())
while True:
asyncio.get_event_loop().run_until_complete(message())
If I create 2 files with the client code as client1.py and client2.py, and send message from the server side, I get the sent data going to either on of the clients.
I would like to:
Server keeps track of the various clients connected
Server is able to direct messages to a specific client out of multiple clients
As I am just starting out with websockets, all input is appreciated.
In this output given, I intended to send all my messages to client 1, yet they got split up between client 1 and 2
"websocket" targets the current connection and if you say "websocket.send(msg)" you're sending a message to the client that has just connected and websocket is an object that is reusable while the client is connected. You can assign the websocket as a variable then send a message some other time as long as the connection is still opened.
NOT RECOMMENED
Requiring user's input from the server is not a good idea because now you're awaiting the server until it receives user inputs. Nothing really happens to the server while it's waiting for user's input and this may crush your server.
RECOMMENED
A client has to tell the server which connection / client to send the message to. I would recommend using a JSON format when sending messages within client's and the server and then convert the String to a python-dict since websocket requires only strings.
Click here to check out my GitHub repository. A websockets server made only Python.
SERVER EXAMPLE
Example on how you can send a packet to a specific client
I'm not familiar with asyncio, so I will try to get to the point with functions/threads;
Usually, my server side listens to one connection and once it accepts it, I have a function 'handler' that is threaded to each connection that gets accepted.
part of my server and handler:
def handler(conn, addr):
global data1
while True:
data = conn.recv(2048)
data1 = json.loads(data.decode())
# treat it as you need
while True:
s.listen(1)
conn, addr = s.accept()
print('Conectado com', addr[0],':', str(addr[1]))
thr = threading.Thread(target = handler, args = (conn, addr)).start()
Now, for the control of the clients and such, I always use a dictionary. The key should be the username or any other particular info. The value of the key is the 'conn' from that user. This way you can get the connection of user 'x' by its specific key.
Something like:
import socket
import time
import datetime as dt
import base64
import os
import json
import threading
HOST = ''
PORT = 12999
global s
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST,PORT))
whoto = {}
def autenticacao():
global data1
global conn
global nomeuser
user1 = data1[1]
passwd = data1[2]
auth = c.execute('SELECT usuario FROM fullinfo WHERE usuario= ? AND password = ?', (user1,passwd)).fetchone()
if auth is not None:
word = 'autenticado'
conn.sendall(word.encode())
nomeuser = auth[0]
whoto[nomeuser] = conn
I'm sorry i'm leaving it unreproducible, but my point is to show the 'algorithm'. This dictionary is what I use to keep record of who is online, the 'adress' (conn) of each client to send messages to single clients and such.
On the example above, I add the user (key) and conn (value) once it's authenticated inside my server.
Hope this helps. Good luck!
I have my Tornado client continuously listening to my Tornado server in a loop, as it is mentioned here - http://tornadoweb.org/en/stable/websocket.html#client-side-support. It looks like this:
import tornado.websocket
from tornado import gen
#gen.coroutine
def test():
client = yield tornado.websocket.websocket_connect("ws://localhost:9999/ws")
client.write_message("Hello")
while True:
msg = yield client.read_message()
if msg is None:
break
print msg
client.close()
if __name__ == "__main__":
tornado.ioloop.IOLoop.instance().run_sync(test)
I'm not able to get multiple instances of clients to connect to the server. The second client always waits for the first client process to end before it connects to the server. The server is set up as follows, with reference from Websockets with Tornado: Get access from the "outside" to send messages to clients and Tornado - Listen to multiple clients simultaneously over websockets.
class WSHandler(tornado.websocket.WebSocketHandler):
clients = set()
def open(self):
print 'new connection'
WSHandler.clients.add(self)
def on_message(self, message):
print 'message received %s' % message
# process received message
# pass it to a thread which updates a variable
while True:
output = updated_variable
self.write_message(output)
def on_close(self):
print 'connection closed'
WSHandler.clients.remove(self)
application = tornado.web.Application([(r'/ws', WSHandler),])
if __name__ == "__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(9999)
tornado.ioloop.IOLoop.instance().start()
But this has not worked - for some reason even after I have made a successful first connection, the second connection just fails to connect i.e. it does not even get added to the clients set.
I initially thought the while True would not block the server from receiving and handling more clients, but it does as without it multiple clients are able to connect. How can I send back continuously updated information from my internal thread without using the while True?
Any help would be greatly appreciated!
To write messages to client in a while loop, you can use the yield None inside the loop. This will pause the while loop and then Tornado's IOLoop will be free to accept new connections.
Here's an example:
#gen.coroutine
def on_message(self):
while True:
self.write_message("Hello")
yield None
Thanks for your answer #xyres! I was able to get it to work by starting a thread in the on_message method that handed processing and the while True to a function outside the WSHandler class. I believe this allowed for the method to run outside of Tornado's IOLoop, unblocking new connections.
This is how my server looks now:
def on_message(self, message):
print 'message received %s' % message
sendThread = threading.Thread(target=send, args=(self, message))
sendThread.start()
def send(client, msg):
# process received msg
# pass it to a thread which updates a variable
while True:
output = updated_variable
client.write_message(output)
Where send is a function defined outside the class which does the required computation for me and writes back to client inside thewhile True.
I have configured MQTT Broker, receiving published messages from another piece of code (not written or accessible by me). I added another topic to the broker configuration and am now trying to publish data to this new topic from a piece of python code. I get the feedback that the message is published by the callback function, but no actual data is received.
Is there anything I am missing?
I am using the following code:
import paho.mqtt.client as mqtt
import time
#=========================================================================
def on_connect(client, userdata, flags, rc) :
print "on_connect()"
#=========================================================================
def on_publish(client, userdata, mid) :
print "on_publish()"
#=========================================================================
def send() :
mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
#host = "localhost"
host = "127.0.0.1"
port = 1883
keepalive = 60
print "\nConnect to '%s', port '%s', keepalive '%s'" % (host, port, keepalive)
mqttc.connect(host=host, port=port, keepalive=keepalive)
time.sleep(3)
mqttc.loop_start()
time.sleep(3)
topic = "data/MY/TOPIC"
msg = "MY_MESSAGE"
print "Publish to '%s' msg '%s'" % (topic, msg)
mqttc.publish(topic, msg, qos=2)
time.sleep(3)
mqttc.loop_stop()
# end send()
#=========================================================================
if __name__ == "__main__":
send()
# end if
Getting the stdout
Connect to '127.0.0.1', port '1883', keepalive '60'
on_connect()
Publish to 'data/MY/TOPIC' msg 'MY MESSAGE'
on_publish()
I am not sure if the loop() functions are necessary, but if I do not embed the publishing in the loop_start() and loop_stop(), I do not get a on_connect callback.
The loop functions are necessary as these are where all the network traffic is processed.
Manually setting up a connection to the broker to just send a single message like this is not a good idea, it would be better to start the client, leave it running (by calling loop_start() and not calling loop_stop()) in the background and then just call the publish method on the mqttc client object.
If you don't want to keep a instance of the client running then you should use the single message publish helper method provided by the paho python library (https://pypi.python.org/pypi/paho-mqtt/1.1#single):
import paho.mqtt.publish as publish
publish.single("paho/test/single", "payload", hostname="iot.eclipse.org")
I have tested the example program on paho-mqtt, and I know that the function loop_forever() can handles reconnecting. But my question is that, although loop_forever() can reconnect, it cannot re-subscribe. It should be a problem when the server suddenly crashes, in this case, the client is still listening, but when the server is restarted, the client can reconnect, but cannot subscribe message any more. I think maybe I should re-write the loop_forever() function, but I am not sure if I were right, and how to do it.
import sys
try:
import paho.mqtt.client as mqtt
except ImportError:
# This part is only required to run the example from within the examples
# directory when the module itself is not installed.
#
# If you have the module installed, just use "import paho.mqtt.client"
import os
import inspect
cmd_subfolder = os.path.realpath(os.path.abspath(os.path.join(os.path.split(inspect.getfile( inspect.currentframe() ))[0],"../src")))
if cmd_subfolder not in sys.path:
sys.path.insert(0, cmd_subfolder)
import paho.mqtt.client as mqtt
def on_connect(mqttc, obj, flags, rc):
print("rc: "+str(rc))
def on_message(mqttc, obj, msg):
print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))
def on_publish(mqttc, obj, mid):
print("mid: "+str(mid))
def on_subscribe(mqttc, obj, mid, granted_qos):
print("Subscribed: "+str(mid)+" "+str(granted_qos))
def on_log(mqttc, obj, level, string):
print(string)
# If you want to use a specific client id, use
# mqttc = mqtt.Client("client-id")
# but note that the client id must be unique on the broker. Leaving the client
# id parameter empty will generate a random id for you.
mqttc = mqtt.Client()
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_subscribe = on_subscribe
# Uncomment to enable debug messages
#mqttc.on_log = on_log
mqttc.connect("m2m.eclipse.org", 1883, 60)
mqttc.subscribe("$SYS/#", 0)
mqttc.loop_forever()
The easy way to deal with this is to do your subscribing in the on_connect callback, then when you reconnect all the subscriptions will be restored as well.
While instantiating your mqtt client you can set the "clean session" flag to false.
mqttc = mqtt.Client(clean_session=False)
Citation from the mosquitto manual:
Clean session / Durable connections
On connection, a client sets the "clean session" flag, which is sometimes also known as the "clean start" flag. If clean session is set to false, then the connection is treated as durable. This means that when the client disconnects, any subscriptions it has will remain and any subsequent QoS 1 or 2 messages will be stored until it connects again in the future. If clean session is true, then all subscriptions will be removed for the client when it disconnects.