We Keep Coding

Mqtt Subscriber, loop_forever() does not work

I am having an issue with the code below.
The code works perfectly at the beginning. First it says Connected to MQTT Broker! and receives data from it. But after a long time (like 6 hours, or 10 hours etc.) it says again Connected to MQTT Broker! and after that id does not receive any other data.
I am trying to make this program work forever, but i don't know what i have done wrong.
Any ideas?
# python3.6
import random
import mysql.connector
from paho.mqtt import client as mqtt_client
import json
# Code for MQTT Connection
broker = 'YOUR_BROKER'
port = 1883
topic = "YOUR_TOPIC"
# generate client ID with pub prefix randomly
client_id = f'python-mqtt-{random.randint(0, 100)}'
username = "THE_USERNAME"
password = "THE_PASSWORD"
# Function to connect on mqtt
def connect_mqtt() -> mqtt_client:
def on_connect(client, userdata, flags, rc):
if rc == 0:
print("Connected to MQTT Broker!")
else:
print("Failed to connect, return code %d\n", rc)
client = mqtt_client.Client(client_id)
client.username_pw_set(username, password)
client.on_connect = on_connect
client.connect(broker, port)
return client
# function to subscribe from mqtt
def subscribeFunc(client: mqtt_client):
def on_messageFunc(client, userdata, msg):
print(f"Received `{msg.payload.decode()}` from `{msg.topic}` topic")
client.subscribe(topic)
client.on_message = on_messageFunc
def run():
client = connect_mqtt()
subscribeFunc(client)
client.loop_forever()
if __name__ == '__main__':
run()
I tried to find the problem but it seems that nothing changed significantly.
I am expecting this program to receive data without stopping.

Network connections may not be 100% reliable (and servers etc are restarted from time to time) so expecting the connection to remain up forever is unrealistic. The issue with your code is that it connects then subscribes; if the connection is dropped it does not resubscribe. As the connection is clean_session=true and subscription qos=0 (the defaults) the broker will forget about any active subscriptions when the connection drops (so the client will reconnect but not receive any more messages).
The Simple solution is to use the approach shown in the docs and subscribe in the on_connect callback (that way the subscription will be renewed after reconnection):
def on_connect(client, userdata, flags, rc):
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("$SYS/#")
client = mqtt.Client()
client.on_connect = on_connect
client.connect("mqtt.eclipseprojects.io", 1883, 60)
You may also want to consider the advice in this answer (as per the comment from #mnikley) because that way the broker will queue up messages for you while the connection is down (otherwise these will be lost).

MQTT Subscriber to ThingsBoard broker in python

Situation:
I have a python virtual sensor (a python program) that submits data via MQTT protocol to my device in ThingsBoard.
I can visualize data on the dashboard, so I'm sure that the data are received.
Problem:
When I try to connect a python subscriber to the thingsboard broker (demo.thingsboard.io) using paho I obtain that the connection code is 0, so the connection is OK, however I see that the dashboard stops visualising the data from the virtual sensor but the subscriber does not receive anything.
The virtual sensor is publishing at v1/devices/me/telemetry and the subscriber is subscribed at the same topic v1/devices/me/telemetry.
How to show the data published by the virtual sensor on my subscriber client?
VIRTUAL SENSOR CODE:
import paho.mqtt.client as paho #mqtt library
import os
import json
import time
from datetime import datetime
ACCESS_TOKEN="vgFztmvT6bps7JCeOEZq" #Token of your device
broker="demo.thingsboard.io" #host name
port=1883 #data listening port
def on_publish(client,userdata,result): #create function for callback
print("data published to thingsboard \n")
pass
client1= paho.Client("control1") #create client object
client1.on_publish = on_publish #assign function to callback
client1.username_pw_set(ACCESS_TOKEN) #access token from thingsboard device
client1.connect(broker,port,keepalive=60) #establish connection
while True:
payload="{"
payload+="\"Humidity\":60,";
payload+="\"Temperature\":25";
payload+="}"
ret= client1.publish("v1/devices/me/telemetry",payload) #topic-v1/devices/me/telemetry
print("Please check LATEST TELEMETRY field of your device")
print(payload);
time.sleep(5)
CLIENT SUBSCRIBER CODE:
import paho.mqtt.client as mqtt
import time
token = "vgFztmvT6bps7JCeOEZq"
broker="demo.thingsboard.io" # host name
port=1883
topic = "v1/devices/me/telemetry"
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc) :
if (rc==0) :
print("connected OK Returned code = ", rc)
else :
print("Bad connection Returned code = ", rc)
def on_message(client, userdata, msg) :
print (msg.topic + " " + str(msg.payload))
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set(token)
client.connect(broker , port, 60)
client.subscribe(topic)
client.loop_forever()

In you publisher client , you are using a topic named:"v1/devices/me/telemetry".
However, you are not subscribing to the same topic using the subscriber client.
Change the following line in your Subscriber client program:
client.subscribe(token)
to
client.subscribe(topic)
This should solve the problem.

I believe TB will only allow for one connection per client so you should use different access_keys for your publisher and subscriber.

Try to use tcp://51.159.155.114 but not demo.thingsboard.io

mqtt client does not receive message in case of thread and rest-api

I have a python script, which based on flask and mqtt. The use case is to receive a request via rest-api then to create a new thread which publishes some messages on mosquitto mqtt and expects a response (see subscribe). My problem is that I don't receive any messages. I think it has something to do with the thread, because without the thread it's working very fine..
Do you know what can be the problem?
Thank you in anticipation!
here the code:
from flask import Flask, Response
import paho.mqtt.client as mqtt
from threading import Thread
import threading
app = Flask(__name__)
lock = threading.Lock()
def on_connect(client, userdata, flags, rc): # The callback for when the client connects to the broker
print("Connected with result code {0}".format(str(rc))) # Print result of connection attempt
client.subscribe("/mytopic")
def on_message(client, userdata, msg): # The callback for when a PUBLISH message is received from the server.
print(msg.topic)
client = mqtt.Client(client_id=client_name, clean_session=True)
client.on_connect = on_connect # Define callback function for successful connection
client.on_message = on_message # Define callback function for receipt of a message
client.username_pw_set(mqtt_user, mqtt_password)
client.loop_start()
client.connect(mqtt_host)
def test(param1, param2):
lock.acquire()
try:
ret = client.publish("/mytopic", "")
while True:
check the response from mqtt => but i don't get the response anymore
....
break
finally:
lock.release()
return result
#app.route('/test/check', methods=['POST'])
def check():
global sessionId
sessionId = sessionId + 1
t = Thread(target=test, args=(sessionId,None))
t.start()
return {'id': sessionId, 'eta': 0}
if __name__ == '__main__':
app.run(debug=True)

There are a couple of problems with this.
Both the client.connect() and the client.subscribe() calls need iterations of the client network loop to run in order to complete properly.
The network loop needs to run at least once every keep alive period the time after the connection has been made in order to stop the broker disconnecting the client as dead. This means if there is a delay between starting the code and the first REST request then the client will be disconnected.
Better to use the client.start_loop() function to run MQTT client network loop continuously in the background on it's own.
You should also remove the call to client.subscribe() that is outside the on_connect() callback.
EDIT:
As hashed out in the comments/chat the following works. It looks like running the flask app in debug mode does some strange things and creates multiple MQTT clients over and over again with the same client id. This leads to the broker constantly kicking the old ones off so messages never get delivered.
from flask import Flask, Response
import paho.mqtt.client as mqtt
import time
from threading import Thread
import threading
app = Flask(__name__)
lock = threading.Lock()
sessionId=0
cont=True
def on_connect(client, userdata, flags, rc): # The callback for when the client connects to the broker
print("Connected with result code {0}".format(str(rc))) # Print result of connection attempt
client.subscribe("mytopic")
def on_message(client, userdata, msg): # The callback for when a PUBLISH message is received from the server.
global cont
print(msg.topic)
cont=False
client = mqtt.Client(client_id="foo", clean_session=True)
client.on_connect = on_connect # Define callback function for successful connection
client.on_message = on_message # Define callback function for receipt of a message
#client.username_pw_set(mqtt_user, mqtt_password)
client.connect("localhost", port=1884)
client.loop_start()
def test(param1, param2):
lock.acquire()
try:
ret = client.publish("mytopic", "foo")
while cont:
time.sleep(5)
print("loop")
finally:
lock.release()
result = "foo"
return result
#app.route('/test/check', methods=['POST'])
def check():
global sessionId
sessionId = sessionId + 1
t = Thread(target=test, args=(sessionId,None))
t.start()
return {'id': sessionId, 'eta': 0}
if __name__ == '__main__':
print("started")
app.run()

Python, MQTT broker, publish message

I have configured MQTT Broker, receiving published messages from another piece of code (not written or accessible by me). I added another topic to the broker configuration and am now trying to publish data to this new topic from a piece of python code. I get the feedback that the message is published by the callback function, but no actual data is received.
Is there anything I am missing?
I am using the following code:
import paho.mqtt.client as mqtt
import time
#=========================================================================
def on_connect(client, userdata, flags, rc) :
print "on_connect()"
#=========================================================================
def on_publish(client, userdata, mid) :
print "on_publish()"
#=========================================================================
def send() :
mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
#host = "localhost"
host = "127.0.0.1"
port = 1883
keepalive = 60
print "\nConnect to '%s', port '%s', keepalive '%s'" % (host, port, keepalive)
mqttc.connect(host=host, port=port, keepalive=keepalive)
time.sleep(3)
mqttc.loop_start()
time.sleep(3)
topic = "data/MY/TOPIC"
msg = "MY_MESSAGE"
print "Publish to '%s' msg '%s'" % (topic, msg)
mqttc.publish(topic, msg, qos=2)
time.sleep(3)
mqttc.loop_stop()
# end send()
#=========================================================================
if __name__ == "__main__":
send()
# end if
Getting the stdout
Connect to '127.0.0.1', port '1883', keepalive '60'
on_connect()
Publish to 'data/MY/TOPIC' msg 'MY MESSAGE'
on_publish()
I am not sure if the loop() functions are necessary, but if I do not embed the publishing in the loop_start() and loop_stop(), I do not get a on_connect callback.

The loop functions are necessary as these are where all the network traffic is processed.
Manually setting up a connection to the broker to just send a single message like this is not a good idea, it would be better to start the client, leave it running (by calling loop_start() and not calling loop_stop()) in the background and then just call the publish method on the mqttc client object.
If you don't want to keep a instance of the client running then you should use the single message publish helper method provided by the paho python library (https://pypi.python.org/pypi/paho-mqtt/1.1#single):
import paho.mqtt.publish as publish
publish.single("paho/test/single", "payload", hostname="iot.eclipse.org")

How Can I Publish File to AWS- IoT using Mosquitto in Python

I am trying to publish a file to AWS IoT using Mosquitto and python. The file i need to publish is a jpg file that is in my local directory. I have secret key and access key that are required to publish data to IoT. I am very new to programming and don't know how to write this program. can someone help me please? I apologize if this is something very basic. Thank you
I have already tried this How can I publish a file using Mosquitto in python? and did not work for me.
This is how i tired to do.
> #!/usr/bin/python
import mosquitto import sys
import ssl
import paho.mqtt.client as mqtt
f = open("data")
imagestring = f.read()
byteArray = bytes(imagestring)
client.publish("photo", byteArray ,0)
#called when a message is received by a topic
def on_message(mqttc, obj, msg):
print("Received message from topic: "+msg.topic+" | QoS: "+str(msg.qos)+"Data Received: "+str(msg.payload))
#creating a client with client-id=mqtt-test
mqttc = mqtt.Client(client_id="mqtt-test")
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
mqttc.on_message = on_message
#Configure network encryption
mqttc.tls_set("/home/username/root-CA.crt",
certfile="/home/username/6fdda68178-certificate.pem.crt",
keyfile="/home/username/6fdda68178-private.pem.key",
tls_version=ssl.PROTOCOL_TLSv1_2,
ciphers=None)
#connecting to aws-account-specific-iot-endpoint
mqttc.connect("A2DL8ZE59089FKF.iot.us-west-2.amazonaws.com", port=8883)
#the topic to publish to
mqttc.subscribe("$aws/things/mqtt-listener/shadow/update/#", qos=1)
#automatically handles reconnecting
mqttc.loop_forever()

It looks like you have some things backwards in your code, or at least hard to understand. Here's an example of some working code to upload a binary file to AWS IOT.
#!/usr/bin/python
import paho.mqtt.client as paho
import os
import socket
import ssl
from time import sleep
from random import uniform
connflag = False
def on_connect(client, userdata, flags, rc):
global connflag
connflag = True
print("Connection returned result: " + str(rc))
mqttc = paho.Client()
mqttc.on_connect = on_connect
awshost = "YOURAWSHOST.iot.us-west-2.amazonaws.com"
awsport = 8883
caPath = "root-CA.crt"
certPath = "YOURCERT.pem.crt"
keyPath = "YOURKEY.pem.key"
mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)
mqttc.connect(awshost, awsport, keepalive=60)
mqttc.loop_start()
while 1==1:
sleep(0.5)
f = open('mybinaryfile')
imagestring = bytearray(f.read())
f.close()
message = '"image": { "bytearray": "' + imagestring + '"} } '
mqttc.publish("$aws/things/rpi/shadow/update", message, qos=1)
Keep in mind that your published messages need to be SMALL - 128KB is the max size. If you have large images, you'll likely need to loop over your image and read it into chunks that are smaller than 128KB (+ overhead), and upload multiple images to AWS IOT for each image until you get the entire thing uploaded.
-Ray