We Keep Coding

How do I save mqtt data that I receive in python output on a text?

I am setting up to receive MQTT data from a subscribed topic on Python and I want to save the data in a text file.
My code is as follows:
import paho.mqtt.client as paho
import time
import sys
import datetime
import time
broker='192.168.1.3' #host name
topic="/gw/ac233fc04e45/status"
def on_message(client, userdata, message):
print("received data is :")
print(str(message.payload.decode("utf-8")) ) #printing Received message
print("")
client= paho.Client("user") #create client object
client.on_message=on_message
print("connecting to broker host",broker)
client.connect(broker)#connection establishment with broker
print("subscribing begins here")
client.subscribe(topic)#subscribe topic test
while 1:
client.loop_start() #contineously checking for message
the output of the program is as follows:
received data is :
[{"timestamp":"2021-04-16T14:16:38Z","type":"Gateway","mac":"AC233FC04E45","gatewayFree":97,"gatewayLoad":0.15},{"timestamp":"2021-04-16T14:16:38Z","type":"Unknown","mac":"AC233FA27B5B","bleName":"","rssi":-34,"rawData":""},{"timestamp":"2021-04-16T14:16:39Z","type":"Unknown","mac":"AC233FA27B6A","bleName":"","rssi":-31,"rawData":""},{"timestamp":"2021-04-16T14:16:38Z","type":"Unknown","mac":"AC233FA28A31","bleName":"","rssi":-42,"rawData":""},{"timestamp":"2021-04-16T14:16:39Z","type":"iBeacon","mac":"AC233FA28A32","bleName":"","ibeaconUuid":"E2C56DB5DFFB48D2B060D0F5A71096E0","ibeaconMajor":0,"ibeaconMinor":0,"rssi":-74,"ibeaconTxPower":-59,"battery":0}]
I need to save time, mac addresses and rssi values to a text file.
Can someone please help me?
Thanks

import json
def on_message(client, userdata, message):
d = json.loads(message.payload.decode("utf-8"))
with open("file.txt", "a+") as file:
file.write("%s,%s,%s" %(d["timestamp"], d["mac"], d["rssi"]))

MQTT Subscriber to ThingsBoard broker in python

Situation:
I have a python virtual sensor (a python program) that submits data via MQTT protocol to my device in ThingsBoard.
I can visualize data on the dashboard, so I'm sure that the data are received.
Problem:
When I try to connect a python subscriber to the thingsboard broker (demo.thingsboard.io) using paho I obtain that the connection code is 0, so the connection is OK, however I see that the dashboard stops visualising the data from the virtual sensor but the subscriber does not receive anything.
The virtual sensor is publishing at v1/devices/me/telemetry and the subscriber is subscribed at the same topic v1/devices/me/telemetry.
How to show the data published by the virtual sensor on my subscriber client?
VIRTUAL SENSOR CODE:
import paho.mqtt.client as paho #mqtt library
import os
import json
import time
from datetime import datetime
ACCESS_TOKEN="vgFztmvT6bps7JCeOEZq" #Token of your device
broker="demo.thingsboard.io" #host name
port=1883 #data listening port
def on_publish(client,userdata,result): #create function for callback
print("data published to thingsboard \n")
pass
client1= paho.Client("control1") #create client object
client1.on_publish = on_publish #assign function to callback
client1.username_pw_set(ACCESS_TOKEN) #access token from thingsboard device
client1.connect(broker,port,keepalive=60) #establish connection
while True:
payload="{"
payload+="\"Humidity\":60,";
payload+="\"Temperature\":25";
payload+="}"
ret= client1.publish("v1/devices/me/telemetry",payload) #topic-v1/devices/me/telemetry
print("Please check LATEST TELEMETRY field of your device")
print(payload);
time.sleep(5)
CLIENT SUBSCRIBER CODE:
import paho.mqtt.client as mqtt
import time
token = "vgFztmvT6bps7JCeOEZq"
broker="demo.thingsboard.io" # host name
port=1883
topic = "v1/devices/me/telemetry"
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc) :
if (rc==0) :
print("connected OK Returned code = ", rc)
else :
print("Bad connection Returned code = ", rc)
def on_message(client, userdata, msg) :
print (msg.topic + " " + str(msg.payload))
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set(token)
client.connect(broker , port, 60)
client.subscribe(topic)
client.loop_forever()

In you publisher client , you are using a topic named:"v1/devices/me/telemetry".
However, you are not subscribing to the same topic using the subscriber client.
Change the following line in your Subscriber client program:
client.subscribe(token)
to
client.subscribe(topic)
This should solve the problem.

I believe TB will only allow for one connection per client so you should use different access_keys for your publisher and subscriber.

Try to use tcp://51.159.155.114 but not demo.thingsboard.io

Solution for continuously running a code part within a MQTT client?

I am trying to send me a message over a mqtt connection and then perform a task on the received messages which are collected in the inbox. The problem is that when I run the code, the messages have not been received yet and the part in the code which shall work on the information in the messages does not run again.
I thought about using a while True: loop for that bit, but then the mqtt script won't be reached at all. This does not work.
This might be a very fundamental question, but how can I achieve running the mqtt client continuously and deal with the collected messages at some point later on?
import paho.mqtt.client as client
hostname = 'test.mosquitto.org'
topic = "personal/topic"
# receiving the data
inbox = []
def on_message(client, userdata, msg):
msg = msg.payload.decode()
inbox.append(msg)
print("Received Message No. %s: %s" % (len(inbox), msg))
def on_connect(client, userdata, flags, rc):
print("Connection returned result: " + str(rc))
client.subscribe(topic)
# this is where I want to perform a task on the message
y = []
X = []
for msg in inbox:
label, data = msg.split(': ')
data = data.split(', ')
y.append(label)
X.append(data)
# the mqtt part
client = client.Client()
client.on_message = on_message
client.on_connect = on_connect
client.connect(hostname)
client.loop_forever()
This is the construction using client.loop_start()at the end of the script (related to #hardillb's answer):
# the mqtt part
client = client.Client()
client.on_message = on_message
client.on_connect = on_connect
client.connect(hostname)
client.loop_start()
while True:
y = []
X = []
for msg in inbox:
label, data = msg.split(': ')
data = data.split(', ')
y.append(label)
X.append(data)
client.disconnect()
client.loop_stop()

The paho client can start it's network worker on a separate thread by using the client.loop_start() function.
You can then start your own processing loop after that, but the on_message callback function will run on the client's network worker thread, so if you want to queue up incoming messages to process later you will need to ensure that when you store the messages it is done in a thread safe manner.

How do I recive back messages through MQTT from lambda function?

Let me introduce my problem to you, first of all I have to say that i'm newbie at AWS and MQTT, I mean (don't be mad at me), my question: I want to publish through a MQTT client to AWS IoT/Lambda a simple JSON {'petition':'Hola', 'n':0}, and once published, the lambda function has to send back the JSON message {'petition': 'Mundo', 'n': 0} and it has to do it 10 times. The code works fine when u publish the message but it does not recive any. Please I need help to solve it.
Here's the squema:
DEVICE ---> MQTT ---> IoT SQL SENTENCE ---> AWS LAMBDA
DEVICE <--- MQTT <--- AWS LAMBDA
NOTE: the device is running in linux terminal and messages has to appear somehow on the screen.
AWS Lambda handler
import json
def lambda_handler(event, context):
# TODO implement
data={}
data["petition"]="Mundo"
data["n"]=event["n"]
mensaje = event['petition']
if mensaje =='Hola':
envio=json.dumps(data)
return data
else:
return 'nada encontrado'
IoT SQL SENTENCE
SELECT * FROM 'Hello_World'
NOTE: those 2 above are implemented using lambda function service.
DEVICE CODE
import paho.mqtt.client as paho
import os
import socket
import ssl
import time
import json
def on_connect(client, userdata, flags, rc):
print("Connection returned result: " + str(rc) )
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
def on_message(client, userdata, msg):
print("topic: "+msg.topic)
print("payload: "+str(msg.payload))
data=json.loads(msg.payload)
if data["n"]<9 and data["petition"]=="Mundo":
data["petition"] = "Hola"
data["n"] = n+1
envio=json.dumps(data)
mqttc.publish(thingName, envio)
time.sleep(4)
def on_publish(client,userdata,missatge): #create function for callback
print("data published \n")
print "Create client instance"
mqttc = paho.Client()
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_message = on_message
#mqttc.on_log = on_log
data={}
awshost = "a20u2bg4i3u0uq.iot.eu-west-2.amazonaws.com"
awsport = 8883
clientId = "39cfba3c326847909aa5f4544211f4a4"
thingName = "Hello_World"
caPath = "/#/rootCA.pem"
certPath = "/#/3218b320c9-certificate.pem.crt"
keyPath = "/#/3218b320c9-private.pem.key"
mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)
print "connecting to browser"
mqttc.connect(awshost, awsport, keepalive=60)
mqttc.loop_start()
time.sleep(2)
print"subscribing...."
mqttc.subscribe(thingName)
print"subscribed"
time.sleep(2)
data["petition"]="Hola"
data["n"]=0
envio=json.dumps(data)
print(envio)
print "Sending..."
mqttc.publish(thingName, envio)
time.sleep(4)
print "Sent"
mqttc.disconnect()
mqttc.loop_stop()

I would really just set up an API call in API Gateway for your callback to the Lambda.

How can I publish and subscribe a image file using Mosquitto in python?

I'm a student who have been studying MQTT.
I've been trying to send a image file in a Raspberry Pi using mosquitto.
This is a pub.py
import paho.mqtt.client as mqtt
def on_public(mosq, userdata, mid):
mosq.disconnect()
client = mqtt.Client()
client.connect("test.mosquitto.org", 1883, 60)
f=open("1.jpg", "rb")
fileContent = f.read()
byteArr = bytes(fileContent)
client.publish("image",byteArr,0)
client.loop(5)
And this is a sub.py
import paho.mqtt.client as mqtt
def on_public(client, userdata, rc):
print("Connect with result code " + str(rc))
client.subscribe(“image”)
def on_public(client, userdata, msg):
print("Topic : " , msg.topic + "\n Image : " + byteArr
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("test.mosquitto.org", 1883, 60)
client.loop(20)
The problem is I couldn't know well how can I subscribe the image which I already pub.
I think the logic is find in my head, but it doesn't work.
I've attempted a lots of ways such using write() or sth like that.
I'm so sorry if It is just a basic coding skill, but I've made a system using MQTT, R-pi.
Please help me I need your hand.

Under normal circumstances messages will only be delivered if the subscribing client is connected and subscribing before the message is published. (For how to get at messages published when the subscriber is disconnected search for Persistent Subscriptions)
Your subscriber app should look something like:
import paho.mqtt.client as mqtt
def on_connect(client, userdata, rc):
print("Connect with result code " + str(rc))
client.subscribe(“image”)
def on_message(client, userdata, msg):
print("Topic : " , msg.topic + "\n Image : " + msg.payload
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("test.mosquitto.org", 1883, 60)
client.loop_forever()
The client.loop_forever() is the important bit to keep subscriber running rather than just for 5 seconds.
This will print the raw bytes to the console which is not going to be all that useful. To write the image to a file try something like this
def on_message(client, userdata, msg)
f = open('/tmp/output.jpg', 'w')
f.write(msg.payload)
f.close()
This would write the file to /tmp/output.jpg