I was working on a prototype for security in MQTT based systems using a Raspberry Pi as my client and Keycloak as my authentication server (AS). I have managed to publish and subscribe to messages via Mosquitto broker without the authentication mechanism in place and have also successfully obtained the access token and token validation output from the keycloak validation endpoint.
However, when I implemented above sequence of tasks via a python script, to authenticate the mqtt client so it can publish/subscribe, I noticed that it is able to establish a connection but at the client end the connection closes. I suspect that this is due to the token not getting passed either due to wrong/invalid parameters or there is an issue with my token validation function
I would be really grateful if members can look at my python script and perhaps point out any errors or provide suggestions to improve my code. TIA!
import paho.mqtt.client as paho
import time
import sys
import json
import requests
from requests.auth import HTTPBasicAuth
broker="broker address"
port= 1884
sub_topic="test"
#-------------------------------------------------Function to get access token-----------------------------------------------------#
def get_token():
url = "https://token endpoint url"
payload = 'grant_type=client_credentials'
headers = {
'Content-Type':'application/x-www-form-urlencoded'
}
response = requests.post(url, headers=headers, data=payload, auth=HTTPBasicAuth("Client_ID","client_secret"))
token = json.loads(response.text)
return token["access_token"]
mytoken = get_token()
print(mytoken)
#-----------------------------------------------------------Callback functions------------------------------------------------------#
def on_connect(client,userdata,flags, rc):
print("Connected with code " +str(rc))
client.subscribe(sub_topic)
def on_subscribe(client, userdata, mid, granted_qos): #create function for callback
print("subscribed with qos",granted_qos, "\n")
pass
def on_message(client, userdata, message):
print("message received " ,str(message.payload.decode("utf-8")))
def on_publish(client,userdata,mid): #create function for callback
print("data published mid=",mid, "\n")
pass
def on_disconnect(client, userdata, rc):
print("client disconnected")
#------------------------------------------------------------Connect to broker------------------------------------------------------#
client= paho.Client("Client",transport="websockets") #create client object
#client.username_pw_set(username= "mytoken", password= "")
client.on_connect = on_connect
client.on_subscribe = on_subscribe #assign function to callback
client.on_publish = on_publish #assign function to callback
client.on_message = on_message #assign function to callback
client.on_disconnect = on_disconnect
print("connecting to broker ",broker,"on port",port)
#client.connect("ws://broker.lailaafreen.com:1884")
client.connect(broker,port,60) #establish connection
#--------------------------------Token validation-----------------------------------#
def get_valid(token):
url = "https://validation endpoint url"
payload = {}
headers = {'content-type': "multipart/form-data", 'Authorization': 'Bearer ' +mytoken}
response = requests.post(url, headers=headers)
valid = json.loads(response.text)
return valid
print(get_valid(get_token()))
#--------------------------------Publish/Subscribe----------------------------------#
client.loop_start()
print("subscribing to ",sub_topic)
#client.subscribe(sub_topic)
time.sleep(3)
client.publish("test","message") #publish
time.sleep(4)
client.disconnect()
UPDATE: Upon obtaining the access token and after attempting to connect to mosquitto broker, using the command:
sudo mosquitto_pub -h broker_addr -p 1883 -i <acces_token> -t test -m "Hi"
I get not authorised message:
Connection error: Connection Refused: not authorised.
Error: The connection was refused.
Below is my mosquitto configuration file:
persistence false
persistence_location /var/lib/mosquitto/
# Mosquitto Log File Destination
log_dest file /var/log/mosquitto/mosquitto.log
log_type all
# Adding Guman Readable Time Stamsps to /var/log/mosquitto/mosquitto.log
log_timestamp true
log_timestamp_format %Y-%m-%dT%H:%M:%S
# =================================================================
# Security
# =================================================================
allow_anonymous false
# =================================================================
# Listeners
# =================================================================
listener 1884
protocol websockets
socket_domain ipv4
listener 1883
protocol mqtt
include_dir /etc/mosquitto/conf.d
Related
We have a Flask app that needs to send messages to different subscriptions on a VernqMQ broker. We are using paho.mqtt library to connect to the broker, and we are using JWT-based authentication, which is validated ok the broker with a custom webhook script. However, the JWT expires every seven days, so the Flask app needs to renew that token and reconnect. We are currently handling this in the on_disconnect event. This setup works, however, after a while, we are starting to get this error message :
can't register client {xxxx} with username 'xxxx' due to register_subscriber_retry_exhausted
The message appears hundreds of times on the broker's log files, but we can't find any information on this error. Here is the code for how we establish/renew the MQTT broker connection within Flask:
def mqtt_on_connect(client, userdata, flags, rc):
if rc == 0:
print("MQTT broker Connection successful")
else:
print("MQTT broker Connection attempt failed.")
def mqtt_on_disconnect(client, userdata, rc):
token = crypto.get_jwt('our_username').decode()
client.username_pw_set("our_username", password=f'Bearer {token}')
try:
client.connect(
app.config['MQTT_BROKER_URL'],
port=app.config['MQTT_BROKER_PORT']
)
except Exception as ex:
print("Could not reconnect to broker.")
print(ex)
def establish_mqtt_connection(app, get_jwt):
token = get_jwt(
'our_username'
).decode()
client_id = str(uuid.uuid4())
if app.config['ENVIRONMENT'] not in {'LOCAL', 'TEST'}:
client_id = str(uuid.uuid4())
mqtt_client = mqtt.Client(
f'control_{client_id}',
clean_session=True,
transport="websockets"
)
mqtt_client.tls_set(
ca_certs=certifi.where(),
tls_version=ssl.PROTOCOL_TLSv1_2
)
mqtt_client.on_connect = mqtt_on_connect
mqtt_client.on_disconnect = mqtt_on_disconnect
mqtt_client.username_pw_set("our_username", password=f'Bearer {token}')
host = app.config['MQTT_BROKER_URL']
mqtt_client._host = host
mqtt_client.loop_start()
try:
mqtt_client.connect(
app.config['MQTT_BROKER_URL'],
port=app.config['MQTT_BROKER_PORT'],
)
except Exception as ex:
print(f'Exception establishing connection to MQTT broker: {ex}')
app.config['MQTT_CLIENT'] = mqtt_client
Why does this error message happen and what can we do to fix it? Is there a problem with how we are setting up the broker connection? Any info on this willbe greatly appreciated.
Additional info: the broker is a single instance of VerneMQ, not a cluster. All configuration values are set to defaults.
Situation:
I have a python virtual sensor (a python program) that submits data via MQTT protocol to my device in ThingsBoard.
I can visualize data on the dashboard, so I'm sure that the data are received.
Problem:
When I try to connect a python subscriber to the thingsboard broker (demo.thingsboard.io) using paho I obtain that the connection code is 0, so the connection is OK, however I see that the dashboard stops visualising the data from the virtual sensor but the subscriber does not receive anything.
The virtual sensor is publishing at v1/devices/me/telemetry and the subscriber is subscribed at the same topic v1/devices/me/telemetry.
How to show the data published by the virtual sensor on my subscriber client?
VIRTUAL SENSOR CODE:
import paho.mqtt.client as paho #mqtt library
import os
import json
import time
from datetime import datetime
ACCESS_TOKEN="vgFztmvT6bps7JCeOEZq" #Token of your device
broker="demo.thingsboard.io" #host name
port=1883 #data listening port
def on_publish(client,userdata,result): #create function for callback
print("data published to thingsboard \n")
pass
client1= paho.Client("control1") #create client object
client1.on_publish = on_publish #assign function to callback
client1.username_pw_set(ACCESS_TOKEN) #access token from thingsboard device
client1.connect(broker,port,keepalive=60) #establish connection
while True:
payload="{"
payload+="\"Humidity\":60,";
payload+="\"Temperature\":25";
payload+="}"
ret= client1.publish("v1/devices/me/telemetry",payload) #topic-v1/devices/me/telemetry
print("Please check LATEST TELEMETRY field of your device")
print(payload);
time.sleep(5)
CLIENT SUBSCRIBER CODE:
import paho.mqtt.client as mqtt
import time
token = "vgFztmvT6bps7JCeOEZq"
broker="demo.thingsboard.io" # host name
port=1883
topic = "v1/devices/me/telemetry"
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc) :
if (rc==0) :
print("connected OK Returned code = ", rc)
else :
print("Bad connection Returned code = ", rc)
def on_message(client, userdata, msg) :
print (msg.topic + " " + str(msg.payload))
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set(token)
client.connect(broker , port, 60)
client.subscribe(topic)
client.loop_forever()
In you publisher client , you are using a topic named:"v1/devices/me/telemetry".
However, you are not subscribing to the same topic using the subscriber client.
Change the following line in your Subscriber client program:
client.subscribe(token)
to
client.subscribe(topic)
This should solve the problem.
I believe TB will only allow for one connection per client so you should use different access_keys for your publisher and subscriber.
Try to use tcp://51.159.155.114 but not demo.thingsboard.io
I'm trying to connect from python script(paho.mqtt.python) to mosquitto broker.
I can connect from terminal using this command:
mosquitto_sub -h localhost -p 8883 -v -t 'owntracks/#' -u owntracks -P 12qwaszx
But when I'm trying to connect via python script I'm getting the error:
Socket error on client <unknown>, disconnecting.
The script I'm using is the example:
(from here: https://owntracks.org/booklet/tech/program/)
import paho.mqtt.client as mqtt
import json
# The callback for when the client successfully connects to the broker
def on_connect(client, userdata, rc):
''' We subscribe on_connect() so that if we lose the connection
and reconnect, subscriptions will be renewed. '''
client.subscribe("owntracks/+/+")
#tried also: client.subscribe("owntracks/#")
# The callback for when a PUBLISH message is received from the broker
def on_message(client, userdata, msg):
topic = msg.topic
try:
data = json.loads(str(msg.payload))
print "TID = {0} is currently at {1}, {2}".format(data['tid'], data['lat'], data['lon'])
except:
print "Cannot decode data on topic {0}".format(topic)
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("localhost", 8883, 60)
# Blocking call which processes all network traffic and dispatches
# callbacks (see on_*() above). It also handles reconnecting.
client.loop_forever()
Here is the content of my config file (I changed the "localhost" from my real IP - tried both of them):
# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example
pid_file /var/run/mosquitto.pid
log_dest file /var/log/mosquitto/mosquitto.log
include_dir /etc/mosquitto/conf.d
listener 8883 "localhost"
persistence true
persistence_location /var/lib/mosquitto/
persistence_file mosquitto.db
log_dest syslog
log_dest stdout
log_dest topic
log_type error
log_type warning
log_type notice
log_type information
connection_messages true
log_timestamp true
allow_anonymous false
password_file /etc/mosquitto/pwfile
Any help would be highly appreciated.
Your python script is attempting to connect to what looks like a TLS secured setup without preparing the client connect method to apply these details to the transaction. Try the following:
def ssl_prep():
ssl_context = ssl.create_default_context()
ssl_context.load_verify_locations(cafile=ca)
ssl_context.load_cert_chain(certfile=mycert, keyfile=priv)
return ssl_context
ca = "PATH_TO_YOUR_CA_FILE"
priv = "PATH_TO_YOUR_PEM_FILE"
mycert = "PATH_TO_YOUR_CERT_FILE"
topics = "YOUR_TOPICS"
broker = "BROKER_URL"
client = mqtt.Client()
ssl_context= ssl_prep()
client.tls_set_context(context=ssl_context)
client.username_pw_set(username="UNAME",password="PASS")
client.connect(broker, port=8883)
By providing the ssl context to the connection attempt before the attempt is made, this should connect assuming you have all the detail specific to your own setup in place.
Try to reduce the KeepAlive time from 1min to 30sec or lower:
client.connect("localhost", 8883, 30)
//Default: connect(host, port=1883, keepalive=60, bind_address=””)
Let me introduce my problem to you, first of all I have to say that i'm newbie at AWS and MQTT, I mean (don't be mad at me), my question: I want to publish through a MQTT client to AWS IoT/Lambda a simple JSON {'petition':'Hola', 'n':0}, and once published, the lambda function has to send back the JSON message {'petition': 'Mundo', 'n': 0} and it has to do it 10 times. The code works fine when u publish the message but it does not recive any. Please I need help to solve it.
Here's the squema:
DEVICE ---> MQTT ---> IoT SQL SENTENCE ---> AWS LAMBDA
DEVICE <--- MQTT <--- AWS LAMBDA
NOTE: the device is running in linux terminal and messages has to appear somehow on the screen.
AWS Lambda handler
import json
def lambda_handler(event, context):
# TODO implement
data={}
data["petition"]="Mundo"
data["n"]=event["n"]
mensaje = event['petition']
if mensaje =='Hola':
envio=json.dumps(data)
return data
else:
return 'nada encontrado'
IoT SQL SENTENCE
SELECT * FROM 'Hello_World'
NOTE: those 2 above are implemented using lambda function service.
DEVICE CODE
import paho.mqtt.client as paho
import os
import socket
import ssl
import time
import json
def on_connect(client, userdata, flags, rc):
print("Connection returned result: " + str(rc) )
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
def on_message(client, userdata, msg):
print("topic: "+msg.topic)
print("payload: "+str(msg.payload))
data=json.loads(msg.payload)
if data["n"]<9 and data["petition"]=="Mundo":
data["petition"] = "Hola"
data["n"] = n+1
envio=json.dumps(data)
mqttc.publish(thingName, envio)
time.sleep(4)
def on_publish(client,userdata,missatge): #create function for callback
print("data published \n")
print "Create client instance"
mqttc = paho.Client()
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_message = on_message
#mqttc.on_log = on_log
data={}
awshost = "a20u2bg4i3u0uq.iot.eu-west-2.amazonaws.com"
awsport = 8883
clientId = "39cfba3c326847909aa5f4544211f4a4"
thingName = "Hello_World"
caPath = "/#/rootCA.pem"
certPath = "/#/3218b320c9-certificate.pem.crt"
keyPath = "/#/3218b320c9-private.pem.key"
mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)
print "connecting to browser"
mqttc.connect(awshost, awsport, keepalive=60)
mqttc.loop_start()
time.sleep(2)
print"subscribing...."
mqttc.subscribe(thingName)
print"subscribed"
time.sleep(2)
data["petition"]="Hola"
data["n"]=0
envio=json.dumps(data)
print(envio)
print "Sending..."
mqttc.publish(thingName, envio)
time.sleep(4)
print "Sent"
mqttc.disconnect()
mqttc.loop_stop()
I would really just set up an API call in API Gateway for your callback to the Lambda.
I have configured MQTT Broker, receiving published messages from another piece of code (not written or accessible by me). I added another topic to the broker configuration and am now trying to publish data to this new topic from a piece of python code. I get the feedback that the message is published by the callback function, but no actual data is received.
Is there anything I am missing?
I am using the following code:
import paho.mqtt.client as mqtt
import time
#=========================================================================
def on_connect(client, userdata, flags, rc) :
print "on_connect()"
#=========================================================================
def on_publish(client, userdata, mid) :
print "on_publish()"
#=========================================================================
def send() :
mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
#host = "localhost"
host = "127.0.0.1"
port = 1883
keepalive = 60
print "\nConnect to '%s', port '%s', keepalive '%s'" % (host, port, keepalive)
mqttc.connect(host=host, port=port, keepalive=keepalive)
time.sleep(3)
mqttc.loop_start()
time.sleep(3)
topic = "data/MY/TOPIC"
msg = "MY_MESSAGE"
print "Publish to '%s' msg '%s'" % (topic, msg)
mqttc.publish(topic, msg, qos=2)
time.sleep(3)
mqttc.loop_stop()
# end send()
#=========================================================================
if __name__ == "__main__":
send()
# end if
Getting the stdout
Connect to '127.0.0.1', port '1883', keepalive '60'
on_connect()
Publish to 'data/MY/TOPIC' msg 'MY MESSAGE'
on_publish()
I am not sure if the loop() functions are necessary, but if I do not embed the publishing in the loop_start() and loop_stop(), I do not get a on_connect callback.
The loop functions are necessary as these are where all the network traffic is processed.
Manually setting up a connection to the broker to just send a single message like this is not a good idea, it would be better to start the client, leave it running (by calling loop_start() and not calling loop_stop()) in the background and then just call the publish method on the mqttc client object.
If you don't want to keep a instance of the client running then you should use the single message publish helper method provided by the paho python library (https://pypi.python.org/pypi/paho-mqtt/1.1#single):
import paho.mqtt.publish as publish
publish.single("paho/test/single", "payload", hostname="iot.eclipse.org")