I have installed the last version of raspbian on my raspberry pi, and I have opened an account AWS IoT on Amazon, then in the IoT web interface I have created a thing, with "RaspberryPi_2" name and create certificate and connect the certificate to the thing, I have followed this guide:
http://blog.getflint.io/blog/get-started-with-aws-iot-and-raspberry-pi
I have then created the script in the guide, to connect and subscribed the raspberry pi, this is my code:
#!/usr/bin/python3
#required libraries
import sys
import ssl
import paho.mqtt.client as mqtt
#called while client tries to establish connection with the server
def on_connect(mqttc, obj, flags, rc):
if rc==0:
print ("Subscriber Connection status code: "+str(rc)+" | Connection status: successful")
elif rc==1:
print ("Subscriber Connection status code: "+str(rc)+" | Connection status: Connection refused")
#called when a topic is successfully subscribed to
def on_subscribe(mqttc, obj, mid, granted_qos):
print("Subscribed: "+str(mid)+" "+str(granted_qos)+"data"+str(obj))
#called when a message is received by a topic
def on_message(mqttc, obj, msg):
print("Received message from topic: "+msg.topic+" | QoS: "+str(msg.qos)+" | Data Received: "+str(msg.payload))
#creating a client with client-id=mqtt-test
mqttc = mqtt.Client(client_id="mqtt-test")
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
mqttc.on_message = on_message
#Configure network encryption and authentication options. Enables SSL/TLS support.
#adding client-side certificates and enabling tlsv1.2 support as required by aws-iot service
mqttc.tls_set("/home/pi/aws_iot/things/raspberryPi_2/certs/aws-iot-rootCA.crt",
certfile="/home/pi/aws_iot/things/raspberryPi_2/certs/0ea2cd7eb6-certificate.pem.crt",
keyfile="/home/pi/aws_iot/things/raspberryPi_2/certs/0ea2cd7eb6-private.pem.key",
tls_version=ssl.PROTOCOL_TLSv1_2,
ciphers=None)
#connecting to aws-account-specific-iot-endpoint
mqttc.connect("A2GF7W5U5A46J1.iot.us-west-2.amazonaws.com", port=8883) #AWS IoT service hostname and portno
#the topic to publish to
mqttc.subscribe("$aws/things/RaspberryPi_2/shadow/update/#", qos=1) #The names of these topics start with $aws/things/thingName/shadow."
#automatically handles reconnecting
mqttc.loop_forever()
but when I do this command:
python3 mqtt_test.py
or this command:
python mqtt_test.py
and press enter, and the cursor flash and doesn't print anything and stay stuck there, someone can help me?
I haven't also understand if the client-id name should be the same of the things name, and the meaning of the subscribe path, for example in a tutorial I have found this:
mqttc.publish("temperature", tempreading, qos=1)
why there isn't the complete path?
or this:
$aws/things/RaspberryPi_2/shadow/update/delta
so I can put everything I want in the path?
thanks
The code is subscribing to a topic but there is no one publishing to it. So, the code also has a on_connect function that would be triggered after a success connection. Is the message "Subscriber Connection status code: ..." being printed? If it is, the message from on_subscribe should also appear. Is it?
If it is not you have a problem before connect to the AWS server. Use netstat command to see where your Raspberry Pi is connected or not and post more debug information in this case.
If the connect and subscribe messages are shown and nothing happens after it, this is normal because you are only subscribing to a topic but not publishing anything.
Regarding topics, think of them as a directory structure. There is no strict rule for topic hierarchy. A "temperature" topic would be temperature topic on the server and a "temperature/livingroom" would be temperature/livingroom, you can subscribe to one, another or both on the same server. The path you choose for your things shall be meaningful to your application. A house, for instance, might be represented as:
house/kitchen/env/temperature
house/kitchen/env/humidity
house/kitchen/lamp/sinklamp
house/kitchen/lamp/mainlap
house/masterbed/env/temperature
house/masterbed/env/humidity
house/masterbed/lamp/readinglampleft
house/masterbed/lamp/readinglampright
house/masterbed/lamp/mainlamp
house/masterbed/lamp/mirrorlamp
And so on.
Let´s say you have a thermostat at master bedroom. It is interested only in temperature but not humidity. It is also interested only in master bedroom temperature. This thermostat shall subscribe to house/masterbed/env/temperature. Opposite to this, a room wide panel that shows state of every thing in the room, would subscribe to house/masterbed/#, meaning "everything after house/masterbed". Read more on wildcards here
The topic you subscribed for: $aws/things/RaspberryPi_2/shadow/update/# means, "every thing after $aws/things/RaspberryPi_2/shadow/update/". Notice that his is a special topic, it starts with $aws, specially, it starts with $ character. In the AWS context this means:
Any topics beginning with $ are considered reserved and are not
supported for publishing and subscribing except when working with the
Thing Shadows service. For more information, see Thing Shadows.
So you need to understand what thing shadow is. This is a AWS specific (and very util) concept. Please read docs on this subject.
Finally, I would you suggest you install a local broker (mosquitto is available on respbian) and try with it before got to AWS. This way you can master mqtt concept without connectivity issues. Later you put AWS to the mix.
Related
EDIT: In an initial version of this question I was using the Python paho-mqtt client version paho-mqtt 1.5.1. The error message I received with that version was on_disconnect 1: Out of memory. error code 1. It was recommended to install the newer version (1.6) of the paho-mqtt client and rerun the code. I did that and still encounter the same spontaneous disconnection, yet the newer version has more descriptive error messages, With the 1.6 version I receive the disconnection error code 7, which means that "The connection was lost".
I am simulating an IoT device. As part of this simulation I have created a Messenger class, which, when instantiated, communicates with a virtual device on Google Cloud's IoT core with the MQTT protocol. The client I am using frequently disconnects leading to lost messages. I am hoping to find a way to prevent the client from randomly disconnecting.
The client used for this communication, which is embedded in the Messenger class, is the python paho mqtt client.
The client id is derived from the project_id, cloud_region, registry_id and device_id using the following code
client_id = "projects/{}/locations/{}/registries/{}/devices/{}".format(project_id, cloud_region, registry_id, device_id)
The client communicates with IoT core in three different ways:
It publishes data to the events topic
sub_topic = "events"
mqtt_topic = "/devices/{}/{}".format(self.device_id, sub_topic)
client.publish(mqtt_topic, payload, qos=1)
It subscribes to the virtual device's config topic
mqtt_config_topic = "/devices/{}/config".format(self.device_id)
sub_status = client.subscribe(mqtt_config_topic, qos=1)
It sends state information to the device's state topic at a frequency no greater than once every 12 seconds. (Google Cloud recommends no more than once every 10 seconds so I added a few more seconds for wiggle room).
sub_topic = "state"
mqtt_topic = "/devices/{}/{}".format(self.device_id, sub_topic)
payload = json.dumps(new_state)
client.publish(mqtt_topic, payload, qos=0)
Initially I noticed that not all of the data sent to be published to the events topic was sent. After examining the log file, I determined the issue was caused by the frequent disconnects of the client. The error code given for the client disconnect is 1, which is technically an "out of memory" error, but also occurs when GCP disconnects the client. I think the issue lies with GCP disconnecting the client, but I am not entirely sure
Here is an excerpt of a log file:
DEBUG Controller4.testA [04/Nov/2021:12:54:57.862] new client created
DEBUG Controller4.testA [04/Nov/2021:12:54:57.865] Publishing state to /devices/<DeviceID>/state
DEBUG Controller4.testA [04/Nov/2021:12:54:57.867] Sending PUBLISH (d0, q0, r0, m2), 'b'/devices/<DeviceID>/state'', ... (197 bytes)
DEBUG Controller4.testA [04/Nov/2021:12:54:57.870] Not publishing state: Previous state update 0:00:00.004303 seconds ago
DEBUG Controller4.testA [04/Nov/2021:12:54:57.870] on_publish client: <paho.mqtt.client.Client object at 0x15bf73670> userdata: None mid: 2
DEBUG Controller4.testA [04/Nov/2021:12:54:57.871] Sending PUBACK (Mid: 1)
DEBUG Controller4.testA [04/Nov/2021:12:54:57.947] Received CONNACK (0, 0)
DEBUG Controller4.testA [04/Nov/2021:12:54:57.951] on_connect Connection Accepted.
DEBUG Controller4.testA [04/Nov/2021:12:54:57.976] Received SUBACK
DEBUG Controller4.testA [04/Nov/2021:12:54:57.976] Subscribed: (1,)
DEBUG Controller4.testA [04/Nov/2021:12:54:57.978] Received PUBLISH (d0, q1, r0, m1), '/devices/<DeviceID>/config', ... (20 bytes)
INFO Controller4.testA [04/Nov/2021:12:54:57.979] Received new state message '{"osmo": 6, "pH": 4}' on topic '/devices/<DeviceID>/config' with Qos 1
DEBUG Controller4.testA [04/Nov/2021:12:54:57.980] Not publishing state: Previous state update 0:00:00.114637 seconds ago
DEBUG Controller4.testA [04/Nov/2021:12:54:57.981] Sending PUBACK (Mid: 1)
DEBUG Controller4.testA [04/Nov/2021:12:55:01.958] on_disconnect 7: The connection was lost. error code 7<paho.mqtt.client.Client object at 0x15bf60b80>None
I am currently working on a virtual private cloud, but I am not sure that matters.
Why does the client keep disconnecting? How can I further troubleshoot this problem?
I have web application based on Node.js and python based hardware. I want to keep status of connectivity between web server and hardware. If Hardware gets disconnected from web application then web application should get event or notification so based on that i can send notification to user. I have used mqtt for data communication, but to keep connection status I can't use MQTT coz it is connected with broker. I don't want to increase more load on server.
Which tools/technology/protocol/method should i use to keep connection status that device is offline or online?. That I also want use when user tries to send data to hardware using web application and if device is not connected with server then user should get notification that device is offline based on connection status.
The following code demonstrates the process I was hinting at in the comments.
The LWT feature tells the broker to publish a message marking the client as offline when it fails to respond in 1.5 times the keepalive period. If the client cleanly disconnects then it needs to mark it's self as offline. The client marks it's self as online when it connects to the broker.
All the status messages have the retained bit set so they will always be delivered when a client subscribes to the status topic.
import paho.mqtt.client as mqtt
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("some/application/topic")
# set status message to online
client.publish("status/client1", payload="online", retain=True)
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
if str(msg.payload) == "shutdown":
# update status to offline as this will be a clean dissconect
client.publish("status/client1", payload="offline", retain=True)
client.disconnect()
client = mqtt.Client(client_id="client1")
client.on_connect = on_connect
client.on_message = on_message
client.will_set("status/client1", payload="offline", retain=True)
client.connect("mqtt.eclipse.org", 1883, 60)
# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_forever()
It will be up to the OP to implement the notification about sending messages to offline clients requested at the end of the question.
I have been struggeling with a strange case of random client MQTT publish failing for certain payloads. It happends randomly when trying to publish some large amount of BASE64 data.
I've finally managed to narrow it down to payloads containing a lot of consequtive forwards slashes (/). I've searched the net to find a good reason why this happends, but havent found anythong. Is it a MQTT feature, a Paho client feature or a broker feature, or just some bug...
Setup:
Python 3.8.8 (Windows 10)
paho-mqtt 1.5.0
mosquitto 1.6.9-1 amd64
On my setup, it fails when I send a payload of 255 '/' to a 1 character topic 'a'. Larger topic length, reduces the possible number of forward slashes.
Code to reproduce error:
import paho.mqtt.client as mqtt_client
import time
address = 'some.server.com'
port = 1883
connected = False
def on_connect(client, userdata, flags, rc):
global connected
connected = True
print("Connected!")
client = mqtt_client.Client()
client.on_connect = on_connect
client.connect(host=address, port=port, keepalive=60)
client.loop_start()
while not connected:
time.sleep(1)
payload = '/'*205
print('Payload: {}'.format(payload))
client.publish(topic='a', payload=payload)
time.sleep(2)
client.loop_stop()
client.disconnect()
print('Done!')
This generates this output:
Connected!
Payload: /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
Connected!
Done!
This produces the following error in /var/log/mosquitto/mosquitto.log for the mosquitto broker:
1616605010: New connection from a.b.c.d on port 1883.
1616605010: New client connected from a.b.c.d as auto-CEF15129-E74C-F00A-A6FA-5B5FDA0CEF1D (p2, c1, k60).
1616605011: Socket error on client auto-CEF15129-E74C-F00A-A6FA-5B5FDA0CEF1D, disconnecting.
1616605012: New connection from a.b.c.d on port 1883.
1616605012: New client connected from a.b.c.d as auto-0149B6DB-5997-9E08-366A-304F21FDF2E1 (p2, c1, k60).
1616605013: Client auto-0149B6DB-5997-9E08-366A-304F21FDF2E1 disconnected.
I observe that the client() connects twice, but do not know why, but this is probably caused by a disconnect...
Any Ideas?
Update 1: I've tested this on Linux Ubunit running Python 3.7.3, and same paho-mqtt version, and this does not produce the same error... Seems like some problem in Windows then.
Update 2:
I also tried running mosquitto_pub and experienced the same error, so this has to be Windows-related (or system related) in some way. Possibly firewall? I will close question if I find manage to solve this.
"C:\Program Files\mosquitto\mosquitto_pub.exe" -h some.server.com -t a -m '/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////'
Update 3 The issue is related to OpenVPN. I closed my open VPN connection and MQTT messages were passing! I'm running OpenVPN Client (Windows version 3.2.2 (1455). I have no idea what causes this conlflict...
I used the following code and I get this error :
Time out error : [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host failed to respond.
I also turned off the firewall in the systems i am using , still i get that error.
#!/usr/bin/env python3
import paho.mqtt.client as mqtt
# This is the Publisher
client = mqtt.Client()
client.connect("10.12.114.103",1883,60)
client.publish("topic/test", "Hello world!");
client.disconnect();
The problem probably isn't with your python code as I tried a slightly modified version with hivemq's public test server and it worked fine. Here is the version I tried:
#!/usr/bin/env python3
import paho.mqtt.client as mqtt
# This is the Publisher
def on_log(client, userdata, level, buf):
print(level, buf)
client = mqtt.Client()
client.on_log = on_log
client.connect("broker.hivemq.com",1883,60)
client.publish("topic/test", "Hello world!");
client.disconnect();
Note that I modified it to return the log output which would be useful in debugging this. The script returned:
16 Sending PUBLISH (dFalse, q0, r0, m1, 'topic/test', ... (12 bytes)
Which is what I would expect from a successful connection. I also checked your code with the mosquitto test server and it worked fine.
It looks like your broker is not accepting your connection attempt. If you tried a public test server and it still didn't work then it suggests that something is interfering with your traffic on port 1883.
I think this is a broker issue, however, I did notice that you are not using any of the network loop functions (like client.loop_start) which, according to the paho documentation, may cause unpredictable behavior. You could try adding a loop like this to see if that helps
client = mqtt.Client()
client.on_log = on_log
client.connect("broker.hivemq.com",1883,60)
client.loop_start
client.publish("topic/test", "Hello world!");
client.disconnect();
client.loop_stop
I'm using this python script to implement a basic Paho(MQTT) subscriber but under certain circumstances it doesn't invoke on_connect.
I tried the following, all with exactly the same code:
Running in a Raspbery pi, broker on desktop Pc (Ubuntu). Works.
Running in a Pc(Ubuntu), broker on the same Pc. Doesn't work.
Running in a Pc(Ubuntu), broker on other Pc(Ubuntu). Doesn't work.
Running in a Pc(Ubuntu), broker.hivemq.com as broker. Works.
Running in a Pc(Ubuntu), broker.hivemq.com (it's IP) as broker. Works.
I also usually do:
mosquitto_sub/pub in terminal in all previous combinations. Works.
Official C++ paho wrapper in all previous combinations. Works.
Also tried all setup methods listed in the official website. So python's paho library doesn't seem to be getting along with my local broker unless I run it from my raspberry. But using the same broker works well with c++'s paho library and if using MQTT from the terminal. Any ideas of what can be happening?
import paho.mqtt.client as mqtt
def on_connect(mqtt_client, obj, flags, rc):
mqtt_client.subscribe("test")
print "on_connect"
def on_message(mqtt_client, obj, msg):
print "on_message"
mqtt_client = mqtt.Client()
mqtt_client.on_connect = on_connect
mqtt_client.on_message = on_message
mqtt_client.connect("127.0.0.1",1883)
mqtt_client.loop_forever()
I had the same problem and changing the version of MQTT protocol in Client's constructor solved it.
mqtt_client = mqtt.Client("", True, None, mqtt.MQTTv31)
Since you do not specify the protocol, it uses the default one: MQTTv311.