We Keep Coding

paho loop issue in python on raspbian

I am attempting to write a python script (for Raspbian) that sends mqtt message on button pushes, and changes LED's on/off when mqtt messages are received.
I can send no worries, and my script structure on 4 RPi's is the same;
import
set variables
while true:
do stuff endlessly
I however can not get a basic script running for paho to receive in this structure.
I tried to follow the paho guide but can not adapt it to an endless while loop application.
Why can't I receive mqtt messages? I cross check with 2 terminals; mosquito sub & pub, they are definitely been sent/received elsewhere on network.
Edit;
New on_connect callback displays "Connected" every 2 seconds, but the script still doesn't receive/print mqtt messages. It still does send them when I push the button.
Attempted to fire up a different MQTT broker in Docker, made no difference.
Edit3; WORKS! :D Last I test I must not have been awake and had the subscribe line commented out in the on_connect callback.
I also removed the "P1" from the mqtt.Client thing which I copied from a tutorial.
Can happierly confirm, it sends and receives mqtt messages. Now I can integrate into my larger script. Thanks hardlib
New code is this;
import sys
sys.path.append('/usr/local/lib/python2.7/dist-packages/paho/mqtt')
import os
import time
import paho.mqtt.client as mqtt
from gpiozero import Button
from time import sleep
def on_message(client, userdata, message):
print("message received " ,str(message.payload.decode("utf-8")))
print("message topic=",message.topic)
print("message qos=",message.qos)
print("message retain flag=",message.retain)
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
client.subscribe("button")
button = Button(25)
broker_address="192.168.1.10"
client = mqtt.Client()
client.on_message=on_message
client.on_connect=on_connect
client.connect(broker_address, 1883, 60)
#client.subscribe("button")
client.loop_start()
while 1:
if button.is_pressed:
print("button pressed")
client.publish("button","ON",1)
sleep(0.1)
sleep(0.1)
print("*")

Can send MQTT message from command line but cannot format message correctly in python

So I am using the esprfid project to control access to a door.
That is all working.
I am using a Raspberry PI running Buster as a message broker and am trying to write a python script to send the open door message.
If I use the following command from a terminal window
mosquitto_pub -h 192.168.0.22 -t maindoor -m '{"cmd":"opendoor","doorip":"192.168.0.45"}'
the broker outputs
Client mosqsub|1227-hackrfid2 received PUBLISH (d0, q0, r0, m0, 'maindoor', ... (42 bytes))
{"cmd":"opendoor","doorip":"192.168.0.45"}
and the door opens.
Following some tutorials I have created the following simple script:
from paho.mqtt import client as mqtt_client
import time
import json
broker = "192.168.0.22"
port = 1883
client_id = "rfidManager"
topic = "-t maindoor -m "
openDoor = json.dumps({"cmd":"opendoor","doorip":"192.168.0.45"})
def connect_mqtt():
def on_connect(client, userdata, flags, rc):
if rc == 0:
print("Connected to MQTT Broker!")
else:
print("Failed to connect, return code %d\n", rc)
client = mqtt_client.Client(client_id)
# client.username_pw_set(username, password)
client.on_connect = on_connect
client.connect(broker, port)
return client
def publish(client,msg):
result = client.publish(topic, msg)
# result: [0, 1]
status = result[0]
if status == 0:
print(f"Send `{msg}` to topic `{topic}`")
else:
print(f"Failed to send message to topic {topic}")
def run():
client = connect_mqtt()
client.loop_start()
publish(client, openDoor)
def main(args):
run()
return 0
if __name__ == '__main__':
import sys
sys.exit(main(sys.argv))
This outputs
Connected to MQTT Broker!
Send {"cmd": "opendoor", "doorip": "192.168.0.45"} to topic -t maindoor -m
when I run it and the broker outputs
Client mosqsub|1227-hackrfid2 received PUBLISH (d0, q0, r0, m0, '-t maindoor -m ', ... (45 bytes))
{"cmd": "opendoor", "doorip": "192.168.0.45"}
I have been trying to sort this on and off for a couple of days and cannot spot the error.
Has anyone got any suggestions as to what to try?
Steve

Publish in on_message with Paho Mqtt Client not working

The publish call in the main method dosen't send a message to the broker, but returns (0,2). on_publish won't be called.
I am not sure how I should look where the error is. I tried qos=2 and run the main method async but it didn't fixed it. The base connection work, because the on_message is called. The mqtt-server works fine with other projects too.
Has anyone some ideas?
import paho.mqtt.client as mqtt
import subprocess
import shutil
import os
import glob
import logging
from datetime import datetime
import RPi.GPIO as GPIO
import time
from multiprocessing import Pool
GPIO.setmode(GPIO.BCM)
direction = 19
pwm = 26
GPIO.setup(direction, GPIO.OUT)
GPIO.setup(pwm, GPIO.OUT)
pwm = GPIO.PWM(pwm,19000)
speed = 0
pwm.start(speed)
running = False
client = mqtt.Client()
#First: pip3 install paho-mqtt
print("Starte Listener")
def start():
GPIO.output(direction, GPIO.LOW)
for dc in range(10, 40, 1):
speed = dc
pwm.ChangeDutyCycle(speed)
time.sleep(0.25)
def end():
GPIO.output(direction, GPIO.LOW)
for dc in list(reversed(range(0,40,1))):
speed = dc
pwm.ChangeDutyCycle(speed)
time.sleep(0.5)
def main(mqttClient):
print("Started Spinning")
running = True
start()
print(mqttClient.publish("scanner","shoot"))
print("Waiting 124 seconds")
time.sleep(124)
print("ending spinning")
end()
print("finished spinning")
running = False
def on_connect2(client, userdata, flags, rc):
client.subscribe("scanner",2)
print("Connected "+str(rc))
# The callback for when a PUBLISH message is received from the server.
def on_message(mqttClient, userdata, msg):
print("Recived message: "+str(msg.payload,'UTF-8'))
if(str(msg.payload,'UTF-8') == "spin" and running == False):
main(mqttClient)
def disconnected():
print("Disconneted")
def on_publish(self, client, userdata, mid):
print("onPublish")
print(client,userdata,mid)
logger = logging.getLogger(__name__)
client.enable_logger(logger)
client.on_connect = on_connect2
client.on_message = on_message
client.on_publish = on_publish
client.on_disconnect = disconnected
client.username_pw_set("....","....")
client.connect("gx1", 1883,60)
client.loop_forever()

This is because you are blocking the MQTT client thread.
When you call client.loop_forever() it takes over the processes main thread and uses it to handle all the MQTT communication. When a new message is received it is picked up from the network stack by the MQTT client thread and turned into the message object that is then passed to the on_message() callback. This function runs on the client thread.
When you call client.publish() this will do one of 2 things
If the message is QOS 0 and smaller than the network MTU it will publish the message.
If the message is QOS 1 or 2 or larger than the network MTU then it will queue the message to be handled by the client thread.
The problem* in your code is that you are blocking the return from main() by 124 seconds which in turn is blocking the return of the on_message() function so the client thread can not get to publishing your message.
If you want to do things the take a long time or block in the on_message() (or any of the callback functions) you should start a separate thread to run them on.
*In theory your message looks like it should fall into case 1 listed above, but there may be other factors that are causing it to queue)

As a quickfix I created a new Client:
def main(mqttClient):
print("Started Spinning")
running = True
start()
mq = mqtt.Client("shot_idicator")
mq.username_pw_set("...","...")
mq.connect("gx1", 1883,60)
mq.publish("scanner","shoot")
mq.disconnect()
mq = None
print("Waiting 124 seconds")
time.sleep(124)
print("ending spinning")
end()
print("finished spinning")
running = False
Now it works but I think this is not how it should be done.

mqtt client does not receive message in case of thread and rest-api

I have a python script, which based on flask and mqtt. The use case is to receive a request via rest-api then to create a new thread which publishes some messages on mosquitto mqtt and expects a response (see subscribe). My problem is that I don't receive any messages. I think it has something to do with the thread, because without the thread it's working very fine..
Do you know what can be the problem?
Thank you in anticipation!
here the code:
from flask import Flask, Response
import paho.mqtt.client as mqtt
from threading import Thread
import threading
app = Flask(__name__)
lock = threading.Lock()
def on_connect(client, userdata, flags, rc): # The callback for when the client connects to the broker
print("Connected with result code {0}".format(str(rc))) # Print result of connection attempt
client.subscribe("/mytopic")
def on_message(client, userdata, msg): # The callback for when a PUBLISH message is received from the server.
print(msg.topic)
client = mqtt.Client(client_id=client_name, clean_session=True)
client.on_connect = on_connect # Define callback function for successful connection
client.on_message = on_message # Define callback function for receipt of a message
client.username_pw_set(mqtt_user, mqtt_password)
client.loop_start()
client.connect(mqtt_host)
def test(param1, param2):
lock.acquire()
try:
ret = client.publish("/mytopic", "")
while True:
check the response from mqtt => but i don't get the response anymore
....
break
finally:
lock.release()
return result
#app.route('/test/check', methods=['POST'])
def check():
global sessionId
sessionId = sessionId + 1
t = Thread(target=test, args=(sessionId,None))
t.start()
return {'id': sessionId, 'eta': 0}
if __name__ == '__main__':
app.run(debug=True)

There are a couple of problems with this.
Both the client.connect() and the client.subscribe() calls need iterations of the client network loop to run in order to complete properly.
The network loop needs to run at least once every keep alive period the time after the connection has been made in order to stop the broker disconnecting the client as dead. This means if there is a delay between starting the code and the first REST request then the client will be disconnected.
Better to use the client.start_loop() function to run MQTT client network loop continuously in the background on it's own.
You should also remove the call to client.subscribe() that is outside the on_connect() callback.
EDIT:
As hashed out in the comments/chat the following works. It looks like running the flask app in debug mode does some strange things and creates multiple MQTT clients over and over again with the same client id. This leads to the broker constantly kicking the old ones off so messages never get delivered.
from flask import Flask, Response
import paho.mqtt.client as mqtt
import time
from threading import Thread
import threading
app = Flask(__name__)
lock = threading.Lock()
sessionId=0
cont=True
def on_connect(client, userdata, flags, rc): # The callback for when the client connects to the broker
print("Connected with result code {0}".format(str(rc))) # Print result of connection attempt
client.subscribe("mytopic")
def on_message(client, userdata, msg): # The callback for when a PUBLISH message is received from the server.
global cont
print(msg.topic)
cont=False
client = mqtt.Client(client_id="foo", clean_session=True)
client.on_connect = on_connect # Define callback function for successful connection
client.on_message = on_message # Define callback function for receipt of a message
#client.username_pw_set(mqtt_user, mqtt_password)
client.connect("localhost", port=1884)
client.loop_start()
def test(param1, param2):
lock.acquire()
try:
ret = client.publish("mytopic", "foo")
while cont:
time.sleep(5)
print("loop")
finally:
lock.release()
result = "foo"
return result
#app.route('/test/check', methods=['POST'])
def check():
global sessionId
sessionId = sessionId + 1
t = Thread(target=test, args=(sessionId,None))
t.start()
return {'id': sessionId, 'eta': 0}
if __name__ == '__main__':
print("started")
app.run()

Python script running in IDLE but not in terminal

There are many instances of this problem here, but all of them list an error when attempting to run the script. When I try to run my script from the terminal, it thinks for about a second, then just goes back to a normal terminal prompt (without an error message).
I am new to Python, so forgive me if I'm simply doing something wrong, but for the life of me . . . this runs perfectly fine when opening/running in IDLE.
import paho.mqtt.client as mqtt
import paho.mqtt.publish as publish
import RPi.GPIO as GPIO
RELAY_PIN_1 = 23
RELAY_PIN_2 = 24
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(RELAY_PIN_1, GPIO.OUT, initial=GPIO.HIGH)
GPIO.setup(RELAY_PIN_2, GPIO.OUT, initial=GPIO.HIGH)
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
client.subscribe("/amber/1/1")
client.subscribe("/amber/1/2")
def on_message(client, userdata, msg):
if msg.topic == "/amber/1/1":
if msg.payload == b'ON':
GPIO.output(RELAY_PIN_1, GPIO.LOW)
elif msg.payload == b'OFF':
GPIO.output(RELAY_PIN_1, GPIO.HIGH)
if msg.topic == "/amber/1/2":
if msg.payload == b'ON':
GPIO.output(RELAY_PIN_2, GPIO.LOW)
elif msg.payload == b'OFF':
GPIO.output(RELAY_PIN_2, GPIO.HIGH)
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("10.0.0.163", 1883, 60)
client.loop_start()
In case anyone is wondering, I have been trying to run this using the command python mqtt.py (with mqtt.py being the file name). Also, I am using Python2.7 in both IDLE and from the terminal.

When you call loop_start() you start the client on another thread, but right after that the script ends and the process terminates and so nothing happens.
If you want the client to keep running, either sleep() in your main thread after loop_start(), or instead, call loop_forever(), which will loop in the current thread and so your script will not terminate.
See the network loop section in the documentation.