I want to implement a Paho MQTT Python Service which is always running, receiving and sending messages. If an error occurs in any instance it should restart.
I implemented two classes which each start a threaded network loop with paho's loop_start(). These classes then have some callback functions which call other classes and so on.
For now i have a simple Python script which calls the classes and loops:
from one import one
from two import two
import time
one()
two()
while True:
if one.is_alive():
print("one is still alive")
else:
print("one died - do something!")
time.sleep(1)
And here my class "one":
import paho.mqtt.client as mqtt
import json
class one():
def __init__(self):
self.__client = mqtt.Client(client_id = "one")
self.__client.connect("localhost", 1883)
self.__client.subscribe("one")
self.__client.on_connect = self.__on_connect
self.__client.on_message = self.__on_message
self.__client.on_disconnect = self.__on_disconnect
self.__client.loop_start()
def __on_connect(self, client, userdata, flags, rc):
print("one: on_connect")
def __on_disconnect(self, client, userdata, flags, rc):
print("one: on_disconnect")
def __on_message(self, client, userdata, message):
str_message = message.payload.decode('utf-8')
message = json.loads(str_message)
print("one: on_message: " + str(message))
def is_alive(self):
return True
However - if I send a package which produces an error (a pickled message instead of json for example) my "is_alive"-function is still returning True but the paho-implementation is not responsive anymore. So no further messages are sent to on_message. So only a part of the class is still responsive!?
Class "two" is still responsive and the script is running in the "while True" still.
How do i properly check the functionality of such a class?
I think you have to build a checker method like class1.isAlive() which tells you if the class is waitng for requests. Also I think you have to build this in the while True loop and react than of failures.
Additionally, you could write your own event with a wait function. Wait is more CPU hungry but it is more responsive. See here for example. But it depends on your python version.
Related
I have been having some interesting issues recently with Python and MQTT.
Basically, my code is subscribing to a topic, and every time there is a new message published, it tries to control a device. Now, this is a blocking function and thus is run in a separate thread, so that on_message() would return immediately.
Additionally, the code publishes a status to a topic every 60 seconds. The code runs fine in the beginning, often a day or two. The device is being controlled via subscribed MQTT messages and the status is published just fine.
Then, it suddenly stops receiving any MQTT messages and also stops publishing them. The publish() function however, does not indicate that there would be problems, and is_connected() returns True. Restarting the program allows it to run another day or two. Below is the full code.
import paho.mqtt.client as mqtt
import json
import threading
class Controller():
def __init__(self):
self.mqtt_client = mqtt.Client()
self.pub_topic = "outgoing"
self.mqtt_client.on_message = self.on_message
self.mqtt_client.connect("192.168.1.1", 1883, 600)
self.mqtt_client.subscribe("incoming")
# This is a blocking function, execution takes approximately 5 minutes.
# The function only runs if there is no existing thread running it yet.
def control_device(self, input_commands):
print("Do some stuff...")
def process_mqtt(self, msg):
mqtt_msg = json.loads(msg.payload.decode('utf-8'))
self.control_device(mqtt_msg)
payload = '{"message": "process started"}'
self.mqtt_client.publish(self.pub_topic, payload)
def on_message(self, client, userdata, msg):
thread = threading.Thread(target=self.process_mqtt, args=(msg,))
thread.start()
# Status is sent to the same topic every 60 seconds
def send_status_msg(self):
if minute_passed:
payload = '{"status": 0}'
self.mqtt_client.publish(self.pub_topic, payload)
def run(self):
while True:
self.mqtt_client.loop()
self.send_status_msg()
if __name__ == "__main__":
c = Controller()
c.run()
Is there something I have not understood about how the MQTT library works? I found some discussion about how you should not publish inside on_message(), but in this case it is put into a separate thread.
I try to setup a mqtt client in python3. This is not the first time im doing this, however i came across a rather odd behaviour.
When trying to call a function, which contains a bug, from one of the callback functions (on_connect or on_message), python does not throw an exception (at least it is not printed), it just stops there. I tied together a short example, that reproduces that behaviour.
Does someone has an idea?
import paho.mqtt.client as mqtt
import re
import os.path
import json
from termcolor import colored
client = mqtt.Client()
def func():
test = 1 + "1"
print("Should never reach that")
def on_connect(client, userdata, flags, rc):
"""Establishes connection to broker
"""
print("Connected to broker with result code " + str(rc))
client.subscribe("test")
def on_message(client,userdata,msg):
print("Recieved message on " + msg.topic)
params = {}
if msg.topic == "test":
print("Invoke func")
func()
if __name__ == "__main__":
client.on_connect = on_connect
client.on_message = on_message
client.connect("localhost",1883,60)
client.loop_forever()
This is the output when sending a message to the topic "test":
Connected to broker with result code 0
Recieved message on test
Invoke func
When calling func() from main, i get the correct TypeError thrown. So something catches this exception in paho. I had a look at an olderproject (python2 though) and tried to recreate the behaviour. There the exception gets thrown correctly. What do i miss?
EDIT
I can catch the exception by putting the func() call in a try block. How ever, it does not stop the execution of the program when not catched. I dont get why
For anybody who comes across this and wonders why all exceptions inside of a mqtt callback are not thrown or at least not visible: In contrast to the python2 version of paho, the clients already catches ALL exceptions that occur when calling on of the user set callback functions. The output of this catch is then outputted to the on_log callback function. If this is not implemented by the user, there will be no visible output. So just add
def on_log(client, userdata, level, buff):
print(buff)
mqttc.on_log = on_log
to your code, to print out the exception.
This will be due to the fact that the on_message function is called by the network thread and it will be wrapping that call in a try block to stop errors in on_message from stopping that thread.
If you want to an error to stop the app then you should use your own try block in on_message and behave appropriately.
You can catch the errors using try + expect and then manually print the error message and pointer to the source of error using the traceback. This will give you mode details than using the on_log function.
import traceback
def on_message(client, userdata, msg):
try:
do_something(msg)
except:
traceback.print_exc()
quit(0)
Currently I am writing an application using the SimpleXMLRPCServer module in Python.
The basic aim of this application is to keep running on a server and keep checking a Queue for any task. If it encounters any new request in the Queue, serve the request.
Snapshot of what I am trying to do :
class MyClass():
"""
This class will have methods which will be exposed to the clients
"""
def __init__(self):
taskQ = Queue.Queue()
def do_some_task(self):
while True:
logging.info("Checking the Queue for any Tasks..")
task = taskQ.get()
# Do some processing based on the availability of some task
Main
if name == "main":
server = SimpleXMLRPCServer.SimpleXMLRPCServer((socket.gethostname(), Port)
classObj = MyClass()
rpcserver.register_function(classObj.do_some_task)
rpcserver.serve_forever()
Once the server is started it remains in the loop forever inside do_some_task method to keep checking the Queue for any task. This is what i wanted to achieve. But now i want to gracefully shutdown the server. In this case i am unable to shutdown the server.
Till now I have Tried using a global flag STOP_SERVER for 'True' and checking its status in the do_some_task while loop to get out of it and stop the server. But no help.
Tried using SHUTDOWN() method of the SimpleXMLRPCServer but it seems it is getting into a infinite loop of somekind.
Could you suggest some proper way to gracefully shutdown the server.
Thanks in advance
You should use handle_request() instead of serve_forever() if you want to close it manualy. Because SimpleXMLRPCServer is implemented as a single thread and the serve_forever() will make the server instance run into an infinite loop.
You can refer to this article. This is an example cited from there:
from SimpleXMLRPCServer import *
class MyServer(SimpleXMLRPCServer):
def serve_forever(self):
self.quit = 0
while not self.quit:
self.handle_request()
def kill():
server.quit = 1
return 1
server = MyServer(('127.0.0.1', 8000))
server.register_function(kill)
server.serve_forever()
By using handle_request(), this code use a state variable self.quit to indicate whether to quit the infinite loop.
The serve_forever function is inherited from a base class in the socketserver module called BaseServer. If you look at this fucntion you'll see it has an attribute called __shutdown_request, and this can be used to break the serving while loop. Because of the double underscore you'll have to access the variable with its mangled name: _BaseServer__shutdown_request.
Putting that all together you can make a very simple quit function as follows:
from xmlrpc.server import SimpleXMLRPCServer
class MyXMLRPCServer(SimpleXMLRPCServer):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.register_function(self.quit)
def quit(self):
self._BaseServer__shutdown_request = True
return 0
I have a Tornado WebSocket Server running in a separate process that is launched by a thread. This thread calls the publish method of my TornadoServer when it gets messages to send via websockets.
Running Tornado on a separate process was the only way I found to start the tornado loop without the thread blocking on this call.
In my thread, I start the tornado process by calling these methods on thread init method:
self.p = tornado_server.TornadoServer()
self.p.daemon = True
self.p.start()
In this thread, I have an infinite loop that tries to get messages from a Queue and if it gets messages, it calls the self.p.publish(client, message).
So far, so good.
On the Tornado process, I basically implemented a publish/subscribe system. When a user opens a webpage, the page sends a "subscription" message for a specific "client" let's say. On the "on_message" callback I append a tuple of the WebSocketHandler instance and the client that the user wants to subscribe to a global list.
Then, the publish method should search in the list for subscribed users to the message's target client and it should call the write_message on the WebSocket stored on that list.
The only thing that it isn't working is that my "clients" list have different scopes or something.
This is the code of my tornado_server file:
#!/usr/bin/python2
import tornado.web, tornado.websocket, tornado.ioloop, multiprocessing
clients = []
class TornadoServer(multiprocessing.Process):
class WebSocketHandler(tornado.websocket.WebSocketHandler):
def on_message(self, message):
global clients
print 'TORNADO - Received message:', str(message)
channel, subtopic = message.split('/')
print 'TORNADO - Subscribing:', str(subtopic)
clients.append((self, subtopic))
def on_close(self):
global clients
for websocket, client in clients:
if self == websocket:
print 'TORNADO - Removed client'
to_remove = (self, client)
clients.remove(to_remove)
def __init__(self):
multiprocessing.Process.__init__(self)
self.application = tornado.web.Application([(r"/tri-anim", WebSocketHandler)])
self.application.listen(1339)
def run(self):
tornado.ioloop.IOLoop.current().start()
def publish(self, client, message):
global clients
for websocket, websocketclient in clients:
if websocketclient == client:
websocket.write_message(str(message))
No matter what I do, clients have always different scopes. When publish is called, the "clients" is always empty. Is there any way to get this working?
You're calling publish in the parent process, but the clients list is only updated in the child process. When using multiprocessing each process gets its own copy of all the variables. If you used threads instead the variables would be shared, but even then you'd need to use IOLoop.instance().add_callback to do a thread-safe handoff between the thread calling publish and the write_message function (which must be called on the IOLoop thread).
I am writing a home automation helpers - they are basically small daemon-like python applications. They can run each as a separate process but since there will be made I decided that I will put up a small dispatcher that will spawn each of the daemons in their own threads and be able to act shall a thread die in the future.
This is what it looks like (working with two classes):
from daemons import mosquitto_daemon, gtalk_daemon
from threading import Thread
print('Starting daemons')
mq_client = mosquitto_daemon.Client()
gt_client = gtalk_daemon.Client()
print('Starting MQ')
mq = Thread(target=mq_client.run)
mq.start()
print('Starting GT')
gt = Thread(target=gt_client.run)
gt.start()
while mq.isAlive() and gt.isAlive():
pass
print('something died')
The problem is that MQ daemon (moquitto) will work fine shall I run it directly:
mq_client = mosquitto_daemon.Client()
mq_client.run()
It will start and hang in there listening to all the messages that hit relevant topics - exactly what I'm looking for.
However, run within the dispatcher makes it act weirdly - it will receive a single message and then stop acting yet the thread is reported to be alive. Given it works fine without the threading woodoo I'm assuming I'm doing something wrong in the dispatcher.
I'm quoting the MQ client code just in case:
import mosquitto
import config
import sys
import logging
class Client():
mc = None
def __init__(self):
logging.basicConfig(format=u'%(filename)s:%(lineno)d %(levelname)-8s [%(asctime)s] %(message)s', level=logging.DEBUG)
logging.debug('Class initialization...')
if not Client.mc:
logging.info('Creating an instance of MQ client...')
try:
Client.mc = mosquitto.Mosquitto(config.DEVICE_NAME)
Client.mc.connect(host=config.MQ_BROKER_ADDRESS)
logging.debug('Successfully created MQ client...')
logging.debug('Subscribing to topics...')
for topic in config.MQ_TOPICS:
result, some_number = Client.mc.subscribe(topic, 0)
if result == 0:
logging.debug('Subscription to topic "%s" successful' % topic)
else:
logging.error('Failed to subscribe to topic "%s": %s' % (topic, result))
logging.debug('Settings up callbacks...')
self.mc.on_message = self.on_message
logging.info('Finished initialization')
except Exception as e:
logging.critical('Failed to complete creating MQ client: %s' % e.message)
self.mc = None
else:
logging.critical('Instance of MQ Client exists - passing...')
sys.exit(status=1)
def run(self):
self.mc.loop_forever()
def on_message(self, mosq, obj, msg):
print('meesage!!111')
logging.info('Message received on topic %s: %s' % (msg.topic, msg.payload))
You are passing Thread another class instance's run method... It doesn't really know what to do with it.
threading.Thread can be used in two general ways: spawn a Thread wrapped independent function, or as a base class for a class with a run method.
In your case it appears like baseclass is the way to go, since your Client class has a run method.
Replace the following in your MQ class and it should work:
from threading import Thread
class Client(Thread):
mc = None
def __init__(self):
Thread.__init__(self) # initialize the Thread instance
...
...
def stop(self):
# some sort of command to stop mc
self.mc.stop() # not sure what the actual command is, if one exists at all...
Then when calling it, do it without Thread:
mq_client = mosquitto_daemon.Client()
mq_client.start()
print 'Print this line to be sure we get here after starting the thread loop...'
Several things to consider:
zeromq hates being initialized in 1 thread and run in another. You can rewrite Client() to be a Thread as suggested, or write your own function that will create a Client and run that function in a thread.
Client() has a class level variable mc. I assume that mosquitto_daemon and gtalk_daemon both use the same Client and so they are in contention for which Client.mc wins.
"while mq.isAlive() and gt.isAlive(): pass" will eat an entire processor because it just keeps polling over and over without sleep. Considering that python is only quasi-threaded (the Global Interpreter Lock (GIL) allows only 1 thread to run at a single time), this will stall out your "daemons".
Also considering the GIL, the orignal daemon implementation is likely to perform better.