I implemented connection to broker like:
app.py
import paho.mqtt.client as mqtt
client = mqtt.Client(client_id='my_client', clean_session=False)
my_client = MyClient(client)
try:
my_client.start()
while True:
try:
client.loop()
except Exception as e:
my_client.start()
except Exception as e:
client.loop_stop()
exit(1)
MyClient.py
class MyClient:
def __init__(self, mqtt=None):
self.mqtt = mqtt
def start(self):
self.mqtt.subscribe('some/topic')
I have part of code where I want to pause topics listening:
self.mqtt.unsubscribe('some/topic')
And later I want to subscribe back to it I want to call start() again like: self.start()
But it never subscribe again. Any idea why?
Calling start() after the exception is thrown won't work as the client is most likely not connected at that point.
You should move your subscriptions to the on_connect callback then it will always re-subscribe after the client has (re)connected
As for your original question, probably better to just set a boolean flag and use it to gate processing the message rather than unsubscribing/subscribing when you want to ignore messages.
Related
I am having a difficult time finding gRPC pub-sub subscriber template for python.
What I am trying is this, but it doesn't seem to work out.
class DaprClientServicer(daprclient_services.DaprClientServicer):
def OnTopicEvent(self, request, context):
if request.topic=="TOPIC_A":
print("Do something")
response = "some response"
return response
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
daprclient_services.add_DaprClientServicer_to_server(DaprClientServicer(), server)
server.add_insecure_port('[::]:50051')
server.start()
try:
while True:
time.sleep(86400)
except KeyboardInterrupt:
server.stop(0)
My publish statement looks something like this:
client.PublishEvent(dapr_messages.PublishEventEnvelope(topic='TOPIC_A', data=data))
Old Answer (Dapr's Python-SDK APIs changed significantly after this)
After doing a bit of research I found that I was skipping on the step.
So how subscriber work is like this:
Subscribe to a topic. (missing step)
Handel the messaged published to the subscribed topic.
Doing this worked for me:
# Our server methods
class DaprClientServicer(daprclient_services.DaprClientServicer):
def GetTopicSubscriptions(self, request, context):
# Dapr will call this method to get the list of topics the app
# wants to subscribe to. In this example, we are telling Dapr
# To subscribe to a topic named TOPIC_A
return daprclient_messages.GetTopicSubscriptionsEnvelope(topics=['TOPIC_A'])
def OnTopicEvent(self, request, context):
logging.info("Event received!!")
return empty_pb2.Empty()
# Create a gRPC server
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
daprclient_services.add_DaprClientServicer_to_server(
DaprClientServicer(), server)
# Start the gRPC server
print('Starting server. Listening on port 50051.')
server.add_insecure_port('[::]:50051')
server.start()
# Since server.start() doesn't block, we need to do a sleep loop
try:
while True:
time.sleep(86400)
except KeyboardInterrupt:
server.stop(0)
I try to setup a mqtt client in python3. This is not the first time im doing this, however i came across a rather odd behaviour.
When trying to call a function, which contains a bug, from one of the callback functions (on_connect or on_message), python does not throw an exception (at least it is not printed), it just stops there. I tied together a short example, that reproduces that behaviour.
Does someone has an idea?
import paho.mqtt.client as mqtt
import re
import os.path
import json
from termcolor import colored
client = mqtt.Client()
def func():
test = 1 + "1"
print("Should never reach that")
def on_connect(client, userdata, flags, rc):
"""Establishes connection to broker
"""
print("Connected to broker with result code " + str(rc))
client.subscribe("test")
def on_message(client,userdata,msg):
print("Recieved message on " + msg.topic)
params = {}
if msg.topic == "test":
print("Invoke func")
func()
if __name__ == "__main__":
client.on_connect = on_connect
client.on_message = on_message
client.connect("localhost",1883,60)
client.loop_forever()
This is the output when sending a message to the topic "test":
Connected to broker with result code 0
Recieved message on test
Invoke func
When calling func() from main, i get the correct TypeError thrown. So something catches this exception in paho. I had a look at an olderproject (python2 though) and tried to recreate the behaviour. There the exception gets thrown correctly. What do i miss?
EDIT
I can catch the exception by putting the func() call in a try block. How ever, it does not stop the execution of the program when not catched. I dont get why
For anybody who comes across this and wonders why all exceptions inside of a mqtt callback are not thrown or at least not visible: In contrast to the python2 version of paho, the clients already catches ALL exceptions that occur when calling on of the user set callback functions. The output of this catch is then outputted to the on_log callback function. If this is not implemented by the user, there will be no visible output. So just add
def on_log(client, userdata, level, buff):
print(buff)
mqttc.on_log = on_log
to your code, to print out the exception.
This will be due to the fact that the on_message function is called by the network thread and it will be wrapping that call in a try block to stop errors in on_message from stopping that thread.
If you want to an error to stop the app then you should use your own try block in on_message and behave appropriately.
You can catch the errors using try + expect and then manually print the error message and pointer to the source of error using the traceback. This will give you mode details than using the on_log function.
import traceback
def on_message(client, userdata, msg):
try:
do_something(msg)
except:
traceback.print_exc()
quit(0)
Summary
I have a client-server application which makes use of Websockets. The backend (server) part is implemented in Python using autobahn.
The server, in addition to serving a Websockets endpoint, runs a series of threads which will feed the Websockets channel with data, though a queue.Queue().
One of these threads has a problem: it crashes at a missing parameter and hangs when resolving the exception.
Implementation details
The server implementation (cut down to highlight the problem):
from autobahn.asyncio.websocket import WebSocketServerProtocol, WebSocketServerFactory
import time
import threading
import arrow
import queue
import asyncio
import json
# backends of components
import dummy
class MyServerProtocol(WebSocketServerProtocol):
def __init__(self):
super().__init__()
print("webserver initialized")
# global queue to handle updates from modules
self.events = queue.Queue()
# consumer
threading.Thread(target=self.push).start()
threading.Thread(target=dummy.Dummy().dummy, args=(self.events,)).start()
def push(self):
""" consume the content of the queue and push it to the browser """
while True:
update = self.events.get()
print(update)
if update:
self.sendMessage(json.dumps(update).encode('utf-8'), False)
print(update)
time.sleep(1)
def worker(self):
print("started thread")
while True:
try:
self.sendMessage(arrow.now().isoformat().encode('utf-8'), False)
except AttributeError:
print("not connected?")
time.sleep(3)
def onConnect(self, request):
print("Client connecting: {0}".format(request.peer))
def onOpen(self):
print("WebSocket connection open.")
def onClose(self, wasClean, code, reason):
print("WebSocket connection closed: {0}".format(reason))
if __name__ == '__main__':
factory = WebSocketServerFactory(u"ws://127.0.0.1:9100")
factory.protocol = MyServerProtocol
loop = asyncio.get_event_loop()
coro = loop.create_server(factory, '0.0.0.0', 9100)
loop.run_until_complete(coro)
loop.run_forever()
The dummy module imported in the code above:
import time
import arrow
class Dummy:
def __init__(self, events):
self.events = events
print("dummy initialized")
def dummy(self):
while True:
self.events.put({
'dummy': {
'time': arrow.now().isoformat()
}
})
time.sleep(1)
The problem
When running the code above and connecting from a client, I get on the output webserver initialized (which proves that the connection was initiated), and WebSocket connection to 'ws://127.0.0.1:9100/' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED on the client.
When debugging the code, I see that the call to threading.Thread(target=dummy.Dummy().dummy, args=(self.events,)).start() crashes and the debugger (PyCharm) leads me to C:\Program Files (x86)\Python36-32\Lib\asyncio\selector_events.py, specifically to the line 236
# It's now up to the protocol to handle the connection.
except Exception as exc:
if self._debug:
The thread hangs when executing if self._debug but I see on the exceptline (thanks to Pycharm) that
exc: __init__() missing 1 required positional argument: 'events'
My question
Why is this parameter missing? It is provided via the threading.Thread(target=dummy.Dummy().dummy, args=(self.events,)).start() call.
As a side question: why does the thread hangs on the if condition?
Notes
there is never a Traceback thrown by my program (due to the hang)
removing this thread call resolves the issue (the client connects correctly)
The events arg is needed for the constructor, not the dummy method. I think you meant something more like:
d = Dummy(self.events)
threading.Thread(d.dummy).start()
I am trying to run the following test using pika 0.10.0 from github:
import logging
import sys
import pika
import threading
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
URL = 'amqp://guest:guest#127.0.0.1:5672/%2F?socket_timeout=0.25'
class BaseConsumer(threading.Thread):
def run(self):
self._connection = None
self._channel = None
self.connect()
self.open_channel()
self.consume()
def connect(self):
parameters = pika.URLParameters(URL)
self._connection = pika.BlockingConnection(parameters)
def open_channel(self):
self._channel = self._connection.channel()
self._channel.exchange_declare(exchange='exc1', exchange_type='topic', passive=False,
durable=False, auto_delete=False, internal=False, arguments=None)
self._channel.queue_declare(queue='test', passive=False, durable=False,
exclusive=False, auto_delete=False, arguments=None)
self._channel.queue_bind(
'test', 'exc1', routing_key='rk', arguments=None)
def consume(self):
self._channel.basic_consume(self.on_message, 'test')
try:
self._channel.start_consuming()
except KeyboardInterrupt:
logging.info("Stop consuming now!")
self._channel.stop_consuming()
self._connection.close()
def on_message(self, channel, method_frame, header_frame, body):
print method_frame.delivery_tag
print body
channel.basic_ack(delivery_tag=method_frame.delivery_tag)
c1 = BaseConsumer()
c1.setDaemon(False)
c1.start()
The script is connecting to my MQ and is apparently able to consume messages from the MQ. The problem is that I have no way of stoping the thread. Pressing CTRL-C on my keyboard only causes "^C" to appear in the console without interrupting the consume.
Question is, how do I make pika stop consuming when it is running inside of a thread ? I would like to note that I am following the guidelines of creating the connection in the consumer thread.
If after starting the thread with c1.start() I also do an infinite while loop and log something from there then pressing CTRL-C will end the while loop but the consumer thread will still ignore any additional CTRL-C.
Side question: is it possible to stop consuming inside the thread with some outside signalling like a threading.Condition or something ? I don't see how i can interfere with start_consuming.
Question: ... from there then pressing CTRL-C will end the while loop
Add a def stop() to your BaseConsumer,
catch the KeyboardInterrupt and call stop().
try:
BaseConsumer.run()
except KeyboardInterrupt:
BaseConsumer.stop()
Read pika: asynchronous_consumer_example
I'm using Pika to process data from RabbitMQ.
As I seemed to run into different kind of problems I decided to write a small test application to see how I can handle disconnects.
I wrote this test app which does following:
Connect to Broker, retry until successful
When connected create a queue.
Consume this queue and put result into a python Queue.Queue(0)
Get item from Queue.Queue(0) and produce it back into the broker queue.
What I noticed were 2 issues:
When I run my script from one host connecting to rabbitmq on another host (inside a vm) then this scripts exits on random moments without producing an error.
When I run my script on the same host on which RabbitMQ is installed it runs fine and keeps running.
This might be explained because of network issues, packets dropped although I find the connection not really robust.
When the script runs locally on the RabbitMQ server and I kill the RabbitMQ then the script exits with error: "ERROR pika SelectConnection: Socket Error on 3: 104"
So it looks like I can't get the reconnection strategy working as it should be. Could someone have a look at the code so see what I'm doing wrong?
Thanks,
Jay
#!/bin/python
import logging
import threading
import Queue
import pika
from pika.reconnection_strategies import SimpleReconnectionStrategy
from pika.adapters import SelectConnection
import time
from threading import Lock
class Broker(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.logging = logging.getLogger(__name__)
self.to_broker = Queue.Queue(0)
self.from_broker = Queue.Queue(0)
self.parameters = pika.ConnectionParameters(host='sandbox',heartbeat=True)
self.srs = SimpleReconnectionStrategy()
self.properties = pika.BasicProperties(delivery_mode=2)
self.connection = None
while True:
try:
self.connection = SelectConnection(self.parameters, self.on_connected, reconnection_strategy=self.srs)
break
except Exception as err:
self.logging.warning('Cant connect. Reason: %s' % err)
time.sleep(1)
self.daemon=True
def run(self):
while True:
self.submitData(self.from_broker.get(block=True))
pass
def on_connected(self,connection):
connection.channel(self.on_channel_open)
def on_channel_open(self,new_channel):
self.channel = new_channel
self.channel.queue_declare(queue='sandbox', durable=True)
self.channel.basic_consume(self.processData, queue='sandbox')
def processData(self, ch, method, properties, body):
self.logging.info('Received data from broker')
self.channel.basic_ack(delivery_tag=method.delivery_tag)
self.from_broker.put(body)
def submitData(self,data):
self.logging.info('Submitting data to broker.')
self.channel.basic_publish(exchange='',
routing_key='sandbox',
body=data,
properties=self.properties)
if __name__ == '__main__':
format=('%(asctime)s %(levelname)s %(name)s %(message)s')
logging.basicConfig(level=logging.DEBUG, format=format)
broker=Broker()
broker.start()
try:
broker.connection.ioloop.start()
except Exception as err:
print err
The main problem with your script is that it is interacting with a single channel from both your main thread (where the ioloop is running) and the "Broker" thread (calls submitData in a loop). This is not safe.
Also, SimpleReconnectionStrategy does not seem to do anything useful. It does not cause a reconnect if the connection is interrupted. I believe this is a bug in Pika: https://github.com/pika/pika/issues/120
I attempted to refactor your code to make it work as I think you wanted it to, but ran into another problem. Pika does not appear to have a way to detect delivery failure, which means that data may be lost if the connection drops. This seems like such an obvious requirement! How can there be no way to detect that basic_publish failed? I tried all kinds of stuff including transactions and add_on_return_callback (all of which seemed clunky and overly complicated), but came up with nothing. If there truly is no way then pika only seems to be useful in situations that can tolerate loss of data sent to RabbitMQ, or in programs that only need to consume from RabbitMQ.
This is not reliable, but for reference, here's some code that solves your multi-thread problem:
import logging
import pika
import Queue
import sys
import threading
import time
from functools import partial
from pika.adapters import SelectConnection, BlockingConnection
from pika.exceptions import AMQPConnectionError
from pika.reconnection_strategies import SimpleReconnectionStrategy
log = logging.getLogger(__name__)
DEFAULT_PROPERTIES = pika.BasicProperties(delivery_mode=2)
class Broker(object):
def __init__(self, parameters, on_channel_open, name='broker'):
self.parameters = parameters
self.on_channel_open = on_channel_open
self.name = name
def connect(self, forever=False):
name = self.name
while True:
try:
connection = SelectConnection(
self.parameters, self.on_connected)
log.debug('%s connected', name)
except Exception:
if not forever:
raise
log.warning('%s cannot connect', name, exc_info=True)
time.sleep(10)
continue
try:
connection.ioloop.start()
finally:
try:
connection.close()
connection.ioloop.start() # allow connection to close
except Exception:
pass
if not forever:
break
def on_connected(self, connection):
connection.channel(self.on_channel_open)
def setup_submitter(channel, data_queue, properties=DEFAULT_PROPERTIES):
def on_queue_declared(frame):
# PROBLEM pika does not appear to have a way to detect delivery
# failure, which means that data could be lost if the connection
# drops...
channel.confirm_delivery(on_delivered)
submit_data()
def on_delivered(frame):
if frame.method.NAME in ['Confirm.SelectOk', 'Basic.Ack']:
log.info('submission confirmed %r', frame)
# increasing this value seems to cause a higher failure rate
time.sleep(0)
submit_data()
else:
log.warn('submission failed: %r', frame)
#data_queue.put(...)
def submit_data():
log.info('waiting on data queue')
data = data_queue.get()
log.info('got data to submit')
channel.basic_publish(exchange='',
routing_key='sandbox',
body=data,
properties=properties,
mandatory=True)
log.info('submitted data to broker')
channel.queue_declare(
queue='sandbox', durable=True, callback=on_queue_declared)
def blocking_submitter(parameters, data_queue,
properties=DEFAULT_PROPERTIES):
while True:
try:
connection = BlockingConnection(parameters)
channel = connection.channel()
channel.queue_declare(queue='sandbox', durable=True)
except Exception:
log.error('connection failure', exc_info=True)
time.sleep(1)
continue
while True:
log.info('waiting on data queue')
try:
data = data_queue.get(timeout=1)
except Queue.Empty:
try:
connection.process_data_events()
except AMQPConnectionError:
break
continue
log.info('got data to submit')
try:
channel.basic_publish(exchange='',
routing_key='sandbox',
body=data,
properties=properties,
mandatory=True)
except Exception:
log.error('submission failed', exc_info=True)
data_queue.put(data)
break
log.info('submitted data to broker')
def setup_receiver(channel, data_queue):
def process_data(channel, method, properties, body):
log.info('received data from broker')
data_queue.put(body)
channel.basic_ack(delivery_tag=method.delivery_tag)
def on_queue_declared(frame):
channel.basic_consume(process_data, queue='sandbox')
channel.queue_declare(
queue='sandbox', durable=True, callback=on_queue_declared)
if __name__ == '__main__':
if len(sys.argv) != 2:
print 'usage: %s RABBITMQ_HOST' % sys.argv[0]
sys.exit()
format=('%(asctime)s %(levelname)s %(name)s %(message)s')
logging.basicConfig(level=logging.DEBUG, format=format)
host = sys.argv[1]
log.info('connecting to host: %s', host)
parameters = pika.ConnectionParameters(host=host, heartbeat=True)
data_queue = Queue.Queue(0)
data_queue.put('message') # prime the pump
# run submitter in a thread
setup = partial(setup_submitter, data_queue=data_queue)
broker = Broker(parameters, setup, 'submitter')
thread = threading.Thread(target=
partial(broker.connect, forever=True))
# uncomment these lines to use the blocking variant of the submitter
#thread = threading.Thread(target=
# partial(blocking_submitter, parameters, data_queue))
thread.daemon = True
thread.start()
# run receiver in main thread
setup = partial(setup_receiver, data_queue=data_queue)
broker = Broker(parameters, setup, 'receiver')
broker.connect(forever=True)