I have a python script that will run on a local machine that needs to access a message queue (RabbitMQ) or receive subscribed events over HTTP. I've researched several solutions, but none seem natively designed to allow desktop clients to access them over HTTP. I'm thinking that using Twisted as a proxy is an option as well. Any guidance or recommendations would be greatly appreciated. Thanks in advance.
I've read this tutorial on RabbitMQ site, and they provide name of some libraries that could solve receiving messages.
Sender: send.py
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='',
routing_key='hello',
body='Hello World!')
print " [x] Sent 'Hello World!'"
connection.close()
Receiver: receive.py
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
print ' [*] Waiting for messages. To exit press CTRL+C'
def callback(ch, method, properties, body):
print " [x] Received %r" % (body,)
channel.basic_consume(callback,
queue='hello',
no_ack=True)
channel.start_consuming()
Now we can try out our programs in a terminal. First, let's send a message using our send.py program:
$ python send.py
[x] Sent 'Hello World!'
The producer program send.py will stop after every run. Let's receive it:
$ python receive.py
[*] Waiting for messages. To exit press CTRL+C
[x] Received 'Hello World!'
Hurray! We were able to send our first message through RabbitMQ. As you might have noticed, the receive.py program doesn't exit. It will stay ready to receive further messages, and may be interrupted with Ctrl-C.
Try to run send.py again in a new terminal.
We've learned how to send and receive a message from a named queue. It's time to move on to part 2 and build a simple work queue.
I've decided to use wamp http://wamp.ws/. Still experimenting with it, but it's working quite well at the moment.
Choice #1
You may be interested in this RabbitHub
Choice #2
If you want it to be on port#80, cant you do port forwarding using a proxy? It could be challenging, but
Choice #3
If your script is not tightly coupled with RMQ message format , you can try celery ( which uses RMQ underneath), then u can try celery Http gateway or celery web hooks if u want any other application to be triggered directly
It might be time consuming to get it up. However, Celery opens up loads of flexibility
Choice #4
For one of my projects, I developed an intermediate web service (Flask Service) to use RMQ
Not ideal, but it served the purpose at that time.
Related
I am trying to write a producer and consumer code in python using pika for rabbitmq. However for my specific case, I need to run producer on a different host and consumer on other.
I have already written a producer code as:
import pika
credentials = pika.PlainCredentials('username', 'password')
parameters = pika.ConnectionParameters('ip add of another host', 5672, '/', credentials)
connection = pika.BlockingConnection()
channel = connection.channel()
channel.queue_declare(queue='test')
channel.basic_publish(exchange='', routing_key='test', body='hello all!')
print (" [x] sent 'Hello all!")
connection.close()
The above producer code is running without any error. I also created a new user and gave administrator credentials to it on rabbitmq-server. However when I run the consumer code on another host running rabbitmq-server, I do not see any output:
import pika
credentials = pika.PlainCredentials('username', 'password')
parameters = pika.ConnectionParameters('localhost', 5672, '/', credentials)
connection = pika.BlockingConnection()
channel = connection.channel()
channel.queue_declare(queue='test')
def callback(ch, method, properties, body):
print(" [x] Recieved %r" % body)
channel.basic_consume(
queue='test', on_message_callback=callback, auto_ack=True)
print (' [x] waiting for messages. To exit press ctrl+c')
channel.start_consume()
So, here i had two hosts on the same network which had rabbitmq installed. However one has 3.7.10 and other had 3.7.16 version of rabbitmq.
The producer is able to send the text without error, but the consumer on another host is not receiving any text.
I do not get any problem when both run on same machine, as i just replace connection settings with localhost. Since user guest is only allowed to connect on localhost by default, i created a new user on consumer host running rabbitmq-server.
Please look if anyone can help me out here...
I have a couple of questions when I see your problem:
Are you 100% sure that on your RabbitMQ management monitoring
you see 2 connections? One from your local host and another from the another host? This will help to debug
Second, Did you check that your ongoing port 5672 on the server that host RabbitMQ is open? Because maybe your producer does not manage to connect What is your cloud provider?
If you don't want to manage those kinds of issues, you should use a service like https://zenaton.com. They host everything for you, and you have integrated monitoring, error handling etc.
Your consumer and producer applications must connect to the same RabbitMQ server. If you have two instances of RabbitMQ running they are independent. Messages do not move from one instance of RabbitMQ to another unless you configure Shovel or Federation.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
You don't seem to be passing the parameters to the BlockingConnection instance.
import pika
rmq_server = "ip_address_of_rmq_server"
credentials = pika.PlainCredentials('username', 'password')
parameters = pika.ConnectionParameters(rmq_server, 5672, '/', credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
Also, your consumer is attaching to the localhost hostname. Make sure this actually resolves and that your RabbitMQ service is listening on the localhost address (127.0.0.1) It may not be bound to that address. I believe that RMQ will bind to all interfaces (and thus all addresses) by default but I'm not sure.
I'm using the gevent-websocket module with Bottle Python framework.
When a client closes the browser, this code
$(window).on('beforeunload', function() { ws.close(); });
helps to close the websocket connection properly.
But if the client's network connection is interrupted, no "close" information can be sent to the server.
Then, often, even 1 minute later, the server still believes the client is connected, and the websocket is still open on the server.
Question: How to detect properly that a websocket is closed because the client is disconnected from network?
Is there a websocket KeepAlive feature available in Python/Bottle/gevent-websocket?
One answer from Web Socket: cannot detect client connection on internet disconnect suggests to use a heartbeat/ping packet every x seconds to tell the server "I'm still alive". The other answer suggests using a setKeepAlive(true). feature. Would this feature be available in gevent-websocket?
Example server code, taken from here:
from bottle import get, template, run
from bottle.ext.websocket import GeventWebSocketServer
from bottle.ext.websocket import websocket
users = set()
#get('/')
def index():
return template('index')
#get('/websocket', apply=[websocket])
def chat(ws):
users.add(ws)
while True:
msg = ws.receive()
if msg is not None:
for u in users:
u.send(msg)
else:
break
users.remove(ws)
run(host='127.0.0.1', port=8080, server=GeventWebSocketServer)
First you need to add a timeout to the receive() method.
with gevent.Timeout(1.0, False):
msg = ws.receive()
Then the loop will not block, if you send even an empty packet and the client doesn't respond, WebsocketError will be thrown and you can close the socket.
I have installed the last version of raspbian on my raspberry pi, and I have opened an account AWS IoT on Amazon, then in the IoT web interface I have created a thing, with "RaspberryPi_2" name and create certificate and connect the certificate to the thing, I have followed this guide:
http://blog.getflint.io/blog/get-started-with-aws-iot-and-raspberry-pi
I have then created the script in the guide, to connect and subscribed the raspberry pi, this is my code:
#!/usr/bin/python3
#required libraries
import sys
import ssl
import paho.mqtt.client as mqtt
#called while client tries to establish connection with the server
def on_connect(mqttc, obj, flags, rc):
if rc==0:
print ("Subscriber Connection status code: "+str(rc)+" | Connection status: successful")
elif rc==1:
print ("Subscriber Connection status code: "+str(rc)+" | Connection status: Connection refused")
#called when a topic is successfully subscribed to
def on_subscribe(mqttc, obj, mid, granted_qos):
print("Subscribed: "+str(mid)+" "+str(granted_qos)+"data"+str(obj))
#called when a message is received by a topic
def on_message(mqttc, obj, msg):
print("Received message from topic: "+msg.topic+" | QoS: "+str(msg.qos)+" | Data Received: "+str(msg.payload))
#creating a client with client-id=mqtt-test
mqttc = mqtt.Client(client_id="mqtt-test")
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
mqttc.on_message = on_message
#Configure network encryption and authentication options. Enables SSL/TLS support.
#adding client-side certificates and enabling tlsv1.2 support as required by aws-iot service
mqttc.tls_set("/home/pi/aws_iot/things/raspberryPi_2/certs/aws-iot-rootCA.crt",
certfile="/home/pi/aws_iot/things/raspberryPi_2/certs/0ea2cd7eb6-certificate.pem.crt",
keyfile="/home/pi/aws_iot/things/raspberryPi_2/certs/0ea2cd7eb6-private.pem.key",
tls_version=ssl.PROTOCOL_TLSv1_2,
ciphers=None)
#connecting to aws-account-specific-iot-endpoint
mqttc.connect("A2GF7W5U5A46J1.iot.us-west-2.amazonaws.com", port=8883) #AWS IoT service hostname and portno
#the topic to publish to
mqttc.subscribe("$aws/things/RaspberryPi_2/shadow/update/#", qos=1) #The names of these topics start with $aws/things/thingName/shadow."
#automatically handles reconnecting
mqttc.loop_forever()
but when I do this command:
python3 mqtt_test.py
or this command:
python mqtt_test.py
and press enter, and the cursor flash and doesn't print anything and stay stuck there, someone can help me?
I haven't also understand if the client-id name should be the same of the things name, and the meaning of the subscribe path, for example in a tutorial I have found this:
mqttc.publish("temperature", tempreading, qos=1)
why there isn't the complete path?
or this:
$aws/things/RaspberryPi_2/shadow/update/delta
so I can put everything I want in the path?
thanks
The code is subscribing to a topic but there is no one publishing to it. So, the code also has a on_connect function that would be triggered after a success connection. Is the message "Subscriber Connection status code: ..." being printed? If it is, the message from on_subscribe should also appear. Is it?
If it is not you have a problem before connect to the AWS server. Use netstat command to see where your Raspberry Pi is connected or not and post more debug information in this case.
If the connect and subscribe messages are shown and nothing happens after it, this is normal because you are only subscribing to a topic but not publishing anything.
Regarding topics, think of them as a directory structure. There is no strict rule for topic hierarchy. A "temperature" topic would be temperature topic on the server and a "temperature/livingroom" would be temperature/livingroom, you can subscribe to one, another or both on the same server. The path you choose for your things shall be meaningful to your application. A house, for instance, might be represented as:
house/kitchen/env/temperature
house/kitchen/env/humidity
house/kitchen/lamp/sinklamp
house/kitchen/lamp/mainlap
house/masterbed/env/temperature
house/masterbed/env/humidity
house/masterbed/lamp/readinglampleft
house/masterbed/lamp/readinglampright
house/masterbed/lamp/mainlamp
house/masterbed/lamp/mirrorlamp
And so on.
Let´s say you have a thermostat at master bedroom. It is interested only in temperature but not humidity. It is also interested only in master bedroom temperature. This thermostat shall subscribe to house/masterbed/env/temperature. Opposite to this, a room wide panel that shows state of every thing in the room, would subscribe to house/masterbed/#, meaning "everything after house/masterbed". Read more on wildcards here
The topic you subscribed for: $aws/things/RaspberryPi_2/shadow/update/# means, "every thing after $aws/things/RaspberryPi_2/shadow/update/". Notice that his is a special topic, it starts with $aws, specially, it starts with $ character. In the AWS context this means:
Any topics beginning with $ are considered reserved and are not
supported for publishing and subscribing except when working with the
Thing Shadows service. For more information, see Thing Shadows.
So you need to understand what thing shadow is. This is a AWS specific (and very util) concept. Please read docs on this subject.
Finally, I would you suggest you install a local broker (mosquitto is available on respbian) and try with it before got to AWS. This way you can master mqtt concept without connectivity issues. Later you put AWS to the mix.
I have an existing Python system that receives messages using Rabbit MQ. What is the absolute easiest way to get these events pushed to a browser using WebSockets using Python? Bonus if the solution works in all major browsers too.
Thanks,
Virgil
Here https://github.com/Gsantomaggio/rabbitmqexample I wrote an complete example that uses tornado and RabbitMQ.
You can find all the instruction from the site:
anyway ..you need:
pip install pika
pip install tornado
First you register your rabbitmq:
def threaded_rmq():
channel.queue_declare(queue="my_queue")
logging.info('consumer ready, on my_queue')
channel.basic_consume(consumer_callback, queue="my_queue", no_ack=True)
channel.start_consuming()
then you register your web-sockets clients:
class SocketHandler(tornado.websocket.WebSocketHandler):
def open(self):
logging.info('WebSocket opened')
clients.append(self)
def on_close(self):
logging.info('WebSocket closed')
clients.remove(self)
When you get a message, you can redirect it to the web-socket page.
def consumer_callback(ch, method, properties, body):
logging.info("[x] Received %r" % (body,))
# The messagge is brodcast to the connected clients
for itm in clients:
itm.write_message(body)
You could use Twisted, txAMQP and Autobahn|Python on the server to write a bridge in probably 50 lines of code, and Autobahn|JS on the browser side. Autobahn implements WebSocket, and WAMP on top, which provides you with Publish & Subscribe (as well as Remote Procedure Calls) over WebSocket.
When using raw WebSocket you would have to invent your own Publish & Subscribe over WebSocket - since I guess that is what you are after: extending the AMQP PubSub to the Web. Or you could check out STOMP.
Disclaimer: I am original author of WAMP and Autobahn.
I have a flask application running with gevent-socketio that I create this way:
server = SocketIOServer(('localhost', 2345), app, resource='socket.io')
gevent.spawn(send_queued_messages_loop, server)
server.serve_forever()
I launch send_queued_messages_loop in a gevent thread that keeps on polling on a gevent.Queue where my program stores data to send it to the socket.io connected clients
I tried different approaches to stop the server (such as using sys.exit) either from the socket.io handler (when the client sends a socket.io message) or from a normal route (when the client makes a request to /shutdown) but in any case, sys.exit seems to fail because of the presence of greenlets.
I tried to call gevent.shutdown() first, but this does not seem to change anything
What would be the proper way to shutdown the server?
Instead of using serve_forever() create a gevent.event.Event and wait for it. To actually initiate shutdown, trigger the event using its set() method:
from gevent.event import Event
stopper = Event()
server = SocketIOServer(('localhost', 2345), app, resource='socket.io')
server.start()
gevent.spawn(send_queued_messages_loop)
try:
stopper.wait()
except KeyboardInterrupt:
print
No matter from where you now want to terminate your process - all you need to do is calling stopper.set().
The try..except is not really necessary but I prefer not getting a stacktrace on a clean CTRL-C exit.