I have a web server running on Django.
Users can create events postponed in time.
These events must be recorded in queue and processed on another server.
Initially I thought to take the Twisted. something like:
#client - django server
factory = pb.PBClientFactory()
reactor.connectTCP(server_ip, server_port, factory)
d = factory.login(credentials.UsernamePassword(login, paswd),)
d.addCallbacks(self.good_connected,self.bad_connected)
d.addCallback(self.add_to_queue)
reactor.run()
def add_to_queue(self, p)
p.callRemote("pickup", data)
#server - twisted server
def perspective_pickup(self, data)
reactor.callLater(timeout, self.pickup_from_queue)
But now I have big doubts about this approach. Maybe do not use twisted? or connect it with Django differently
Run twisted inside of Django is not a good idea anyway. So, try Celery or run HTTP server with twisted and use urllib on django side to send data to twisted server.
Related
I have an existing Python system that receives messages using Rabbit MQ. What is the absolute easiest way to get these events pushed to a browser using WebSockets using Python? Bonus if the solution works in all major browsers too.
Thanks,
Virgil
Here https://github.com/Gsantomaggio/rabbitmqexample I wrote an complete example that uses tornado and RabbitMQ.
You can find all the instruction from the site:
anyway ..you need:
pip install pika
pip install tornado
First you register your rabbitmq:
def threaded_rmq():
channel.queue_declare(queue="my_queue")
logging.info('consumer ready, on my_queue')
channel.basic_consume(consumer_callback, queue="my_queue", no_ack=True)
channel.start_consuming()
then you register your web-sockets clients:
class SocketHandler(tornado.websocket.WebSocketHandler):
def open(self):
logging.info('WebSocket opened')
clients.append(self)
def on_close(self):
logging.info('WebSocket closed')
clients.remove(self)
When you get a message, you can redirect it to the web-socket page.
def consumer_callback(ch, method, properties, body):
logging.info("[x] Received %r" % (body,))
# The messagge is brodcast to the connected clients
for itm in clients:
itm.write_message(body)
You could use Twisted, txAMQP and Autobahn|Python on the server to write a bridge in probably 50 lines of code, and Autobahn|JS on the browser side. Autobahn implements WebSocket, and WAMP on top, which provides you with Publish & Subscribe (as well as Remote Procedure Calls) over WebSocket.
When using raw WebSocket you would have to invent your own Publish & Subscribe over WebSocket - since I guess that is what you are after: extending the AMQP PubSub to the Web. Or you could check out STOMP.
Disclaimer: I am original author of WAMP and Autobahn.
I am trying to learn Twisted, a Python framework, and I want to put a basic application online that, when it receive a message sends it back. I decided to use Heroku to host it, and I followed the instructions on their docs.
import os
from twisted.internet import protocol, reactor
class Echo(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)
class EchoFactory(protocol.Factory):
def buildProtocol(self, addr):
return Echo()
port = int(os.environ.get('PORT', 5000))
reactor.listenTCP(port, EchoFactory(), interface = '0.0.0.0')
reactor.run()
Its all working except (and I know this is a stupid question), how do I send a message to it? When I am working locally I just do telnet localhost <port>, but now I have no idea.
Also, since heroku connects to a random port how can I know what port it connects my app to?
Thanks.
I'm not very familiar with Twisted, but I'm not sure what you're trying to do is supported on Heroku. Heroku currently only supports HTTP[S] requests and not raw TCP. There are more details in the answers to this question.
If you wanted to connect to your app, you should use the myapp.herokuapp.com host name or any custom domain that you've added.
"Pure Python applications, such as headless processes and evented web frameworks like Twisted, are fully supported on Cedar."
Reference: https://devcenter.heroku.com/articles/python-support
I have a flask application running with gevent-socketio that I create this way:
server = SocketIOServer(('localhost', 2345), app, resource='socket.io')
gevent.spawn(send_queued_messages_loop, server)
server.serve_forever()
I launch send_queued_messages_loop in a gevent thread that keeps on polling on a gevent.Queue where my program stores data to send it to the socket.io connected clients
I tried different approaches to stop the server (such as using sys.exit) either from the socket.io handler (when the client sends a socket.io message) or from a normal route (when the client makes a request to /shutdown) but in any case, sys.exit seems to fail because of the presence of greenlets.
I tried to call gevent.shutdown() first, but this does not seem to change anything
What would be the proper way to shutdown the server?
Instead of using serve_forever() create a gevent.event.Event and wait for it. To actually initiate shutdown, trigger the event using its set() method:
from gevent.event import Event
stopper = Event()
server = SocketIOServer(('localhost', 2345), app, resource='socket.io')
server.start()
gevent.spawn(send_queued_messages_loop)
try:
stopper.wait()
except KeyboardInterrupt:
print
No matter from where you now want to terminate your process - all you need to do is calling stopper.set().
The try..except is not really necessary but I prefer not getting a stacktrace on a clean CTRL-C exit.
My project use bottle and HBase, client connect to HBase via python thrift client, code simplify like this
#!/usr/bin/env python
from bottle import route, run, default_app, request
client = HBaseClient()
#route('/', method='POST')
def index():
data = client.getdata()
return data
Now the issue is if client disconnect, our request will be failed. So it requires to make sure client keep alive.
One solution is using connection pool, is there any connection pool I can refer to?
Any other solution for this issue?
Looks happybase can deal this issue
HappyBase has a connection pool that tries to deal with broken connections to some extent: http://happybase.readthedocs.org/en/latest/user.html#using-the-connection-pool
I am trying to learn Twisted, a Python framework, and I want to put a basic application online that, when it receive a message sends it back. I decided to use Heroku to host it, and I followed the instructions on their docs.
import os
from twisted.internet import protocol, reactor
class Echo(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)
class EchoFactory(protocol.Factory):
def buildProtocol(self, addr):
return Echo()
port = int(os.environ.get('PORT', 5000))
reactor.listenTCP(port, EchoFactory(), interface = '0.0.0.0')
reactor.run()
Its all working except (and I know this is a stupid question), how do I send a message to it? When I am working locally I just do telnet localhost <port>, but now I have no idea.
Also, since heroku connects to a random port how can I know what port it connects my app to?
Thanks.
I'm not very familiar with Twisted, but I'm not sure what you're trying to do is supported on Heroku. Heroku currently only supports HTTP[S] requests and not raw TCP. There are more details in the answers to this question.
If you wanted to connect to your app, you should use the myapp.herokuapp.com host name or any custom domain that you've added.
"Pure Python applications, such as headless processes and evented web frameworks like Twisted, are fully supported on Cedar."
Reference: https://devcenter.heroku.com/articles/python-support