I am learning how to use Twisted AMP. I am developing a program that sends data from a client to a server and inserts the data in a SQLite3 DB. The server then sends back a result to the client which indicates success or error (try and except might not be the best way to do this but it is only a temporary solution while I work out the main problem). In order to do this I modified an example I found that originally did a sum and returned the result, so I realize that this might not be the most efficient way to do what I am trying to do. In particular I am trying to do some timings on multiple insertions (i.e. send the data to the server multiple times for multiple insertions) and I have included the code I have written. It works but clearly it is not a good way to send multiple data for insertion since I am performing multiple connections before running the reactor.
I have tried several ways to get around this including passing the ClientCreator to reactor.callWhenRunning() but you cannot do this with a deferred.
Any suggestions, advice or help with how to do this would be much appreciated. Here is the code.
Server:
from twisted.protocols import amp
from twisted.internet import reactor
from twisted.internet.protocol import Factory
import sqlite3, time
class Insert(amp.Command):
arguments = [('data', amp.Integer())]
response = [('insert_result', amp.Integer())]
class Protocol(amp.AMP):
def __init__(self):
self.conn = sqlite3.connect('biomed1.db')
self.c =self.conn.cursor()
self.res=None
#Insert.responder
def dbInsert(self, data):
self.InsertDB(data) #call the DB inserter
result=self.res # send back the result of the insertion
return {'insert_result': result}
def InsertDB(self,data):
tm=time.time()
print "insert time:",tm
chx=data
PID=2
device_ID=5
try:
self.c.execute("INSERT INTO btdata4(co2_data, patient_Id, sensor_Id) VALUES ('%s','%s','%s')" % (chx, PID, device_ID))
except Exception, err:
print err
self.res=0
else:
self.res=1
self.conn.commit()
pf = Factory()
pf.protocol = Protocol
reactor.listenTCP(1234, pf)
reactor.run()
Client:
from twisted.internet import reactor
from twisted.internet.protocol import ClientCreator
from twisted.protocols import amp
import time
class Insert(amp.Command):
arguments = [('data', amp.Integer())]
response = [('insert_result', amp.Integer())]
def connected(protocol):
return protocol.callRemote(Insert, data=5555).addCallback(gotResult)
def gotResult(result):
print 'insert_result:', result['insert_result']
tm=time.time()
print "stop", tm
def error(reason):
print "error", reason
tm=time.time()
print "start",tm
for i in range (10): #send data over ten times
ClientCreator(reactor, amp.AMP).connectTCP(
'127.0.0.1', 1234).addCallback(connected).addErrback(error)
reactor.run()
End of Code.
Thank you.
Few things which will improve your Server code.
First and foremost: The use of direct database access functions is discouraged in twisted, as they normally causes block. Twisted has nice abstraction for database access which provides twisted approach to db connection - twisted.adbapi
Now on to reuse of db connection: If you want to reuse certain assets (like database connection) across a number of Protocol instances, you should initialize those in constructor of Factory or if you dont fancy initiating such things at a launch time, create an resource access method, which will initiate resource upon first method call then assign it to class variable and return that on subsequent calls.
When Factory creates a specific Protocol intance, it will add a reference to itself inside the protocol, see line 97 of twisted.internet.protocol
Then within your Protocol instance, you can access shared database connection instance like:
self.factory.whatever_name_for_db_connection.doSomething()
Reworked Server code (I dont have python, twisted or even decent IDE available, so this is pretty much untested, some errors are to be expected)
from twisted.protocols import amp
from twisted.internet import reactor
from twisted.internet.protocol import Factory
import time
class AMPDBAccessProtocolFactory(Factory):
def getDBConnection(self):
if 'dbConnection' in dir(self):
return self.dbConnection
else:
self.dbConnection = SQLLiteTestConnection(self.dbURL)
return self.dbConnection
class SQLLiteTestConnection(object):
"""
Provides abstraction for database access and some business functions.
"""
def __init__(self,dbURL):
self.dbPool = adbapi.ConnectionPool("sqlite3" , dbURL, check_same_thread=False)
def insertBTData4(self,data):
query = "INSERT INTO btdata4(co2_data, patient_Id, sensor_Id) VALUES (%s,%s,%s)"
tm=time.time()
print "insert time:",tm
chx=data
PID=2
device_ID=5
dF = self.dbPool.runQuery(query,(chx, PID, device_ID))
dF.addCallback(self.onQuerySuccess,insert_data=data)
return dF
def onQuerySuccess(self,insert_data,*r):
"""
Here you can inspect query results or add any other valuable information to be parsed at client.
For the test sake we will just return True to a customer if query was a success.
original data available at kw argument insert_data
"""
return True
class Insert(amp.Command):
arguments = [('data', amp.Integer())]
response = [('insert_result', amp.Integer())]
class MyAMPProtocol(amp.AMP):
#Insert.responder
def dbInsert(self, data):
db = self.factory.getDBConnection()
dF = db.insertBTData4(data)
dF.addErrback(self.onInsertError,data)
return dF
def onInsertError(self, error, data):
"""
Here you could do some additional error checking or inspect data
which was handed for insert here. For now we will just throw the same exception again
so that the client gets notified
"""
raise error
if __name__=='__main__':
pf = AMPDBAccessProtocolFactory()
pf.protocol = MyAMPProtocol
pf.dbURL='biomed1.db'
reactor.listenTCP(1234, pf)
reactor.run()
Now on to the client. IF AMP follows the overall RPC logic (cant test it currently) it should be able to peruse the same connection across a number of calls. So I have created a ServerProxy class which will hold that perusable protocol instance and provide abstraction for calls:
from twisted.internet import reactor
from twisted.internet.protocol import ClientCreator
from twisted.protocols import amp
import time
class Insert(amp.Command):
arguments = [('data', amp.Integer())]
response = [('insert_result', amp.Integer())]
class ServerProxy(object):
def connected(self,protocol):
self.serverProxy = protocol # assign protocol as instance variable
reactor.callLater(5,self.startMultipleInsert) #after five seconds start multiple insert procedure
def remote_insert(self,data):
return self.serverProxy.callRemote(Insert, data)
def startMultipleInsert(self):
for i in range (10): #send data over ten times
dF = self.remote_insert(i)
dF.addCallback(self.gotInsertResult)
dF.addErrback(error)
def gotInsertResult(self,result):
print 'insert_result:', str(result)
tm=time.time()
print "stop", tm
def error(reason):
print "error", reason
def main():
tm=time.time()
print "start",tm
serverProxy = ServerProxy()
ClientCreator(reactor, amp.AMP).connectTCP('127.0.0.1', 1234).addCallback(serverProxy.connected).addErrback(error)
reactor.run()
if __name__=='__main__':
main()
Related
Is there a straightforward way to asynchronously chain GRPC calls in Python?
This feels like the kind of things that "should" be feasible, but I can't seem to find it.
Here's a rough idea of what I feel I should be able to do:
class MyServer(my_grpc.MyServicer):
def __init__(self, child_stub):
self.child_stub_ = child_stub
def MyMethod(self, request, context):
child_result = self.child_stub_.ChildMethod.future(my_grpc.ChildMethodParams())
child_result.add_done_callback(something_that_completes_MyMethod)
return presumably_something
Is there something I'm missing here? It feels like this would be a common use-case, yet I can't seem to find anything related to it in the docs.
Edit: I believe you're trying to send two response for one request on a unary request/response setup, which I don't believe to be possible. Instead you should do a unary request and streaming responses, which will allow for many responses.
client
import grpc
import test_pb2_grpc as pb_grpc
import test_pb2 as pb2
def test():
channel = grpc.insecure_channel('localhost:50051')
stub = pb_grpc.TestStub(channel=channel)
for response in stub.Produce(pb2.Empty()):
print(response.result)
if __name__ == '__main__':
test()
Server
import test_pb2_grpc as pb_grpc
import test_pb2 as pb2
import time
import grpc
from concurrent import futures
class test_servcie(pb_grpc.TestServicer):
def Produce(self, request, context):
my_method_results = [50, 200]
for result in my_method_results:
yield pb2.Resp(result=result)
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
pb_grpc.add_TestServicer_to_server(test_servcie(), server)
server.add_insecure_port('[::]:50051')
print("service started")
server.start()
try:
while True:
time.sleep(3600)
except KeyboardInterrupt:
server.stop(0)
if __name__ == '__main__':
serve()
proto
syntax = "proto3";
package api;
service Test {
rpc Produce (Empty) returns (stream Resp);
}
message Empty {}
message Resp{
int32 result = 1;
}
App Description
So I'm trying to create an application that does real-time sentiment analysis on tweets(as close to real time as I'm able to get it) and these tweets have to be based on user input. So in the main page of my application, I have a simple search bar where the user can enter a topic they would like to perform sentiment analysis on and when they press enter, it would take them to another page where they see a line chart displaying all the data in real time.
Problem 1
The first problem I'm facing at the moment is that I don't know how I can get tweepy to change what it is tracking when two or more people make a request. If I were to have global streaming that I simply disconnect and reconnect every time the user makes a new query, then it is also going to disconnect for other users as well which I don't want. On the other hand, if I were to allocate a streaming object for each user that connects, then this strategy should work. This still poses a problem. Twitter does not allow you to hold more than one connection at a time it seems given this StackOverflow post.
Does Tweepy support running multiple Streams to collect data?
If I still were to go along with this, I risk getting my IP banned. So both of these solutions are no good.
Problem 2
The last problem I'm having is figuring out who the message belongs to. At the moment, I'm using RabbitMQ to store all incoming messages in one single queue called twitter_topic_feed. For every tweet that I receive from tweepy, I publish it in that queue. Then RabbiMQ consumes the message and sends it to every available connection. Obviously, that behaviour is not what I'm looking for. Consider two users who search for pizza and sports. Both users will receive tweets pertaining to football and pizza when one user asked for sports tweets and the other asked for pizza tweets.
One idea is to create a queue with a unique identifier for each available connection. The identifier would have the form {Search Term}_{Hash ID}.
For generating the hash ID, I can use the UUID package that is available in python and create the ID when the connection opens and delete it when it closes. Of course, when they close the connection I also need to delete the queue. I'm not sure how well this solution would scale. If we were to have 10,000 connections, we would have 10,000 queues and each queue could potentially have a lot of messages stored in it. Seems like it would be very memory intensive.
Design
tornado Framework for WebSockets,
tweepy API for streaming tweets
RabbitMQ For publishing messages to the queue whenever tweepy receives a new tweet. RabbitMQ will then consume that message and send it to the WebSocket.
Attempt(What I currently have so far)
TweetStreamListener uses the tweepy API to listen for tweets based on the user's input. Whatever tweet it gets, it calculates the polarity of that tweet and publishes it to rabbitMQ twitter_topic_feed queue.
import logging
from tweepy import StreamListener, OAuthHandler, Stream, API
from sentiment_analyzer import calculate_polarity_score
from constants import SETTINGS
auth = OAuthHandler(
SETTINGS["TWITTER_CONSUMER_API_KEY"], SETTINGS["TWITTER_CONSUMER_API_SECRET_KEY"])
auth.set_access_token(
SETTINGS["TWITTER_ACCESS_KEY"], SETTINGS["TWITTER_ACCESS_SECRET_KEY"])
api = API(auth, wait_on_rate_limit=True)
class TweetStreamListener(StreamListener):
def __init__(self):
self.api = api
self.stream = Stream(auth=self.api.auth, listener=self)
def start_listening(self):
pass
def on_status(self, status):
if not hasattr(status, 'retweeted_status'):
polarity = calculate_polarity_score(status.text)
message = {
'polarity': polarity,
'timestamp': status.created_at
}
# TODO(Luis) Need to figure who to send this message to.
logging.debug("Message received from Twitter: {0}".format(message))
# limit handling
def on_limit(self, status):
logging.info(
'Limit threshold exceeded. Status code: {0}'.format(status))
def on_timeout(self, status):
logging.error('Stream disconnected. continuing...')
return True # Don't kill the stream
"""
Summary: Callback that executes for any error that may occur. Whenever we get a 420 Error code, we simply
stop streaming tweets as we have reached our rate limit. This is due to making too many requests.
Returns: False if we are sending too many tweets, otherwise return true to keep the stream going.
"""
def on_error(self, status_code):
if status_code == 420:
logging.error(
'Encountered error code 420. Disconnecting the stream')
# returning False in on_data disconnects the stream
return False
else:
logging.error('Encountered error with status code: {}'.format(
status_code))
return True # Don't kill the stream
WS_Handler is in charge of maintaining a list of open connections and sending any message that it receives back to every client(This behaviour is something I don't want).
import logging
import json
from uuid import uuid4
from tornado.web import RequestHandler
from tornado.websocket import WebSocketHandler
class WSHandler(WebSocketHandler):
def check_origin(self, origin):
return True
#property
def sess_id(self):
return self._sess_id
def open(self):
self._sess_id = uuid4().hex
logging.debug('Connection established.')
self.application.pc.register_websocket(self._sess_id, self)
# When messages arrives via RabbitMQ, write it to websocket
def on_message(self, message):
logging.debug('Message received: {0}'.format(message))
self.application.pc.redirect_incoming_message(
self._sess_id, json.dumps(message))
def on_close(self):
logging.debug('Connection closed.')
self.application.pc.unregister_websocket(self._sess_id)
The PikaClient module contains the PikaClient that will allows to keep track of inbound and outbound channels as well as keeping track of the websockets that currently running.
import logging
import pika
from constants import SETTINGS
from pika import PlainCredentials, ConnectionParameters
from pika.adapters.tornado_connection import TornadoConnection
pika.log = logging.getLogger(__name__)
class PikaClient(object):
INPUT_QUEUE_NAME = 'in_queue'
def __init__(self):
self.connected = False
self.connecting = False
self.connection = None
self.in_channel = None
self.out_channels = {}
self.websockets = {}
def connect(self):
if self.connecting:
return
self.connecting = True
# Setup rabbitMQ connection
credentials = PlainCredentials(
SETTINGS['RABBITMQ_USERNAME'], SETTINGS['RABBITMQ_PASSWORD'])
param = ConnectionParameters(
host=SETTINGS['RABBITMQ_HOST'], port=SETTINGS['RABBITMQ_PORT'], virtual_host='/', credentials=credentials)
return TornadoConnection(param, on_open_callback=self.on_connected)
def run(self):
self.connection = self.connect()
self.connection.ioloop.start()
def stop(self):
self.connected = False
self.connecting = False
self.connection.ioloop.stop()
def on_connected(self, unused_Connection):
self.connected = True
self.in_channel = self.connection.channel(self.on_conn_open)
def on_conn_open(self, channel):
self.in_channel.exchange_declare(
exchange='tornado_input', exchange_type='topic')
channel.queue_declare(
callback=self.on_input_queue_declare, queue=self.INPUT_QUEUE_NAME)
def on_input_queue_declare(self, queue):
self.in_channel.queue_bind(
callback=None, exchange='tornado_input', queue=self.INPUT_QUEUE_NAME, routing_key="#")
def register_websocket(self, sess_id, ws):
self.websockets[sess_id] = ws
self.create_out_channel(sess_id)
def unregister_websocket(self, sess_id):
self.websockets.pop(sess_id)
if sess_id in self.out_channels:
self.out_channels[sess_id].close()
def create_out_channel(self, sess_id):
def on_output_channel_creation(channel):
def on_output_queue_declaration(queue):
channel.basic_consume(self.on_message, queue=sess_id)
self.out_channels[sess_id] = channel
channel.queue_declare(callback=on_output_queue_declaration,
queue=sess_id, auto_delete=True, exclusive=True)
self.connection.channel(on_output_channel_creation)
def redirect_incoming_message(self, sess_id, message):
self.in_channel.basic_publish(
exchange='tornado_input', routing_key=sess_id, body=message)
def on_message(self, channel, method, header, body):
sess_id = method.routing_key
if sess_id in self.websockets:
self.websockets[sess_id].write_message(body)
channel.basic_ack(delivery_tag=method.delivery_tag)
else:
channel.basic_reject(delivery_tag=method.delivery_tag)
Server.py is the main entry point of the application.
import logging
import os
from tornado import web, ioloop
from tornado.options import define, options, parse_command_line
from client import PikaClient
from handlers import WSHandler, MainHandler
define("port", default=3000, help="run on the given port.", type=int)
define("debug", default=True, help="run in debug mode.", type=bool)
def main():
parse_command_line()
settings = {
"debug": options.debug,
"static_path": os.path.join(os.path.dirname(__file__), "web/static")
}
app = web.Application(
[
(r"/", MainHandler),
(r"/stream", WSHandler),
],
**settings
)
# Setup PikaClient
app.pc = PikaClient()
app.listen(options.port)
logging.info("Server running on http://localhost:3000")
try:
app.pc.run()
except KeyboardInterrupt:
app.pc.stop()
if __name__ == "__main__":
main()
I have written a script that gets the account value from my stock trading, which looks like this (working):
from ib.opt import ibConnection, message
def account_summary_handler(msg):
print(msg.tag, msg.value, msg.currency)
connection.cancelAccountSummary(1)
connection.disconnect()
connection = ibConnection(port=7497, clientId=100)
connection.register(account_summary_handler, 'AccountSummary')
connection.connect()
connection.reqAccountSummary(1, 'All', 'NetLiquidation')
It's a realtime API, so once the connection is open, updates are streaming, and my_account_handler gets called on every update.
I want to get the account balance from another synchronous script, something like, called something like:
myaccount.account_summary()
I've written something that looks like this (untested):
class IBAccount(object):
def __init__(self):
self.clientId=100
self.port=7497
def connect(self):
self.connection = ibConnection(port=self.port, clientId=self.clientId)
def account_summary(self):
self.connection.register(self.account_summary_handler, 'AccountSummary')
connection.connect()
connection.reqAccountSummary(1, 'All', 'NetLiquidation')
def account_summary_handler(self, msg):
self.connection.cancelAccountSummary(1)
self.connection.disconnect()
return msg.value
I believe I need account_summary() to block, and return the actual account value, rather than have it come back through a different function.
My question is:
How do I get account_summary() to return the account value?
In addition, if I'm using an inappropriate design pattern for this code and shouldn't use a class, please advise.
The solution was to make it stated:
from ib.opt import ibConnection, message
class IBstate(object):
def __init__(self):
self.clientId=100
self.port=7496
self.netLiquidation = None
self.connection = ibConnection(port=self.port, clientId=self.clientId)
self.connection.connect()
self.connection.register(self.account_summary_handler, 'AccountSummary')
self.connection.reqAccountSummary(1, 'All', 'NetLiquidation')
def account_summary_handler(self, msg):
self.netLiquidation = msg
This runs in the background and updates the variables on the object as they change on the remote end.
Hello I am working on develop a rpc server based on twisted to serve several microcontrollers which make rpc call to twisted jsonrpc server. But the application also required that server send information to each micro at any time, so the question is how could be a good practice to prevent that the response from a remote jsonrpc call from a micro be confused with a server jsonrpc request which is made for a user.
The consequence that I am having now is that micros are receiving bad information, because they dont know if netstring/json string that is comming from socket is their response from a previous requirement or is a new request from server.
Here is my code:
from twisted.internet import reactor
from txjsonrpc.netstring import jsonrpc
import weakref
creds = {'user1':'pass1','user2':'pass2','user3':'pass3'}
class arduinoRPC(jsonrpc.JSONRPC):
def connectionMade(self):
pass
def jsonrpc_identify(self,username,password,mac):
""" Each client must be authenticated just after to be connected calling this rpc """
if creds.has_key(username):
if creds[username] == password:
authenticated = True
else:
authenticated = False
else:
authenticated = False
if authenticated:
self.factory.clients.append(self)
self.factory.references[mac] = weakref.ref(self)
return {'results':'Authenticated as %s'%username,'error':None}
else:
self.transport.loseConnection()
def jsonrpc_sync_acq(self,data,f):
"""Save into django table data acquired from sensors and send ack to gateway"""
if not (self in self.factory.clients):
self.transport.loseConnection()
print f
return {'results':'synced %s records'%len(data),'error':'null'}
def connectionLost(self, reason):
""" mac address is searched and all reference to self.factory.clientes are erased """
for mac in self.factory.references.keys():
if self.factory.references[mac]() == self:
print 'Connection closed - Mac address: %s'%mac
del self.factory.references[mac]
self.factory.clients.remove(self)
class rpcfactory(jsonrpc.RPCFactory):
protocol = arduinoRPC
def __init__(self, maxLength=1024):
self.maxLength = maxLength
self.subHandlers = {}
self.clients = []
self.references = {}
""" Asynchronous remote calling to micros, simulating random calling from server """
import threading,time,random,netstring,json
class asyncGatewayCalls(threading.Thread):
def __init__(self,rpcfactory):
threading.Thread.__init__(self)
self.rpcfactory = rpcfactory
"""identifiers of each micro/client connected"""
self.remoteMacList = ['12:23:23:23:23:23:23','167:67:67:67:67:67:67','90:90:90:90:90:90:90']
def run(self):
while True:
time.sleep(10)
while True:
""" call to any of three potential micros connected """
mac = self.remoteMacList[random.randrange(0,len(self.remoteMacList))]
if self.rpcfactory.references.has_key(mac):
print 'Calling %s'%mac
proto = self.rpcfactory.references[mac]()
""" requesting echo from selected micro"""
dataToSend = netstring.encode(json.dumps({'method':'echo_from_micro','params':['plop']}))
proto.transport.write(dataToSend)
break
factory = rpcfactory(arduinoRPC)
"""start thread caller"""
r=asyncGatewayCalls(factory)
r.start()
reactor.listenTCP(7080, factory)
print "Micros remote RPC server started"
reactor.run()
You need to add a enough information to each message so that the recipient can determine how to interpret it. Your requirements sounds very similar to those of AMP, so you could either use AMP instead or use the same structure as AMP to identify your messages. Specifically:
In requests, put a particular key - for example, AMP uses "_ask" to identify requests. It also gives these a unique value, which further identifies that request for the lifetime of the connection.
In responses, put a different key - for example, AMP uses "_answer" for this. The value matches up with the value from the "_ask" key in the request the response is for.
Using an approach like this, you just have to look to see whether there is an "_ask" key or an "_answer" key to determine if you've received a new request or a response to a previous request.
On a separate topic, your asyncGatewayCalls class shouldn't be thread-based. There's no apparent reason for it to use threads, and by doing so it is also misusing Twisted APIs in a way which will lead to undefined behavior. Most Twisted APIs can only be used in the thread in which you called reactor.run. The only exception is reactor.callFromThread, which you can use to send a message to the reactor thread from any other thread. asyncGatewayCalls tries to write to a transport, though, which will lead to buffer corruption or arbitrary delays in the data being sent, or perhaps worse things. Instead, you can write asyncGatewayCalls like this:
from twisted.internet.task import LoopingCall
class asyncGatewayCalls(object):
def __init__(self, rpcfactory):
self.rpcfactory = rpcfactory
self.remoteMacList = [...]
def run():
self._call = LoopingCall(self._pokeMicro)
return self._call.start(10)
def _pokeMicro(self):
while True:
mac = self.remoteMacList[...]
if mac in self.rpcfactory.references:
proto = ...
dataToSend = ...
proto.transport.write(dataToSend)
break
factory = ...
r = asyncGatewayCalls(factory)
r.run()
reactor.listenTCP(7080, factory)
reactor.run()
This gives you a single-threaded solution which should have the same behavior as you intended for the original asyncGatewayCalls class. Instead of sleeping in a loop in a thread in order to schedule the calls, though, it uses the reactor's scheduling APIs (via the higher-level LoopingCall class, which schedules things to be called repeatedly) to make sure _pokeMicro gets called every ten seconds.
I'm trying to write a dead-simple interface for an IRC library, like so:
import simpleirc
connection = simpleirc.Connect('irc.freenode.net', 6667)
channel = connection.join('foo')
find_command = re.compile(r'google ([a-z]+)').findall
for msg in channel:
for t in find_command(msg):
channel.say("http://google.com/search?q=%s" % t)
Working from their example, I'm running into trouble (code is a bit lengthy, so I pasted it here). Since the call to channel.__next__ needs to be returned when the callback <IRCClient instance>.privmsg is called, there doesn't seem to be a clean option. Using exceptions or threads seems like the wrong thing here, is there a simpler (blocking?) way of using twisted that would make this possible?
In general, if you're trying to use Twisted in a "blocking" way, you're going to run into a lot of difficulties, because that's neither the way it's intended to be used, nor the way in which most people use it.
Going with the flow is generally a lot easier, and in this case, that means embracing callbacks. The callback-style solution to your question would look something like this:
import re
from twisted.internet import reactor, protocol
from twisted.words.protocols import irc
find_command = re.compile(r'google ([a-z]+)').findall
class Googler(irc.IRCClient):
def privmsg(self, user, channel, message):
for text in find_command(message):
self.say(channel, "http://google.com/search?q=%s" % (text,))
def connect():
cc = protocol.ClientCreator(reactor, Googler)
return cc.connectTCP(host, port)
def run(proto):
proto.join(channel)
def main():
d = connect()
d.addCallback(run)
reactor.run()
This isn't absolutely required (but I strongly suggest you consider trying it). One alternative is inlineCallbacks:
import re
from twisted.internet import reactor, protocol, defer
from twisted.words.protocols import irc
find_command = re.compile(r'google ([a-z]+)').findall
class Googler(irc.IRCClient):
def privmsg(self, user, channel, message):
for text in find_command(message):
self.say(channel, "http://google.com/search?q=%s" % (text,))
#defer.inlineCallbacks
def run():
cc = protocol.ClientCreator(reactor, Googler)
proto = yield cc.connectTCP(host, port)
proto.join(channel)
def main():
run()
reactor.run()
Notice no more addCallbacks. It's been replaced by yield in a decorated generator function. This could get even closer to what you asked for if you had a version of Googler with a different API (the one above should work with IRCClient from Twisted as it is written - though I didn't test it). It would be entirely possible for Googler.join to return a Channel object of some sort, and for that Channel object to be iterable like this:
#defer.inlineCallbacks
def run():
cc = protocol.ClientCreator(reactor, Googler)
proto = yield cc.connectTCP(host, port)
channel = proto.join(channel)
for msg in channel:
msg = yield msg
for text in find_command(msg):
channel.say("http://google.com/search?q=%s" % (text,))
It's only a matter of implementing this API on top of the ones already present. Of course, the yield expressions are still there, and I don't know how much this will upset you. ;)
It's possible to go still further away from callbacks and make the context switches necessary for asynchronous operation to work completely invisible. This is bad for the same reason it would be bad for sidewalks outside your house to be littered with invisible bear traps. However, it's possible. Using something like corotwine, itself based on a third-party coroutine library for CPython, you can have the implementation of Channel do the context switching itself, rather than requiring the calling application code to do it. The result might look something like:
from corotwine import protocol
def run():
proto = Googler()
transport = protocol.gConnectTCP(host, port)
proto.makeConnection(transport)
channel = proto.join(channel)
for msg in channel:
for text in find_command(msg):
channel.say("http://google.com/search?q=%s" % (text,))
with an implementation of Channel that might look something like:
from corotwine import defer
class Channel(object):
def __init__(self, ircClient, name):
self.ircClient = ircClient
self.name = name
def __iter__(self):
while True:
d = self.ircClient.getNextMessage(self.name)
message = defer.blockOn(d)
yield message
This in turn depends on a new Googler method, getNextMessage, which is a straightforward feature addition based on existing IRCClient callbacks:
from twisted.internet import defer
class Googler(irc.IRCClient):
def connectionMade(self):
irc.IRCClient.connectionMade(self)
self._nextMessages = {}
def getNextMessage(self, channel):
if channel not in self._nextMessages:
self._nextMessages[channel] = defer.DeferredQueue()
return self._nextMessages[channel].get()
def privmsg(self, user, channel, message):
if channel not in self._nextMessages:
self._nextMessages[channel] = defer.DeferredQueue()
self._nextMessages[channel].put(message)
To run this, you create a new greenlet for the run function and switch to it, and then start the reactor.
from greenlet import greenlet
def main():
greenlet(run).switch()
reactor.run()
When run gets to its first asynchronous operation, it switches back to the reactor greenlet (which is the "main" greenlet in this case, but it doesn't really matter) to let the asynchronous operation complete. When it completes, corotwine turns the callback into a greenlet switch back into run. So run is granted the illusion of running straight through, like a "normal" synchronous program. Keep in mind that it is just an illusion, though.
So, it's possible to get as far away from the callback-oriented style that is most commonly used with Twisted as you want. It's not necessarily a good idea, though.