I've made an IRC bot in Python and I've been trying to figure out a way to wait for an IRC command and return the message to a calling function for a while now. I refuse to use an external library for various reasons including I'm trying to learn to make these things from scratch. Also, I've been sifting through documentation for existing ones and they're way too comprehensive. I'me trying to make a simple one.
For example:
def who(bot, nick):
bot.send('WHO %s' % nick)
response = ResponseWaiter('352') # 352 - RPL_WHOREPLY
return response.msg
Would return a an object of my Message class that parses IRC messages to the calling function:
def check_host(bot, nick, host):
who_info = who(bot, nick)
if who_info.host == host:
return True
return False
I have looked at the reactor pattern, observer pattern, and have tried implementing a hundred different event system designs for this to no avail. I'm completely lost.
Please either provide a solution or point me in the right direction. There's got to be a simple way to do this.
So what I've done is use grab messages from my generator (a bot method) from the bot's who method. The generator looks like this:
def msg_generator(self):
''' Provides messages until bot dies '''
while self.alive:
for msg in self.irc.recv(self.buffer).split(('\r\n').encode()):
if len(msg) > 3:
try: yield Message(msg.decode())
except Exception as e:
self.log('%s %s\n' % (except_str, str(e)))
And now the bot's who method looks like this:
def who(self, nick):
self.send('WHO %s' % nick)
for msg in self.msg_generator():
if msg.command == '352':
return msg
However, it's now taking control of the messages, so I need some way of relinquishing the messages I'm not using for the who method to their appropriate handlers.
My bot generally handles all messages with this:
def handle(self):
for msg in self.msg_generator():
self.log('◀ %s' % (msg))
SpaghettiHandler(self, msg)
So any message that my SpaghettiHandler would be handling is not handled while the bot's who method uses the generator to receive messages.
It's working.. and works fast enough that it's hard to lose a message. But if my bot were to be taking many commands at the same time, this could become a problem. I'm pretty sure I'll find a solution in this direction, but I didn't create this as the answer because I'm not sure it's a good way, even when I have it set to relinquish messages that don't pertain to the listener.
Related
I wanted to make a Telegram "quick response" function for my application, so that I would be able to call, for example
answer = get_answer(question, user_id) and the bot will send the question to the person and return their answer. I know that this is done using the next_step_handler in PyTelegramBotAPI, but I don't have and don't want to have bot.polling() enabled, because I want the bot to be active only when the function is called. Is there a way to do that?
I was thinking of something maybe like that, but this obviously doesn't work, sp consider it a pseudo code:
def get_answer(question, user_id):
msg = bot.send_message(user_id, question)
bot.polling(none_stop=True, interval=0)
return bot.register_next_step_handler(msg, __handle_answer__)
def __handle_answer__(msg):
bot.stop_polling()
return msg.text
Just to be clear, I don't mind having polling enabled, as long as it is disabled after the person answers the question.
I'm trying to convert a synchronous flow in Python code which is based on callbacks to an A-syncronious flow using asyncio.
Basically the code interacts a lot with TCP/UNIX sockets. It reads data from the sockets, manipulates it to make decisions and writes stuff back to the other side. This is going on over multiple sockets at once and data is shared between the contexts to make decisions sometimes.
EDIT :: The code currently is mostly based on registering a callback to a central entity for a specific socket, and having that entity run the callback when the relevant socket is readable (something like "call this function when that socket has data to be read"). Once the callback is called - a bunch of stuff happens, and eventually a new callback is registered for when new data is available. The central entity runs a select over all sockets registered to figure out which callbacks should be called.
I'm trying to do this without refactoring my entire code and making this as seamless as possible to the programmer - so I was trying to think about it like so - all code should run the same way as it does today - but whenever the current code does a socket.recv() to get new data - the process would yield execution to other tasks. When the read returns, it should go back to handling the data from the same point using the new data it got.
To do this, I wrote a new class called AsyncSocket - which interacts with the IO streams of asyncIO and placed the Async/await statements almost solely in there - thinking that I would implement the recv method in my class to make it look like a "regular IO socket" to the rest of my code.
So far - this is my understanding of what A-sync programming should allow.
Now to the problem :
My code awaits for clients to connect - when it does, each client's context is allowed to read and write from it's own connection.
I've simplified to flow to the following to clarify the problem:
class AsyncSocket():
def __init__(self,reader,writer):
self.reader = reader
self.writer = writer
def recv(self,numBytes):
print("called recv!")
data = self.read_mitigator(numBytes)
return data
async def read_mitigator(self,numBytes):
print("Awaiting of AsyncSocket.reader.read")
data = await self.reader.read(numBytes)
print("Done Awaiting of AsyncSocket.reader.read data is %s " % data)
return data
def mit2(aSock):
return mit3(aSock)
def mit3(aSock):
return aSock.recv(100)
async def echo_server(reader, writer):
print ("New Connection!")
aSock = AsyncSocket(reader,writer) # create a new A-sync socket class and pass it on the to regular code
while True:
data = await some_func(aSock) # this would eventually read from the socket
print ("Data read is %s" % (data))
if not data:
break
writer.write(data) # echo everything back
async def main(host, port):
server = await asyncio.start_server(echo_server, host, port)
await server.serve_forever()
asyncio.run(main('127.0.0.1', 5000))
mit2() and mit3() are synchronous functions that do stuff with the data on the way back before returning to the main client's loop - but here I'm just using them as empty functions.
The problem starts when I play with the implementation of some_func().
A pass through implementation (edit: kind-of-works) - but still has issues :
def some_func(aSock):
try:
return (mit2(aSock)) # works
except:
print("Error!!!!")
While an implementation which reads the data and does something with it - like adding a suffix before returning, throws an error:
def some_func(aSock):
try:
return (mit2(aSock) + "something") # doesn't work
except:
print("Error!!!!")
The error (as far as I understand it) means it's not really doing what it should:
New Connection!
called recv!
/Users/user/scripts/asyncServer.py:36: RuntimeWarning: coroutine 'AsyncSocket.read_mitigator' was never awaited
return (mit2(aSock) + "something") # doesn't work
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Error!!!!
Data read is None
And the echo server obviously doesn't work.
Obviously my code looks more like option #2 with a lot more stuff in some_func(),mit2() and mit3() - but I can't get this to work. I'm fairly new in using asyncio/async/await - so what (rather basic concept I guess) am I missing?
This code won't work as envisioned:
def recv(self,numBytes):
print("called recv!")
data = self.read_mitigator(numBytes)
return data
async def read_mitigator(self,numBytes):
...
You cannot call an async function from a sync function and get the result, you must await it, which ensures that you return to the event loop in case the data is not yet ready. This mismatch between async and sync code is sometimes referred to as the issue of function color.
Since your code is already using non-blocking sockets and an event loop, a good approach to porting it to asyncio might be to first switch to the asyncio event loop. You can use event loop methods like sock_recv to request data:
def start():
loop = asyncio.get_event_loop()
sock = make_socket() # make sure it's non-blocking
future_data = loop.sock_recv(sock, 1024)
future_data.add_done_callback(continue_read)
# return to the event loop - when some data is ready
# continue_read will be invoked
def continue_read(future):
data = future.result()
print('got', data)
# ... do something with data, e.g. process it
# and call sock_sendall with the response
asyncio.get_event_loop().call_soon(start())
asyncio.get_event_loop().run_forever()
Once you have the program working in that mode, you can start moving to coroutines, which allow the code to look like sync code, but work in exactly the same way:
async def start():
loop = asyncio.get_event_loop()
sock = make_socket() # make sure it's non-blocking
data = await loop.sock_recv(sock, 1024)
# data is available "immediately", meaning the coroutine gets
# automatically suspended when awaiting data that is not yet
# ready, and automatically re-scheduled when the data is ready
print('got', data)
asyncio.run(start())
The next step can be eliminating make_socket and switching to asyncio streams.
I'm trying to convert a simple syncronous server to an asyncronous version, the server receives post requestes and it retrieves the response from an external web service (amazon sqs). Here's the syncronous code
def post(self):
zoom_level = self.get_argument('zoom_level')
neLat = self.get_argument('neLat')
neLon = self.get_argument('neLon')
swLat = self.get_argument('swLat')
swLon = self.get_argument('swLon')
data = self._create_request_message(zoom_level, neLat, neLon, swLat, swLon)
self._send_parking_spots_request(data)
#....other stuff
def _send_parking_spots_request(self, data):
msg = Message()
msg.set_body(json.dumps(data))
self._sqs_send_queue.write(msg)
Reading Tornado documentation and some threads here I ended with this code using coroutines:
def post(self):
zoom_level = self.get_argument('zoom_level')
neLat = self.get_argument('neLat')
neLon = self.get_argument('neLon')
swLat = self.get_argument('swLat')
swLon = self.get_argument('swLon')
data = self._create_request_message(zoom_level, neLat, neLon, swLat, swLon)
self._send_parking_spots_request(data)
self.finish()
#gen.coroutine
def _send_parking_spots_request(self, data):
msg = Message()
msg.set_body(json.dumps(data))
yield gen.Task(write_msg, self._sqs_send_queue, msg)
def write_msg(queue, msg, callback=None):
queue.write(msg)
Comparing the performances using siege I get that the second version is even worse than the original one, so probably there's something about coroutines and Torndado asyncronous programming that I didn't understand at all.
Could you please help me with this?
Edit: self._sqs_send_queue it's a queue object retrieved from boto interface and queue.write(msg) returns the message that has been written on the queue
tornado relies on you converting all your I/O to be non-blocking. Simply sticking the same code you were using before inside of a gen.Task will not improve performance at all, because the I/O itself is still going to block the event loop. Additionally, you need to make your post method a coroutine, and call _send_parking_spots_requests using yield for the code to behave properly. So, a "correct" solution would look something like this:
#gen.coroutine
def post(self):
...
yield self._send_parking_spots_request(data) # wait (without blocking the event loop) until the method is done
self.finish()
#gen.coroutine
def _send_parking_spots_request(self, data):
msg = Message()
msg.set_body(json.dumps(data))
yield gen.Task(write_msg, self._sqs_send_queue, msg)
def write_msg(queue, msg, callback=None):
yield queue.write(msg, callback=callback) # This has to do non-blocking I/O.
In this example, queue.write would need to be some API that sends your request using non-blocking I/O, and executes callback when a response is received. Without knowing exactly what queue in your original example is, I can't specify exactly how that can be implemented in your case.
Edit: Assuming you're using boto, you may want to check out bototornado, which implements the exact same API I described above:
def write(self, message, callback=None):
"""
Add a single message to the queue.
:type message: Message
:param message: The message to be written to the queue
:rtype: :class:`boto.sqs.message.Message`
:return: The :class:`boto.sqs.message.Message` object that was written.
PyOSC very courteously handles all kinds of exceptions, but unfortunately makes it difficult to debug. How do I defeat this?
For instance, I have a coding bug, which it reports as:
OSCServer: NameError on request from localhost:50542: global name 'cell' is not defined
Normally in python, this would throw up an exception with a helpful trace. But in PyOSC, this is all I get.
How do I turn off PyOSC's exception handling?
UPDATE: After a month, this question received zero comments or answers. Guess there aren't that many people using Python, open sound control, and the PyOSC module. I finally answered my own question below.
Okay, I figured it out. Since PyOSC is only lightly documented, the source code serves as the documentation.
Here is the code that traps exceptions:
def handle_error(self, request, client_address):
"""Handle an exception in the Server's callbacks gracefully.
Writes the error to sys.stderr and, if the error_prefix (see setSrvErrorPrefix()) is set,
sends the error-message as reply to the client
"""
(e_type, e) = sys.exc_info()[:2]
self.printErr("%s on request from %s: %s" % (e_type.__name__, getUrlStr(client_address), str(e)))
if self.print_tracebacks:
import traceback
traceback.print_exc() # XXX But this goes to stderr!
if len(self.error_prefix):
self.sendOSCerror("%s: %s" % (e_type.__name__, str(e)), client_address)
And here's is the simple way you turn trace on when you create the OSCServer instance:
import OSC
myoscserver = OSC.OSCServer( (host, port) )
myoscserver.print_tracebacks = True
This works for OSCClient as well:
myoscclient = OSC.OSCClient()
myoscclient.connect( (host, port) )
myoscclient.print_tracebacks = True
Hope that helps.
I'm trying to write a dead-simple interface for an IRC library, like so:
import simpleirc
connection = simpleirc.Connect('irc.freenode.net', 6667)
channel = connection.join('foo')
find_command = re.compile(r'google ([a-z]+)').findall
for msg in channel:
for t in find_command(msg):
channel.say("http://google.com/search?q=%s" % t)
Working from their example, I'm running into trouble (code is a bit lengthy, so I pasted it here). Since the call to channel.__next__ needs to be returned when the callback <IRCClient instance>.privmsg is called, there doesn't seem to be a clean option. Using exceptions or threads seems like the wrong thing here, is there a simpler (blocking?) way of using twisted that would make this possible?
In general, if you're trying to use Twisted in a "blocking" way, you're going to run into a lot of difficulties, because that's neither the way it's intended to be used, nor the way in which most people use it.
Going with the flow is generally a lot easier, and in this case, that means embracing callbacks. The callback-style solution to your question would look something like this:
import re
from twisted.internet import reactor, protocol
from twisted.words.protocols import irc
find_command = re.compile(r'google ([a-z]+)').findall
class Googler(irc.IRCClient):
def privmsg(self, user, channel, message):
for text in find_command(message):
self.say(channel, "http://google.com/search?q=%s" % (text,))
def connect():
cc = protocol.ClientCreator(reactor, Googler)
return cc.connectTCP(host, port)
def run(proto):
proto.join(channel)
def main():
d = connect()
d.addCallback(run)
reactor.run()
This isn't absolutely required (but I strongly suggest you consider trying it). One alternative is inlineCallbacks:
import re
from twisted.internet import reactor, protocol, defer
from twisted.words.protocols import irc
find_command = re.compile(r'google ([a-z]+)').findall
class Googler(irc.IRCClient):
def privmsg(self, user, channel, message):
for text in find_command(message):
self.say(channel, "http://google.com/search?q=%s" % (text,))
#defer.inlineCallbacks
def run():
cc = protocol.ClientCreator(reactor, Googler)
proto = yield cc.connectTCP(host, port)
proto.join(channel)
def main():
run()
reactor.run()
Notice no more addCallbacks. It's been replaced by yield in a decorated generator function. This could get even closer to what you asked for if you had a version of Googler with a different API (the one above should work with IRCClient from Twisted as it is written - though I didn't test it). It would be entirely possible for Googler.join to return a Channel object of some sort, and for that Channel object to be iterable like this:
#defer.inlineCallbacks
def run():
cc = protocol.ClientCreator(reactor, Googler)
proto = yield cc.connectTCP(host, port)
channel = proto.join(channel)
for msg in channel:
msg = yield msg
for text in find_command(msg):
channel.say("http://google.com/search?q=%s" % (text,))
It's only a matter of implementing this API on top of the ones already present. Of course, the yield expressions are still there, and I don't know how much this will upset you. ;)
It's possible to go still further away from callbacks and make the context switches necessary for asynchronous operation to work completely invisible. This is bad for the same reason it would be bad for sidewalks outside your house to be littered with invisible bear traps. However, it's possible. Using something like corotwine, itself based on a third-party coroutine library for CPython, you can have the implementation of Channel do the context switching itself, rather than requiring the calling application code to do it. The result might look something like:
from corotwine import protocol
def run():
proto = Googler()
transport = protocol.gConnectTCP(host, port)
proto.makeConnection(transport)
channel = proto.join(channel)
for msg in channel:
for text in find_command(msg):
channel.say("http://google.com/search?q=%s" % (text,))
with an implementation of Channel that might look something like:
from corotwine import defer
class Channel(object):
def __init__(self, ircClient, name):
self.ircClient = ircClient
self.name = name
def __iter__(self):
while True:
d = self.ircClient.getNextMessage(self.name)
message = defer.blockOn(d)
yield message
This in turn depends on a new Googler method, getNextMessage, which is a straightforward feature addition based on existing IRCClient callbacks:
from twisted.internet import defer
class Googler(irc.IRCClient):
def connectionMade(self):
irc.IRCClient.connectionMade(self)
self._nextMessages = {}
def getNextMessage(self, channel):
if channel not in self._nextMessages:
self._nextMessages[channel] = defer.DeferredQueue()
return self._nextMessages[channel].get()
def privmsg(self, user, channel, message):
if channel not in self._nextMessages:
self._nextMessages[channel] = defer.DeferredQueue()
self._nextMessages[channel].put(message)
To run this, you create a new greenlet for the run function and switch to it, and then start the reactor.
from greenlet import greenlet
def main():
greenlet(run).switch()
reactor.run()
When run gets to its first asynchronous operation, it switches back to the reactor greenlet (which is the "main" greenlet in this case, but it doesn't really matter) to let the asynchronous operation complete. When it completes, corotwine turns the callback into a greenlet switch back into run. So run is granted the illusion of running straight through, like a "normal" synchronous program. Keep in mind that it is just an illusion, though.
So, it's possible to get as far away from the callback-oriented style that is most commonly used with Twisted as you want. It's not necessarily a good idea, though.