The create_connection function from Python 3's asyncio module takes as it's first parameter a protocol factory. The documentation has the following note:
Note protocol_factory can be any kind of callable, not necessarily a class. For example, if you want to use a pre-created protocol instance, you can pass lambda: my_protocol.
So you can pass in an instance using a lambda like so:
create_connection(lambda: Protocol(a, b, c))
An alternative would be to define __call__ to return self such that you could just pass the instance without defining a lambda.
protocol = Protocol(a, b, c)
create_connection(protocol)
Is there any reason to use a lambda as the documentation suggests over defining __call__ on the class?
Notice the difference between these two lines:
loop.create_connection(MyProtocol, '127.0.0.1', 8888) # Option #1
loop.create_connection(MyProtocol(), '127.0.0.1', 8888) # Option #2
Here is the echo client example from asyncio docs, modified to work with the Option #1:
class MyEchoClientProtocol(asyncio.Protocol):
def connection_made(self, transport):
message = "hello"
transport.write(message.encode())
print('Data sent: {!r}'.format(message))
def data_received(self, data):
print('Data received: {!r}'.format(data.decode()))
def connection_lost(self, exc):
print('The server closed the connection')
print('Stop the event loop')
loop.stop()
loop = asyncio.get_event_loop()
coro = loop.create_connection(MyEchoClientProtocol, '127.0.0.1', 8765)
loop.run_until_complete(coro)
loop.run_forever()
loop.close()
If you choose to use Option #2, you will need to implement MyProtocol.__call__(self) which works on instances of MyProtocol.
Although this might work OK for create_connection, since your __call__ will be called only once, this does not work well for the protocol_factory parameter of create_server:
...
# Each client connection will create a new protocol instance
coro = loop.create_server(EchoServerClientProtocol, '127.0.0.1', 8888)
...
Here protocol_factory is called multiple times to create new Protocol instances. Using EchoServerClientProtocol() and defining def __call__(self): return self will reuse only one instance of Protocol!
Short answer:
The lambda should be used in preference because it is more readable - it can be understood easily without having to scrutinise the Protocol class code.
Explanation:
BaseEventLoop.create_connection yields from BaseEventLoop._create_connection_transport ref, which instantiates a protocol object from the Protocol class as follows:
protocol = protocol_factory()
We can present the problem in a simplified manner without the event loop code to demonstrate how the Protocol is being instantiated:
class Prococol:
pass
def create_connection(Protocol):
protocol = Protocol()
create_connection(Protocol)
So, "protocol = Protocol()" needs to work with the parameters. This can be by using a lambda:
class Protocol:
def __init__(self, a):
self.a = a
def create_connection(Protocol):
protocol = Protocol()
create_connection(lambda: Protocol(1))
Or the alternate suggestion that the OP suggested would be making the object a callable:
class Protocol:
def __init__(self, a):
self.a = a
def __call__(self):
return self
def create_connection(Protocol):
protocol = Protocol()
create_connection(Protocol(1))
Functionally both will work, and thus it is a question of what is better practice. I would argue that the lambda approach is better, because looking the final line create_connection(lambda: Protocol(1)) makes it clear that we are passing to the create_connection function that returns an object when called, whereas passing an a callable object makes the code less readable - because one needs to scrutinise the Protocol class for ascertain that the instantiated object is also a callable entity.
Udi answer to this question says that using def __call__(self): return self, will not work with create_server (which as an aside is not what the question asked) as it will reuse one instance of an instantiated object. This observation is correct, but what is omitted from that answer is that the callable can easily be adjusted to work with the create_server. For example:
class Protocol:
def __init__(self, a):
self.a = a
def __call__(self):
return Protocol(self.a)
The bottom line is using __call__ should work as will the lambda approach. The reason why lambda should be used in preference is for readability reasons.
Related
Problem
I am using a library to facilitate client side websocket communication with a server.
The websocket library allows you to specify call back functions for when the socket opens, closes, errors or receives a message
If I set my callback functions to be instance functions of another class, then they need to be passed the self parameter when the are called.
I had understood that if you call a class instance method it will always be passed self as the first parameter. However,my callbacks are not getting passed self
For example
from websocket import WebSocketApp
import websocket
class X(object):
def run(self):
self.ws = WebSocketApp('wss://api.bitfinex.com/ws/2'
,on_open=self.on_open
,on_message=self.on_message
,on_error=self.on_error
,on_close=self.on_close)
websocket.enableTrace(True)
self.ws.run_forever()
def on_open(self, ws):
print('open')
def on_close(self, ws):
print('close')
def on_message(self, ws, message):
print('message')
def on_error(self, ws, error):
print('error')
if __name__=='__main__':
x = X().run()
Output
error from callback <bound method X.on_open of <__main__.X object at 0x7fd7635e87f0>>: on_open() missing 1 required positional argument: 'ws'
File "/home/arran/.local/lib/python3.6/site-packages/websocket/_app.py", line 343, in _callback
callback(*args)
I am probably missing something basic here. But any help would be greatly appreciated
Edit
Looks like this might be a version specific issue with the websocket-client library https://github.com/home-assistant/home-assistant/issues/17532
I have downgraded to an earlier version and fixed my problem.
I would still be curious to know how this issue can arise though. My understanding was that class instance methods will always be passed self as the first parameter
It looks to be an issue with the WebSocket class not passing the ws argument that your on_open method expects. I tried to reproduce it with my own dummy class, and it works fine.
class WS:
def __init__(self, on_call):
self.on_call = on_call
def call(self):
print("hi")
self.on_call(self)
class X:
def on_call(self, ws):
print(ws)
def run(self):
self.ws = WS(self.on_call)
self.ws.call()
X().run()
hi
<__main__.WS instance at 0x029AB698>
I am probably missing something basic here.
No, you were spot on. However, the on_open callback does not get called with the ws argument, although it should according to the documentation:
class WebSocketApp(object):
(...)
on_open: callable object which is called at opening websocket.
this function has one argument. The argument is this class object.
(...)
This is a known bug that was closed despite some discussion around the way it was fixed.
would still be curious to know how this issue can arise though.
I guess it's an honest mistake in an attempted bug fix. As there is no test for your particular scenario, it did not get caught.
I have downgraded to an earlier version and fixed my problem
Please kindly submit a bug report or write a pull request to fix the problem.
My understanding was that class instance methods will always be passed self as the first parameter
Yes, your understanding is correct. Here is an example mirroring what you tried.
class Server(object):
def __init__(self, callback):
self.callback = callback
def run(self):
self.callback(5)
class Client(object):
def on_message(self, n):
print("got", n)
client = Client()
server = Server(client.on_message)
server.run()
I'm trying to override a python class (first time doing this), and I can't seem to override this method. When I run this, my recv method doesn't run. It runs the superclasses's method instead. What am I doing wrong here? (This is python 2.7 by the way.)
import socket
class PersistentSocket(socket.socket):
def recv(self, count):
print("test")
return super(self.__class__, self).recv(count)
if __name__ == '__main__':
s = PersistentSocket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 2300))
print(s.recv(1)
The socket type (officially socket.SocketType, though socket.socket happens to be the same object) makes the strange choice of implementing recv and a few other methods as instance attributes, rather than as normal methods in the class dict. In socket.SocketType.__init__, it sets a self.recv instance attribute that overrides the recv method you tried to define.
Picking on the explanation from #user2357112, one thing that seems to have helped is to do a delattr(self, 'recv') on the class constructor (inheriting from socket.SocketType) and then define you own recv method; for example:
class PersistentSocket(socket.SocketType):
def __init__(self):
"""As usual."""
delattr(self, 'recv')
def recv(self, buffersize=1024, flags=0):
"""Your own implementation here."""
return None
I am using a simpleWebSocket server class and have a 1 second interval timer that I would like to call methods in a couple of different classes.
the wsscb() class is the handler for the SimpleWebSocketServer(), how can I call a method from the wss() object from another object such as the udt() timer ?
Calling wss.wsscb().myfunc() results in an error: "AttributeError: 'SimpleWebSocketServer' object has no attribute 'wsscb'"
calling wsscb.myfunc() results in: TypeError: unbound method myfunc() must be called with wsscb instance as first argument (got nothing instead)
class wsscb(WebSocket):
def __init__(self, server, sock, address):
WebSocket.__init__(self, server, sock, address)
def myfunc(self):
self.send('some data')
def handleMessage(self):
pass
def handleConnected(self):
pass
class udt(Thread):
def __init__(self, event):
Thread.__init__(self)
self.stopped = event
def run(self):
while not self.stopped.wait(1.00):
wss.wsscb().myfunc()
xxx.yyy()().anotherfunc()
## Main
wss = SimpleWebSocketServer('', 4545,wsscb)
## Start Timer
stopFlag = Event()
self.udt = udt(stopFlag)
self.udt.start()
wss.serveforever()
There are a couple problems.
wss.wsscb() isn't valid. Typing that means you're trying to call a function in wss called wsscb(). wss is a SimpleWebSocketServer, and there is no function called wsscb(). A function is not the same as calling an object.
wsscb() won't work either, because in your class, you're saying it's takes a WebSocket object, which I assume takes some parameters, so you need to pass it those.
I think it would be best to make a subclass of SimpleWebSocketServer (instead of WebSocket), and put your custom function in there. Your comment says "wsscb() is a subclass of SimpleSocketServer", but it is not. It's a subclass of WebSocket.
You also never created an object of type wsscb.
If you can explain what you're specifically trying to achieve, and what myfunc() is, we may be able to help more
Also, you really shouldn't subclass Thread. Scrap the udt class you made and instead
def myfunc(wsscb_object):
while True:
time.sleep(1)
wsscb_object.myfunc()
#whatever else you want
wsscb_object = wsscb(#pass the parameters)
thread = Thread(target=myfunc, args=(some_socket))
thread.start()
You may also want to read up more on inheritance:
python subclasses
http://www.jesshamrick.com/2011/05/18/an-introduction-to-classes-and-inheritance-in-python/
Using inheritance in python
I'm implementing a python application which is using ThreadingTCPServer and a custom subclass of BaseRequestHandler. The problem with this is that the ThreadingTCPServer seems to automatically spawn threads and create instances of the handler, calling their handle() function. However this leaves me with no way to pass data to the handler other than using global variables or class variables, both of which seem hackish. Is there any better way to do it?
Ideally this should be something like:
class ThreadedTCPServer(ThreadingTCPServer):
def process_request(self, *args, **kwargs):
ThreadingTCPServer.process_request(self, data, *args, **kwargs)
with the handler like
class ThreadedTCPRequestHandler(BaseRequestHandler):
def handle(self,data):
#do something with data
I stumbled upon the very same thing. My solution was the following:
class ThreadedTCPRequestHandler(SocketServer.StreamRequestHandler):
def handle(self):
print(self.server.mycustomdata)
class ThreadedTCPServer(SocketServer.ThreadingTCPServer):
pass
server = ThreadedTCPServer((args.host, args.port), ThreadedTCPRequestHandler)
server.mycustomdata = 'foo.bar.z'
server.serve_forever()
The RequestHandler is called with a server object as a third parameter, and it is saved as self.server attribute, so you can access it. If you would set this attribute to a callable, you could easily call it, too:
def handle(self):
mycustomdata = self.server.mycustomdata()
The first answer worked for me, but I think it is cleaner to alter the __init__ method and pass the attribute in the constructor:
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
def __init__(self, host_port_tuple, streamhandler, Controllers):
super().__init__(host_port_tuple, streamhandler)
self.Controllers = Controllers
Note the third parameter 'Controllers' in the constructor, then the call to super without that parameter, then setting the new attribute Controllers to the property self.Controllers. The rest of the class is unchanged. Then, in your Requesthandler, you get access to the parameter using the 'server' attribute, as described above:
def handle(self):
self.Controllers = self.server.Controllers
<rest of your code>
It's much the same as the answer above but I find it a little cleaner because the constructor is overloaded and you simply add the attribute you want in the constructor:
server = ServerInterface.ThreadedTCPServer((HOST, PORT), ServerInterface.ThreadedTCPRequestHandler, Controllers)
Since handle is implemented by your BaseRequest subclass, it can get the data from itself without having it passed by the caller. (handle could also be a callable attribute of the request instance, such as a lambda—explicit user_data arguments are normally unnecessary in idiomatically designed python.)
Looking at the SocketServer code, it should be straightforward to override finish_request to pass the additional data to your BaseRequestHandler subtype constructor which would store it in the instance for handle to use.
Python newbie here. I'm writing an SMTP server using Twisted and twisted.mail.smtp. I'd like to log incoming connections and possibly dump them when there are too many concurrent connections. Basically, I want ConsoleMessageDelivery.connectionMade() method to be called in the following, when a new connection is made:
class ConsoleMessageDelivery:
implements(smtp.IMessageDelivery)
def connectionMade(self):
# This never gets called
def receivedHeader(self, helo, origin, recipients):
myHostname, clientIP = helo
headerValue = "by %s from %s with ESMTP ; %s" % (myHostname, clientIP, smtp.rfc822date())
# email.Header.Header used for automatic wrapping of long lines
return "Received: %s" % Header(headerValue)
def validateFrom(self, helo, origin):
# All addresses are accepted
return origin
def validateTo(self, user):
if user.dest.local == "console":
return lambda: ConsoleMessage()
raise smtp.SMTPBadRcpt(user)
class ConsoleMessage:
implements(smtp.IMessage)
def __init__(self):
self.lines = []
def lineReceived(self, line):
self.lines.append(line)
def eomReceived(self):
return defer.succeed(None)
def connectionLost(self):
# There was an error, throw away the stored lines
self.lines = None
class ConsoleSMTPFactory(smtp.SMTPFactory):
protocol = smtp.ESMTP
def __init__(self, *a, **kw):
smtp.SMTPFactory.__init__(self, *a, **kw)
self.delivery = ConsoleMessageDelivery()
def buildProtocol(self, addr):
p = smtp.SMTPFactory.buildProtocol(self, addr)
p.delivery = self.delivery
return p
connectionMade is part of twisted.internet.interfaces.IProtocol, not part of twisted.mail.smtp.IMessageDelivery. There's no code anywhere in the mail server implementation that cares about a connectionMade method on a message delivery implementation.
A better place to put per connection logic is in the factory. And specifically, a good way to approach this is with a factory wrapper, to isolate the logic about connection limits and logging from the logic about servicing SMTP connections.
Twisted comes with a few factory wrappers. A couple in particular that might be interesting to you are twisted.protocols.policies.LimitConnectionsByPeer and twisted.protocols.policies.LimitTotalConnectionsFactory.
Unfortunately, I don't know of any documentation explaining twisted.protocols.policies. Fortunately, it's not too complicated. Most of the factories in the module wrap another arbitrary factory to add some piece of behavior. So, for example, to use LimitConnectionsByPeer, you do something like this:
from twisted.protocols.policies import LimitConnectionsByPeer
...
factory = ConsoleSMTPFactory()
wrapper = LimitConnectionsByPeer(ConsoleSMTPFactory(...))
reactor.listenTCP(465, wrapper)
This is all that's needed to get LimitConnectionsByPeer to do its job.
There's only a little bit more complexity involved in writing your own wrapper. First, subclass WrappingFactory. Then implement whichever methods you're interested in customizing. In your case, if you want to reject connections from a certain IP, that would mean overriding buildProtocol. Then, unless you also want to customize the protocol that is constructed (which you don't in this case), call the base implementation and return its result. For example:
from twisted.protocols.policies import WrappingFactory
class DenyFactory(WrappingFactory):
def buildProtocol(self, clientAddress):
if clientAddress.host == '1.3.3.7':
# Reject it
return None
# Accept everything else
return WrappingFactory.buildProtocol(self, clientAddress)
These wrappers stack, so you can combine them as well:
from twisted.protocols.policies import LimitConnectionsByPeer
...
factory = ConsoleSMTPFactory()
wrapper = LimitConnectionsByPeer(DenyFactory(ConsoleSMTPFactory(...)))
reactor.listenTCP(465, wrapper)