An example of my code is as follows. I would like to arbitrarly send data at various points in the program. Twisted seems great for listening and then reacting, but how to I simply send data.
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
import os
class listener(DatagramProtocol):
def __init__(self):
def datagramReceived(self, data, (host, port)):
print "GOT " + data
def send_stuff(data):
self.transport.write(data, (host, port))
reactor.listenUDP(10000, listener())
reactor.run()
##Some things happen in the program independent of the connection state
##Now how to I access send_stuff
Your example already includes some code that sends data:
def send_stuff(data):
self.transport.write(data, (host, port))
In other words, the answer to your question is "call send_stuff" or even "call transport.write".
In a comment you asked:
#Now how to I access send_stuff
There's nothing special about how you "access" objects or methods when you're using Twisted. It's the same as in any other Python program you might write. Use variables, attributes, containers, function arguments, or any of the other facilities to maintaining references to objects.
Here are some examples:
# Save the listener instance in a local variable
network = listener()
reactor.listenUDP(10000, network)
# Use the local variable to connect a GUI event to the network
MyGUIApplication().connect_button("send_button", network.send_stuff)
# Use the local variable to implement a signal handler that sends data
def report_signal(*ignored):
reactor.callFromThread(network.send_stuff, "got sigint")
signal.signal(signal.SIGINT, report_signal)
# Pass the object referenced by the local variable to the initializer of another
# network-related object so it can save the reference and later call methods on it
# when it gets events it wants to respond to.
reactor.listenUDP(20000, AnotherDatagramProtocol(network))
And so on.
Related
I am writing a TCP messaging application based on a client server model consisting of one server and multiple clients communicating concurrently. I know quite little about multi-threading and so in an effort to get a better understanding ofh ow to do this, I sourced some code online.
OPS CODE:1/2
def recv_handler():
global t_lock
global clients
global serverSocket
print('Server is up.')
while True:
# create a new connection for a new client
connection_socket, client_address = serverSocket.accept()
# create a new function handler for the client
socket_handler = connection_handler(connection_socket, client_address)
# create a new thread for the client socket
socket_thread = threading.Thread(name=str(client_address), target=socket_handler)
socket_thread.daemon = False
socket_thread.start()
OPS CODE:2/2
# return a function as connection handler for a specific socket for multi threading
def connection_handler(connection_socket, client_address):
def real_connection_handler():
while True:
data = connection_socket.recv(1024)
....
more logic
return real_connection_handler
Superficially, I think I understand what's happening here:
recv_handler is constantly listening for new client connections, and upon request creates a new dedicated socket for each new client connection.This happens by connection_socket, client_address = serverSocket.accept().
Once a new new socket has been created, the socket and the address are passed to the
connection_handler(connection_socket, client_address) function, which returns the object real_connection_handler().
Then a new thread is created with the real_connection_handler object as the target function in the thread constructor.
Why does OP do this? Specifically, why does OP set target to the inner function (real_connection_handler) instead of the outer function (connection_handler)? Why is he nesting functions here? Why not just do:
MY CODE SUGGESTION 1/2
def recv_handler():
global t_lock
global clients
global serverSocket
print('Server is up.')
while True:
# create a new connection for a new client
connection_socket, client_address = serverSocket.accept()
# create a new thread for the client socket
socket_thread = threading.Thread(name=str(client_address),target=connection_handler(connection_socket, client_address))
socket_thread.daemon = False
socket_thread.start()
MY CODE SUGGESTION 2/2
def connection_handler(connection_socket, client_address):
while True:
data = connection_socket.recv(1024)
....
more logic
Does my suggestion code invite competition over resources? Like forming a queue in accessing the connection_handler function? For example, if a very first client_connection_request_1 (call it conn_1) were to be established with the server and remain active in the connection_handler function while a second client_connection_request_2 (call it conn_2) were to be established, would conn_2 only gain access to the connection_handler function once conn_1 exited (i.e. stopped sending messages)?
This is my first ever post so I hope it's clear. I'm confused by what he's done and python is also new to me, so my question my in turn be confusing. I appreciate any clarity. Thanks in advance.
I will be hosting a service that will be acting somewhat like a proxy for something I am a client to.
So I want my ProxyService (a twisted.protocol server) to takes lots of actors (clients). On the server side of things, I need a global connection (only need 1 connection to it for all clients) to an ExistingService (code I didn't write, and I'm a client to it).
When the ExistingService says something interesting, I need to broadcast it to all actors.
When an actor says something to my ProxyService, I need to check if it looks good to me. If it does, I need to inform the ExistingService.
I think I know how to solve this using global variables, but just wondering if better way to push the messages.
You have the basic design well established.
It's a basic "man in the middle" approach.
There are many ways to implement it, but this should get you started:
from twisted.internet import endpoints, protocol, reactor
class ProxyClient(protocol.Protocol):
def connectionMade(self):
print('[x] proxy connection made to server')
self.factory.proxy_proto = self
def connectionLost(self, reason):
print('[ ] proxy connection to server lost: {0}'.format(reason))
self.factory.proxy_proto = None
def dataReceived(self, data):
print('==> received {0} from server'.format(data))
print('<== transmitting data to all actors')
for actor in self.factory.actors:
actor.transport.write(data)
class Actor(protocol.Protocol):
def connectionMade(self):
print('[x] actor connection established')
self.factory.actors.add(self)
def connectionLost(self, reason):
print('[ ] actor disconnected: {0}'.format(reason))
self.factory.actors.remove(self)
def dataReceived(self, data):
print('==> received {0} from actor'.format(data))
proxy_connection = self.factory.proxy_factory.proxy_proto
if proxy_connection is not None:
print('<== transmitting data to server through the proxy')
proxy_connection.transport.write(data)
else:
print('[ ] proxy connection to server has not been established')
def setup_servers():
PROXY_HOST = '127.0.0.1'
PROXY_PORT = 9000
proxy_factory = protocol.ClientFactory()
proxy_factory.protocol = ProxyClient
proxy_factory.proxy_proto = None
proxy_factory.actors = set()
proxy_client = endpoints.TCP4ClientEndpoint(reactor, port=PROXY_PORT, host=PROXY_HOST)
proxy_client.connect(proxy_factory)
ACTOR_HOST = '127.0.0.1'
ACTOR_PORT = 8000
actor_factory = protocol.Factory()
actor_factory.protocol = Actor
actor_factory.proxy_factory = proxy_factory
actor_factory.actors = proxy_factory.actors
actor_server = endpoints.TCP4ServerEndpoint(reactor, port=ACTOR_PORT, interface=ACTOR_HOST)
actor_server.listen(actor_factory)
def main():
setup_servers()
reactor.run()
main()
The core logic that allows the data received from the server to be proxied to actors is proxy_factory.actors = set() and actor_factory.actors = proxy_factory.actors.
Most "list-like" containers, for lack of better words, are "global" and this example gives context into each connection's factory objects.
When an actor connects to the server, an Actor protocol is appended to the set and when data is received, each protocol in the set will get the data.
See the respective dataReceived() methods of each protocol object on how that works.
The example above doesn't use global variables at all, but that's not to say that you couldn't use them.
See how far you can get using this method of passing around variables that give context into other objects.
Also, certain situations weren't explicitly handled, such as caching received data in the event the server or actors haven't connected yet.
Hopefully there's enough information here for you to determine the best course of action based on your needs.
There's some room for streamlining the syntax to make it shorter as well.
As a side note. An alternative to global variables is picobox. It's a dependency injector library but I've found that it satisfies most my needs when I require parameters from external sources.
I was trying to build a server. Beside accept connection from clients as normal servers do, my server will connect other server as a client either.
I've set the protocol and endpoint like below:
p = FooProtocol()
client = TCP4ClientEndpoint(reactor, '127.0.0.1' , 8080) # without ClientFactory
Then, after call reactor.run(), the server will listen/accept new socket connections. when new socket connections are made(in connectionMade), the server will call connectProtocol(client, p), which acts like the pseudocode below:
while server accept new socket:
connectProtocol(client, p)
# client.client.connect(foo_client_factory) --> connecting in this way won't
# cause memory leak
As the connections to the client are made, the memory is gradually consumed(explicitly calling gc doesn't work).
Do I use the Twisted in a wrong way?
-----UPDATE-----
My test programe: Server waits clients to connect. When connection from client is made, server will create 50 connections to other server
Here is the code:
#! /usr/bin/env python
import sys
import gc
from twisted.internet import protocol, reactor, defer, endpoints
from twisted.internet.endpoints import TCP4ClientEndpoint, connectProtocol
class MyClientProtocol(protocol.Protocol):
def connectionMade(self):
self.transport.loseConnection()
class MyClientFactory(protocol.ClientFactory):
def buildProtocol(self, addr):
p = MyClientProtocol()
return p
class ServerFactory(protocol.Factory):
def buildProtocol(self, addr):
p = ServerProtocol()
return p
client_factory = MyClientFactory() # global
client_endpoint = TCP4ClientEndpoint(reactor, '127.0.0.1' , 8080) # global
times = 0
class ServerProtocol(protocol.Protocol):
def connectionMade(self):
global client_factory
global client_endpoint
global times
for i in range(50):
# 1)
p = MyClientProtocol()
connectProtocol(client_endpoint, p) # cause memleak
# 2)
#client_endpoint.connect(client_factory) # no memleak
times += 1
if times % 10 == 9:
print 'gc'
gc.collect() # doesn't work
self.transport.loseConnection()
if __name__ == '__main__':
server_factory = ServerFactory()
serverEndpoint = endpoints.serverFromString(reactor, "tcp:8888")
serverEndpoint.listen(server_factory)
reactor.run()
This program doesn't do any Twisted log initialization. This means it runs with the "log beginner" for its entire run. The log beginner records all log events it observes in a LimitedHistoryLogObserver (up to a configurable maximum).
The log beginner keeps 2 ** 16 (_DEFAULT_BUFFER_MAXIMUM) events and then begins throwing out old ones, presumably to avoid consuming all available memory if a program never configures another observer.
If you hack the Twisted source to set _DEFAULT_BUFFER_MAXIMUM to a smaller value - eg, 10 - then the program no longer "leaks". Of course, it's really just an object leak and not a memory leak and it's bounded by the 2 ** 16 limit Twisted imposes.
However, connectProtocol creates a new factory each time it is called. When each new factory is created, it logs a message. And the application code generates a new Logger for each log message. And the logging code puts the new Logger into the log message. This means the memory cost of keeping those log messages around is quite noticable (compared to just leaking a short blob of text or even a dict with a few simple objects in it).
I'd say the code in Twisted is behaving just as intended ... but perhaps someone didn't think through the consequences of that behavior complete.
And, of course, if you configure your own log observer then the "log beginner" is taken out of the picture and there is no problem. It does seem reasonable to expect that all serious programs will enable logging rather quickly and avoid this issue. However, lots of short throw-away or example programs often don't ever initialize logging and rely on print instead, making them subject to this behavior.
Note This problem was reported in #8164 and fixed in 4acde626 so Twisted 17 will not have this behavior.
I'm trying to pass a TCP connection to a Twisted subprocess with adoptStreamConnection, but I can't figure out how to get the Process disposed in the main process after doing that.
My desired flow looks like this:
Finish writing any data the Protocol transport has waiting
When we know the write buffer is empty send the AMP message to transfer the socket to the subprocess
Dispose the Protocol instance in the main process
I tried doing nothing, loseConnection, abortConnection, and monkey patching _socketClose out and using loseConnection. See code here:
import weakref
from twisted.internet import reactor
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.python.sendmsg import getsockfam
from twisted.internet.protocol import Factory, Protocol
import twisted.internet.abstract
class EchoProtocol(Protocol):
def dataReceived(self, data):
self.transport.write(data)
class EchoFactory(Factory):
protocol = EchoProtocol
class TransferProtocol(Protocol):
def dataReceived(self, data):
self.transport.write('main process still listening!: %s' % (data))
def connectionMade(self):
self.transport.write('this message should make it to the subprocess\n')
# attempt 1: do nothing
# everything works fine in the adopt (including receiving the written message), but old protocol still exists (though isn't doing anything)
# attempt 1: try calling loseConnection
# we lose connection before the adopt opens the socket (presumably TCP disconnect message was sent)
#
# self.transport.loseConnection()
# attempt 2: try calling abortConnection
# result is same as loseConnection
#
# self.transport.abortConnection()
# attempt 3: try monkey patching the socket close out and calling loseConnection
# result: same as doing nothing-- adopt works (including receiving the written message), old protocol still exists
#
# def ignored(*args, **kwargs):
# print 'ignored :D'
#
# self.transport._closeSocket = ignored
# self.transport.loseConnection()
reactor.callLater(0, adopt, self.transport.fileno())
class ServerFactory(Factory):
def buildProtocol(self, addr):
p = TransferProtocol()
self.ref = weakref.ref(p)
return p
f = ServerFactory()
def adopt(fileno):
print "does old protocol still exist?: %r" % (f.ref())
reactor.adoptStreamConnection(fileno, getsockfam(fileno), EchoFactory())
port = 1337
endpoint = TCP4ServerEndpoint(reactor, port)
d = endpoint.listen(f)
reactor.run()
In all cases the Protocol object still exists in the main process after the socket has been transferred. How can I clean this up?
Thanks in advance.
Neither loseConnection nor abortConnection tell the reactor to "forget" about a connection; they close the connection, which is very different; they tell the peer that the connection has gone away.
You want to call self.transport.stopReading() and self.transport.stopWriting() to remove the references to it from the reactor.
Also, it's not valid to use a weakref to test for the remaining existence of an object unless you call gc.collect() first.
As far as making sure that all the data has been sent: the only reliable way to do that is to have an application-level acknowledgement of the data that you've sent. This is why protocols that need a handshake that involves changing protocols - say, for example, STARTTLS - have a specific handshake where the initiator says "I'm going to switch" (and then stops sending), then the peer says "OK, you can switch now". Another way to handle that in this case would be to hand the data you'd like to write to the subprocess via some other channel, instead of passing it to transport.write.
I am reading the post (http://pymotw.com/2/asynchat/#module-asynchat) on how to use module asynchat. Here is the code for server
import asyncore
import logging
import socket
from asynchat_echo_handler import EchoHandler
class EchoServer(asyncore.dispatcher):
"""Receives connections and establishes handlers for each client.
"""
def __init__(self, address):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind(address)
self.address = self.socket.getsockname()
self.listen(1)
return
def handle_accept(self):
# Called when a client connects to our socket
client_info = self.accept()
EchoHandler(sock=client_info[0])
# We only want to deal with one client at a time,
# so close as soon as we set up the handler.
# Normally you would not do this and the server
# would run forever or until it received instructions
# to stop.
self.handle_close()
return
def handle_close(self):
self.close()
Why "EchoHandler(sock=client_info[0])" is enough? The object created does not have a name. How to call the method defined in the object of EchoHandler ?
"EchoHandler(sock=client_info[0])" is enough because no method call to it is needed in this example. You just connect the handler to the client and forget about it. It will answer the calls from the client itself and closes when the client closes.
When you want to call it later just do it like jedwards wrote.