I have the following situation:
SomeServer(S) <-> (C)MyApp(S) <-> (C)User
(S) represents a server socket
(C) represents a client socket
Essentially, MyApp initiates communication with SomeServer (SomeServer(S) <-> (C)MyApp) and once some authentication routines are successful MyApp(S) starts waiting for (C)User to connect. As soon as User connects, MyApp relays data from SomeServer to User. This happens in both directions.
I have SomeServer(S) <-> (C)MyApp working perfectly, but I'm not able to get MyApp(S) <-> (C)User working. I get as far as User connecting to MyApp(S), but can't get data relayed!
Ok, I hope that's some what clear ;) Now let me show my code for MyApp. Btw the implementation of SomeServer and User are not relevant for solving my question, as neither can be modified.
I have commented my code indicating where I'm experiencing issues. Oh, I should also mention that I have no problem scrapping the whole "Server Section" for some other code if necessary. This is a POC, so my main focus is getting the functionality working rather than writing efficient code. Thanks for you time.
''' MyApp.py module '''
import asyncore, socket
import SSL
# Client Section
# Connects to SomeServer
class MyAppClient(asyncore.dispatcher):
def __init__(self, host, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect((host, port))
connectionPhase = 1
def handle_read(self):
print "connectionPhase =", self.connectionPhase
# The following IF statements may not make sense
# as I have removed code irrelevant to this question
if self.connectionPhase < 3: # authentication phase
data = self.recv(1024)
print 'Received:', data
# Client/Server authentication is handled here
# Everything from this point on happens over
# an encrypted socket using SSL
# Start the RelayServer listening on localhost 8080
# self.socket is encrypted and is the socket communicating
# with SomeServer
rs = RelayServer(('localhost', 8080), self.socket)
print 'RelayServer started'
# connectionPhase = 3 when this IF loop is done
elif self.connectionPhase == 3: # receiving data for User
data = self.recv(1024)
print 'Received data - forward to User:', data
# Forward this data to User
# Don't understand why data is being read here
# when the RelayServer was instantiated above
# Server Section
# Connects to User
class RelayConnection(asyncore.dispatcher):
def __init__(self, client, sock):
asyncore.dispatcher.__init__(self)
self.client = client
print "connecting to %s..." % str(sock)
def handle_connect(self):
print "connected."
# Allow reading once the connection
# on the other side is open.
self.client.is_readable = True
# For some reason this never runs, i.e. data from SomeServer
# isn't read here, but instead in MyAppClient.handle_read()
# don't know how to make it arrive here instead as it should
# be relayed to User
def handle_read(self):
self.client.send(self.recv(1024))
class RelayClient(asyncore.dispatcher):
def __init__(self, server, client, sock):
asyncore.dispatcher.__init__(self, client)
self.is_readable = False
self.server = server
self.relay = RelayConnection(self, sock)
def handle_read(self):
self.relay.send(self.recv(1024))
def handle_close(self):
print "Closing relay..."
# If the client disconnects, close the
# relay connection as well.
self.relay.close()
self.close()
def readable(self):
return self.is_readable
class RelayServer(asyncore.dispatcher):
def __init__(self, bind_address, MyAppClient_sock):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind(bind_address)
self.MyAppClient_sock = MyAppClient_sock
print self.MyAppClient_sock
self.listen(1)
def handle_accept(self):
conn, addr = self.accept()
RelayClient(self, conn, self.MyAppClient_sock)
if __name__ == "__main__":
# Connect to host
# First connection stage
connectionPhase = 1
c = MyAppClient('host', port) # SomeServer's host and port
asyncore.loop()
EDIT:
#samplebias I replaced my complete module with your code (not shown) and I have re-added all the bits and pieces that I need for authentication etc.
At this point I'm getting the same result, as with my own code above. What I mean is that MyApp (or Server in your code) is connected to SomeServer and passing data back and forth. Everything is fine thus far. When User (or client application) connects to localhost 8080, this code is run:
if not self.listener:
self.listener = Listener(self.listener_addr, self)
BUT, this is not run
# if user is attached, send data
elif self.user:
print 'self.user'
self.user.send(data)
So, Server is not relaying data to User. I added print statements throughout the User class to see what is run and init is the only thing. handle_read() never runs.
Why is this?
The code is a bit hard to follow, and I'm sure there are a few bugs. For
example in handle_read() you're passing MyAppClient's raw socket self.socket to
RelayServer. You end up with both MyAppClient and RelayConnection working on the same socket.
Rather than attempt to suggest bug fixes to the original code I put together
an example which does what your code intents and is cleaner and easier to follow.
I've tested it talking to an IMAP server and it works, but omits some
things for brevity (error handling, proper close() handling in all cases, etc).
Server initiates the connection to "someserver". Once it connects
it starts the Listener.
Listener listens on port 8080 and accepts only 1 connection, creates a User,
and passes it a reference to Server. Listener rejects all other
client connections while User is active.
User forwards all data to Server, and vice versa. The comments
indicate where the authentication should be plugged in.
Source:
import asyncore
import socket
class User(asyncore.dispatcher_with_send):
def __init__(self, sock, server):
asyncore.dispatcher_with_send.__init__(self, sock)
self.server = server
def handle_read(self):
data = self.recv(4096)
# parse User auth protocol here, authenticate, set phase flag, etc.
# if authenticated, send data to server
if self.server:
self.server.send(data)
def handle_close(self):
if self.server:
self.server.close()
self.close()
class Listener(asyncore.dispatcher_with_send):
def __init__(self, listener_addr, server):
asyncore.dispatcher_with_send.__init__(self)
self.server = server
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.set_reuse_addr()
self.bind(listener_addr)
self.listen(1)
def handle_accept(self):
conn, addr = self.accept()
# this listener only accepts 1 client. while it is serving 1 client
# it will reject all other clients.
if not self.server.user:
self.server.user = User(conn, self.server)
else:
conn.close()
class Server(asyncore.dispatcher_with_send):
def __init__(self, server_addr, listener_addr):
asyncore.dispatcher_with_send.__init__(self)
self.server_addr = server_addr
self.listener_addr = listener_addr
self.listener = None
self.user = None
def start(self):
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect(self.server_addr)
def handle_error(self, *n):
self.close()
def handle_read(self):
data = self.recv(4096)
# parse SomeServer auth protocol here, set phase flag, etc.
if not self.listener:
self.listener = Listener(self.listener_addr, self)
# if user is attached, send data
elif self.user:
self.user.send(data)
def handle_close(self):
if self.user:
self.user.server = None
self.user.close()
self.user = None
if self.listener:
self.listener.close()
self.listener = None
self.close()
self.start()
if __name__ == '__main__':
app = Server(('someserver', 143), ('localhost', 8080))
app.start()
asyncore.loop()
Related
I have to send data only to a connection, as I can do?
server:
import asyncore, socket, threading
class EchoHandler(asyncore.dispatcher_with_send):
def __init__(self,sock):
asyncore.dispatcher.__init__(self,sock=sock);
self.out_buffer = ''
def handle_read(self):
datos = self.recv(1024);
if datos:
print(datos);
self.sock[0].send("signal");
class Server(asyncore.dispatcher):
def __init__(self,host='',port=6666):
asyncore.dispatcher.__init__(self);
self.create_socket(socket.AF_INET, socket.SOCK_STREAM);
self.set_reuse_addr();
self.bind((host,port));
self.listen(1);
def handle_accept(self):
self.sock,self.addr = self.accept();
if self.addr:
print self.addr[0];
handler = EchoHandler(self.sock);
def handle_close(self):
self.close();
cliente = Server();
asyncore.loop()
this line is an example fails, but I want to send data to zero sock:
self.sock[0].send("probando");
for example, if I have 5 sockets choose who to send the data
Explanation
You tried to get sock from list and execute its send method. This causes error, because EchoHandler neither has sock attribute nor it's a list of sockets. The right method is to get instance of EchoHandler you want (based on, eg. IP address, or slots assigned by some user-defined protocol) and then use its send method - here (with dispatcher_with_send) its also better to use special buffer for that than send.
EchoHandler instantion is created on every accept of connection - from then it is an established channel for communication with the given host. Server listens for any non-established connection, while EchoHandlers use socks (given by Server in handle_accept) for established ones, so there are as many EchoHandler instances as connections.
Solution
You need to make some list of connections (EchoHandler instantions; we'll use buffer, not socket's send() directly) and give them opportunity to delete their entries on close:
class Server(asyncore.dispatcher):
def __init__(self, host='', port=6666):
...
self.connections = []
def handle_accept(self):
...
handler = EchoHandler(self.sock, self);
self.connections.append(self.sock)
...
def remove_channel(self, sock):
if sock in self.connections:
self.connections.remove(sock)
class EchoHandler(asyncore.dispatcher_with_send):
def __init__(self, sock, server):
...
self.server = server
def handle_read(self):
datos = self.recv(1024);
if datos:
print(datos);
self.out_buffer += 'I echo you: ' + datos
def handle_close(self):
self.server.remove_channel(self)
self.close()
EchoHandler is now aware of server instance and can remove its socket from list. This echo example is now fully functional, and with working socket list we can proceed to asynchronous sending.
But, at this point you can use this list as you wanted - cliente.connections[0].out_buffer += 'I am data' will do the work, but probably you'd want some better controlling of this. If yes, go ahead.
'For whom, by me'
In order to send data asynchronously, we need to separate asyncore from our control thread, in which we'll enter what to send and to whom.
class ServerThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.daemon = True # if thread is a daemon, it'll be killed when main program exits
self.cliente = Server()
self.start()
def run(self):
print 'Starting server thread...'
asyncore.loop()
thread = ServerThread()
while True:
msg = raw_input('Enter IP and message divided by semicolon: ')
if msg == 'exit':
break
ip, data = msg.split('; ')
for sock in thread.cliente.connections:
if sock.addr[0] == ip:
sock.out_buffer += data
break
This will work and wait for destination IP and data. Remember to have client connected.
As I said, you can use anything to indicate which socket is which. It can be a class with fields for eg. IP and username, so you could send data only to peers whose usernames start with 'D'.
But...
This solution is a bit rough and needs better knowledge of asyncore module if you want to send data nicely (here it has some delay due to how select() works) and make good use of this socket wrapper.
Here and here are some resources.
Syntax note
Although your code will now work, your code has some not-nice things. Semicolons on instructions ends don't cause errors, but making nearly every variable of class attribute can lead to them. For example here:
def handle_accept(self):
self.sock,self.addr = self.accept();
if self.addr:
print self.addr[0];
handler = EchoHandler(self.sock);
self.sock and self.addr might be used in that class for something other (eg. socket-related thing; addresses) and overriding them could make trouble. Methods used for requests should never save state of previous actions.
I hope Python will be good enough for you to stay with it!
Edit: sock.addr[0] can be used instead of sock.socket.getpeername()[0] but it requires self.addr not to be modified, so handle_accept() should look like this:
def handle_accept(self):
sock, addr = self.accept()
if addr:
print addr[0]
handler = EchoHandler(sock, self)
self.connections.append(handler)
I'm trying to create a small program that will log information output from a device via TCP
Basically this just streams data out, that i want to capture, and dump into a database for dealing with later
but the device reboots so i need to be able to reconnect when the socket closes with out any human interference
so this is what i have so far
import socket, time, logging, sys, smtplib # Import socket module
logging.basicConfig(filename='Tcplogger.log',level=logging.DEBUG,format='%(asctime)s : %(levelname)s : %(message)s')
logging.info('|--------------------------------------|')
logging.info('|--------------- TCP Logger Starting---|')
logging.info('|--------------------------------------|')
host = '127.0.0.01' # host or Ip address
port = 12345 # output port
retrytime = 1 # reconnect time
reconnectattemps = 10 # Number of time to try and reconnect
class TPCLogger:
def __init__(self):
logging.debug('****Trying connection****')
print('****Trying connection****')
self.initConnection()
def initConnection(self):
s = socket.socket()
try:
s.connect((host, port))
logging.debug('****Connected****')
except IOError as e:
while 1:
reconnectcount = 0;
logging.error(format(e.errno)+' : '+format(e.strerror))
while 1:
reconnectcount = reconnectcount + 1
logging.error('Retrying connection to Mitel attempt : '+str(reconnectcount))
try:
s.connect((host, port))
connected = True
logging.debug('****Connected****')
except IOError as e:
connected = False
logging.error(format(e.errno)+' : '+format(e.strerror))
if reconnectcount == reconnectattemps:
logging.error('******####### Max Reconnect attempts reached logger will Terminate ######******')
sys.exit("could Not connect")
time.sleep(retrytime)
if connected == True:
break
break
while 1:
s.recv(1034)
LOGGER= TCPLogger()
Which all works fine on start up if a try to connect and its not there it will retry the amount of times set by reconnectattemps
but he is my issue
while 1:
s.recv(1034)
when this fails i want to try to reconnect
i could of course type out or just copy my connection part again but what i want to be able todo is call a function that will handle the connection and retry and hand me back the connection object
for example like this
class tcpclient
#set some var
host, port etc....
def initconnection:
connect to socket and retry if needed
RETURN SOCKET
def dealwithdata:
initconnection()
while 1:
try:
s.recv
do stuff here copy to db
except:
log error
initconnection()
I think this is possible but im really not geting how the class/method system works in python so i think im missing something here
FYI just in case you didn't notice iv very new to python. any other comments on what i already have are welcome too :)
Thanks
Aj
Recommendation
For this use-case I would recommend something higher-level than sockets. Why? Controlling all these exceptions and errors for yourself can be irritating when you just want to retrieve or send data and maintain connection.
Of course you can achieve what you want with your plain solution, but you mess with code a bit more, methinks. Anyway it'll look similarly to class amustafa wrote, with handling socket errors to close/reconnect method, etc.
Example
I made some example for easier solution using asyncore module:
import asyncore
import socket
from time import sleep
class Client(asyncore.dispatcher_with_send):
def __init__(self, host, port, tries_max=5, tries_delay=2):
asyncore.dispatcher.__init__(self)
self.host, self.port = host, port
self.tries_max = tries_max
self.tries_done = 0
self.tries_delay = tries_delay
self.end = False # Flag that indicates whether socket should reconnect or quit.
self.out_buffer = '' # Buffer for sending.
self.reconnect() # Initial connection.
def reconnect(self):
if self.tries_done == self.tries_max:
self.end = True
return
print 'Trying connecting in {} sec...'.format(self.tries_delay)
sleep(self.tries_delay)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
try:
self.connect((self.host, self.port))
except socket.error:
pass
if not self.connected:
self.tries_done += 1
print 'Could not connect for {} time(s).'.format(self.tries_done)
def handle_connect(self):
self.tries_done = 0
print 'We connected and can get the stuff done!'
def handle_read(self):
data = self.recv(1024)
if not data:
return
# Check for terminator. Can be any action instead of this clause.
if 'END' in data:
self.end = True # Everything went good. Shutdown.
else:
print data # Store to DB or other thing.
def handle_close(self):
print 'Connection closed.'
self.close()
if not self.end:
self.reconnect()
Client('localhost', 6666)
asyncore.loop(timeout=1)
reconnnect() method is somehow core of your case - it's called when connection is needed to be made: when class initializes or connection brokes.
handle_read() operates any recieved data, here you log it or something.
You can even send data using buffer (self.out_buffer += 'message'), which will remain untouched after reconnection, so class will resume sending when connected again.
Setting self.end to True will inform class to quit when possible.
asyncore takes care of exceptions and calls handle_close() when such events occur, which is convenient way of dealing with connection failures.
You should look at the python documentation to understand how classes and methods work. The biggest difference between python methods and methods in most other languages is the addition of the "self" tag. The self represents the instance that a method is called against and is automatically fed in by the python system. So:
class TCPClient():
def __init__(self, host, port, retryAttempts=10 ):
#this is the constructor that takes in host and port. retryAttempts is given
# a default value but can also be fed in.
self.host = host
self.port = port
self.retryAttempts = retryAttempts
self.socket = None
def connect(self, attempt=0):
if attempts<self.retryAttempts:
#put connecting code here
if connectionFailed:
self.connect(attempt+1)
def diconnectSocket(self):
#perform all breakdown operations
...
self.socket = None
def sendDataToDB(self, data):
#send data to db
def readData(self):
#read data here
while True:
if self.socket is None:
self.connect()
...
Just make sure you properly disconnect the socket and set it to None.
python 2.6
Windows 7
I am trying to put together an as simple as possible tutorial of how to write cooperative multitasking programs. As an example application I've written a chat server with python's asyncore backend. I think this will be a valuable resource for the community. However, I have not gotten it to work yet, hence this post.
The structure is as follows. An instance of ChatServer runs on a remote computer. It's socket listens on REMOTE_PORT. When it detects an incoming connection, it spawns an instance of ChatHandler to mediate communication with that connection. Now, who is that connection? On the user's local machine we run an instance of ChatDaemon. This guy listens on LOCAL_PORT. When you connect to him like this
import socket
s = socket.socket()
s.connect(('localhost',LOCAL_PORT))
he detects the connection and spawns two things, a LocalListener and a Connection. The Connection connects to the server, answering our question from above. The LocalListener simply waits for data coming in from the user. If you send data
s.send("Hello, world!")
the LocalListener picks it up, and gives it to the Connection, which then sends it to the ChatServer. The server then puts the data in each ChatHandler's buffer to be sent out to all connected clients. When the Connection receives that data, it passes it to the Daemon who prints it to the screen.
(The Daemon layer seems overly complex but without it you have to do other complicated things to prevent hot loops in asyncore's select() loop whilst keeping latency for the user sending data low. I don't want to go down that road.)
The problem is that the connection to the Daemon does not seem to be made. My exact steps are
In one python session
d = ChatDaemon('localhost')
d.start()
When I do this I see the message "Chat daemon binding to 'localhost: 7668' as expected.
In another python session
import socket
s = socket.socket()
s.connect(('localhost',7668))
When I do this, I do not see the "Got new local connection" line printed.
I have edited my etc/hosts file to map 'localhost' to 127.0.0.1, and I installed the Microsoft Loopback adapter.
EDIT: I have found and fixed the problem. The code below should now be acceptable as a very simple chat implementation using asyncore.
Here is the source
import asyncore
import socket
import sys
LOCAL_HOST = 'localhost'
LOCAL_PORT = 7668
REMOTE_HOST = 'localhost'
REMOTE_PORT = 7667
class LocalListener(asyncore.dispatcher):
"""Receive data from user, putting into cxn's buffer"""
def __init__(self, sock, cxn):
self.cxn = cxn
asyncore.dispatcher.__init__(self, sock)
def writable(self):
return False
def readable(self):
return True
def handle_read(self):
data = self.recv(4096)
if data:
self.cxn.buf = self.cxn.buf + data
class Connection(asyncore.dispatcher):
"""Mediates between user and server"""
def __init__(self, host, port, master):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect((host,port))
self.buf=""
def writable(self):
return len(self.buf) > 0
def readable(self):
return True
def handle_read(self):
data = self.recv(4096)
if data:
self.master.newMessage(data)
def handle_write(self):
sent = self.send(self.buf)
self.buf = self.buf[sent:]
class ChatDaemon(asyncore.dispatcher):
"""Listen for local connections and dispatch in/out data"""
ADDRESS_FAMILY = socket.AF_INET
SOCKET_TYPE = socket.SOCK_STREAM
def __init__(self, remoteHost, remotePort=REMOTE_PORT,
localHost=LOCAL_HOST, localPort=LOCAL_PORT):
self.remoteHost = remoteHost
self.remotePort = remotePort
self.localHost = localHost
self.localPort = localPort
self.buffer = ""
asyncore.dispatcher.__init__(self)
def writable(self):
return False
def readable(self):
return True
def newMessage(self, data):
print data
def start(self):
"""Listen for user connection on local port"""
self.create_socket(self.ADDRESS_FAMILY, self.SOCKET_TYPE)
print("Chat deamon binding to '%s': %s"%(self.localHost,self.localPort))
self.bind((self.localHost,self.localPort))
self.listen(1)
asyncore.loop()
def handle_accept(self):
"""Spawn local reader and remote reader/writer"""
print "Got new local connection"
(connSock, localAddress) = self.accept()
print("New connection address is %s"%localAddress)
#Make a server connection
cxn = Connection(self.remoteHost, self.remotePort, self)
#Connect to local user
LocalListener(connSock, cxn)
### SERVER ###
class ChatHandler(asyncore.dispatcher):
def __init__(self, sock, map, server):
self.server = server
self.buffer = ''
asyncore.dispatcher.__init__(self, sock, map)
def writable(self):
return len(self.buffer) > 0
def readable(self):
return True
def handle_read(self):
"""Notify server of any new incoming data"""
data = self.recv(4096)
if data:
self.server.newMessage(data, self)
def handle_write(self):
"""send some amount of buffer"""
sent = self.send(self.buffer)
self.buffer = self.buffer[sent:]
class ChatServer(asyncore.dispatcher):
"""Receive and forward chat messages
When a new connection is made we spawn a dispatcher for that
connection.
"""
ADDRESS_FAMILY = socket.AF_INET
SOCKET_TYPE = socket.SOCK_STREAM
def __init__(self, host=REMOTE_HOST, port=REMOTE_PORT):
self.map = {}
self.address = (host,port)
self.clients = []
asyncore.dispatcher.__init__(self, map=self.map)
def serve(self):
"""Bind to socket and start asynchronous loop"""
self.create_socket(self.ADDRESS_FAMILY, self.SOCKET_TYPE)
self.bind(self.address)
self.listen(1)
asyncore.loop(map=self.map)
def writable(self):
return False
def readable(self):
return True
def newMessage(self, data, fromWho):
"""Put data in all clients' buffers"""
for client in self.clients:
client.buf = client.buf + data
def handle_accept(self):
"""Deal with newly accepted connection"""
print 'got new connection'
(connSock, clientAddress) = self.accept()
self.clients.append(ChatHandler(connSock, self.map, self))
The problem was that in ChatDaemon I forget the "return" keywords in readable and writable
I am trying to learn how to use sockets and a useful asynchronous backend. I've started in python with asyncore. After reading various online posts I've written a very simple chat server and connection client, reproduced below.
It seems to work. I open a python interactive session and type
> import chatserver
> server = chatserver.EchoServer('localhost', 7667)
> server.serve()
Then I open another IPython interactive session and type
> import chatserver
> cxn = chatserver.Connection()
> cxn._connect('localhost', 7667)
When I do that, I get a log output in the server window indicating that a connection has been made. Good. Then I type
> cxn.say('hi')
Nothing happens for a while, and then log messages show up for the server and client as expected.
Why is this delay ocurring?
Am I using the log functionality correctly?
I used threading to make it so that I could use the interactive session while the asyncore loop does it's thing for the Connection. Did I do this in a reasonable way?
(optional) If I don't include the line self.out_buffer="" in the Connection._connect function I get an error saying that .out_buffer does not exist. What's up with this?
import asyncore
import socket
import logging
import threading
logging.basicConfig(level=logging.DEBUG, format="%(created)-15s %(msecs)d %(levelname)8s %(thread)d %(name)s %(message)s")
log = logging.getLogger(__name__)
class Connection(asyncore.dispatcher_with_send):
def __init__(self):
asyncore.dispatcher.__init__(self)
def _connect(self, host, port, timeout=5, password=None):
self.host = host
self.port = port
self.out_buffer=""
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect((host, port))
#Run the asyncore loop in its own thread so that we can use the interactive session
self.loop = threading.Thread(target=asyncore.loop)
self.loop.daemon = True
self.loop.start()
def say(self, msg):
self.out_buffer = msg
def handle_read(self):
data = self.recv(4096)
log.debug('Received %s'%data)
class EchoHandler(asyncore.dispatcher_with_send):
def handle_read(self):
log.debug("handle_read")
data = self.recv(1024)
log.debug("after recv")
if data:
log.debug("got data: %s"%data)
self.out_buffer = data
else:
log.debug("got null data")
class EchoServer(asyncore.dispatcher):
SOCKET_TYPE = socket.SOCK_STREAM
ADDRESS_FAMILY = socket.AF_INET
def __init__(self, host, port):
self.address = (host,port)
asyncore.dispatcher.__init__(self)
self.create_socket(self.ADDRESS_FAMILY, self.SOCKET_TYPE)
log.debug("bind address=%s %s"%(host,port))
self.bind(self.address)
self.listen(1)
def fileno(self):
return self.socket.fileno()
def serve(self):
asyncore.loop()
#Start asyncore loop in new thread
# self.loop = threading.Thread(target=asyncore.loop)
# self.loop.daemon = True
# self.loop.start()
def handle_accept(self):
"""Deal with a newly accepted client"""
(connSock, clientAddress) = self.accept()
log.info("conn made: clientAddress=%s %s"%(clientAddress[0], clientAddress[1]))
#Make a handler for this connection
EchoHandler(connSock)
def handle_close(self):
self.close()
Looking at the asyncore docs you are relying on asyncore.dispatcher_with_send to call send() and the default timeout for asyncore.loop() is 30 seconds. This may explain the delay.
It turns out the problem was as Eero suggested.
I made two changes:
In EchoServer
asyncore.loop() to asyncore.loop(timeout=0.1)
In Connection
self.loop = threading.Thread(target=asyncore.loop) to self.loop = threading.Thread(target=asyncore.loop, kwargs={'timeout':0.1})
The response is now much faster. This seems like a hack though so if someone can explain a way to get the same effect in a proper way please contribute.
Maybe someone here will have a response for this thing which is just driving me insane.
To make it simple, I'm making a kind of proxy. Whenever it receives something, it forwards everything to a server, and sends back the response. So there is one socket always listening on port 4557 for clients, and for each incoming connection, there is a new socket created on a random port to connect to the server port 4556.
Clients <==> Proxy <==> Server
Also, there another socket which is instantiated and listening for requests coming from the server and to be forwarded to the corresponding client.
Here is an example:
Client A connects to proxy on port 4557
Proxy creates a socket to Server on port 4556
Along with that, it creates a socket listening on port 40100
Client sends stuff, forwarded to Server
Client disconnects. Close client connection and socket to server
Some time later, Server sends stuff to proxy on port 40100
Everything's forwarded to Client A (port 40100 corresponding to Client A)
And so on..
So far in my tests, I use a simple python script for sending a unique tcp packet to the proxy, along with a dump server showing received data and echoing back.
So the issue is that when a connection to the proxy is closed, the connection to the Server should also be closed with "sock.close()". However it just seems to be completely ignored. The socket remains as ESTABLISHED.
About the code now.
A few notes.
DTN and Node are respectively Server and Clients.
runCallback is called in a loop until thread dies.
finalCallback is called when the thread is dying.
Associations between remote hosts (Client), proxy ports (to Server) and proxies are kept in the dictionaries: TCPProxyHostRegister (RemoteHost => Proxy), TCPProxyPortRegister (Port => Proxy), TCPPortToHost (Port => RemoteHost).
The first class is TCPListenerThread.
It just listen on a specific port and instantiate proxies (one for each Client=>Server couple and Server=>Client couple) and forward them connections.
class TCPListenerThread(StoppableThread):
def __init__(self, tcp_port):
StoppableThread.__init__(self)
self.tcp_port = tcp_port
self.sock = socket.socket( socket.AF_INET, # Internet
socket.SOCK_STREAM ) # tcp
self.sock.bind( (LOCAL_ADDRESS, self.tcp_port) )
self.sock.listen(1)
def runCallback(self):
print "Listen on "+str(self.tcp_port)+".."
conn, addr = self.sock.accept()
if isFromDTN(addr):
tcpProxy = getProxyFromPort(tcp_port)
if not tcpProxy:
tcpProxy = TCPProxy(host, True)
else:
host = addr[0]
tcpProxy = getProxyFromHost(host)
if not tcpProxy:
tcpProxy = TCPProxy(host, False)
tcpProxy.handle(conn)
def finalCallback(self):
self.sock.close()
Now comes the TCP Proxy:
It associates a remote host (Client) with a port connecting to Server.
If it's a connection coming from a new Client, it will create a new listener (see above) for the Server and create a socket ready to forward everything to Server.
class TCPProxy():
def __init__(self, remote, isFromDTN):
#remote = port for Server or Remote host for Client
self.isFromDTN = isFromDTN
self.conn = None
#add itself to proxy registries
#If listening from a node
if not isFromDTN:
#Set node remote host
self.remoteHost = remote
TCPProxyHostRegister[self.remoteHost] = self
#Set port to DTN interface + listener
self.portToDTN = getNewTCPPort()
TCPPortToHost[self.portToDTN] = self.remoteHost
newTCPListenerThread(self.portToDTN)
#Or from DTN
else:
self.portToDTN = remote
TCPProxyPortRegister[self.portToDTN] = self
self.remoteHost = getRemoteHostFromPortTCP(self.portToDTN)
def handle(self, conn):
print "New connection!"
#shouldn't happen, but eh
if self.conn != None:
self.closeConnections()
self.conn = conn
#init socket with remote
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
if self.isFromDTN:
self.sock.connect((self.remoteHost, 4556)) #TODO: handle dynamic port..
else:
self.sock.connect((DTN_Address, DTN_TCPPort))
#handle connection in a thread
self.handlerThread = newTCPHandlerThread(self)
#handle reply in a therad
self.replyThread = newTCPReplyThread(self)
def closeConnections(self):
try:
if self.conn != None:
print "Close connections!"
self.sock.close()
self.conn.close()
self.conn = None
self.handlerThread.kill()
self.replyThread.kill()
except Exception, err:
print str(err)
#pass
def forward(self, data):
print "TCP forwarding data: "+data
self.sock.send(data)
def forwardBack(self, data):
print "TCP forwarding data back: "+data
self.conn.send(data)
In this proxy class, I instantiate two classes, TCPHandlerThread and TCPReplyThread. They are responsible for forwarding to Server, and forwarding back to Client, respectively.
class TCPHandlerThread(StoppableThread):
def __init__(self, proxy):
StoppableThread.__init__(self)
self.proxy = proxy
def runCallback(self):
test = False
while 1:
data = self.proxy.conn.recv(BUFFER_SIZE)
if test:
self.proxy.sock.close()
test = True
if not data:
break
print "TCP received data:", data
self.proxy.forward(data)
self.kill()
def finalCallback(self):
self.proxy.closeConnections()
class TCPReplyThread(StoppableThread):
def __init__(self, proxy):
StoppableThread.__init__(self)
self.proxy = proxy
def runCallback(self):
while 1:
data = self.proxy.sock.recv(BUFFER_SIZE)
if not data:
break
print "TCP received back data: "+data
self.proxy.forwardBack(data)
self.kill()
def finalCallback(self):
self.proxy.closeConnections()
You see that whenever a connection is closed, the thread dies and the other connection (Client/Server to proxy or Proxy to Server/Client) should be closed in Proxy.closeConnections()
I noticed that when closeConnections() is "data = self.proxy.conn.recv(BUFFER_SIZE)", it goes well, but when it's called even right after the latter statement, it goes wrong.
I wiresharked TCP, and the proxy doesn't send any "bye signal". The socket state doesn't go to TIME_WAIT or whatever, it just remains ESTABLISHED.
Also, I tested it on Windows and Ubuntu.
On Windows it goes exactly as I explained
On Ubuntu, it works well for usually (not always), 2 connections, and the third time I connect with the same client in exactly the same way to the proxy, it goes wrong again exactly as explained.
Here are the three files i'm using so that you can have a look at the whole code. I'm sorry the proxy file might not be really easy to read. Was SUPPOSED to be a quick dev.
http://hognerud.net/stackoverflow/
Thanks in advance..
It's surely something stupid. Please don't hit me too hard when you see it :(
First I'm sorry that I currently have not the time to actually run and test your code.
But the idea came to my mind, that your problem might actually have something todo with using blocking mode vs. non-blocking mode on the socket. In that case you should checkout the "socket" module help in the python documentation, especially socket.setblocking().
My guess is, that the proxy.conn.recv() function only returns, when actually BUFFER_SIZE bytes where received by the socket. Because of this the thread is blocked until enough data was received and therefore the socket doesn't get closed.
As I said first, this is currently just a guess, so please don't vote me down if it doesn't solve the problem...