I'm running a Twisted server with the LineReceiver protocol. Sometimes clients will disconnect silently, so Twisted keeps the connection open. And because the server doesn't send anything unless requested of it, there's never a TCP timeout. In other words, some connections are never closed server-side.
How can I have Twisted close a connection that's been inactive for a few hours?
You can schedule timed events using reactor.callLater. Based on this, there's a helper for adding timeouts to protocols, twisted.protocols.policies.TimeoutMixin.
Another approach is to use TCP keep-alives, which you can enable using the transport's setTcpKeepAlive method.
And another approach is to use application-level keep-alives. Essentially send a ''noop'' once in a while. It doesn't need a response. If the connection has been lost, the extra data in the send buffer will cause the TCP stack to eventually notice.
See also the FAQ entry.
Related
I have a basic implementation of a TCP client using python sockets, all the client does is connect to a server and send heartbeats every X seconds. The problem is that I don't want to send the server a heartbeat if the connection is closed, but I'm not sure how to detect this situation without actually sending a heartbeat and catch an exception. When I turn off the server, in the traffic capture I see FIN/ACK arriving and the client sends an ACK back, this is when I want my code to do something (or at least change some internal state of the connection). Currently, what happens is after the server went down and X seconds passed since last heartbeat the client will try to send another heartbeat, only then I see RST packet in the traffic capture and get an exception of broken pipe (errno 32). Clearly python socket handles the transport layer and the heartbeats are part of application layer, the problem I want to solve is not to send the redundant heartbeat after FIN/ACK arrived from server, any simple way to know the connection state with python socket?
I've written a tcp client using python and twisted, it connects to a server and communicate in a simple string based protocol (Defined by the server manufacturer). The TCP/IP connection should persist, and reconnect in case of failure.
When some sort of network error occurs (I assume on the server side or on some node along the way), it takes a very long time for the client to realize that and initiate a new connection, much more than a few minutes.
Is there a way to speed that up? Some sort of built in TCP/IP keep alive functionality that can detect the disconnect sooner?
I can implement a keep alive mechanism myself, and look for timeouts, not sure that's the best practice in this case. What do you think? Also, when using reactor.connectTCP() and reactor.run() with a ClientFactory, what's the best way to force a re-connection?
Application level keep-alives for TCP-based protocols are a good idea. You should probably implement this. This gives you complete and precise control over the timeout semantics you want from your application.
TCP itself has a keepalive mechanism. You can enable this with an ITCPTransport method call from your protocol. For example:
class YourProtocol(Protocol):
def connectionMade(self):
self.transport.setTcpKeepAlive(True)
The exact semantics of this keepalive are platform and configuration dependent. It's entirely possible this is already enabled and is what's detecting your connection lose. Thirty minutes is a pretty plausible amount of time for this mechanism to notice a lost connection.
As stated in by Jean-Paul Calderone, you can either implement an application level keepalive or use the TCP keepalive mechanism. The application level keepalive is the preferred method as it gives you more fine-grained control.
The TCP keepalive mechanism lives on the OS level and the defaults are OS dependant, but are configurable. For example the default linux TCP keepalive works in the following way:
After 2 hours send a keepalive probe.
If this fails, send another probe every 75 seconds.
After 9 consecutive fails, mark the connection as closed. This will be picked up by the server and it will trigger whatever cleanup mechanisms it has in place.
See: https://en.wikipedia.org/wiki/Keepalive#TCP_keepalive and http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html
So while the TCP keepalive will eventually reap your dead connections, it will take quite a long time to kick in.
I'm working in a NetHack clone that is supposed to be playing through Telnet, like many NetHack servers. As I've said, this is a clone, so it's being written from scratch, on Python.
I've set up my socket server reusing code from a SMTP server I wrote a while ago and all of suddenly my attention jumped to this particular line of code:
s.listen(15)
My server was designed to be able to connect to 15 simultaneous clients just in case the data exchange with any took too long, but ideally listen(1) or listen(2) would be enough. But this case is different.
As it happens with Alt.org when you telnet their NetHack servers, people connected to my server should be able to play my roguelike remotely, through a single telnet session, so I guess this connection should not be interrupted. Yet, I've read here that
[...] if you are really holding more than 128 queued connect requests you are
a) taking too long to process them or b) need a heavy-weight
distributed server or c) suffering a DDoS attack.
What is the better practice to carry out here? Should I keep every connection open until the connected user disconnects or is there any other way? Should I go for listen(128) (or whatever my system's socket.SOMAXCONN is) or is that a bad practice?
number in listen(number) request limits number of pending connect requests.
Connect request is pending from initial SYN request received by OS until you called accept socket method. So number does not limits open (established) connection number but it limits number of connections in SYN_RECV state.
It is bad idea not to answer on incoming connection because:
Client will retransmit SYN requests until answer SYN is received
Client can not distinguish situation when your server is not available and it just in queue.
Better idea is to answer on connection but send some message to client with rejection reason and then close connection.
How can I create a UDP server in Python that is possible to know when a client has disconnected? The server needs to be fast because I will use in an MMORPG. Never did a UDP server then I have a little trouble.
There is no such thing as a connection in UDP. Because of this, it becomes your responsibility to detect if the client has disconnected. Generally speaking, your protocol should implement a way to notify the server that it is ending its session. Additionally, you will need to implement some type of timeout functionality such that after a certain period of interactivity, the session is ended.
Note that UDP is more difficult to work with than TCP because packets are not always guaranteed to be delivered. Depending on what you are doing, you might need to implement some type of check to ensure that packets that are not delivered are sent again. TCP does this for you, but it also has the side effect of making the protocol slower.
This answer provides some more considerations: https://stackoverflow.com/a/57489/4250606
UDP is not connection-based. Since no connection exists when using UDP, there is nothing to disconnect. Since there is nothing to disconnect, you can't ever know when something disconnects. It never will because it was never connected in the first place.
I wrote a server based on Twisted, and I encountered a problem, some of the clients are disconnected not gracefully. For example, the user pulls out the network cable.
For a while, the client on Windows is disconnected (the connectionLost is called, and it is also written in Twisted). And on the Linux server side, my connectionLost of twisted is never triggered. Even it try to writes data to client when the connection is lost. Why Twisted can't detect those non-graceful disconnection (even write data to client) on Linux? How to makes Twisted detect non-graceful disconnections? Because the feature Twisted can't detect non-graceful, I have lots of zombie user on my server.
---- Update ----
I thought it might be the feature of socket of unix-like os, so, what is the behavior of socket on unix-like for handling situation like this?
Thanks.
Victor Lin.
You're describing the behavior of TCP connections on an unreliable network. Twisted is merely exposing this behavior: after all, when you set up a TCP connection with Twisted, it is nothing more than a TCP connection.
You're mistaken when you say that the connectionLost callback isn't invoked even if you try to send data over it. After two minutes, the underlying TCP connection will disappear and Twisted will inform you of this by calling connectionLost.
If you need to detect this condition more quickly than that, then you can implement your own timeouts using reactor.callLater.
Seconding what Jean-Paul said, if you need more fine grained TCP connection management, just use reactor.CallLater. We have exactly that implementation on a Twisted/wxPython trading platform, and it works a treat. You might also want to tweak the behaviour of the ReconnectingClientFactory in order to achieve the results I understand your looking for.