I'm using a TCP socket to connect to a certain web page, as it will be running in a loop trying to create socket, connect, send data, and receive then close the socket. the problem is s.connect() times out at random iterations and specific web site. i read about TCP time_wait so
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
was added with no luck. then one second of sleep after each iteration, but still no luck.
i should note that it works fine in another slower network, but not in the fast one i'm trying to run the code.
i think following pseudo code can clarify it more:
while(1)
{time.sleep(1)
s = socket.socket() #TCP socket creation
s.settimeout(2) # 2 seconds for time-out which is more than needed
s.connect()
s.send(message) #which is a HTTP get request
s.recv() # which is a HTTP response
s.close()}
my code routine follows these steps, each of socket methods are implemented correctly, but i get time-outs on connect. i am sure about server functioning correctly. i suspected it might have something to do with TCP time_wait , but comments don't seem to agree
A connect times out if the TCP handshake with the server is not finished within a specific time, usually because the server does not respond in time. There might be various reasons for is, like the server being down, getting the wrong IP address for the server during DNS lookup, server being overloaded etc.
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Adding this on the client side on the connection has no use.
Adding this on the server side will help if the problem was caused by a crashed server since it will allow the server to bind to the listener address again without waiting and thus the server is faster available again. But, I have no idea if this is really the problem you are facing.
Related
I have some code which will connect to a host and do nothing but listen for incoming data until either the client is shut down or the host send a close statement. For this my code works well.
However when the host dies without sending a close statement, my client keeps listening for incoming data forever as expected. To resolve this I made the socket timeout every foo seconds and start the process of checking if the connection is alive or not. From the Python socket howto I found this:
One very nasty problem with select: if somewhere in those input lists of sockets is one which has died a nasty death, the select will fail. You then need to loop through every single damn socket in all those lists and do a select([sock],[],[],0) until you find the bad one. That timeout of 0 means it won’t take long, but it’s ugly.
# Example code written for this question.
from select import select
from socket include socket, AF_INET, SOCK_STREAM
socket = socket(AF_INET, SOCK_STREAM)
socket.connect(('localhost', 12345))
socklist = [socket,]
attempts = 0
def check_socklist(socks):
for sock in socklist:
(r, w, e) = select([sock,], [], [], 0)
...
...
...
while True:
(r, w, e) = select(socklist, [], [], 60)
for sock in r:
if sock is socket:
msg = sock.recv(4096)
if not msg:
attempts +=1
if attempts >= 10:
check_socket(socklist)
break
else:
attempts = 0
print msg
This text creates three questions.
I was taught that to check if a connection is alive or not, one has to write to the socket and see if a response returns. If not, the connection has to be assumed it is dead. In the text it says that to check for bad connections, one single out each socket, pass it to select's first parameter and set the timeout to zero. How will this confirm that the socket is dead or not?
Why not test if the socket is dead or alive by trying to write to the socket instead?
What am I looking for when the connection is alive and when it is dead? Select will timeout at once, so having no data there will prove nothing.
I realize there are libraries like gevent, asyncore and twisted that can help me with this, but I have chosen to do this my self to get a better understanding of what is happening and to get more control over the source my self.
If a connected client crashes or exits, but its host OS and computer are still running, then its OS's TCP stack will send your server a FIN packet to let your computer's TCP stack know that the TCP connection has been closed. Your Python app will see this as select() indicating that the client's socket is ready-for-read, and then when you call recv() on the socket, recv() will return 0. When that happens, you should respond by closing the socket.
If the connected client's computer never gets a chance to send a FIN packet, on the other hand (e.g. because somebody reached over and yanked its Ethernet cord or power cable out of the socket), then your server won't realize that the TCP connection is defunct for quite a while -- possibly forever. The easiest way to avoid having a "zombie socket" is simply to have your server send some dummy data on the socket every so often, e.g. once per minute or something. The client should know to discard the dummy data. The benefit of sending the dummy data is that your server's TCP stack will then notice that it's not getting any ACK packets back for the data packet(s) it sent, and will resend them; and after a few resends your server's TCP stack will give up and decide that the connection is dead, at which point you'll see the same behavior that I described in my first paragraph.
If you write something to a socket and then wait for an answer to check the connection, the server should support this "ping" messages. It is not alway the case. Otherwise the server app may crash itself or disconnect your client if the server doesn't wait this message.
If select failed in the way you described, the socket framework knows which socket is dead. You just need to find it. But if a socket is dead by that nasty death like server's app crash, it doesn't mean mandatory that client's socket framework will detect that. E.g. in the case when a client is waiting some messages from the server and the server crashes, in some cases the client can wait forever. For example Putty, to avoid this scenario, can use application's protocol-level ping (SSH ping option) of the server to check the connection; SSH server can use TCP keepalive to check the connection and to prevent network equipment from dropping connections without activity.
(see p.1).
You are right that select's timeout and having no data proves nothing. As documentation says you have to check every socket when select fails.
I'm building an RPC Server in golang that uses msgpack.
The client is built in python using the mprpc library (msgpack over TCP with gevent).
My issue is, being an absolute noob in networking, I discovered that I can't use the same address/port with multiple clients running at once on the same computer (socket already bound i guess, it just stalls and timeouts).
I have looked around quite a bit but I'm not sure what I should be doing to be able to have multiple clients on the same machine talk to a server (msgpack back and forth). Is this a case where I need to use ZeroMQ ? Or requests over HTTP ?
Thanks !
TCP is a connection-oriented protocol. This means that only the server needs to have a fixed, known port. The client can use any port it wants, because nobody is making a connection to the client.
So, how does the server know how to talk to the client? Whenever it accepts a connection, it's told who the connection is from. But usually, you don't even need that, because the socket keeps track of who the connection is from. Just recv and send on that socket, and you're talking to the right client.
You should probably read the Socket Programming HOWTO in the Python docs, or some other tutorial, but briefly…
A server starts like this:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', 12345))
sock.listen(5)
while True:
csock, addr = sock.accept()
It binds a port and listens and loops around accepting connections and doing something with them.
A client, on the other hand, just does this:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('localhost', 12345))
… or, equivalently:
sock = socket.create_connection(('localhost', 12345))
It doesn't call bind, it just creates a connection, letting the sockets library pick an arbitrary port on the appropriate interface for that connection. Unless you've got thousands of sockets already open, it should always be able to find a free port for you.
If you want to have two way connection, then HTTP is not suitable for this. Because HTTP is designed in a way that the server only responds to a request, which prevents server to issue a request itself. There are other solutions that provide two way connection(server to client and client to server in same time).
WebSocket is the first thing that comes to my mind. Of course ZeroMQ also can do this.
I am trying to do some simple sockets programming in Python. I have a UDP server in Python that accepts an input and sends a response. I'm having trouble with the client code.
sock = socket.socket(
socket.AF_INET, socket.SOCK_DGRAM
)
sock.bind(('0.0.0.0', 0))
sock.settimeout(2)
sock.sendto(json.dumps({
'operation': operation,
'operands': [operand1, operand2]
}), (host, port))
print sock.recvfrom(4096)
This code works perfectly when the server is running. However, when I try it without the server running, the code throws an exception immediately instead of blocking on recvfrom().
socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host
My desired functionality would be to timeout after some time.
Can anyone explain what I am doing wrong? I use recvfrom in the server code and it blocks, so I'm a little puzzled at what the difference is.
Much of the former answer doesn't apply, since you have a SOCK_DGRAM type, i. e. UDP socket. UDP is a connectionless service, even though the error message talks about connection. The exception occurs due to a notification from the destination host in response to the sendto datagram, telling that the port is unreachable; this notification is processed (perhaps even arrives) not before you call recvfrom. There is no automatic retry; you have to try the recvfrom (maybe as well the sendto) again and implement the desired timeout yourself.
There are plenty possible reasons for that.
Socket error [10054] means connection reset by peer.
An existing connection was forcibly closed by the remote host. This normally results if the peer application on the remote host is suddenly stopped, the host is rebooted, the host or remote network interface is disabled, or the remote host uses a hard close (see setsockopt for more information on the SO_LINGER option on the remote socket). This error may also result if a connection was broken due to keep-alive activity detecting a failure while one or more operations are in progress. Operations that were in progress fail with WSAENETRESET. Subsequent operations fail with WSAECONNRESET.
Check this link for details.
You need to look into it to figure out what actually happened. I don't have your environment so I cannot locate what the real problem is.
Hope this helps.
I have a client-server "snake" game working really well with TCP connections, and I would like to try it the UDP way.
I wonder how it is supposed to be used ? I know how UDP works, how to make a simple ECHO example, but I wonder how to do the following :
For instance with TCP, every TICK (1/15 second) server sends to the client the new Snake head position.
With UDP, am I supposed to do something like this :
Client SIDE :
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
serverAddr = (('localhost', PORT))
while 1:
client.sendto('askForNewHead', serverAddr)
msg, addrServer = client.recvfrom(1024)
game.addPosition(msg)
Server SIDE :
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind((HOST, PORT))
while 1:
data, addr = server.recvfrom(1024)
server.sendto(headPosition, addr)
So here Client has to ask server to get the new head position, and then server sends the answer. I managed to make it work this way, but I can't figure out if it is a good way of doing.
Seems weird that client has to ask udp for an update while with my TCP connection, client has just to wait untill he receives a message.
There are differences between TCP and UDP but not the way you describe. Like with TCP the client can recvfrom to get messages from the server without asking each time for new data. The differences are:
With TCP the initial connect includes a packet exchange between client and server. Unless the client socket was already bound to an IP and port it will be bound to the clients IP and a free port will be allocated. Because of the handshake between client and server the server knows where to contact the client and thus can send data to the packet without getting data from the client before.
With UDP there is no initial handshake. Unless already bound, the socket will be bound to clients IP and a free port when sending the first packet to the server. Only when receiving this packet the server knows the IP and port of the client and can send data back.
Which means, that you don't need to 'askForNewHead' all the time. Instead the client has to send only a single packet to the server so that the server knows where to send all future packets.
But there are other important differences between TCP and UDP:
With UDP packets may be lost or could arrive in a different order. With TCP you have a guaranteed delivery.
With UDP there is no real connection, only an exchange of packets between two peers. With TCP you have the start and end of a connection. This is relevant for packet filters in firewalls or router, which often need to maintain the state of a connection. Because UDP has no end-of-connection the packet filters will just use a simple timeout, often as low as 30 seconds. Thus, if the client is inside a home network and waits passively for data from server, it might wait forever if the packet filter closed the state because of the timeout. To work around this data have to be transmitted in regular intervals so that the state does not time out.
One often finds the argument, that UDP is faster then TCP. This is plain wrong. But you might see latency problems if packets get lost because TCP will notice packet loss and send the packet again and also reduce wire speed to loose less packets. With UDP instead you have to deal with the packet loss and other congestion problems yourself. There are situations like real time audio, where it is ok to loose some packets but low latency is important. These are situations where UDP is good, but in most other situations TCP is better.
UDP is different to TCP, and I believe with python the client does have to ask for an update from the server.
Although it is fun to learn and use a different way of communicating over the internet, for python I would really recommend sticking with TCP.
You don't have to ask the server for a update. But since UDP is connection-less the server can send head-positions without being asked. But the client should send i'm-alive-packets to the server, but this could happen every 10 seconds or so.
I have some code which will connect to a host and do nothing but listen for incoming data until either the client is shut down or the host send a close statement. For this my code works well.
However when the host dies without sending a close statement, my client keeps listening for incoming data forever as expected. To resolve this I made the socket timeout every foo seconds and start the process of checking if the connection is alive or not. From the Python socket howto I found this:
One very nasty problem with select: if somewhere in those input lists of sockets is one which has died a nasty death, the select will fail. You then need to loop through every single damn socket in all those lists and do a select([sock],[],[],0) until you find the bad one. That timeout of 0 means it won’t take long, but it’s ugly.
# Example code written for this question.
from select import select
from socket include socket, AF_INET, SOCK_STREAM
socket = socket(AF_INET, SOCK_STREAM)
socket.connect(('localhost', 12345))
socklist = [socket,]
attempts = 0
def check_socklist(socks):
for sock in socklist:
(r, w, e) = select([sock,], [], [], 0)
...
...
...
while True:
(r, w, e) = select(socklist, [], [], 60)
for sock in r:
if sock is socket:
msg = sock.recv(4096)
if not msg:
attempts +=1
if attempts >= 10:
check_socket(socklist)
break
else:
attempts = 0
print msg
This text creates three questions.
I was taught that to check if a connection is alive or not, one has to write to the socket and see if a response returns. If not, the connection has to be assumed it is dead. In the text it says that to check for bad connections, one single out each socket, pass it to select's first parameter and set the timeout to zero. How will this confirm that the socket is dead or not?
Why not test if the socket is dead or alive by trying to write to the socket instead?
What am I looking for when the connection is alive and when it is dead? Select will timeout at once, so having no data there will prove nothing.
I realize there are libraries like gevent, asyncore and twisted that can help me with this, but I have chosen to do this my self to get a better understanding of what is happening and to get more control over the source my self.
If a connected client crashes or exits, but its host OS and computer are still running, then its OS's TCP stack will send your server a FIN packet to let your computer's TCP stack know that the TCP connection has been closed. Your Python app will see this as select() indicating that the client's socket is ready-for-read, and then when you call recv() on the socket, recv() will return 0. When that happens, you should respond by closing the socket.
If the connected client's computer never gets a chance to send a FIN packet, on the other hand (e.g. because somebody reached over and yanked its Ethernet cord or power cable out of the socket), then your server won't realize that the TCP connection is defunct for quite a while -- possibly forever. The easiest way to avoid having a "zombie socket" is simply to have your server send some dummy data on the socket every so often, e.g. once per minute or something. The client should know to discard the dummy data. The benefit of sending the dummy data is that your server's TCP stack will then notice that it's not getting any ACK packets back for the data packet(s) it sent, and will resend them; and after a few resends your server's TCP stack will give up and decide that the connection is dead, at which point you'll see the same behavior that I described in my first paragraph.
If you write something to a socket and then wait for an answer to check the connection, the server should support this "ping" messages. It is not alway the case. Otherwise the server app may crash itself or disconnect your client if the server doesn't wait this message.
If select failed in the way you described, the socket framework knows which socket is dead. You just need to find it. But if a socket is dead by that nasty death like server's app crash, it doesn't mean mandatory that client's socket framework will detect that. E.g. in the case when a client is waiting some messages from the server and the server crashes, in some cases the client can wait forever. For example Putty, to avoid this scenario, can use application's protocol-level ping (SSH ping option) of the server to check the connection; SSH server can use TCP keepalive to check the connection and to prevent network equipment from dropping connections without activity.
(see p.1).
You are right that select's timeout and having no data proves nothing. As documentation says you have to check every socket when select fails.