So I'm making a basic "ping" application using UDP for an assignment, and everything is working except the implementation of socket.settimeout().
I can't seem to figure out why, but it has to do with the bound socket. It might just be a quirk in Python but I'd like to confirm it before I document it.
I'm not looking for a functional code answer (that'd be cheating), but rather why what I have is broken. (e.g: some undocumented reason that Python doesn't like client/server on same machine etc)
Python Socket Timeout Details: http://docs.python.org/2/library/socket.html#socket.socket.settimeout
In the code represented below, the communication with server running on same machine is successful, but only when the client does not bind to the socket. But if it does not bind to the socket, the timeout system fails (this being tested by turning off the server, in which all ten timeouts get printed immediately and at once).
Note: Code is not ideal, but this is a networking theory class and not a programming class. It just has to work in the end. I could hand it in right now and get an A, but I want to understand why the timeout function does not work.
EDIT: To clarify an issue, the use of Bind in the client was after seeing the server code had it before I realized UDP doesn't need it, but it happened to make the timeout function work properly, but breaks the normal operation.
Could the socket.settimeout() declaration only work for TCP maybe?
Client Code (which has the timeout):
import socket
import time
import select
data = "Re-verify our range to target... one ping only. \n"
addrDest = ("127.0.0.1",8675)
addrLocal = ("127.0.0.1",12345)
totalTime = 0
averageTime = 0
totalPings = 0
#timeout_seconds = 1.0
UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
UDPSock.bind(addrLocal)
# adding bind here even though it's UDP makes timeout work,
# but breaks normal functionality
UDPSock.settimeout(1)
while (totalPings < 10):
totalPings = (totalPings + 1)
start = time.clock()
str_list = []
str_list.append(str(totalPings))
str_list.append(" ")
str_list.append(str(start))
str_list.append(" ")
str_list.append(data)
dataOut = ''.join(str_list)
UDPSock.sendto(dataOut,addrDest)
try:
dataIn,addrIn = UDPSock.recvfrom(1024)
print dataIn.strip(),"\n",addrIn
elapsed = ((time.clock() - start) * 1000)
print elapsed, " ms round trip"
except socket.error:
print "Connection timed out for Packet #", totalPings
Server Code:
import socket
UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
# (to all IP addresses on this system)
listen_addr = ("",8675)
UDPSock.bind(listen_addr)
# Report on all data packets received and
# where they came from in each case (as this is
# UDP, each may be from a different source and it's
# up to the server to sort this out!)
while True:
data,addr = UDPSock.recvfrom(1024)
print data.strip(),addr
UDPSock.sendto(data,addr)
Why do you need to bind to local address of the client? Will the client act as a server too at any point? If not there is no need to bind the client at all. You need a specific port only if you need your client to act as a server, if you don't call bind it will create a random port no ranging from (0 - 1023 are reserved) 1024 - 65535 (if I remember correctly) and that will be Source Port in the UDP Packet, Source Address is the Address where client runs.
According to Berkley Sockets
bind() assigns a socket to an address. When a socket is created using socket(),
it is only given a protocol family, but not assigned an address. This association with an address must be performed with the bind() system call before the socket can accept connections to other hosts
If this is a Networking class project and you are trying to implement Client-Server architecture then you should never call bind from within your client code because Client should never act as a Server and Client should connect to a listening Server not Server connecting to Client.
Update:
Bind may be required to be called from a TCP Client-Server design but not from a UDP Client-Server model because UDP is a send and forget design and doesn't have low level packet send success acknowledgement. A UDP packet will have Source Address and Port within itself.
I found the cause of the problem by removing the exception handling. There is a socket error when the server is turned off, specifically "socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host" when it tries to read from the socket.
This apparently ignores the timeout function when the socket is not bound in Python (which is why the timeout worked when I bound it).
If I run the server, but just have it not send any data (by commenting the last line), the program times out correctly when it does not receive its packet back.
I am also going to use a more specific exception handler
In the end it's just a quirk in Python and the fact that UDP is connection-less.
Also, someone mentioned the use of "select" to solve this problem. I looked into it, and I would end up with an if...else statement block which kinda works, but the native exceptions are preferred.
Thanks all.
-Jimmy
Related
I have some code which will connect to a host and do nothing but listen for incoming data until either the client is shut down or the host send a close statement. For this my code works well.
However when the host dies without sending a close statement, my client keeps listening for incoming data forever as expected. To resolve this I made the socket timeout every foo seconds and start the process of checking if the connection is alive or not. From the Python socket howto I found this:
One very nasty problem with select: if somewhere in those input lists of sockets is one which has died a nasty death, the select will fail. You then need to loop through every single damn socket in all those lists and do a select([sock],[],[],0) until you find the bad one. That timeout of 0 means it won’t take long, but it’s ugly.
# Example code written for this question.
from select import select
from socket include socket, AF_INET, SOCK_STREAM
socket = socket(AF_INET, SOCK_STREAM)
socket.connect(('localhost', 12345))
socklist = [socket,]
attempts = 0
def check_socklist(socks):
for sock in socklist:
(r, w, e) = select([sock,], [], [], 0)
...
...
...
while True:
(r, w, e) = select(socklist, [], [], 60)
for sock in r:
if sock is socket:
msg = sock.recv(4096)
if not msg:
attempts +=1
if attempts >= 10:
check_socket(socklist)
break
else:
attempts = 0
print msg
This text creates three questions.
I was taught that to check if a connection is alive or not, one has to write to the socket and see if a response returns. If not, the connection has to be assumed it is dead. In the text it says that to check for bad connections, one single out each socket, pass it to select's first parameter and set the timeout to zero. How will this confirm that the socket is dead or not?
Why not test if the socket is dead or alive by trying to write to the socket instead?
What am I looking for when the connection is alive and when it is dead? Select will timeout at once, so having no data there will prove nothing.
I realize there are libraries like gevent, asyncore and twisted that can help me with this, but I have chosen to do this my self to get a better understanding of what is happening and to get more control over the source my self.
If a connected client crashes or exits, but its host OS and computer are still running, then its OS's TCP stack will send your server a FIN packet to let your computer's TCP stack know that the TCP connection has been closed. Your Python app will see this as select() indicating that the client's socket is ready-for-read, and then when you call recv() on the socket, recv() will return 0. When that happens, you should respond by closing the socket.
If the connected client's computer never gets a chance to send a FIN packet, on the other hand (e.g. because somebody reached over and yanked its Ethernet cord or power cable out of the socket), then your server won't realize that the TCP connection is defunct for quite a while -- possibly forever. The easiest way to avoid having a "zombie socket" is simply to have your server send some dummy data on the socket every so often, e.g. once per minute or something. The client should know to discard the dummy data. The benefit of sending the dummy data is that your server's TCP stack will then notice that it's not getting any ACK packets back for the data packet(s) it sent, and will resend them; and after a few resends your server's TCP stack will give up and decide that the connection is dead, at which point you'll see the same behavior that I described in my first paragraph.
If you write something to a socket and then wait for an answer to check the connection, the server should support this "ping" messages. It is not alway the case. Otherwise the server app may crash itself or disconnect your client if the server doesn't wait this message.
If select failed in the way you described, the socket framework knows which socket is dead. You just need to find it. But if a socket is dead by that nasty death like server's app crash, it doesn't mean mandatory that client's socket framework will detect that. E.g. in the case when a client is waiting some messages from the server and the server crashes, in some cases the client can wait forever. For example Putty, to avoid this scenario, can use application's protocol-level ping (SSH ping option) of the server to check the connection; SSH server can use TCP keepalive to check the connection and to prevent network equipment from dropping connections without activity.
(see p.1).
You are right that select's timeout and having no data proves nothing. As documentation says you have to check every socket when select fails.
So, I have a server socket defined as server_sock the current code looks as follow
# define a variable server_sock
server_sock.bind(("", PORT_ANY))
server_sock.listen(1)
port = server_sock.getsockname()[1]
client_sock, client_info = server_sock.accept()
print("[+] Accepted connection from ", client_info)
server_sock.close()
client_sock.close()
# define a variable sock
sock.connect((host, port)) # This will be the client socket
Now, this code will create a server_sock , listen for incoming connections and after any client is connected it will close those socket and act as client by using another sock.
What I am planning to do is to first let the code run as a server (i.e server_sock should wait for connections) for specific timeout (Let's assume 10 seconds).
After the 10 seconds, the server_sock should get closed by itself and then the next piece of code (i.e client part) should start.
So, it's loosely something like change from Server Mode to Client Mode after a specific Time-Out.
I am having a hard time to solve this issue. Usually server_sock.accept() line would be stuck until it hits a new connection else it won't proceed.
So, how can I implement something which breaks that whole thing after a specific time-out.
Note that I am running this code cross platform on Windows and UNIX. So, I have been looking to some signal specific answers but Windows doesn't support some Signal.
EDIT:
Many people are saying to use settimeout() on the socket. But that doesn't answer the behavior I need.
Because Let's say if I get a connection from a device during the server_sock mode, I would like to continue communication thereafter rather than abruptly closing the socket.
The settimeout() will close socket no matter what actions are being performed. So, that fails to answer my case
You can simply use a timeout function.
# define a socket variable server_sock
server_sock.settimeout(10.0) #setting timeout for 10 sec
# rest of your code
I'm trying to make a python function that scans a range of addresses. I started a socket and pass the socket as an argument to the function that connects to it:
def scan(socket, address, port):
c = socket.connect_ex((address, port))
print(c)
then I call scan for each address, each in its own thread. I'm getting Error 114: Operation already in progress..
Do I need to start a new socket for each connection? I'm trying to read about socket reusage, and I found that there exists flags like SO_ADDREUSE or something like that. I tried to insert but it didn't work.
I'm trying to think how a socket works. I think the moment I create one, it choses a tcp source port, and then when I create a connection, it sends to a destination port. I think I can't reuse the same socket because the source port would be the same for all destination ports, so the clients would answer to the same port and would cause confusion.
So do I need to create a new socket for each connection?
You can not connect stream socket multiple times.
One of the connect possible errors is EISCONN.
The socket is already connected.
This goes for stream sockets.
man bind also has this:
[EINVAL] The socket is already bound to an address, and the
protocol does not support binding to a new address; or
the socket has been shut down.
Again, this goes for stream sockets.
From the man connect:
Generally, stream sockets may successfully connect() only once; datagram sockets may use connect() multiple times to change their association.
I made emphasis on the important line.
stream sockets can not be connected multiple times. datagram sockets can be connected multiple times. Generally speaking, BSD sockets have multiple protocols, types, domains avaible. You shall read documentation for your particular case.
P.S Get yourself familiar with the readings that were suggested in the comment to your question. That will explain enough to manipulate socket family of functions.
Do I need to start a new socket for each connection?
Yes.
I'm trying to read about socket reusage
There is no such thing as 'socket reusage'. There is port reuse. Not the same thing. You cannot reconnect an existing socket once you've tried to connect it, even if the connect attempt failed.
I found that there exists flags like SO_ADDREUSE or something like that
SO_REUSEADDR means to reuse the port. Not the socket.
I'm trying to think how a socket works. I think the moment I create one, it choses a tcp source port,
Between creating a socket using the socket() system call and using it to create an outgoing connection with the connect() system call, there is an opportunity to optionally use the bind() system call to set source IP address and/or port if you want to. If you don't use bind(), the operating system will automatically bind the socket to the first available port in the appropriate range when you use the connect() system call. In this case, the source IP address is normally selected to match the network interface that provides the shortest route to the specified destination according to the routing table.
At least, that's how it works at the system call level. Some programming languages or libraries may choose to combine some of these operations into one.
To your actual question, man 7 ip says:
A TCP local socket address that has been bound is unavailable for some
time after closing, unless the SO_REUSEADDR flag has been set. Care
should be taken when using this flag as it makes TCP less reliable.
The idea is to delay the re-use of a port until any possible re-sent copies of packages that belonged to the closed connection have for sure expired on the network.
According to the bind() man page, trying to re-bind a socket that is already bound to an address will result in an EINVAL error. So "recycling" a socket using bind(socket, INADDR_ANY, 0) (after ending a connection that used SO_REUSEADDR) does not seem to be possible.
And even if that would be possible, when you're using multiple threads on a modern multi-core system, you end up (very probably) doing multiple things in parallel. A socket can be used for just one outgoing connection at a time. Each of your scan threads will need its own socket.
I send mouse coordinates from python server to python client via socket. Mouse coordinates are send every time when mouse movement event is catch on the server which means quite often (dozen or so per second).
Problem is when I use python server and python client on different hosts. Then only part of messages are delivered to the client.
e.g. 3 first messages are delivered, 4 messages aren't delivered, 4 messages are delivered etc...
Everything is fine when server and client are on the same host (localhost).
Everything is fine when server and client are on different hosts but instead of python client I use standard windows Telnet client to read messages from the server.
I noticed that when I use time.sleep(0.4) break between each message that is send then all messages are delivered. Problem is I need that information in real time not with such delay. Is it possible to achieve that in Python using sockets?
Below python client code that I use:
import pickle
import socket
import sys
host = '192.168.1.222'
port = 8888
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
print "Faile. Error:" + str(msg[0]), "Error message : " + msg[1]
sys.exit()
mySocket = socket.socket()
mySocket.connect((host,port))
while 1:
data = mySocket.recv(1024)
if not data: break
load_data = pickle.loads(data)
print 'parametr x: ' + str(load_data[0])
print 'parametr y : ' + str(load_data[1])
mySocket.close()
You are using TCP (SOCK_STREAM) which is a reliable protocol which (contrary to UDP) does not loose any messages, even if the recipient is not reading the data fast enough. Instead TCP will reduce the sending speed.
This means that the problem must be somewhere in your application code.
One possibility is that the problem is in your sender, i.e. that you use socket.send and do not check if all the bytes you've intended to send are really got send. But this check needs to be done since socket.send might only send part of the data if the socket buffer of the OS is full which can happen if the client does not read the data fast enough.
Another possibility is that your socket.recv call receives more data than your pickle.loads needs and that the rest of the data gets discarded (not sure if pickle.loads will throw an exception if too much data are provided). Note that TCP is not a message but a stream protocol so it might be that you have more that socket.recv will return a buffer which contains more than one pickled object but you only read the first. The chance that this will happen on a network is higher than on localhost because by default the TCP layer will try to concatenate multiple send buffers into a single TCP packet for better use of the connection (i.e. less overhead). And the chance is high that these will then be received within the same recv call. By putting a sleep(0.4) on the sender side you've effectively switch off this optimization of TCP, see NAGLE algorithm for details.
Thus the correct way to implement what you want would be:
Make sure that all data are delivered at the server, i.e. check the return of socket.send.
Make sure that you unpack all messages you receive. To do this you probable need to add some message layer on top of the TCP stream to find out where the message boundary is.
I have some code which will connect to a host and do nothing but listen for incoming data until either the client is shut down or the host send a close statement. For this my code works well.
However when the host dies without sending a close statement, my client keeps listening for incoming data forever as expected. To resolve this I made the socket timeout every foo seconds and start the process of checking if the connection is alive or not. From the Python socket howto I found this:
One very nasty problem with select: if somewhere in those input lists of sockets is one which has died a nasty death, the select will fail. You then need to loop through every single damn socket in all those lists and do a select([sock],[],[],0) until you find the bad one. That timeout of 0 means it won’t take long, but it’s ugly.
# Example code written for this question.
from select import select
from socket include socket, AF_INET, SOCK_STREAM
socket = socket(AF_INET, SOCK_STREAM)
socket.connect(('localhost', 12345))
socklist = [socket,]
attempts = 0
def check_socklist(socks):
for sock in socklist:
(r, w, e) = select([sock,], [], [], 0)
...
...
...
while True:
(r, w, e) = select(socklist, [], [], 60)
for sock in r:
if sock is socket:
msg = sock.recv(4096)
if not msg:
attempts +=1
if attempts >= 10:
check_socket(socklist)
break
else:
attempts = 0
print msg
This text creates three questions.
I was taught that to check if a connection is alive or not, one has to write to the socket and see if a response returns. If not, the connection has to be assumed it is dead. In the text it says that to check for bad connections, one single out each socket, pass it to select's first parameter and set the timeout to zero. How will this confirm that the socket is dead or not?
Why not test if the socket is dead or alive by trying to write to the socket instead?
What am I looking for when the connection is alive and when it is dead? Select will timeout at once, so having no data there will prove nothing.
I realize there are libraries like gevent, asyncore and twisted that can help me with this, but I have chosen to do this my self to get a better understanding of what is happening and to get more control over the source my self.
If a connected client crashes or exits, but its host OS and computer are still running, then its OS's TCP stack will send your server a FIN packet to let your computer's TCP stack know that the TCP connection has been closed. Your Python app will see this as select() indicating that the client's socket is ready-for-read, and then when you call recv() on the socket, recv() will return 0. When that happens, you should respond by closing the socket.
If the connected client's computer never gets a chance to send a FIN packet, on the other hand (e.g. because somebody reached over and yanked its Ethernet cord or power cable out of the socket), then your server won't realize that the TCP connection is defunct for quite a while -- possibly forever. The easiest way to avoid having a "zombie socket" is simply to have your server send some dummy data on the socket every so often, e.g. once per minute or something. The client should know to discard the dummy data. The benefit of sending the dummy data is that your server's TCP stack will then notice that it's not getting any ACK packets back for the data packet(s) it sent, and will resend them; and after a few resends your server's TCP stack will give up and decide that the connection is dead, at which point you'll see the same behavior that I described in my first paragraph.
If you write something to a socket and then wait for an answer to check the connection, the server should support this "ping" messages. It is not alway the case. Otherwise the server app may crash itself or disconnect your client if the server doesn't wait this message.
If select failed in the way you described, the socket framework knows which socket is dead. You just need to find it. But if a socket is dead by that nasty death like server's app crash, it doesn't mean mandatory that client's socket framework will detect that. E.g. in the case when a client is waiting some messages from the server and the server crashes, in some cases the client can wait forever. For example Putty, to avoid this scenario, can use application's protocol-level ping (SSH ping option) of the server to check the connection; SSH server can use TCP keepalive to check the connection and to prevent network equipment from dropping connections without activity.
(see p.1).
You are right that select's timeout and having no data proves nothing. As documentation says you have to check every socket when select fails.