I'm new to socket programming (and somewhat to Python too) and I'm having trouble getting the select timeout to work the way I want to (on the server side). Before clients connect, timeout works just fine. I give it a value of 1 second and the timeout expires in my loop every 1 second.
Once a client connects, however, it doesn't wait 1 second to tell me the timeout expires. It just loops as fast as it can and tells me the timeout expires. Here's a snippet of my code:
while running:
try:
self.timeout_expired = False
inputready, outputready, exceptready = select.select(self.inputs, self.outputs, [], self.timeout)
except select.error, e:
break
except socket.error, e:
break
if not (inputready):
# Timeout expired
print 'Timeout expired'
self.timeout_expired = True
# Additional processing follows here
I'm not sure if this is enough code to see where my problem is, so please let me know if you need to see more. Basically, after a client connects, it at least appears that it ignores the timeout of 1 second and just runs as fast as it can, continuously telling me "Timeout expired". Any idea what I'm missing?
Thanks much!!
Edit: I should clarify..."inputready" represents input from a client connecting or sending data to the server, as well as stdin from the server. The other variables returned from select are only server-side variables, and is what I'm trying to do is detect whether the CLIENT took too long to reply, so I'm only checking if inputready is empty.
It is only a timeout if inputready, outputready, and exceptready are ALL empty. My guess is you have added the client socket to both self.inputs and self.outputs. Since the output socket is usually writable, it will always show up in outputready. Only add the client socket to self.outputs if you are ready to output something.
"When the timeout expires, select() returns three empty lists.
...To use a timeout requires adding the extra argument to the select() call and handling the empty lists after select() returns."
readable, writable, exceptional = select.select(inputs, outputs, inputs,timeout)
if not (readable or writable or exceptional):
print(' timed out, do some other work here', file=sys.stderr)
[https://pymotw.com/3/select/index.html][1]
Related
To simplify things, assume a TCP client-server app where the client sends a request and the server responds. The server uses sendall to respond to each client.
Now assume a bad client that sends requests to the server but doesn't really handle the responses. I.e. the client never calls socket.recv. (It doesn't have to be a bad client btw...it may be a slow consumer on the other end).
What ends up happening, is that the server keeps sending responses using sendall, until I'm assuming a buffer gets full, and then at some point sendall blocks and never returns.
This seems like a common problem to me so what would be the recommended solution?
Is there something like a try-send that would raise or return an EWOULDBLOCK (or similar) if the recipient's buffer is full? I'd like to avoid non-blocking select type calls if possible (happy to go that way if there are no alternatives).
Thank you in advance.
Following rveed's comment, here's a solution that works for my case:
def send_to_socket(self, sock: socket.socket, message: bytes) -> bool:
try:
sock.settimeout(10.0) # protect against bad clients / slow consumers by making this timeout (instead of blocking)
res = sock.sendall(message)
sock.settimeout(None) # put back to blocking (if needed for subsequent calls to recv, etc. using this socket)
if res is not None:
return False
return True
except socket.timeout as st:
# do whatever you need to here
return False
except Exception as ex:
# handle other exceptions here
return False
If needed, instead of setting the timeout to None afterwards (i.e. back to blocking), you can store the previous timeout value (using gettimeout) and restore to that.
def mp_worker(row):
ip = row[0]
ip_address = ip
tcp_port = 2112
buffer_size = 1024
# Read the reset message sent from the sign when a new connection is established
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
print('Connecting to terminal: {0}'.format(ip_address))
s.connect((ip_address, tcp_port))
#Putting a breakpoint on this call in debug makes the script work
s.send(":08a8RV;")
#data = recv_timeout(s)
data = s.recv(buffer_size)
strip = data.split("$", 1)[-1].rstrip()
strip = strip[:-1]
print(strip)
termStat = [ip_address, strip]
terminals.append(termStat)
except Exception as exc:
print("Exception connecting to: " + ip_address)
print(exc)
The above code is the section of the script that is causing the problem. It's a pretty simple function that connects to a socket based on a passed in IP from a DB query and receives a response that indicates the hardware's firmware version.
Now, the issue is that when I run it in debug with a breakpoint on the socket I get the entire expected response from the hardware, but if I don't have a breakpoint in there or I full on Run the script it only responds with part of the expected message. I tried both putting a time.sleep() in after the send to see if it would get the entire response and I tried using the commented out recv_timeout() method in there which uses a non-blocking socket and timeout to try to get an entire response, both with the exact same results.
As another note, this works in a script with everything in one main code block, but I need this part separated into a function so I can use it with the multiprocessing library. I've tried running it on both my local Windows 7 machine and on a Unix server with the same results.
I'll expands and reiterate on what I've put into a comment moment ago. I am still not entirely sure what is behind the different behavior in either scenario (apart from timing guess apparently disproved by an attempt to include sleep.
However, it's somewhat immaterial as stream sockets do not guarantee you get all the requested data at once and in chunks as requested. This is up for an application to deal with. If the server closes the socket after full response was sent, you could replace:
data = s.recv(buffer_size)
with recv() until zero bytes were received, this would be equivalent of getting 0 (EOF) from from the syscall:
data = ''
while True:
received = s.recv(buffer_size)
if len(received) == 0:
break
data += received
If that is not the case, you would have to rely on fixed or known (sent in the beginning) size you want to consider together. Or deal with this on protocol level (look for characters, sequences used to signal message boundaries.
I just recently found out a solution here, and thought I'd post it in case anyone else has issue, I just decided to try and call socket.recv() before calling socket.send() and then calling socket.recv() again afterwards and it seems to have fixed the issue; I couldn't really tell you why it works though.
data = s.recv(buffer_size)
s.send(":08a8RV;")
data = s.recv(buffer_size)
I'm having an issue with Python's socket module that I haven't been able to find anywhere else.
I'm building a simple TCP chat client, and while it successfully connects to the server initially, the script hangs endlessly on sock.recv() despite the fact that I explicitly set a timeout length.
I've tried using different timeout values and including setblocking(False) but no matter what I do it keeps acting like the socket is in blocking mode.
Here are the relevant parts of my code:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(1)
def listen_to_server():
global connected
while connected:
ready_to_read, ready_to_write, in_error = select.select([sock], [], [])
if ready_to_read:
try:
data = sock.recv(1024)
except socket.timeout:
print('TIMEOUT')
if not data:
append_to_log('Disconnected from server.\n')
connected = False
else:
append_to_log(str(data))
Any suggestions would be helpful, I'm at a total loss here.
You've mixed two things the socket timeout and the select.
When you set socket timeout then you are telling to that socket: if I try do some operation (e.g. recv()) and it won't be finished until my limit then raise timeout exception.
The select takes file descriptors (on Windows only sockets) and start checking if the rlist (the first parameter) contains any socket ready to read (aka some data have arrived). If any data arrived then the program continues.
Now your code do this:
Set timeout for socket operations
Select start waiting for data (if you don't send them then they never arrives)
and that's it. You stuck at the select.
You should just call the recv() without select. Than your timeout should be applied.
If you need manage multiple sockets at once than you have to use select and set the 4th parameter timeout.
I coded a basic socket system with "select". I want get the list of connected clients instantly.
When the timeout of "select" has passed and several clients come after, it's the drama..
Example - Concerns:
I have 3 clients with one that connects before the timeout, 2 others are connected after the timeout, so I'm going to refresh my list if it took into account two other clients after the timeout.
1st result: I display my variable "list", I see the first socket that is connected before the timeout + one of the other socket who is connected after the timeout. Total: 2 of 3 clients
2nd result: I still re-display my variable "list", and the three clients are there ....
But I want the list without having to re-display the list every time for every customer you can imagine I have 10 clients and I have to show my liste10 times
So I thought to use the asyncore module who is more fluid, what do you think? Do you have a solution for me (easier)? Should I use the multi-threading or stayed on asyncore or select module?
EDIT CODE SOURCE:
import socket, select
hote = ''
port = 81
mainConnection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mainConnection.bind((hote, port))
mainConnection.listen(5)
print("Listen to {}".format(port))
client_online = []
while True:
connection_access, wlist, xlist = select.select([mainConnection], [], [], 10)
for connexion in connection_access:
connection_client, infos_connexion = connexion.accept()
client_online.append(connection_client)
refresh = input(">>> ")
while True:
try:
refresh = int(refresh)
except ValueError:
print("Not allowed")
refresh = int(refresh)
else:
break
if refresh == 1:
print("List client : {}".format(client_online))
There are three major problems with your code:
You call input in your loop. This function will block until ENTER is pressed.
If a non-integer is input from the console, you will get an exception. You handle that exception, but you handle it wrongly. Instead or asking for input again, you simply try to perform the same operation that caused the exception again.
You only check for incoming connection in your select call. You never check if any of the connected sockets have sent anything.
The major problem here for you is the call to input as it will completely stop your program until input from the console is entered.
Your post is very unclear but I can tell you that the problem is that you aren't understanding how to use select.
The code you posted only calls select one time. The program gets to the select() call and waits for mainConnection to be readable (or for the timeout). If mainConnection becomes readable before the timeout, select() returns with one readable file descriptor which you then process in your for loop. But that's it. select is never called again and so your program never checks for any more incoming connections.
In almost every application select should be in a loop. Each time through the loop the program waits in the select() call until one or more sockets is ready for reading or writing. When that happens, select gives you the file descriptors that are ready and it's your job to have other code actually do something. For example, if select returns a socket's file descriptor as readable it's your job to call .recv() on that socket.
You can certainly use asyncore. In fact, I think you should study the source code for asyncore to learn how to properly use select.
I've written a simple multi-threaded game server in python that creates a new thread for each client connection. I'm finding that every now and then, the server will crash because of a broken-pipe/SIGPIPE error. I'm pretty sure it is happening when the program tries to send a response back to a client that is no longer present.
What is a good way to deal with this? My preferred resolution would simply close the server-side connection to the client and move on, rather than exit the entire program.
PS: This question/answer deals with the problem in a generic way; how specifically should I solve it?
Assuming that you are using the standard socket module, you should be catching the socket.error: (32, 'Broken pipe') exception (not IOError as others have suggested). This will be raised in the case that you've described, i.e. sending/writing to a socket for which the remote side has disconnected.
import socket, errno, time
# setup socket to listen for incoming connections
s = socket.socket()
s.bind(('localhost', 1234))
s.listen(1)
remote, address = s.accept()
print "Got connection from: ", address
while 1:
try:
remote.send("message to peer\n")
time.sleep(1)
except socket.error, e:
if isinstance(e.args, tuple):
print "errno is %d" % e[0]
if e[0] == errno.EPIPE:
# remote peer disconnected
print "Detected remote disconnect"
else:
# determine and handle different error
pass
else:
print "socket error ", e
remote.close()
break
except IOError, e:
# Hmmm, Can IOError actually be raised by the socket module?
print "Got IOError: ", e
break
Note that this exception will not always be raised on the first write to a closed socket - more usually the second write (unless the number of bytes written in the first write is larger than the socket's buffer size). You need to keep this in mind in case your application thinks that the remote end received the data from the first write when it may have already disconnected.
You can reduce the incidence (but not entirely eliminate) of this by using select.select() (or poll). Check for data ready to read from the peer before attempting a write. If select reports that there is data available to read from the peer socket, read it using socket.recv(). If this returns an empty string, the remote peer has closed the connection. Because there is still a race condition here, you'll still need to catch and handle the exception.
Twisted is great for this sort of thing, however, it sounds like you've already written a fair bit of code.
Read up on the try: statement.
try:
# do something
except socket.error, e:
# A socket error
except IOError, e:
if e.errno == errno.EPIPE:
# EPIPE error
else:
# Other error
SIGPIPE (although I think maybe you mean EPIPE?) occurs on sockets when you shut down a socket and then send data to it. The simple solution is not to shut the socket down before trying to send it data. This can also happen on pipes, but it doesn't sound like that's what you're experiencing, since it's a network server.
You can also just apply the band-aid of catching the exception in some top-level handler in each thread.
Of course, if you used Twisted rather than spawning a new thread for each client connection, you probably wouldn't have this problem. It's really hard (maybe impossible, depending on your application) to get the ordering of close and write operations correct if multiple threads are dealing with the same I/O channel.
I face with the same question. But I submit the same code the next time, it just works.
The first time it broke:
$ packet_write_wait: Connection to 10.. port 22: Broken pipe
The second time it works:
[1] Done nohup python -u add_asc_dec.py > add2.log 2>&1
I guess the reason may be about the current server environment.
My answer is very close to S.Lott's, except I'd be even more particular:
try:
# do something
except IOError, e:
# ooops, check the attributes of e to see precisely what happened.
if e.errno != 23:
# I don't know how to handle this
raise
where "23" is the error number you get from EPIPE. This way you won't attempt to handle a permissions error or anything else you're not equipped for.