I am trying to stop recv from waiting endlessly for input.
First I tried:
recv = bytes('','UTF-8')
while True:
data_recv = self.socketclient.recv(1024*768)
if not data_recv:
break
else:
recv += data_recv
return recv
On Serverside I send a picture and then the server just waits after host.sendall(string).
So I thought after a couple of receives (since the picture is bigger the client has to receive more often) it will detect data_recv == false and stops but it doesn't happen.
My second try was with select()
do_read = False
recv = bytes('','UTF-8')
while True:
read_sockets,write_sockets,error_sockets = select.select([self.socketclient],[],[])
do_read = bool(read_sockets)
print (do_read)
if do_read:
print ("read")
data_recv = self.socketclient.recv(640*480)
recv += data_recv
else:
break
return recv
With this he only reads True from print(do_read) and then also just stops and waits endlessly. Here too I had expected to read False at some point.
So my question is how do I manage it, that my client reads the entire string from the socket and if nothing is send anymore it stops?
I had some success with self.socketclient.settimeout() but I rather would not use it since it will always waste time in the programm and it is more like a workaround.
You have to stop your while loop when all data is received. So after each recv you have to check if received data length is the length of file you requested. If it is, then you break the loop and do something with that data.
recv has no knowledge if sending of data is done, because it doesn't know how long data is. It is just listening forever until you or other endpoint close connection.
You can also use non-blocking socket (which is better by the way).
Related
I am using Python's select library to asynchronously read data off of two sockets. Since the size of the packets that I receive can vary, and I don't know if there is a maximum possible size for the data that I am reading, I have implemented a function called recvAll(sock) to get all of the data off of a socket:
def recvAll(sock):
buffer = ''
data = []
try:
while True:
buffer = sock.recv(8192)
if not buffer:
break
data.append(buffer)
except error, (errorCode, message):
if errorCode != 10035:
print 'error: ', str(errorCode), ' ', message
return "".join(data)
I am calling the select library like this:
rlist, wlist, elist = select.select([sock1, sock2], [], [])
for sock in rlist:
if sock == sock1:
#data1 = sock.recv(8192)
data1 = recvAll(sock)
else:
#data2 = sock.recv(8192)
data2 = recvAll(sock)
In the for loop for each socket I process, if I call sock.recv directly, I am able to properly get data1 and data2. However, if I first pass sock to recvAll I am only able to get data1. It does not appear that recvAll is being called on sock2 at all. Why is this the case?
Most likely what is going on is that, since your socket is not set as non-blocking ( socket.setblocking(0) ), your recvAll function is blocking, waiting for more input from the socket. It won't return 0 until the socket is closed by the other end, or an error occurs.
The way to fix this is to structure your code to combine the functions of recvAll with your select. Each time your select returns with an indication that there is data waiting on the socket, read from the socket ONLY ONCE, append the data to the buffer for that socket, then loop back into select.
After each recv, look at what you got and decide what to do next.. eg. for some protocols, if the buffer contains a \n, that would be an indication that you got a complete message and need to do something with it. Or, in your case, it seems that closing of the socket is the indication that your message is complete.. so you should look for recv returning a zero length string....
If the socket was in fact closed, then you need to remove it from the list of sockets you are passing into select.
I'm a bit confused about how to keep calling recv() when using select(). This code isn't complete but it demonstrates the issue. Lets assume we are receiving a decent amount of data from each connection (10,20mb).
Should you keep looping using recv() until you get the desired number of bytes after the call to select()?
while True:
r,w,e = select.select(r_ready, w_ready, [], timeout)
for client in r:
if client == sock:
acceptConnection(sock)
else:
chunks = []
bytesRead = 0
while bytesRead < desiredBytes:
chunk = client.recv(1024)
bytesRead += len(chunk)
Or should you only call recv() once after each select() loop?
clientBuffers = {}
while True:
r,w,e = select.select(r_ready, w_ready, [], timeout)
for client in r:
if client == sock:
acceptConnection(sock)
else:
chunk = client.recv(1024)
clientBuffers[client].append(chunk)
Should you keep looping using recv() until you get the desired number
of bytes after the call to select()?
In general, no; because you have no way of knowing how long that will take. (e.g. for all you know, the client might not send (or the network might not deliver) the entire sequence of desired bytes until an hour after it sends the first bytes in the sequence; which means that if you stay in a loop calling recv() until you get all of the bytes, then it's possible that all of the other clients will not get any response from your server for a very long time -- clearly not desirable behavior for a multi-client server!)
Instead, just get as many bytes from recv() as you currently can, and if you didn't receive enough bytes to take action yet, then store the received bytes in a buffer somewhere for later and go back to your regular select() call. select() should be the only place in your event loop that you ever block. Making all of your sockets non-blocking is highly recommended, in order to guarantee that you won't ever accidentally block inside a recv() call.
The python3 socket programming howto presents this code snippet
class MySocket:
"""demonstration class only
- coded for clarity, not efficiency
"""
def __init__(self, sock=None):
if sock is None:
self.sock = socket.socket(
socket.AF_INET, socket.SOCK_STREAM)
else:
self.sock = sock
def connect(self, host, port):
self.sock.connect((host, port))
def mysend(self, msg):
totalsent = 0
while totalsent < MSGLEN:
sent = self.sock.send(msg[totalsent:])
if sent == 0:
raise RuntimeError("socket connection broken")
totalsent = totalsent + sent
def myreceive(self):
chunks = []
bytes_recd = 0
while bytes_recd < MSGLEN:
chunk = self.sock.recv(min(MSGLEN - bytes_recd, 2048))
if chunk == b'':
raise RuntimeError("socket connection broken")
chunks.append(chunk)
bytes_recd = bytes_recd + len(chunk)
return b''.join(chunks)
where the send loop is interrupted if the socket send method returns 0.
The logic behind this snippet is that when the send method returns '0 bytes sent', the sending side of a socket connection should give up its efforts to send data. This is for sure true for the recv method, where zero bytes read for a socket in blocking mode should be interpreted as EOF, and therefore the reading side should give up.
However I cannot understand under which situations the send method could return zero. My understanding of python sockets is that send returns immediately due to buffering at the OS level. If the buffer is full send will block, or if the connections is closed at the remote side, an exception is raised.
Finally suppose send returns zero without raising an exception: does this really indicate that all future send calls will return zero?
I've done some testing (although using only socket connected to ::1 on OS X) and was not able to find a situation in which send returns 0.
Edit
The HOWTO states:
But if you plan to reuse your socket for further transfers, you need
to realize that there is no EOT on a socket. I repeat: if a socket
send or recv returns after handling 0 bytes, the connection has been
broken. If the connection has not been broken, you may wait on a recv
forever, because the socket will not tell you that there’s nothing
more to read (for now).
It is pretty easy to find a situation in which recv returns 0: when the remote (sending) side calls socket.shutdown(SHUT_WR), further recv on the receiving side will return 0 and not raise any exception.
I'm looking for a concrete example where you can show that receiving 0 zero from send indicates a broken connection (which will continue to return 0 on send.)
Upon seeing the question I was somehow stunned, because a send C call can return 0 bytes and the connection is of course still alive (the socket cannot simply send more bytes at that given moment in time)
https://github.com/python/cpython/blob/master/Modules/socketmodule.c
I decided to "use the source" and unless I am very wrong (which can always be and often is) this is a bug in the HOWTO.
Chain:
send is an alias for sock_send
sock_send calls in turn sock_call
sock_call calls in turn sock_call_ex
sock_call calls in turn sock_send_impl (which has been passed down the chain starting with sock_send)
Unwinding:
sock_send_impl returns true or false (1 or 0) with return (ctx->result >= 0)
sock_call_ex returns
-1 if sock_send_impl returns false
0 if sock_send_impl returns true
sock_call returns this value transparently.
sock_send
returns NULL for a -1 (because an error has been set and an exception will be raised)
returns ctx->result for 0from sock_call
And ctx->result is the number of bytes written by the C call send in sock_send_impl.
The chain shows that if 0 bytes have been sent, there is no error and this actually is a potential real life socket situation.
If my logic is wrong, someone please let me know.
I might be wrong, but I think you are looking for an impossible situation...
As #mementum has shown in his answer, it is theoretically possible for a socket to return zero when there is no error, but also no data sent.
However, as shown elsewhere on SO this can only happen in very specific scenarios. In my experience (and also covered in the comments to the accepted answer) you would only ever get a zero result on a non-blocking socket when the network is congested. Now Python sockets are blocking by default, which means that the kernel should wait until there is room to take some more data then return how many bytes were queued. By definition, this can never be zero.
So, putting it all together, since your snippet doesn't reset the socket type - e.g. using the set_blocking function - it is using blocking sockets and so cannot return zero and thus cannot hit the path mementum identified.
This is backed up by the fact that you have been unable to trigger the specific line of code no matter what you do.
I'm new to socket programming (and somewhat to Python too) and I'm having trouble getting the select timeout to work the way I want to (on the server side). Before clients connect, timeout works just fine. I give it a value of 1 second and the timeout expires in my loop every 1 second.
Once a client connects, however, it doesn't wait 1 second to tell me the timeout expires. It just loops as fast as it can and tells me the timeout expires. Here's a snippet of my code:
while running:
try:
self.timeout_expired = False
inputready, outputready, exceptready = select.select(self.inputs, self.outputs, [], self.timeout)
except select.error, e:
break
except socket.error, e:
break
if not (inputready):
# Timeout expired
print 'Timeout expired'
self.timeout_expired = True
# Additional processing follows here
I'm not sure if this is enough code to see where my problem is, so please let me know if you need to see more. Basically, after a client connects, it at least appears that it ignores the timeout of 1 second and just runs as fast as it can, continuously telling me "Timeout expired". Any idea what I'm missing?
Thanks much!!
Edit: I should clarify..."inputready" represents input from a client connecting or sending data to the server, as well as stdin from the server. The other variables returned from select are only server-side variables, and is what I'm trying to do is detect whether the CLIENT took too long to reply, so I'm only checking if inputready is empty.
It is only a timeout if inputready, outputready, and exceptready are ALL empty. My guess is you have added the client socket to both self.inputs and self.outputs. Since the output socket is usually writable, it will always show up in outputready. Only add the client socket to self.outputs if you are ready to output something.
"When the timeout expires, select() returns three empty lists.
...To use a timeout requires adding the extra argument to the select() call and handling the empty lists after select() returns."
readable, writable, exceptional = select.select(inputs, outputs, inputs,timeout)
if not (readable or writable or exceptional):
print(' timed out, do some other work here', file=sys.stderr)
[https://pymotw.com/3/select/index.html][1]
I have the following python code that expects data coming from the serial port, and writes it to a file.
import time
import serial
def write_log ( text ):
f = open('logger.log', 'a')
f.write( text )
f.close()
ser = serial.Serial()
ser.port = "/dev/ttyS0"
ser.baudrate = 4800
ser.open()
if ser.isOpen():
while 1:
while ser.inWaiting() <= 0:
time.sleep(1)
response = ser.read(ser.inWaiting())
if len ( response ):
write_log( response )
print response
It works to an extent, as after some time, it starts to hang, bringing the CPU all the way up, and not writing anything (or sometimes writing only pieces of text) to the .log file.
The process here is pretty intensive, as my serial port will be writing an 8 bytes string every second, and this python script is supposed to then receive it, and write its contents to the log file.
I'm thinking the problem here is the fact that I'm opening and closing the file too much, and this is somehow making the whole process slow. I'm no python wizz, so any help or advice on improving this code would be greatly appreciated.
Thanks in advance,
You have an infinite loop in your code, and you don't break out from it when there is a problem - or the serial device is no longer open.
Probably use:
while ser.isOpen():
while ser.inWaiting() <= 0:
time.sleep(1)
response = ser.read(ser.inWaiting())
if len(response):
write_log(response)
print response
or even:
while ser.isOpen() && ser.inWaiting() <= 0:
time.sleep(1)
response = ser.read(ser.inWaiting())
if len(response):
write_log(response)
print response
I'm not sure about the sleep, either; you'd do better just waiting in the read for data to become available.
As I think about it, not knowing the methods available in the serial class, the more I think the main loop should be attempting to read from the serial device, hanging happily if there is nothing currently available, and only terminating when the input method indicates there is no more input to come - the device has been closed on you, or has failed in some way.
I think the problem here is that you are polling for updates.
Python's serial.read() function actually blocks the current thread until something becomes readable on the thread, until timeout. What you should do, therefore, is break out another thread to handle serial IO. Have it loop indefinitely, checking a condition (the master thread wants you to stay listening and the serial port is still available). This thread will do something like:
while ser.isOpen() && thisthread_should_keep_doing_this:
response = ser.read(ser.inWaiting())
if len(response):
write_log(response)
print response
Then, when you want it to exit, your master thread sets thisthread_should_keep_doing_this=False and when your slave thread has finished reading, it kills itself.
Two things:
Do have the read timeout relatively frequently.
Do not "remote kill" the thread. Pass it a message and have it kill itself. Killing threads remotely creates an awful mess.
See http://pyserial.sourceforge.net/examples.html#miniterm