We Keep Coding

MicroPython usockets not timing out

For various reasons, I am trying to have my ESP32 device with MicroPython poll all 256 options of 192.168.1.*:79 to find a 'host' PC. In doing so, the ESP32 attempts to create a socket and connect it to each possible address, i.e.:
while not connected:
try:
addr = generate_next_address()
s = usocket.socket()
s.connect(addr)
except OSError:
s.close()
continue
print("Found a connection!")
connected = True
When attempting to send a connection to a device that refuses the connect(), it is very quick to throw the exception and move onward. However, the problem is when it starts encountering devices that either don't respond or don't exist, it waits for a significant time before timing out.
Now, I've tried every variation of using usocket.settimeout(), usocket.setblocking(), uselect.poll(), and time.delay(), but I was unable to get anything to change the timeout period.
By setting blocking to false, the script immediately attempts all 256 addresses and then breaks out of the while loop, disallowing an opportunity to connect properly. Having blocking on completely ignores any timeout setting I attempt, continuing to take 15-20 seconds to timeout, as opposed to 1.
Is there something I'm not understanding about how this works? Is there a solution that is obvious but I have missed?

readable socket times out on recv

I have a 'jobs' server which accepts requests from a client (there are 8 clients sending requests from another machine). The server then submits a 'job' (a 'job' is just an executable which writes a results file to disk), and on a 'jobs manager' thread waits until the job is done. When a job is done it sends a message to the client that a results files is ready to be copied back to the client.
On the main thread I use select to read incoming connections from clients, as well as jobs requests:
readable, writable, exceptional = select.select(inputs, [], [])
where inputs is a list of accepted connections (sockets), and this list also includes the server socket. All sockets are set to non-blocking. To my best understanding, if this call to select returns a non-empty readable, it means some elements of inputs has incoming data waiting to be read.
I am reading data using the following logic (SIZE is a constant):
for s in readable:
if s is not server:
try:
socket_ok = True
data = s.recv(SIZE)
except socket.error as e:
print ('ERROR socket error: ' + str(e) )
socket_ok = False
except Exception as e:
print ('ERROR error reading from socket: ' + str(e))
socket_ok = False
if not socket_ok:
# do something
I have 2 problems:
Sometimes I get a [Errno 110] Connection timed out exception, and I don't understand why - if I have a readable socket, doesn't it mean it has some data to be read?
How to deal with this exception - the #do something part. I can do a 'cleanup' - delete the running jobs which were requested by the timed-out socket, and remove the dead socket from the list. But I have no way of letting the client know that it should stop waiting for these jobs' results. Ideally I would like to reconnect somehow, because the jobs themselves keep running and produce results which I don't want to throw away.
EDIT I realized now that the jobs manager thread also have access to the sockets via a Queue instance - if a job is finished, the thread sends a 'job done' message through the relevant socket - so maybe the send and recv methods of the same socket cause some kind of race condition? But anyway, I don't see how this can cause a 'connection timed out' error.

A solution that was just a guess and seems to work: On the client side, I am using a blocking recv method to get message from the server that the job is done. Since a job can take a long time (e.g if the cluster running the jobs is low on resources), I guessed that maybe the socket waiting was the cause of the time-out. So instead of using recv in blocking mode, I use it with time-out of 5 seconds, so I can send a dummy message to the server every 5 seconds to keep the connection alive until a message is received. Now I don't get the exception (on the server side) any more.

Why is adding a while loop causing unexpected behavior?

I have this client Python program, sending to a Java server program.
My client was working fine before adding the while loop. I needed to add a while loop to keep sending user input; now the text does not appear on the other end until I kill the program (the Python client), then all the text I entered in Python will appear on the other end, but only after killing the program! What causes this?
import socket
HOST = "192.168.0.76"
PORT = 8080
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
while (True):
sock.sendall(raw_input(""))

Taking a quick look at the socket docs, I noticed that there is a setsockopt() function. If you look at the docs (man setsockopt), there's a SO_SNDBUF flag, which controls the buffer size while sending. If you didn't send enough, it probably doesn't flush until there's more data.
However, there doesn't appear to be a way to flush sockets (due to TCP frame sizes), and this related answer might give you some more hints: Python Socket Flush

change the loop to:
while True:
input = raw_input("")
sock.send(input)

Python: interrupt s.accept()

I have this code:
host, port = sys.argv[1:3]
port=int(port)
s = socket.socket()
s.bind((host,port))
s.listen(5)
while True:
conn, addr = s.accept()
threading.Thread(target=handle,args=(conn,)).start()
I need to stop my code using Ctrl-C, but Python doesn't receive Ctrl-C when it waits for new connection (s.accept()). How can I solve this problem?

In order to stop the socket connection, you can call the shutdown method like so:
s.shutdown(socket.SHUT_WR)
(This SHUT_WR stops all new writes and reads)
However, while your code is running, it is suspended while trying to make the TCP connection. In order to stop it via Ctrl-C, you'll need to run the socket on another thread, giving your main thread the ability to wake up to the interrupt and send the shutdown message.

You can use shutdown() or close() for your need
A wonderful explanation (from AIX 4.3 Communications Programming Concepts) is given below
Once a socket is no longer required, the calling program can discard the socket by applying a close subroutine to the socket descriptor. If a reliable delivery socket has data associated with it when a close takes place, the system continues to attempt data transfer. However, if the data is still undelivered, the system discards the data. Should the application program have no use for any pending data, it can use the shutdown subroutine on the socket prior to closing it.

python socket server/client protocol with unstable client connection

I have a threaded python socket server that opens a new thread for each connection.
The thread is a very simple communication based on question and answer.
Basically client sends initial data transmission, server takes it run an external app that does stuff to the transmission and returns a reply that the server will send back and the loop will begin again until client disconnects.
Now because the client will be on a mobile phone thus an unstable connection I get left with open threads no longer connected and because the loop starts with recv it is rather difficult to break on lost connectivity this way.
I was thinking on adding a send before the recv to test if connection is still alive but this might not help at all if the client disconnects after my failsafe send as the client sends a data stream every 5 seconds only.
I noticed the recv will break sometimes but not always and in those cases I am left with zombie threads using resources.
Also this could be a solid vulnerability for my system to be DOSed.
I have looked through the python manual and Googled since thursday trying to find something for this but most things I find are related to client and non blocking mode.
Can anyone point me in the right direction towards a good way on fixing this issue?
Code samples:
Listener:
serversocket = socket(AF_INET, SOCK_STREAM)
serversocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
serversocket.bind(addr)
serversocket.listen(2)
logg("Binded to port: " + str(port))
# Listening Loop
while 1:
clientsocket, clientaddr = serversocket.accept()
threading.Thread(target=handler, args=(clientsocket, clientaddr,port,)).start()
# This is useless as it will never get here
serversocket.close()
Handler:
# Socket connection handler (Threaded)
def handler(clientsocket, clientaddr, port):
clientsocket.settimeout(15)
# Loop till client closes connection or connection drops
while 1:
stream = ''
while 1:
ending = stream[-6:] # get stream ending
if ending == '.$$$$.':
break
try:
data = clientsocket.recv(1)
except:
sys.exit()
if not data:
sys.exit()
# this is the usual point where thread is closed when a client closes connection normally
stream += data
# Clear the line ending
stream = base64.b64encode(stream[:-6])
# Send data to be processed
re = getreply(stream)
# Send response to client
try:
clientsocket.send(re + str('.$$$$.'))
except:
sys.exit()
As you can see there are three conditions that at least one should trigger exit if connection fails but sometimes they do not.

Sorry, but I think that threaded idea in this case is not good. As you do not need to process/do a lot of stuff in these threads (workers?) and most of the time these threads are waiting for socket (is the blocking operation, isn't it?) I would advice to read about event-driven programming. According to sockets this pattern is extremly useful, becouse you can do all stuff in one thread. You are communicate with one socket at a time, but the rest of connections are just waiting to data so there is almost no loss. When you send several bytes you just check that maybe another connection requires carrying. You can read about select
and epoll.
In python there is several libraries to play with this nicly:
libev (c library wrapper) - pyev
tornado
twisted
I used tornado in some projects and it is done this task very good. Libev is nice also, but is a c-wrapper so it is a little bit low-level (but very nice for some tasks).

So you should use socket.settimeout(float) with the clientsocket like one of the comments suggested.
The reason you don't see any difference is, when you call socket.recv(bufsize[, flags]) and the timeout runs out an socket.timeout exception is thrown and you catch that exception and exit.
try:
data = clientsocket.recv(1)
except:
sys.exit()
should be somthing like:
try:
data = clientsocket.recv(1)
except timeout:
#timeout occurred
#handle it
clientsocket.close()
sys.exit()