For various reasons, I am trying to have my ESP32 device with MicroPython poll all 256 options of 192.168.1.*:79 to find a 'host' PC. In doing so, the ESP32 attempts to create a socket and connect it to each possible address, i.e.:
while not connected:
try:
addr = generate_next_address()
s = usocket.socket()
s.connect(addr)
except OSError:
s.close()
continue
print("Found a connection!")
connected = True
When attempting to send a connection to a device that refuses the connect(), it is very quick to throw the exception and move onward. However, the problem is when it starts encountering devices that either don't respond or don't exist, it waits for a significant time before timing out.
Now, I've tried every variation of using usocket.settimeout(), usocket.setblocking(), uselect.poll(), and time.delay(), but I was unable to get anything to change the timeout period.
By setting blocking to false, the script immediately attempts all 256 addresses and then breaks out of the while loop, disallowing an opportunity to connect properly. Having blocking on completely ignores any timeout setting I attempt, continuing to take 15-20 seconds to timeout, as opposed to 1.
Is there something I'm not understanding about how this works? Is there a solution that is obvious but I have missed?
Related
Pseudo-code to better explain question:
#!/usr/bin/env python2.7
import pycurl, threading
def threaded_work():
conn = pycurl.Curl()
conn.setopt(pycurl.TIMEOUT, 10)
# Make a request to host #1 just to open the connection to it.
conn.setopt(pycurl.URL, 'https://host1.example.com/')
conn.perform_rs()
while not condition_that_may_take_very_long:
conn.setopt(pycurl.URL, 'https://host2.example.com/')
print 'Response from host #2: ' + conn.perform_rs()
# Now, after what may be a very long time, we must request host #1 again with a (hopefully) already established connection.
conn.setopt(pycurl.URL, 'https://host1.example.com/')
print 'Response from host #1, hopefully with an already established connection from above: ' + conn.perform_rs()
conn.close()
for _ in xrange(30):
# Multiple threads must work with host #1 and host #2 individually.
threading.Thread(target = threaded_work).start()
I am omitting extra, only unnecessary details for brevity so that the main problem has focus.
As you can see, I have multiple threads that must work with two different hosts, host #1 and host #2. Mostly, the threads will be working with host #2 until a certain condition is met. That condition may take hours or even longer to be met, and will be met at different times in different threads. Once the condition (condition_that_may_take_very_long in the example) is met, I would like host #1 to be requested as fast as possible with the connection that I have already established at the start of the threaded_work method. Is there any efficient way to efficiently accomplish this (open to the suggestion of using two PycURL handles, too)?
Pycurl uses libcurl. libcurl keeps connections alive by default after use, so as long as you keep the handle alive and use that for the subsequent transfer, it will keep the connection alive and ready for reuse.
However, due to modern networks and network equipment (NATs, firewalls, web servers), connections without traffic are often killed off relatively soon so having an idle connection and expecting it to actually work after "hours", is a very slim chance and rare occurance. Typically, libcurl will then discover that the connection has been killed in the mean time and create a new one to use at the next use.
Additionally, and in line with what I've described above, since libcurl 7.65.0 it now defaults to not reusing connections anymore that are older than 118 seconds. Changeable with the CURLOPT_MAXAGE_CONN option. The reason is that they barely ever work so by avoiding having to keep them around, detect them to be dead and reissue the request, this is an optimization.
I am building a port scanning program ((irrelevant to the question, just explaining the background)), and I know the IP of the host, but not what ports are open. Hence, the scan.
It is in the early stages of development, so the error handling is bad, but not bad enough to make why Python does this explainable.
It tries to connect to, say, 123.456.7.8, 1. Obviously it's a ridiculous port to be open, so it throws an error. The error is No Route to Host or the such, right? Wrong! It is instead Operation Timed Out!.
Okay, let's increase the timeout in case my calculations were incorrect.
.
..
...
....All that did was rinse and repeat!
About 20 minutes later, the timeout is at 20 seconds, and it still is timing out. Really? Why does python raise a timed out error though, instead of No route to host! or similar?
I need to distinguish between time outs and connection failures, because there is a difference between late and nowhere. This prevents me from doing so, creating an infinite loop of hurry up and wait.
Whatever shall I do? Wherever shall I go?
Python socket module is a thin wrapper around your platform's socket API. The issue is unrelated to Python.
It is not necessary that you get No Route to Host error. Moreover it is common that a firewall just drops received packets (for a filtered port) that may manifest as a timeout error in your code. See Drop vs. Reject (ignore the conclusion but read the explanation of what is happening).
To workaround, make multiple concurrent connections and set a fixed timeout or use raw-sockets and send the packets yourself (you could use scapy, to investigate the behavior).
I am using exscripts module which has a call conn.connect('IP address').
It tries to open a telnet session to that IP.
It will generate an error after connection times out.
The timeout exception is set somewhere in the code of the module or it would be what the default for telnet is. (not sure)
This timeout time is too long and slowing down the script if 1 device is not reachable. Is there something we can do with the try except here ? Like
Try for 3 secs:
then process the code
except:
print " timed out"
We changed the API. Mike Pennington only recently introduced the new connect_timeout parameter for that specific use case.
New solution (current master, latest release on pypi 2.1.451):
conn = Telnet(connect_timeout=3)
We changed the API because you usually don't want to wait for unreachable devices, but want to wait for commands to finish (some take a little longer).
I think you can use
conn = Telnet(timeout=3)
I dont know whether timeout in seconds. If microseconds, try 3000
I have a threaded python socket server that opens a new thread for each connection.
The thread is a very simple communication based on question and answer.
Basically client sends initial data transmission, server takes it run an external app that does stuff to the transmission and returns a reply that the server will send back and the loop will begin again until client disconnects.
Now because the client will be on a mobile phone thus an unstable connection I get left with open threads no longer connected and because the loop starts with recv it is rather difficult to break on lost connectivity this way.
I was thinking on adding a send before the recv to test if connection is still alive but this might not help at all if the client disconnects after my failsafe send as the client sends a data stream every 5 seconds only.
I noticed the recv will break sometimes but not always and in those cases I am left with zombie threads using resources.
Also this could be a solid vulnerability for my system to be DOSed.
I have looked through the python manual and Googled since thursday trying to find something for this but most things I find are related to client and non blocking mode.
Can anyone point me in the right direction towards a good way on fixing this issue?
Code samples:
Listener:
serversocket = socket(AF_INET, SOCK_STREAM)
serversocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
serversocket.bind(addr)
serversocket.listen(2)
logg("Binded to port: " + str(port))
# Listening Loop
while 1:
clientsocket, clientaddr = serversocket.accept()
threading.Thread(target=handler, args=(clientsocket, clientaddr,port,)).start()
# This is useless as it will never get here
serversocket.close()
Handler:
# Socket connection handler (Threaded)
def handler(clientsocket, clientaddr, port):
clientsocket.settimeout(15)
# Loop till client closes connection or connection drops
while 1:
stream = ''
while 1:
ending = stream[-6:] # get stream ending
if ending == '.$$$$.':
break
try:
data = clientsocket.recv(1)
except:
sys.exit()
if not data:
sys.exit()
# this is the usual point where thread is closed when a client closes connection normally
stream += data
# Clear the line ending
stream = base64.b64encode(stream[:-6])
# Send data to be processed
re = getreply(stream)
# Send response to client
try:
clientsocket.send(re + str('.$$$$.'))
except:
sys.exit()
As you can see there are three conditions that at least one should trigger exit if connection fails but sometimes they do not.
Sorry, but I think that threaded idea in this case is not good. As you do not need to process/do a lot of stuff in these threads (workers?) and most of the time these threads are waiting for socket (is the blocking operation, isn't it?) I would advice to read about event-driven programming. According to sockets this pattern is extremly useful, becouse you can do all stuff in one thread. You are communicate with one socket at a time, but the rest of connections are just waiting to data so there is almost no loss. When you send several bytes you just check that maybe another connection requires carrying. You can read about select
and epoll.
In python there is several libraries to play with this nicly:
libev (c library wrapper) - pyev
tornado
twisted
I used tornado in some projects and it is done this task very good. Libev is nice also, but is a c-wrapper so it is a little bit low-level (but very nice for some tasks).
So you should use socket.settimeout(float) with the clientsocket like one of the comments suggested.
The reason you don't see any difference is, when you call socket.recv(bufsize[, flags]) and the timeout runs out an socket.timeout exception is thrown and you catch that exception and exit.
try:
data = clientsocket.recv(1)
except:
sys.exit()
should be somthing like:
try:
data = clientsocket.recv(1)
except timeout:
#timeout occurred
#handle it
clientsocket.close()
sys.exit()
I'm using Python 3.2.3 on Windows 7, and one piece of code I have connects to a server with a blocking socket, with a user-specified timeout value. The code is simply:
testconn = socket.create_connection((host, port), timeout)
The code works fine, apart from the odd fact that timing out seems to take longer than it should on invalid requests. I tried connecting to www.google.com:59855 deliberately (random port should mean it should try connecting until it reaches the timeout), with a timeout of 5 seconds, but it seemed to take 15 seconds at least to timeout.
Are there any possible reasons for this, and/or any fixes? (It's not a huge problem if it's not fixable, but a solution would be appreciated nevertheless.) Thanks in advance.
This isn't an issue specific to Python 3 or Windows. Take a look at the docs for create_connection(): http://docs.python.org/library/socket.html#socket.create_connection
The important snippet is:
if host is a non-numeric hostname, it will try to resolve it for both
AF_INET and AF_INET6, and then try to connect to all possible
addresses in turn until a connection succeeds.
It resolves the name using socket.getaddrinfo. If you run
socket.getaddrinfo('google.com', 59855, 0, socket.SOCK_STREAM)
You'll probably get a few results returned. When you call socket.create_connection, it will iterate over all of those results, each waiting for timeout seconds until it fails. Because it waits timeout seconds for EACH result, the total time is obviously going to be greater than timeout.
If you call create_connection with an IP address rather than host name, e.g.
testconn = socket.create_connection(('74.125.226.201', 59855), timeout=5)
you should get your 5 second timeout.
And if you're really curious, take a look at the source for create_connection. It's pretty simple and you can see the loop that is causing your problems:
https://github.com/python/cpython/blob/3.2/Lib/socket.py#L408