Python socket error resilience / workaround - python

I have a script running that is testing a series of urls for availability.
This is one of the functions.
def checkUrl(url): # Only downloads headers, returns status code.
p = urlparse(url)
conn = httplib.HTTPConnection(p.netloc)
conn.request('HEAD', p.path)
resp = conn.getresponse()
return resp.status
Occasionally, the VPS will lose connectivity, the entire script crashes when that occurs.
File "/usr/lib/python2.6/httplib.py", line 914, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.6/httplib.py", line 951, in _send_request
self.endheaders()
File "/usr/lib/python2.6/httplib.py", line 908, in endheaders
self._send_output()
File "/usr/lib/python2.6/httplib.py", line 780, in _send_output
self.send(msg)
File "/usr/lib/python2.6/httplib.py", line 739, in send
self.connect()
File "/usr/lib/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib/python2.6/socket.py", line 561, in create_connection
raise error, msg
socket.error: [Errno 101] Network is unreachable
I'm not at all familiar with handling errors like this in python.
What is the appropriate way to keep the script from crashing when network connectivity is temporarily lost?
Edit:
I ended up with this - feedback?
def checkUrl(url): # Only downloads headers, returns status code.
try:
p = urlparse(url)
conn = httplib.HTTPConnection(p.netloc)
conn.request('HEAD', p.path)
resp = conn.getresponse()
return resp.status
except IOError, e:
if e.errno == 101:
print "Network Error"
time.sleep(1)
checkUrl(url)
else:
raise
I'm not sure I fully understand what raise does though..

If you just want to handle this Network is unreachable 101, and let other exceptions throw an error, you can do following for example.
from errno import ENETUNREACH
try:
# tricky code goes here
except IOError as e:
# an IOError exception occurred (socket.error is a subclass)
if e.errno == ENETUNREACH:
# now we had the error code 101, network unreachable
do_some_recovery
else:
# other exceptions we reraise again
raise

Problem with your solution as it stands is you're going to run out of stack space if there are too many errors on a single URL (> 1000 by default) due to the recursion. Also, the extra stack frames could make tracebacks hard to read (500 calls to checkURL). I'd rewrite it to be iterative, like so:
def checkUrl(url): # Only downloads headers, returns status code.
while True:
try:
p = urlparse(url)
conn = httplib.HTTPConnection(p.netloc)
conn.request('HEAD', p.path)
resp = conn.getresponse()
return resp.status
except IOError as e:
if e.errno == 101:
print "Network Error"
time.sleep(1)
except:
raise
Also, you want the last clause in your try to be a bare except not an else. Your else only gets executed if control falls through the try suite, which can never happen, since the last statement of the try suite is return.
This is very easy to change to allow a limited number of retries. Just change the while True: line to for _ in xrange(5) or however many retries you wish to accept. The function will then return None if it can't connect to the site after 5 attempts. You can have it return something else or raise an exception by adding return or raise SomeException at the very end of the function (indented the same as the for or while line).

put try...except: around your code to catch exceptions.
http://docs.python.org/tutorial/errors.html

Related

requests.exceptions.SSLError - Failing to use python module

I am using the python module udemy-dl which i have installed via pypi.org/project/udemy-dl. When i run the script , i keep getting a SSL Error. I have looked through many questions on Stackoverflow and none of them have seemed to work. I get the following on my terminal:
[INFO-835] Downloading to: /Users/dev/the-complete-python-web-course-learn-by-building-8-apps
[INFO-107] Trying to log in ...
Traceback (most recent call last):
File "/Users/dev/homebrew/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/Users/dev/homebrew/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/Users/dev/homebrew/lib/python2.7/site-packages/udemy_dl/dev.py", line 8, in <module>
main()
File "/Users/dev/homebrew/lib/python2.7/site-packages/udemy_dl/udemy_dl.py", line 837, in main
udemy_dl(username, password, link, lecture_start, lecture_end, save_links, safe_file_names, just_list, output_dest)
File "/Users/dev/homebrew/lib/python2.7/site-packages/udemy_dl/udemy_dl.py", line 658, in udemy_dl
login(username, password)
File "/Users/dev/homebrew/lib/python2.7/site-packages/udemy_dl/udemy_dl.py", line 109, in login
csrf_token = get_csrf_token()
File "/Users/dev/homebrew/lib/python2.7/site-packages/udemy_dl/udemy_dl.py", line 95, in get_csrf_token
response = session.get('https://www.udemy.com/join/login-popup')
File "/Users/dev/homebrew/lib/python2.7/site-packages/udemy_dl/udemy_dl.py", line 66, in get
return self.session.get(url, headers=self.headers)
File "/Users/dev/homebrew/lib/python2.7/site-packages/requests/sessions.py", line 488, in get
return self.request('GET', url, **kwargs)
File "/Users/dev/homebrew/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/Users/dev/homebrew/lib/python2.7/site-packages/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/Users/dev*emphasized text*/homebrew/lib/python2.7/site-packages/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
I see that in adapters.py , is where the exception is raised :
def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
"""Sends PreparedRequest object. Returns Response object.
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
:param stream: (optional) Whether to stream the request content.
:param timeout: (optional) How long to wait for the server to send
data before giving up, as a float, or a :ref:`(connect timeout,
read timeout) <timeouts>` tuple.
:type timeout: float or tuple
:param verify: (optional) Whether to verify SSL certificates.
:param cert: (optional) Any user-provided SSL certificate to be trusted.
:param proxies: (optional) The proxies dictionary to apply to the request.
:rtype: requests.Response
"""
conn = self.get_connection(request.url, proxies)
self.cert_verify(conn, request.url, verify, cert)
url = self.request_url(request, proxies)
self.add_headers(request)
chunked = not (request.body is None or 'Content-Length' in request.headers)
if isinstance(timeout, tuple):
try:
connect, read = timeout
timeout = TimeoutSauce(connect=connect, read=read)
except ValueError as e:
# this may raise a string formatting error.
err = ("Invalid timeout {0}. Pass a (connect, read) "
"timeout tuple, or a single float to set "
"both timeouts to the same value".format(timeout))
raise ValueError(err)
else:
timeout = TimeoutSauce(connect=timeout, read=timeout)
try:
if not chunked:
resp = conn.urlopen(
method=request.method,
url=url,
body=request.body,
headers=request.headers,
redirect=False,
assert_same_host=False,
preload_content=False,
decode_content=False,
retries=self.max_retries,
timeout=timeout
)
# Send the request.
else:
if hasattr(conn, 'proxy_pool'):
conn = conn.proxy_pool
low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)
try:
low_conn.putrequest(request.method,
url,
skip_accept_encoding=True)
for header, value in request.headers.items():
low_conn.putheader(header, value)
low_conn.endheaders()
for i in request.body:
low_conn.send(hex(len(i))[2:].encode('utf-8'))
low_conn.send(b'\r\n')
low_conn.send(i)
low_conn.send(b'\r\n')
low_conn.send(b'0\r\n\r\n')
# Receive the response from the server
try:
# For Python 2.7+ versions, use buffering of HTTP
# responses
r = low_conn.getresponse(buffering=True)
except TypeError:
# For compatibility with Python 2.6 versions and back
r = low_conn.getresponse()
resp = HTTPResponse.from_httplib(
r,
pool=conn,
connection=low_conn,
preload_content=False,
decode_content=False
)
except:
# If we hit any problems here, clean up the connection.
# Then, reraise so that we can handle the actual exception.
low_conn.close()
raise
except (ProtocolError, socket.error) as err:
raise ConnectionError(err, request=request)
except MaxRetryError as e:
if isinstance(e.reason, ConnectTimeoutError):
# TODO: Remove this in 3.0.0: see #2811
if not isinstance(e.reason, NewConnectionError):
raise ConnectTimeout(e, request=request)
if isinstance(e.reason, ResponseError):
raise RetryError(e, request=request)
if isinstance(e.reason, _ProxyError):
raise ProxyError(e, request=request)
raise ConnectionError(e, request=request)
except ClosedPoolError as e:
raise ConnectionError(e, request=request)
except _ProxyError as e:
raise ProxyError(e)
except (_SSLError, _HTTPError) as e:
if isinstance(e, _SSLError):
raise SSLError(e, request=request)
The script you use will verify certain certificates with the main site and it allows connection only when the certificates are verified. Possible workaround.
1.) You need to download the certificates given by the website and pass the same to requests call verify='path/to/ssl/certificate/' (or)
2.) find the requests call in the script and set verify=False

Python FTP socket timeout handling

I'm having a problem handling socket timeouts with python35 ftplib. When a socket timeout error occurs, for some reason I am unable to catch the exception and the script raises the error anyway and exits. Here is the relevant code block:
try:
ftp = FTP(self.config.base_url, timeout=400)
ftp.login()
ftp.cwd(self.config.root_path)
ftp.retrbinary("RETR {0}".format(os.path.join(self.root_path, file_path)), fp.write, 1024)
ftp.quit()
except socket.timeout:
self.download_file(file_path)
For some reason this script will still error out with a socket timeout exception, how is that possible? A general catch-all doesn't work either. Here is the stack trace of the error:
File "ftp.py", line 82, in __init__
self.ftp = FTP(self.config.base_url, timeout=400)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 118, in __init__
self.connect(host)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 156, in connect
self.welcome = self.getresp()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 235, in getresp
resp = self.getmultiline()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 221, in getmultiline
line = self.getline()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 203, in getline
line = self.file.readline(self.maxline + 1)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 576, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
As you can see, this is the socket.timeout error being thrown, so how is that not caught? I was unable to find any useful info on how to resolve this issue after many hours of internet research, any insight into this issue would be much appreciated.
For reference, here is the relevant code block of socket.py:
def readinto(self, b):
"""Read up to len(b) bytes into the writable buffer *b* and return
the number of bytes read. If the socket is non-blocking and no bytes
are available, None is returned.
If *b* is non-empty, a 0 return value indicates that the connection
was shutdown at the other end.
"""
self._checkClosed()
self._checkReadable()
if self._timeout_occurred:
raise OSError("cannot read from timed out object")
while True:
try:
return self._sock.recv_into(b)
except timeout:
self._timeout_occurred = True
raise
except error as e:
if e.args[0] in _blocking_errnos:
return None
raise
UPDATE:
It seems the issue is that ftplib behaves unexpectedly if the timeout passed to the FTP constructor is greater than the default for a socket timeout. At the time of this edit there is an open python issue to address this behavior: http://bugs.python.org/issue30956
try to add
ftp.set_pasv(False)
disabling the passive mode it should work

What does this python exception mean

I see the following exception sometimes when I try to hit my home page.
ERROR:root:HTTPConnectionPool(host='0.0.0.0', port=8003): Max retries exceeded with url:
/snapshot/?app=cdnstats&key=28736ba5fbe151d5ff6678015c8f6ade (Caused by <class 'socket.error'>:
[Errno 61] Connection refused)
Traceback (most recent call last):
File "/Users/rokumar/CDNStats/cdnstats/app/core/views.py", line 257, in get_snapshot_data
data = templates.render_snapshot(controllers.get_snapshot_data())
File "/Users/rokumar/CDNStats/cdnstats/util.py", line 260, in decorated
expiry, mem_args, func, args, kwargs)
File "/Users/rokumar/CDNStats/cdnstats/util.py", line 227, in get_data_from_meminstance
data = func(*args, **kwargs)
File "/Users/rokumar/CDNStats/cdnstats/app/core/controllers.py", line 255, in get_snapshot_data
return util.call_get_api(config.CDNSTATS_API_URL + 'snapshot/?', data)
File "/Users/rokumar/CDNStats/cdnstats/util.py", line 123, in call_get_api
raise ex
The following is the piece of code generating the exception.
def call_get_api(url, data):
try:
data = data.copy()
data['key'] = request.args.get('key')
data['app'] = config.APPNAME
query = soft_urlencode(data)
response = requests.get(url + query)
if response.status_code == 200:
return response.json()
else:
apiexception = APIException(response.content)
apiexception.status_code = response.status_code
raise apiexception
except UnicodeEncodeError as ex:
print ex
raise ex
except Exception as ex:
raise ex
I see the exception intermittently and my app slows down heavily. I don;t really understand the exception or what is wrong. The exception says max retries exceeded but I do not have any retry logic going on.
in urlopen, try setting retries=False or retries=1. The default is 3 so that will probably be your retry logic going on there.

SSLError (Read operation timed out) in Python requests

I have a python API script and my script sometimes gets terminated on this line despite using try/except. Here is the code:
try:
r = requests.post(URL, data=params, headers=headers, timeout=self.request_timeout)
try:
response = r.json()
except Exception, e:
message = "ERROR_0104! Unexpected error occured. The error is: "
message += str(e)
print message
aux_func.write_log(message)
return 'Switch'
except requests.exceptions.RequestException:
print "Exception occurred on 'API requests post' procedure."
counter += 1
continue
...
The error occurs on the second line of above shown code. This is the error:
r = requests.post(URL, data=params, headers=headers, timeout=self.request_timeout)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 394, in send
r.content
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 679, in content
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 616, in generate
decode_content=True):
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/response.py", line 236, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/response.py", line 183, in read
data = self._fp.read(amt)
File "/usr/lib/python2.7/httplib.py", line 543, in read
return self._read_chunked(amt)
File "/usr/lib/python2.7/httplib.py", line 585, in _read_chunked
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib/python2.7/ssl.py", line 305, in recv
return self.read(buflen)
File "/usr/lib/python2.7/ssl.py", line 224, in read
return self._sslobj.read(len)
ssl.SSLError: The read operation timed out
I presume something within the Requests module is causing this, but I don't know what.
The read operation has timed out, as it says.
It times out, however, with an ssl.SSLError. This is not what your except is catching. If you want to catch and retry, you need to catch the right error.
except Exception, e does not work with >= Python 3
You have to make it except Exception as e
I saw that there was some confusion here regarding what the solution is because of lack of enough details. I posted the answer on the comment to the top post but the formatting is not great in the comment section and I will post a properly formatted answer here.
The problem, as Veedrac has mentioned is that I was not catching all the possible exceptions in the code that I posted in the question. My code only catches "requests.exceptions.RequestException", and any other exception will cause the code to exit abruptly.
Instead, I'm gonna re-write the code like this:
try:
r = requests.post(URL, data=params, headers=headers, timeout=self.request_timeout)
try:
response = r.json()
except Exception, e:
message = "ERROR_0104! Unexpected error occured. The error is: "
message += str(e)
print message
aux_func.write_log(message)
return 'Switch'
except requests.exceptions.RequestException:
print "Exception occurred on 'API requests post' procedure."
counter += 1
continue
except Exception, e:
print "Exception {0} occurred".format(e)
continue
All I did was add an extra generic exception catcher at the end which will catch all other unaccounted for exceptions.
I hope this helps.
Thanks.

Check if the internet cannot be accessed in Python

I have an app that makes a HTTP GET request to a particular URL on the internet. But when the network is down (say, no public wifi - or my ISP is down, or some such thing), I get the following traceback at urllib2.urlopen:
70, in get
u = urllib2.urlopen(req)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 391, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 409, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 1161, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 1136, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
I want to print a friendly error to the user telling him that his network maybe down instead of this unfriendly "nodename nor servname provided" error message. Sure I can catch URLError, but that would catch every url error, not just the one related to network downtime.
I am not a purist, so even an error message like "The server example.com cannot be reached; either the server is indeed having problems or your network connection is down" would be nice. How do I go about selectively catching such errors? (For a start, if DNS resolution fails at urllib2.urlopen, that can be reasonably assumed as network inaccessibility? If so, how do I "catch" it in the except block?)
You should wrap the request in a try/except statement so that you catch the fault and then let them know.
try:
u = urllib2.urlopen(req)
except HTTPError as e:
#inform them of the specific error here (based off the error code)
except URLError as e:
#inform them of the specific error here
except Exception as e:
#inform them that a general error has occurred
urllib2 - The Missing Manual has a good section on how to handle URLError and HTTPError exceptions and how to differentiate the conditions that caused them.
How about catching URLError, then testing the reason attribute? If the reason isn't one you're interested in, re-throw the URLError and handle it somewhere else.
Alternatively, you could try httplib2. Its ServerNotFoundError exception would probably suit your needs.

Categories