I'm using urllib.request to download files from the internet. However sometimes I get Connection Reset by Peer and I want to retry.
I tried the following, but it seems that e.errno contains socket error and not an actual errno:
while True:
try:
filename, headers = urllib.request.urlretrieve(url)
break
except IOError as e:
if e.errno != errno.ECONNRESET:
raise
except Exception as e:
raise
Any suggestions?
Well this part is not needed, first of all.
except Exception as e:
raise
And the arguments of the IOError is the type of error (socket error) and the error given to it. This error, in turn, is not the original error, but that error is in the args, so...
except IOError as e:
if e.args[1].args[0].errno != errno.ECONNRESET:
raise
Should work. I don't have a server that will reset on me, so I can't test it 100% But it works with ECONNREFUSED. :-)
Related
I want to create a connection timeout exception using urlopen.
try:
urllib2.urlopen("http://example.com", timeout = 5)
except urllib2.URLError, e:
raise MyException("There was an error: %r" % e)
This is the code
I want to create a timeout that this code would bring an exception.
Thank You in advance.
You need to catch socket.timeout exception, check example below.
import urllib2
import socket
class MyException(Exception):
pass
try:
urllib2.urlopen("http://example.com", timeout = 1)
except socket.timeout, e:
# For Python 2.7
raise MyException("There was an error: %r" % e)
I strongly recommend using Requests library for making requests, it will make your life easier.
Every now and then setting up a tunnel using sshtunnel.py fails because the gateway (ssh_host) complains the first time I connect to it. I would like to give it a few retries before giving up:
for attempt in range(5):
try:
forwarder.start()
except Exception, e:
print 'Error (trying again in five seconds):\n' + format(e.message))
time.sleep(5)
else:
break
else:
print 'Failed to setup a connection to the gateway'
sys.exit(1)
However, the error is not 'detected'. I took a peek in the sshtunnel.py code and found that the following code catches the related Paramiko exception:
except paramiko.ssh_exception.AuthenticationException:
self.logger.error('Could not open connection to gateway')
return
How do I catch this in my try:?
An SSHTunnel.py project member advised me to add forwarder._check_is_started() to my code:
for attempt in range(5):
try:
forwarder.start()
forwarder._check_is_started()
except BaseSSHTunnelForwarderError as e:
print 'Error (trying again in five seconds):\n' + format(e.message))
time.sleep(5)
else:
break
else:
print 'Failed to setup a connection to the gateway'
sys.exit(1)
So I have a pretty generic logging statement after a request:
try:
r = requests.get(testUrl, timeout=10.0)
except Exception, err:
logger.error({"message": err.message})
This works great for everything I've thrown at it except TimeoutError. When the request times out the err I get back is a tuple that it tries and fails to serialize.
My question is how do I catch just this one type of error? For starters TimeoutError is not something I have access to. I have tried adding from exceptions import * but with no luck. I've also tried importing OSError because the docs say TimeoutError is a subclass, but I was unable to access TimeoutError after importing OSError.
TimeoutError docs
I plan to either list my exceptions in order:
except TimeoutError, err:
#handle this specific error
except Exception, err:
#handle all other errors
or just check for type:
except Exception, err:
if isinstance(err, TimeoutError):
#handle specific error
#handle all other errors
Python 2.7.3 & Django 1.5
You can handle requests.Timeout exception:
try:
r = requests.get(testUrl, timeout=10.0)
except requests.Timeout as err:
logger.error({"message": err.message})
except requests.RequestException as err:
# handle other errors
Example:
>>> import requests
>>> url = "http://httpbin.org/delay/2"
>>> try:
... r = requests.get(url, timeout=1)
... except requests.Timeout as err:
... print(err.message)
...
HTTPConnectionPool(host='httpbin.org', port=80): Read timed out. (read timeout=1)
I want to create socket errors (By doing things, obviously) but I've no idea how I should test if my script handles errors properly (If it dectes them.)
Currently, my code is this:
except socket.error as err:
print "Connection lost, waiting..."
time.sleep(5)
In theory, it should handle all the socket errors, print and then sleep (It's a part of a while loop.).
Any idea of how can I test it to see how it handles errors?
Use the raise statement:
try:
raise socket.error
except socket.error as err:
print "Connection lost, waiting..."
time.sleep(5)
Yet another example:
try:
raise AttributeError
except AttributeError:
print 'Sorry'
#Sorry
Also take a look at here and here
I have a FTP connection from which I am downloading many files and processing them in between. I'd like to be able to check that my FTP connection hasn't timed out in between. So the code looks something like:
conn = FTP(host='blah')
conn.connect()
for item in list_of_items:
myfile = open('filename', 'w')
conn.retrbinary('stuff", myfile)
### do some parsing ###
How can I check my FTP connection in case it timed out during the ### do some parsing ### line?
Send a NOOP command. This does nothing but check that the connection is still going and if you do it periodically it can keep the connection alive.
For example:
conn.voidcmd("NOOP")
If there is a problem with the connection then the FTP object will throw an exception. You can see from the documentation that exceptions are thrown if there is an error:
socket.error and IOError: These are raised by the socket connection and are most likely the ones you are interested in.
exception ftplib.error_reply: Exception raised when an unexpected reply is received from the server.
exception ftplib.error_temp: Exception raised when an error code signifying a temporary error (response codes in the range 400–499) is received.
exception ftplib.error_perm: Exception raised when an error code signifying a permanent error (response codes in the range 500–599) is received.
exception ftplib.error_proto: Exception raised when a reply is received from the server that does not fit the response specifications of the File Transfer Protocol, i.e. begin with a digit in the range 1–5.
Therefore you can use a try-catch block to detect the error and handle it accordingly.
For example this sample of code will catch an IOError, tell you about it and then retry the operation:
retry = True
while (retry):
try:
conn = FTP('blah')
conn.connect()
for item in list_of_items:
myfile = open('filename', 'w')
conn.retrbinary('stuff', myfile)
### do some parsing ###
retry = False
except IOError as e:
print "I/O error({0}): {1}".format(e.errno, e.strerror)
print "Retrying..."
retry = True