How to handle exceptions with python library requests?
For example how to check is PC connected to internet?
When I try
try:
requests.get('http://www.google.com')
except ConnectionError:
# handle the exception
it gives me error name ConnectionError is not defined
Assuming you did import requests, you want requests.ConnectionError. ConnectionError is an exception defined by requests. See the API documentation here.
Thus the code should be:
try:
requests.get('http://www.google.com')
except requests.ConnectionError:
# handle the exception
The original link to the Python v2 API documentation from the original answer no longer works.
As per the documentation, I have added the below points:
In the event of a network problem (refused connection e.g internet issue), Requests will raise a ConnectionError exception.
try:
requests.get('http://www.google.com')
except requests.ConnectionError:
# handle ConnectionError the exception
In the event of the rare invalid HTTP response, Requests will raise an HTTPError exception.
Response.raise_for_status() will raise an HTTPError if the HTTP request returned an unsuccessful status code.
try:
r = requests.get('http://www.google.com/nowhere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
#handle the HTTPError request here
In the event of times out of request, a Timeout exception is raised.
You can tell Requests to stop waiting for a response after a given number of seconds, with a timeout arg.
requests.get('https://github.com/', timeout=0.001)
# timeout is not a time limit on the entire response download; rather,
# an exception is raised if the server has not issued a response for
# timeout seconds
All exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException. So a base handler can look like,
try:
r = requests.get(url)
except requests.exceptions.RequestException as e:
# handle all the errors here
The original link to the Python v2 documentation no longer works, and now points to the new documentation.
Actually, there are much more exceptions that requests.get() can generate than just ConnectionError. Here are some I've seen in production:
from requests import ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError
try:
r = requests.get(url, timeout=6.0)
except (ConnectTimeout, HTTPError, ReadTimeout, Timeout, ConnectionError):
continue
Include the requests module using import requests .
It is always good to implement exception handling. It does not only help to avoid unexpected exit of script but can also help to log errors and info notification. When using Python requests I prefer to catch exceptions like this:
try:
res = requests.get(adress,timeout=30)
except requests.ConnectionError as e:
print("OOPS!! Connection Error. Make sure you are connected to Internet. Technical Details given below.\n")
print(str(e))
continue
except requests.Timeout as e:
print("OOPS!! Timeout Error")
print(str(e))
continue
except requests.RequestException as e:
print("OOPS!! General Error")
print(str(e))
continue
except KeyboardInterrupt:
print("Someone closed the program")
for clarity, that is
except requests.ConnectionError:
NOT
import requests.ConnectionError
You can also catch a general exception (although this isn't recommended) with
except Exception:
Related
I'm using following code, to get a big API response using python requests library :
try :
r = requests.get("download_link")
data = r.content
except requests.exceptions.ConnectionError as ex:
print('Issue file download...'+str(ex))
except Exception as ex:
print('Issue file download...'+str(ex))
Python requests version : 2.18.4,
Python version : 3-x
Here the use case is, as soon as the API is called, I'm disconnecting the internet, and this is not throwing any error(program is stuck at the API call). The API called would take 30+ secs to give complete response based on network speed, and there is high possibility of internet disconnection during this API call.
I would like to catch error when internet disconnects during the API call. Could someone please guide me on how to handle it.
Since requests >= 2.4.0, you can use the timeout argument in seconds:
try :
timeout = 10
r = requests.get("download_link", timeout=10)
data = r.content
except requests.exceptions.ConnectionError as ex:
print('Issue file download...'+str(ex))
except requests.exceptions.ReadTimeout as ex:
print('Issue file download...'+str(ex))
except Exception as ex:
print('Issue file download...'+str(ex))
I've been having some issues with this piece of code, and I cannot figure out why.
When I run:
try:
r = requests.get('https://httpbin.org/get',
timeout=10,
headers={'Cache-Control': 'nocache', 'Pragma': 'nocache'})
r.raise_for_status()
return r.json()
except (requests.exceptions.RequestException, ValueError):
return False
NOTE: Host changed for privacy, the actual service is way
more erratic/buggy.
I will occasionally get this error:
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='https://httpbin.org/get', port=80): Read timed out. (read timeout=10)
I can't understand what has gone wrong;
I seem to be properly catching requests.exceptions.RequestException which is a superset/parent of requests.exceptions.ReadTimeout..
EDIT: It seems updating requests has fixed it.
It would be nice to have your error replicated, so far I tried code below,
but it still does catch a timeout.
import requests
try:
r = requests.get('http://httpstat.us/200?sleep=1000',
timeout=0.01,
headers={'Cache-Control': 'nocache', 'Pragma': 'nocache'})
r.raise_for_status()
print(r.json())
except (requests.exceptions.RequestException, ValueError) as e:
print('Error caught!')
print(e)
prints:
Error caught!
HTTPConnectionPool(host='192.168.1.120', port=8080): Read timed out. (read timeout=0.01)
Even in a minimal form your are still catching requests.exceptions.ReadTimeout:
try:
raise requests.exceptions.ReadTimeout
except requests.exceptions.RequestException:
print('Caught ReadTimeout')
My best guess is that your exception arises in some other part of code, but not in this example.
I am using the following code to resolve redirects to return a links final url
def resolve_redirects(url):
return urllib2.urlopen(url).geturl()
Unfortunately I sometimes get HTTPError: HTTP Error 429: Too Many Requests. What is a good way to combat this? Is the following good or is there a better way.
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
return urllib2.urlopen(url).geturl()
Also, what would happen if there is an exception in the except block?
It would be better to make sure the HTTP code is actually 429 before re-trying.
That can be done like this:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError, e:
if e.code == 429:
time.sleep(5);
return resolve_redirects(url)
raise
This will also allow arbitrary numbers of retries (which may or may not be desired).
https://docs.python.org/2/howto/urllib2.html#httperror
This is a fine way to handle the exception, though you should check to make sure you are always sleeping for the appropriate amount of time between requests for the given website (for example twitter limits the amount of requests per minute and has this amount clearly shown in their api documentation). So just make sure you're always sleeping long enough.
To recover from an exception within an exception, you can simply embed another try/catch block:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
return "Failed twice :S"
Edit: as #jesse-w-at-z points out, you should be returning an URL in the second error case, the code I posted is just a reference example of how to write a nested try/catch.
Adding User-Agent to request header solved my issue:
from urllib import request
from urllib.request import urlopen
url = 'https://www.example.com/abc.json'
req = request.Request(url)
req.add_header('User-Agent', 'abc-bot')
response = request.urlopen(req)
In a script I am creating I am posting a lot of data to a REST API.
The script is quite modularized and at the top level somewhere I am catching a URLError. I need to know what is in the body of the response, because there will be an error message in there.
Is there a method on URLError that I can use?
try:
(calling some function that throws URLError)
except urllib2.URLError, e:
print "Error: " + str(e.body_or_something)
Yes there is. You have an access to the response via e.readlines():
try:
(calling some function that throws URLError)
except urllib2.URLError, e:
print e.readlines()
See the documenet: https://docs.python.org/2/library/urllib2.html#urllib2.URLError
exception urllib2.URLError The handlers raise this exception (or
derived exceptions) when they run into a problem. It is a subclass of
IOError.
reason The reason for this error. It can be a message string or
another exception instance (socket.error for remote URLs, OSError for
local URLs).
exception urllib2.HTTPError Though being an exception (a subclass of
URLError), an HTTPError can also function as a non-exceptional
file-like return value (the same thing that urlopen() returns). This
is useful when handling exotic HTTP errors, such as requests for
authentication.
code An HTTP status code as defined in RFC 2616. This numeric value
corresponds to a value found in the dictionary of codes as found in
BaseHTTPServer.BaseHTTPRequestHandler.responses.
reason The reason for this error. It can be a message string or
another exception instance.
So you can access the response body when the request raise urllib2.HTTPError.
Try this:
try:
(calling some function that throws URLError)
except urllib2.HTTPError as e:
body = e.readlines()
print e.code, e.reason, body
except urllib2.URLError as e:
print e.reason
except:
sys.excepthook(*sys.exc_info())
What is a good pattern to avoid code duplication when dealing with different exception types in Python, eg. I want to treat URLError and HTTPError simlar but not quite:
try:
page = urlopen(request)
except URLError, err:
logger.error("An error ocurred %s", err)
except HTTPError, err:
logger.error("An error occured %s", err)
logger.error("Error message: %s", err.read())
In this example, I would like to avoid the duplication of the first logger.error call. Given URLError is the parent of HTTPError one could do something like this:
except URLError, err:
logger.error("An error occurred %s", err)
try:
raise err
except HTTPError, err:
# specialization for http errors
logger.error("Error message: %s", err.read())
except:
pass
Another approach would be to use isinstance eg. if URLError and HTTPError would not be in a chain of inheritance:
except (URLError, HTTPError), err:
logger.error("An error occured %s", err)
if isinstance(err, HTTPError):
logger.error("Error message: %s", err.read())
Which one should I prefer, is there another better approach?
I think that your third example is the best solution.
It's the shortest version
It avoids duplication
It is clear to read and easy to follow, much unlike the second version.
You might want to use the newer except FooError as err syntax, though, if you're on Python 2.6 or higher.
Also, in your example, the first version isn't quite correct since the URLError handler already catches the HTTPError, so the except HTTPError part is never reached. You'd have to switch the two excepts around. Another reason not to use this.