I am using requests to get some data from a server, which is done in a while loop. However, every now and then, one of two errors occur. The first is that the status_code of the return from the request is not equal to 200, and this prints out an error message. The second is that a ConnectionError exception is raised.
If I receive either error, I want to keep attempting to get the data. However, I'm not sure how to do this for both types of error.
I know how to handle the ConnectionError exception, for example:
def get_data(self, path):
# Keep trying until the connection attempt is successful
while True:
# Attempt a request
try:
request_return = requests.get(path, timeout=30)
break
# Handle a connection error
except ConnectionError as e:
pass
# Return the data
return request_return.json()
But how can I also handle the status_code in a similar manner? Is it something to do with the raise_for_status() method?
Seems like you could just adjust your try/except to look like this:
try:
request_return = requests.get(path, timeout=30)
if request_return.status_code == 200:
break
except ConnectionError as e:
pass
If you prefer, you can use request_return.status_code == requests.codes.ok as well.
If you're set on handling the request as an exception (for whatever reason), raise_for_status() returns an HTTPError, so you can amend your try/except like this:
try:
request_return = requests.get(path, timeout=30)
request_return.raise_for_status()
break
except ConnectionError as e:
pass
except HTTPError as e:
pass
You can test the status code and leave the loop only on a 200 like:
Code:
if request_return.status_code == 200:
break
Probably should limit the number of retries:
import requests
def get_data(path):
# Keep trying until the connection attempt is successful
retries = 5
while retries > 0:
# Attempt a request
try:
request_return = requests.get(path, timeout=3)
if request_return.status_code == 200:
break
# Handle a connection error
except ConnectionError as e:
pass
retries -= 1
if retries == 0:
""" raise an error here """
# Return the data
return request_return.json()
get_data('https://stackoverflow.com/rep')
Related
I've been having some issues with this piece of code, and I cannot figure out why.
When I run:
try:
r = requests.get('https://httpbin.org/get',
timeout=10,
headers={'Cache-Control': 'nocache', 'Pragma': 'nocache'})
r.raise_for_status()
return r.json()
except (requests.exceptions.RequestException, ValueError):
return False
NOTE: Host changed for privacy, the actual service is way
more erratic/buggy.
I will occasionally get this error:
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='https://httpbin.org/get', port=80): Read timed out. (read timeout=10)
I can't understand what has gone wrong;
I seem to be properly catching requests.exceptions.RequestException which is a superset/parent of requests.exceptions.ReadTimeout..
EDIT: It seems updating requests has fixed it.
It would be nice to have your error replicated, so far I tried code below,
but it still does catch a timeout.
import requests
try:
r = requests.get('http://httpstat.us/200?sleep=1000',
timeout=0.01,
headers={'Cache-Control': 'nocache', 'Pragma': 'nocache'})
r.raise_for_status()
print(r.json())
except (requests.exceptions.RequestException, ValueError) as e:
print('Error caught!')
print(e)
prints:
Error caught!
HTTPConnectionPool(host='192.168.1.120', port=8080): Read timed out. (read timeout=0.01)
Even in a minimal form your are still catching requests.exceptions.ReadTimeout:
try:
raise requests.exceptions.ReadTimeout
except requests.exceptions.RequestException:
print('Caught ReadTimeout')
My best guess is that your exception arises in some other part of code, but not in this example.
I have the below flask code :
from flask import Flask,request,jsonify
import requests
from werkzeug.exceptions import InternalServerError, NotFound
import sys
import json
app = Flask(__name__)
app.config['SECRET_KEY'] = "Secret!"
class InvalidUsage(Exception):
status_code = 400
def __init__(self, message, status_code=None, payload=None):
Exception.__init__(self)
self.message = message
if status_code is not None:
self.status_code = status_code
self.payload = payload
def to_dict(self):
rv = dict(self.payload or ())
rv['message'] = self.message
rv['status_code'] = self.status_code
return rv
#app.errorhandler(InvalidUsage)
def handle_invalid_usage(error):
response = jsonify(error.to_dict())
response.status_code = error.status_code
return response
#app.route('/test',methods=["GET","POST"])
def test():
url = "https://httpbin.org/status/404"
try:
response = requests.get(url)
if response.status_code != 200:
try:
response.raise_for_status()
except requests.exceptions.HTTPError:
status = response.status_code
print status
raise InvalidUsage("An HTTP exception has been raised",status_code=status)
except requests.exceptions.RequestException as e:
print e
if __name__ == "__main__":
app.run(debug=True)
My question is how do i get the exception string(message) and other relevant params from the requests.exceptions.RequestException object e ?
Also what is the best way to log such exceptions . In case of an HTTPError exceptions i have the status code to refer to.
But requests.exceptions.RequestException catches all request exceptions . So how do i differentiate between them and also what is the best way to log them apart from using print statements.
Thanks a lot in advance for any answers.
RequestException is a base class for HTTPError, ConnectionError, Timeout, URLRequired, TooManyRedirects and others (the whole list is available at the GitHub page of requests module). Seems that the best way of dealing with each error and printing the corresponding information is by handling them starting from more specific and finishing with the most general one (the base class). This has been elaborated widely in the comments in this StackOverflow topic. For your test() method this could be:
#app.route('/test',methods=["GET","POST"])
def test():
url = "https://httpbin.org/status/404"
try:
# some code...
except requests.exceptions.ConnectionError as ece:
print("Connection Error:", ece)
except requests.exceptions.Timeout as et:
print("Timeout Error:", et)
except requests.exceptions.RequestException as e:
print("Some Ambiguous Exception:", e)
This way you can firstly catch the errors that inherit from the RequestException class and which are more specific.
And considering an alternative for printing statements - I'm not sure if that's exactly what you meant, but you can log into console or to a file with standard Python logging in Flask or with the logging module itself (here for Python 3).
This is actually not a question about using the requests library as much as it is a general Python question about how to extract the error string from an exception instance. The answer is relatively straightforward: you convert it to a string by calling str() on the exception instance. Any properly written exception handler (in requests or otherwise) would have implemented an __str__() method to allow an str() call on an instance. Example below:
import requests
rsp = requests.get('https://httpbin.org/status/404')
try:
if rsp.status_code >= 400:
rsp.raise_for_status()
except requests.exceptions.RequestException as e:
error_str = str(e)
# log 'error_str' to disk, a database, etc.
print('The error was:', error_str)
Yes, in this example, we print it, but once you have the string you have additional options. Anyway, saving this to test.py results in the following output given your test URL:
$ python3 test.py
The error was: 404 Client Error: NOT FOUND for url: https://httpbin.org/status/404
I am trying to handle the exceptions from the http responses.
The PROBLEM with my code is that I am forced to use and IF condition to catch http error codes
if page.status_code != requests.codes.ok:
page.raise_for_status()
I do not believe this is the right way to do it, I am trying the FOLLOWING
import requests
url = 'http://someurl.com/404-page.html'
myHeaders = {'User-agent': 'myUserAgent'}
s = requests.Session()
try:
page = s.get(url, headers=myHeaders)
#if page.status_code != requests.codes.ok:
# page.raise_for_status()
except requests.ConnectionError:
print ("DNS problem or refused to connect")
# Or Do something with it
except requests.HTTPError:
print ("Some HTTP response error")
#Or Do something with it
except requests.Timeout:
print ("Error loading...too long")
#Or Do something with it, perhaps retry
except requests.TooManyRedirects:
print ("Too many redirect")
#Or Do something with it
except requests.RequestException as e:
print (e.message)
#Or Do something with it
else:
print ("nothing happen")
#Do something if no exception
s.close()
This ALWAYS prints "nothing happen", How I would be able to catch all possible exceptions related to GET URL?
You could catch a RequestException if you want to catch all the exceptions:
import requests
try:
r = requests.get(........)
except requests.RequestException as e:
print(e.message)
I am writing some small python app which uses requests to get and post data to an html page.
now the problem I am having is that if I can't reach the html page the code stops with a max retries exceeded. I want to be able to do some things if I can't reach the server.
is such a thing possible?
here is sample code:
import requests
url = "http://127.0.0.1/"
req = requests.get(url)
if req.status_code == 304:
#do something
elif req.status_code == 404:
#do something else
# etc etc
# code here if server can`t be reached for whatever reason
You want to handle the exception requests.exceptions.ConnectionError, like so:
try:
req = requests.get(url)
except requests.exceptions.ConnectionError as e:
# Do stuff here
You may want to set a suitable timeout when catching ConnectionError:
url = "http://www.stackoverflow.com"
try:
req = requests.get(url, timeout=2) #2 seconds timeout
except requests.exceptions.ConnectionError as e:
# Couldn't connect
See this answer if you want to change the number of retries.
How to handle exceptions with python library requests?
For example how to check is PC connected to internet?
When I try
try:
requests.get('http://www.google.com')
except ConnectionError:
# handle the exception
it gives me error name ConnectionError is not defined
Assuming you did import requests, you want requests.ConnectionError. ConnectionError is an exception defined by requests. See the API documentation here.
Thus the code should be:
try:
requests.get('http://www.google.com')
except requests.ConnectionError:
# handle the exception
The original link to the Python v2 API documentation from the original answer no longer works.
As per the documentation, I have added the below points:
In the event of a network problem (refused connection e.g internet issue), Requests will raise a ConnectionError exception.
try:
requests.get('http://www.google.com')
except requests.ConnectionError:
# handle ConnectionError the exception
In the event of the rare invalid HTTP response, Requests will raise an HTTPError exception.
Response.raise_for_status() will raise an HTTPError if the HTTP request returned an unsuccessful status code.
try:
r = requests.get('http://www.google.com/nowhere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
#handle the HTTPError request here
In the event of times out of request, a Timeout exception is raised.
You can tell Requests to stop waiting for a response after a given number of seconds, with a timeout arg.
requests.get('https://github.com/', timeout=0.001)
# timeout is not a time limit on the entire response download; rather,
# an exception is raised if the server has not issued a response for
# timeout seconds
All exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException. So a base handler can look like,
try:
r = requests.get(url)
except requests.exceptions.RequestException as e:
# handle all the errors here
The original link to the Python v2 documentation no longer works, and now points to the new documentation.
Actually, there are much more exceptions that requests.get() can generate than just ConnectionError. Here are some I've seen in production:
from requests import ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError
try:
r = requests.get(url, timeout=6.0)
except (ConnectTimeout, HTTPError, ReadTimeout, Timeout, ConnectionError):
continue
Include the requests module using import requests .
It is always good to implement exception handling. It does not only help to avoid unexpected exit of script but can also help to log errors and info notification. When using Python requests I prefer to catch exceptions like this:
try:
res = requests.get(adress,timeout=30)
except requests.ConnectionError as e:
print("OOPS!! Connection Error. Make sure you are connected to Internet. Technical Details given below.\n")
print(str(e))
continue
except requests.Timeout as e:
print("OOPS!! Timeout Error")
print(str(e))
continue
except requests.RequestException as e:
print("OOPS!! General Error")
print(str(e))
continue
except KeyboardInterrupt:
print("Someone closed the program")
for clarity, that is
except requests.ConnectionError:
NOT
import requests.ConnectionError
You can also catch a general exception (although this isn't recommended) with
except Exception: