My question is closely related to this one.
I'm using the Requests library to hit an HTTP endpoint.
I want to check if the response is a success.
I am currently doing this:
r = requests.get(url)
if 200 <= response.status_code <= 299:
# Do something here!
Instead of doing that ugly check for values between 200 and 299, is there a shorthand I can use?
The response has an ok property. Use that:
if response.ok:
...
The implementation is just a try/except around Response.raise_for_status, which is itself checks the status code.
#property
def ok(self):
"""Returns True if :attr:`status_code` is less than 400, False if not.
This attribute checks if the status code of the response is between
400 and 600 to see if there was a client error or a server error. If
the status code is between 200 and 400, this will return True. This
is **not** a check to see if the response code is ``200 OK``.
"""
try:
self.raise_for_status()
except HTTPError:
return False
return True
I am a Python newbie but I think the easiest way is:
if response.ok:
# whatever
The pythonic way to check for requests success would be to optionally raise an exception with
try:
resp = requests.get(url)
resp.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err)
EAFP: It’s Easier to Ask for Forgiveness than Permission: You should just do what you expect to work and if an exception might be thrown from the operation then catch it and deal with that fact.
Related
How should I structure the code so I won't get this error:
UnboundLocalError: local variable 'r' referenced before assignment.
If I want to ensure I get a 200 response before returning r.json(), where should I place this code — inside or outside the try block?
if r.status_code == requests.code['ok']
My function:
def get_req():
url = 'https://www.example.com/search'
data = {'p': 'something'}
try:
r = requests.get(url, params=data)
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err)
return r.json()
I don't see any way you can get that error. I see no references to r that can occur unless your call to requests.get() succeeded and set r properly. Are you sure you're seeing that happen with the code you're showing us?
Why do you want to check for status code 200? You're already calling raise_for_status(), which basically does that. raise_for_status() checks for some other codes that mean success, but you probably want your code to treat those as success for your purpose as well. So I don't think you need to check for 200 explicitly.
So by the time you call r.json() and return, that should be what you want to do.
UPDATE: Now that you've removed the sys.exit(), you have to do something specific in the error case. In the comments I've given you the four possibilities I see. The simplest one would be to declare your method as returning None if the request fails. This returns the least information to the caller, but you are printing an error already, so that might be fine. For that case, your code would look like this:
def get_req():
url = 'https://www.example.com/search'
data = {'p': 'something'}
try:
r = requests.get(url, params=data)
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err)
return None
return r.json()
This code is "correct" if you define returning None on failure of the request, but throwing an exception in some other cases, as being the expected behavior. You might want to catch Exception instead or as a separate case, if you never want this method to throw an exception. I don't know if it's possible for requests.get() to throw some other exception than HTTPError. Maybe it isn't, in which case this code will never throw an exception as is. IMO, it's better to assume it can, and deal with that case explicitly. The code is much more readable that way and does not require future readers to know what exceptions requests.get() is able to throw to understand the behavior of this code in all cases.
your function should look like this:
def get_req():
url = 'https://www.example.com/search'
data = {'p': 'something'}
try:
r = requests.get(url, params=data)
r.raise_for_status()
if r.status_code == requests.code['ok']:
return r.json()
except requests.exceptions.HTTPError as err:
print(err)
sys.exit(1)
hope this helps
As seen here, max-retries can be set for requests.Session(), but I only need the head.status_code to check if a url is valid and active.
Is there a way to just get the head within a mount session?
import requests
def valid_active_url(url):
try:
site_ping = requests.head(url, allow_redirects=True)
except requests.exceptions.ConnectionError:
print('Error trying to connect to {}.'.format(url))
try:
if (site_ping.status_code < 400):
return True
else:
return False
except Exception:
return False
return False
Based on docs am thinking I need to either:
see if the session.mount method results return a status code (which I haven't found yet)
roll my own retry method, perhaps with a decorator like this or this or a (less eloquent) loop like this.
In terms of the first approach I have tried:
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=3)
s.mount('http://redirected-domain.com', a)
resp = s.get('http://www.redirected-domain.org')
resp.status_code
Are we only using s.mount() to get in and set max_retries? Seems to be a redundancy, aside from that the http connection would be pre-established.
Also resp.status_code returns 200 where I am expecting a 301 (which is what requests.head returns.
NOTE: resp.ok might be all I need for my purposes here.
After a mere two hours of developing the question, the answer took five minutes:
def valid_url(url):
if (url.lower() == 'none') or (url == ''):
return False
try:
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=5)
s.mount(url, a)
resp = s.head(url)
return resp.ok
except requests.exceptions.MissingSchema:
# If it's missing the schema, run again with schema added
return valid_url('http://' + url)
except requests.exceptions.ConnectionError:
print('Error trying to connect to {}.'.format(url))
return False
Based on this answer it looks like the head request will be slightly less resource intensive than the get, particularly if the url contains a large amount of data.
The requests.adapters.HTTPAdapter is the built in adaptor for the urllib3 library that underlies the Requests library.
On another note, I'm not sure what the correct term or phrase for what I'm checking here is. A url could still be valid if it returns an error code.
What is the easiest way to check whether the response received from a requests post was "200 OK" or an error has occurred?
I tried doing something like this:
....
resp = requests.post(my_endpoint_var, headers=header_var, data=post_data_var)
print(resp)
if resp == "<Response [200]>":
print ('OK!')
else:
print ('Boo!')
The output on the screen is:
Response [200] (including the "<" and ">")
Boo!
So even though I am getting a 200, my check in the if statement is somehow not matching?
According to the docs, there's a status_code property on the response-object. So you can do the following:
if resp.status_code == 200:
print ('OK!')
else:
print ('Boo!')
EDIT:
As others have pointed out, a simpler check would be
if resp.ok:
print ('OK!')
else:
print ('Boo!')
if you want to consider all 2xx response codes and not 200 explicitly.
You may also want to check Peter's answer for a more python-like way to do this.
Just check the response attribute resp.ok. It is True for all 2xx responses, but False for 4xx and 5xx. However, the pythonic way to check for success would be to optionally raise an exception with Response.raise_for_status():
try:
resp = requests.get(url)
resp.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err)
EAFP: It’s Easier to Ask for Forgiveness than Permission: You should just do what you expect to work and if an exception might be thrown from the operation then catch it and deal with that fact.
Much simpler check would be
if resp.ok :
print ('OK!')
else:
print ('Boo!')
Since any 2XX class response is considered successful in HTTP, I would use:
if 200 <= resp.status_code <= 299:
print ('OK!')
else:
print ('Boo!')
I'm surprised no one has mentioned this so:
If you want to check for exactly a 200 response:
if resp.status_code == requests.codes.ok:
The status_code of a response contains the HTTP status returned.
requests.codes.ok is exactly 200.
If you want to check if the status code is "ok", and not an error:
if resp.ok:
The ok attribute of a response checks that the status code of the response is less than 400.
Make sure you know what it is you want to check, for example a 201 HTTP Created is a successful response that you would be omitting if you only check for exactly 200.
resp.status_code will return the status code as an integer.
See http://docs.python-requests.org/en/master/
try:
if resp.status_code == 200:
print ('OK!')
else:
print ('Boo!)
In easy case:
import requests
response = requests.get(url)
if not response:
#handle error here
else:
#handle normal response
I am using the following code to resolve redirects to return a links final url
def resolve_redirects(url):
return urllib2.urlopen(url).geturl()
Unfortunately I sometimes get HTTPError: HTTP Error 429: Too Many Requests. What is a good way to combat this? Is the following good or is there a better way.
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
return urllib2.urlopen(url).geturl()
Also, what would happen if there is an exception in the except block?
It would be better to make sure the HTTP code is actually 429 before re-trying.
That can be done like this:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError, e:
if e.code == 429:
time.sleep(5);
return resolve_redirects(url)
raise
This will also allow arbitrary numbers of retries (which may or may not be desired).
https://docs.python.org/2/howto/urllib2.html#httperror
This is a fine way to handle the exception, though you should check to make sure you are always sleeping for the appropriate amount of time between requests for the given website (for example twitter limits the amount of requests per minute and has this amount clearly shown in their api documentation). So just make sure you're always sleeping long enough.
To recover from an exception within an exception, you can simply embed another try/catch block:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
return "Failed twice :S"
Edit: as #jesse-w-at-z points out, you should be returning an URL in the second error case, the code I posted is just a reference example of how to write a nested try/catch.
Adding User-Agent to request header solved my issue:
from urllib import request
from urllib.request import urlopen
url = 'https://www.example.com/abc.json'
req = request.Request(url)
req.add_header('User-Agent', 'abc-bot')
response = request.urlopen(req)
I have a program which uses the requests module to send a get request which (correctly) responds with a 304 "Not Modified". After making the request, I check to make sure response.status_code == requests.codes.ok, but this check fails. Does requests not consider a 304 as "ok"?
There is a property called ok in the Response object that returns True if the status code is not a 4xx or a 5xx.
So you could do the following:
if response.ok:
# 304 is included
The code of this property is pretty simple:
#property
def ok(self):
try:
self.raise_for_status()
except HTTPError:
return False
return True
You can check actual codes in the source. ok means 200 only.
You can check the implementation of requests.status code here source code.The implementation allows you to access all/any kind of status_codes as follow:
import requests
import traceback
url = "https://google.com"
req = requests.get(url)
try:
if req.status_code == requests.codes['ok']: # Check the source code for all the codes
print('200')
elif req.status_code == requests.codes['not_modified']: # 304
print("304")
elifreq.status_code == requests.codes['not_found']: # 404
print("404")
else:
print("None of the codes")
except:
traceback.print_exc(file=sys.stdout)
In conclusion, you can access any request-response like demonstrated. I am sure there are better ways but this worked for me.
.ok "..If the status code is between 200 and 400, this will return True."
mentioned in source code as:
"""Returns True if :attr:status_code is less than 400, False if not.
This attribute checks if the status code of the response is between
400 and 600 to see if there was a client error or a server error. If
the status code is between 200 and 400, this will return True. This
is not a check to see if the response code is 200 OK.
"""