Need to capture the response body for a HTTP error in python. Currently using the python request module's raise_for_status(). This method only returns the Status Code and description. Need a way to capture the response body for a detailed error log.
Please suggest alternatives to python requests module if similar required feature is present in some different module. If not then please suggest what changes can be done to existing code to capture the said response body.
Current implementation contains just the following:
resp.raise_for_status()
I guess I'll write this up quickly. This is working fine for me:
try:
r = requests.get('https://www.google.com/404')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err.request.url)
print(err)
print(err.response.text)
you can do something like below, which returns the content of the response, in unicode.
response.text
or
try:
r = requests.get('http://www.google.com/nothere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err)
sys.exit(1)
# 404 Client Error: Not Found for url: http://www.google.com/nothere
here you'll get the full explanation on how to handle the exception. please check out Correct way to try/except using Python requests module?
You can log resp.text if resp.status_code >= 400.
There are some tools you may pick up such as Fiddler, Charles, wireshark.
However, those tools can just display the body of the response without including the reason or error stack why the error raises.
Related
There are many situations where I know an error will occur and want to pass additional data to Sentry. I still want the exception to get raised (so the rest of my code stops) and I only want one error in sentry.
For example, let's say I'm making an HTTP call and, in the event that the HTTP call fails, I want an error including the response text sent to Sentry:
import requests
resp = requests.post(url, json=payload)
if resp.ok:
return resp.json()
try:
text = resp.json()
except json.JSONDecodeError:
text = resp.text
# TODO: add `text` to Sentry error
resp.raise_for_status()
How do I do this using the Sentry Python SDK?
Rejected solutions:
Sentry's logging: this results in two errors in sentry (one for the log statement and one for the raised exception)
capture_expection: this results in two errors in sentry (one for the captured exception and one for the raised exception)
Adding extra details to the exception message: this breaks Sentry's error grouping because each error has a unique name
Large or Unpredictable Data: set_context
If you need to send a lot of data or you don't know the contents of your data, the function you are looking for is Sentry's set_context. You want to call this function right before your exception gets raised. Note that context objects are limited in size to 8kb.
Note: you should only call set_context in a situation where an exception will definitely get raised, or else the extra information you set may get added to other (irrelevant) errors in Sentry.
For example:
import requests
import sentry_sdk
resp = requests.post(url, json=payload)
if resp.ok:
return resp.json()
try:
text = resp.json()
except json.JSONDecodeError:
text = resp.text
# vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
sentry_sdk.set_context("Payload", {"text": text})
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
resp.raise_for_status()
This attaches it as additional data to your Sentry error, listed just after the breadcrumbs:
Small, Predictable Data: set_tag
If your data is small and predictable (such as the HTTP status code), you can use sentry's set_tag. It's best to do this within a push_scope block so that the tag is just set for the area of your code that may go wrong. Note that tags keys are limited to 32 characters and tag values are limited in size to 200 characters.
Tags show up at the top of the view of a sentry error.
For example:
import requests
from sentry_sdk import push_scope
resp = requests.post(url, json=payload)
if resp.ok:
return resp.json()
# vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
with push_scope() as scope:
sentry_sdk.set_tag("status", resp.status_code)
resp.raise_for_status()
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I am using python 3.8, Flask 1.1.2
While trying to handle errors can't figure out a way to return status code and break code when error is found.
When everything runs fine, program return statement is as follows
return jsonify({'status':'success', 'prediction':pred}), 200
which allow me to access status_code
response = requests.post(url_path, json=data)
print(response.status_code)
>>> 200
However when error arise before reaching end of code I've tried to handle error like this:
code....
try:
code
except KeyError:
return jsonify({'error_message':'something wrong with input'}), 10
code...
return jsonify({"status":"success!", "best_actions":final_actions}), 200
When except statement is executed it outputs ConnectionError: ('Connection aborted.', BadStatusLine('HTTP/1.0 10 UNKNOWN\r\n')) which seems to happen when python client receives empty response according to Connection aborted.', BadStatusLine("''",) on server?
changing except statement like:
expect KeyError:
return jsonify({'error_message':'something wrong with input'})
allow me to obtain response.json() however cannot get response.status_code.
returning with http status_code works:
expect KeyError:
return jsonify({'error_message':'something wrong with input'}), 1xx
above code works fine however I am trying to create custom status_codes therefore I can add detailed reason and solution in my documentation.
Any help or guide to helpful resource would be greatly appreciated, thanks!
How should I structure the code so I won't get this error:
UnboundLocalError: local variable 'r' referenced before assignment.
If I want to ensure I get a 200 response before returning r.json(), where should I place this code — inside or outside the try block?
if r.status_code == requests.code['ok']
My function:
def get_req():
url = 'https://www.example.com/search'
data = {'p': 'something'}
try:
r = requests.get(url, params=data)
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err)
return r.json()
I don't see any way you can get that error. I see no references to r that can occur unless your call to requests.get() succeeded and set r properly. Are you sure you're seeing that happen with the code you're showing us?
Why do you want to check for status code 200? You're already calling raise_for_status(), which basically does that. raise_for_status() checks for some other codes that mean success, but you probably want your code to treat those as success for your purpose as well. So I don't think you need to check for 200 explicitly.
So by the time you call r.json() and return, that should be what you want to do.
UPDATE: Now that you've removed the sys.exit(), you have to do something specific in the error case. In the comments I've given you the four possibilities I see. The simplest one would be to declare your method as returning None if the request fails. This returns the least information to the caller, but you are printing an error already, so that might be fine. For that case, your code would look like this:
def get_req():
url = 'https://www.example.com/search'
data = {'p': 'something'}
try:
r = requests.get(url, params=data)
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err)
return None
return r.json()
This code is "correct" if you define returning None on failure of the request, but throwing an exception in some other cases, as being the expected behavior. You might want to catch Exception instead or as a separate case, if you never want this method to throw an exception. I don't know if it's possible for requests.get() to throw some other exception than HTTPError. Maybe it isn't, in which case this code will never throw an exception as is. IMO, it's better to assume it can, and deal with that case explicitly. The code is much more readable that way and does not require future readers to know what exceptions requests.get() is able to throw to understand the behavior of this code in all cases.
your function should look like this:
def get_req():
url = 'https://www.example.com/search'
data = {'p': 'something'}
try:
r = requests.get(url, params=data)
r.raise_for_status()
if r.status_code == requests.code['ok']:
return r.json()
except requests.exceptions.HTTPError as err:
print(err)
sys.exit(1)
hope this helps
I am using the following code to resolve redirects to return a links final url
def resolve_redirects(url):
return urllib2.urlopen(url).geturl()
Unfortunately I sometimes get HTTPError: HTTP Error 429: Too Many Requests. What is a good way to combat this? Is the following good or is there a better way.
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
return urllib2.urlopen(url).geturl()
Also, what would happen if there is an exception in the except block?
It would be better to make sure the HTTP code is actually 429 before re-trying.
That can be done like this:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError, e:
if e.code == 429:
time.sleep(5);
return resolve_redirects(url)
raise
This will also allow arbitrary numbers of retries (which may or may not be desired).
https://docs.python.org/2/howto/urllib2.html#httperror
This is a fine way to handle the exception, though you should check to make sure you are always sleeping for the appropriate amount of time between requests for the given website (for example twitter limits the amount of requests per minute and has this amount clearly shown in their api documentation). So just make sure you're always sleeping long enough.
To recover from an exception within an exception, you can simply embed another try/catch block:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
return "Failed twice :S"
Edit: as #jesse-w-at-z points out, you should be returning an URL in the second error case, the code I posted is just a reference example of how to write a nested try/catch.
Adding User-Agent to request header solved my issue:
from urllib import request
from urllib.request import urlopen
url = 'https://www.example.com/abc.json'
req = request.Request(url)
req.add_header('User-Agent', 'abc-bot')
response = request.urlopen(req)
I'm trying to write a small program that will simply display the header information of a website. Here is the code:
import urllib2
url = 'http://some.ip.add.ress/'
request = urllib2.Request(url)
try:
html = urllib2.urlopen(request)
except urllib2.URLError, e:
print e.code
else:
print html.info()
If 'some.ip.add.ress' is google.com then the header information is returned without a problem. However if it's an ip address that requires basic authentication before access then it returns a 401. Is there a way to get header (or any other) information without authentication?
I've worked it out.
After try has failed due to unauthorized access the following modification will print the header information:
print e.info()
instead of:
print e.code()
Thanks for looking :)
If you want just the headers, instead of using urllib2, you should go lower level and use httplib
import httplib
conn = httplib.HTTPConnection(host)
conn.request("HEAD", path)
print conn.getresponse().getheaders()
If all you want are HTTP headers then you should make HEAD not GET request. You can see how to do this by reading Python - HEAD request with urllib2.