We Keep Coding

How to get the exception string from requests.exceptions.RequestException

I have the below flask code :
from flask import Flask,request,jsonify
import requests
from werkzeug.exceptions import InternalServerError, NotFound
import sys
import json
app = Flask(__name__)
app.config['SECRET_KEY'] = "Secret!"
class InvalidUsage(Exception):
status_code = 400
def __init__(self, message, status_code=None, payload=None):
Exception.__init__(self)
self.message = message
if status_code is not None:
self.status_code = status_code
self.payload = payload
def to_dict(self):
rv = dict(self.payload or ())
rv['message'] = self.message
rv['status_code'] = self.status_code
return rv
#app.errorhandler(InvalidUsage)
def handle_invalid_usage(error):
response = jsonify(error.to_dict())
response.status_code = error.status_code
return response
#app.route('/test',methods=["GET","POST"])
def test():
url = "https://httpbin.org/status/404"
try:
response = requests.get(url)
if response.status_code != 200:
try:
response.raise_for_status()
except requests.exceptions.HTTPError:
status = response.status_code
print status
raise InvalidUsage("An HTTP exception has been raised",status_code=status)
except requests.exceptions.RequestException as e:
print e
if __name__ == "__main__":
app.run(debug=True)
My question is how do i get the exception string(message) and other relevant params from the requests.exceptions.RequestException object e ?
Also what is the best way to log such exceptions . In case of an HTTPError exceptions i have the status code to refer to.
But requests.exceptions.RequestException catches all request exceptions . So how do i differentiate between them and also what is the best way to log them apart from using print statements.
Thanks a lot in advance for any answers.

RequestException is a base class for HTTPError, ConnectionError, Timeout, URLRequired, TooManyRedirects and others (the whole list is available at the GitHub page of requests module). Seems that the best way of dealing with each error and printing the corresponding information is by handling them starting from more specific and finishing with the most general one (the base class). This has been elaborated widely in the comments in this StackOverflow topic. For your test() method this could be:
#app.route('/test',methods=["GET","POST"])
def test():
url = "https://httpbin.org/status/404"
try:
# some code...
except requests.exceptions.ConnectionError as ece:
print("Connection Error:", ece)
except requests.exceptions.Timeout as et:
print("Timeout Error:", et)
except requests.exceptions.RequestException as e:
print("Some Ambiguous Exception:", e)
This way you can firstly catch the errors that inherit from the RequestException class and which are more specific.
And considering an alternative for printing statements - I'm not sure if that's exactly what you meant, but you can log into console or to a file with standard Python logging in Flask or with the logging module itself (here for Python 3).

This is actually not a question about using the requests library as much as it is a general Python question about how to extract the error string from an exception instance. The answer is relatively straightforward: you convert it to a string by calling str() on the exception instance. Any properly written exception handler (in requests or otherwise) would have implemented an __str__() method to allow an str() call on an instance. Example below:
import requests
rsp = requests.get('https://httpbin.org/status/404')
try:
if rsp.status_code >= 400:
rsp.raise_for_status()
except requests.exceptions.RequestException as e:
error_str = str(e)
# log 'error_str' to disk, a database, etc.
print('The error was:', error_str)
Yes, in this example, we print it, but once you have the string you have additional options. Anyway, saving this to test.py results in the following output given your test URL:
$ python3 test.py
The error was: 404 Client Error: NOT FOUND for url: https://httpbin.org/status/404

How to mock requests methods called dynamically using getattr

I have class which called requests method using getattr like this:
import requests
class CustomRequests(object):
def __init__(self):
pass
def _do_requests(self, method='GET', url='', expected_status=200):
make_request = getattr(requests, method.lower())
url = url if url else 'http://example.com'
try:
response = make_request(method, url=url)
except response.exceptions.RequestException as exception:
raise exception
if response.status_code != expected_status:
raise ValueError
def get(self, *args, **kwargs):
self._do_requests(method='GET', *args, **kwargs)
I am trying to test the api using mock and responses lib like this:
import responses
#responses.activate
def test_get_method(self):
responses.add('GET', url='http://test_this_api.com', status=200)
custom_request = CustomRequest()
response_data = custom_request.get(method='GET')
AssertIsNotNone(response_data)
Is there any better or right way to test this method.
Getting this error:
message = message.format(**values)
KeyError: 'method'

There's no need to use getattr. requests.get, requests.post, etc. are just convenience methods for requests.request, which lets you pass the HTTP method as a parameter:
requests.request('GET', url) # equivalent to requests.get(url)
Also:
Your try/except is pointless, since all you do is re-raise the exception.
It doesn't make sense to raise a ValueError when the response status doesn't match what you expected. ValueError is for "when a built-in operation or function receives an argument that has the right type but an inappropriate value." Create your own exception class, e.g. UnexpectedHTTPStatusError.
Since the whole point of your CustomRequests class seems to be to raise an exception when the status code of the response doesn't match what the user expected, your tests should assert that an exception was actually raised with assertRaises().

Custom response during falcon middleware exception

I'm writing Falcon middleware for my application. When i get any errors i want to raise error, break process and return my custom response, that looks like:
{
"status": 503,
"message": "No Token found. Token is required."
}
But standard Falcon error implementation does not allow me to set custom fields to my response.
How to solve this problem most properly?

After a lot of time spent, I solved this problem in such interesting way. I put my code in a try/catch block, and when an error is caught I decided not to raise Falcon error, and just tried to write return keyword after setting response status and body, because the method is void, so it does not return anything. Now it looks like:
resp.status = falcon.HTTP_403
resp.body = body
return

I was still looking for an example and here is for anyone who still need it:
from falcon.http_error import HTTPError
class MyHTTPError(HTTPError):
"""Represents a generic HTTP error.
"""
def __init__(self, status, error):
super(MyHTTPError, self).__init__(status)
self.status = status
self.error = error
def to_dict(self, obj_type=dict):
"""Returns a basic dictionary representing the error.
"""
super(MyHTTPError, self).to_dict(obj_type)
obj = self.error
return obj
using:
error = {"error": [{"message": "Auth token required", "code": "INVALID_HEADER"}]}
raise MyHTTPError(falcon.HTTP_400, error)

Create custom exception class explained in falcon docs, search for add_error_handler
class RaiseUnauthorizedException(Exception):
def handle(ex, req, resp, params):
resp.status = falcon.HTTP_401
response = json.loads(json.dumps(ast.literal_eval(str(ex))))
resp.body = json.dumps(response)
Add custom exception class to falcon API object
api = falcon.API()
api.add_error_handler(RaiseUnauthorizedException)

raise falcon.HTTPError(falcon.HTTP_503, 'No Token found. Token is required.')

Ensure the POST data is valid JSON

I am developping a JSON API with Python Flask.
What I want is to always return JSON, with a error message indicating any error that occured.
That API also only accept JSON data in the POST body, but Flask by default return a HTML error 400 if it can't read the data as JSON.
Preferably, I d also like to not force the user to send the Content-Type header, and if raw or text content-type, try to parse the body as JSON nonetheless.
In short, I need a way to validate that the POST body's is JSON, and handle the error myself.
I've read about adding decorator to request to do that, but no comprehensive example.

You have three options:
Register a custom error handler for 400 errors on the API views. Have this error return JSON instead of HTML.
Set the Request.on_json_loading_failed method to something that raises a BadRequest exception subclass with a JSON payload. See Custom Errors in the Werkzeug exceptions documentation to see how you can create one.
Put a try: except around the request.get_json() call, catch the BadRequest exception and raise a new exception with a JSON payload.
Personally, I'd probably go with the second option:
from werkzeug.exceptions import BadRequest
from flask import json, Request, _request_ctx_stack
class JSONBadRequest(BadRequest):
def get_body(self, environ=None):
"""Get the JSON body."""
return json.dumps({
'code': self.code,
'name': self.name,
'description': self.description,
})
def get_headers(self, environ=None):
"""Get a list of headers."""
return [('Content-Type', 'application/json')]
def on_json_loading_failed(self):
ctx = _request_ctx_stack.top
if ctx is not None and ctx.app.config.get('DEBUG', False):
raise JSONBadRequest('Failed to decode JSON object: {0}'.format(e))
raise JSONBadRequest()
Request.on_json_loading_failed = on_json_loading_failed
Now, every time request.get_json() fails, it'll call your custom on_json_loading_failed method and raise an exception with a JSON payload rather than a HTML payload.

Combining the options force=True and silent=True make the result of request.get_json be None if the data is not parsable, then a simple if allow you to check the parsing.
from flask import Flask
from flask import request
#app.route('/foo', methods=['POST'])
def function(function = None):
print "Data: ", request.get_json(force = True, silent = True);
if request.get_json() is not None:
return "Is JSON";
else:
return "Nope";
if __name__ == "__main__":
app.run()
Credits to lapinkoira and Martijn Pieters.

You can try to decode JSON object using python json library.
The main idea is to take plain request body and try to convert to JSON.E.g:
import json
...
# somewhere in view
def view():
try:
json.loads(request.get_data())
except ValueError:
# not a JSON! return error
return {'error': '...'}
# do plain stuff

Implementing API exception flask-restful

I am trying to catch the Exception which is raised when the url provided is a messy and wrong url and then return the error response as JSON. This is what i did to implement this logic.
The exception is raised inside the Analysis class when the key_id is not a valid key for S3.
def url_error(status_code, message, reason):
response = jsonify({
'status': status_code,
'message': message,
'reason': reason
})
response.status_code = status_code
return response
class RowColumnCount(Resource):
def get(self, key_id):
try:
rc = Analysis(key_id=key_id)
except S3ResponseError as e:
return url_error(e.status, e.message, e.reason)
json_response = json.loads(rc.count_rows_columns())
return json_response
The above code works fine but its kinda getting repetitive for 50 different Resource classes. Each Resource class should handle this specific error. How to make it a decorator, such that code repetitiveness is reduced.
I am using Flask, Flask-Restful, Python 3.4.3

There are a couple of ways you can achieve what you're trying to do but I think the cleanest way is to extend the Resource class as described in the Flask-Restful docs here and create a new decorator whose job is to catch the S3ResponseError and return the appropriate response. You can then subclass all your resources from your new base Resource class.
Also I would suggest you specify an API level json_output method as described here and here so that way all you have to do is return a dict from any of your resources and they'll be converted to JSON appropriately.