We Keep Coding

Can I factor out parameter checks and responses in Flask?

I'm currently refactoring a larger code base which uses Flask. I see a lot of duplication around parameter checks. It's similar to this:
#api.route("/foo/bar/<string:content_id>")
def get_foo(content_id):
try:
content = get_content(content_id)
except:
return jsonify({"status": 404, "error": "not found"}), 404
...
#api.route("/foo/bar/<string:content_id>/update")
def update_foo(content_id):
try: # This is exactly the same!
content = get_content(content_id) # It's just copy-pasted
except: # and this way a lot of duplication
return jsonify({"status": 404, "error": "not found"}), 404 # is added! I want to reduce that.
...
The checks are longer and there are way more of them, but I hope this illustrates the issue well.
Is it possible to factor this check out? Maybe by using a decorator? How would that look like?
What I want
I would love to have something like this:
def get_content_validator():
try:
content = get_content(content_id)
except:
return jsonify({"status": 404, "error": "not found"}), 404
return content
#api.route("/foo/bar/<get_content_validator(string:content_id)>")
def get_foo(content_id):
...
#api.route("/foo/bar/<get_content_validator(string:content_id)>/update")
def update_foo(content):
...
The issue with that is that in one case the value is returned and the normal execution of get_foo / update_foo is done and in other cases the function body of get_foo / update_foo is not executed. This could be possible by checking the return type. If it's a "response" return type, the response is given. In all other cases it's just returned. But that sounds as if it could result in other issues.

I'd just make a decorator:
from functools import wraps
def validated_content(func):
#wraps(func)
def wrapper(content_id, *args, **kwargs):
try:
content = get_content(content_id)
except:
return jsonify(...)
return func(content, *args, **kwargs)
return wrapper
And then your handlers become:
#api.route("...")
#validated_content
def get_foo(content):
# do stuff with content

How to mock out custom throttle class for testing in django-rest-framework?

I have a custom throttle class like: (print statements are for debugging :)) in api.throttle.py
print("befor CustomThrottle class")
class CustomThrottle(BaseThrottle):
def __init__(self):
super().__init__()
print("initializing CustomThrottle", self)
self._wait = 0
def allow_request(self, request, view):
print("CustomThrottle.allow_request")
if request.method in SAFE_METHODS:
return True
# some checking here
if wait > 0:
self._wait = wait
return False
return True
def wait(self):
return self._wait
my api.views.py is like:
from api.throttle import CustomThrottle
print("views module")
class SomeView(APIView):
print("in view")
throttle_classes = [CustomThrottle]
def post(self, request, should_exist):
# some processing
return Response({"message": "Done."})
and my tests is api/tests/test_views.py:
#patch.object(api.views.CustomThrottle, "allow_request")
def test_can_get_confirmation_code_for_registered_user(self, throttle):
throttle.return_value = True
response = self.client.post(path, data=data)
self.assertEqual(
response.status_code,
status.HTTP_200_OK,
"Should be successful",
)
#patch("api.views.CustomThrottle")
def test_learn(self, throttle):
throttle.return_value.allow_request.return_value = True
response = self.client.post(path, data=data)
self.assertEqual(
response.status_code,
status.HTTP_200_OK,
"Should be successful",
)
the first test passes correctly but still the CustomThrottle class is instantiated but the allow_request method is mocked; the second test shows that neither CustomThrottle class is mocked nor the allow_request method and it fails with status 429 (CustomThrottle has a throttle rate of 2 minutes).

After spending some time and testing different scenarios (Putting debug like print statements here and there :) ) I finally found out why it's not working, But it may not be as correct as I think so any better answer is welcome.
I defined list of throttle classes like throttle_classes = [CustomThrottle] so when server starts it's imported and evaluated so there's a reference to the actual and unmocked version of CustomThrottle in my SomeView class and when processing the responses it will use that for instantiation and so my test fails, but when I patch it like patch.object(CustomThrottle, "allow_request") when view actually needs to check throttles it will call something like CustomThrottle().allow_request(...) (pay attention to parentheses for object creation); the object has not a method like allow_request in itself so it searches that in it's class and use the mocked version correctly.

If you want also to mock CustomThrottle class, and his properties, you also need to patch in places where do you need it, because of patch decorator applies monkey patching in places where you calling it.
Regarding your case, you able to do something like
from unittest.mock import MagicMock, patch
#patch('api.views.CustomThrottle')
#patch('api.views.PhoneConfirmationThrottle')
def test_learn(self, phone_throttle, custom_throttle):
mocked_custom_throttle_instance = MagicMock()
mocked_custom_throttle_instance.allow_request = True
mocked_custom_throttle_instance.status = ...
# Do some stuff which do you need with the mocked class
# and then return an instance for your patched object
custom_throttle.return_value = mocked_custom_throttle_instance
response = self.client.post(path, data=data)
self.assertEqual(
response.status_code,
status.HTTP_200_OK,
"Should be successful",
)
It will replace your real object with the mocked one, also check out unittest.mock documentation, it will help you in the future.

How to access the instance of a class from an inner decorator class?

I have a class that handles the API calls to a server. Certain methods within the class require the user to be logged in. Since it is possible for the session to run out, I need some functionality that re-logins the user once the session timed out. My idea was to use a decorator. If I try it like this
class Outer_Class():
class login_required():
def __init__(self, decorated_func):
self.decorated_func = decorated_func
def __call__(self, *args, **kwargs):
try:
response = self.decorated_func(*args, **kwargs)
except:
print('Session probably timed out. Logging in again ...')
args[0]._login()
response = self.decorated_func(*args, **kwargs)
return response
def __init__(self):
self.logged_in = False
self.url = 'something'
self._login()
def _login(self):
print(f'Logging in on {self.url}!')
self.logged_in = True
#this method requires the user to be logged in
#login_required
def do_something(self, param_1):
print('Doing something important with param_1')
if (): #..this fails
raise Exception()
I get an error. AttributeError: 'str' object has no attribute '_login'
Why do I not get a reference to the Outer_Class-instance handed over via *args? Is there another way to get a reference to the instance?
Found this answer How to get instance given a method of the instance? , but the decorated_function doesn't seem to have a reference to it's own instance.
It works fine, when Im using a decorator function outside of the class. This solves the problem, but I like to know, if it is possible to solve the this way.

The problem is that the magic of passing the object as the first hidden parameter only works for a non static method. As your decorator returns a custom callable object which is not a function, it never receives the calling object which is just lost in the call. So when you try to call the decorated function, you only pass it param_1 in the position of self. You get a first exception do_something() missing 1 required positional argument: 'param_1', fall into the except block and get your error.
You can still tie the decorator to the class, but it must be a function to have self magic work:
class Outer_Class():
def login_required(decorated_func):
def inner(self, *args, **kwargs):
print("decorated called")
try:
response = decorated_func(self, *args, **kwargs)
except:
print('Session probably timed out. Logging in again ...')
self._login()
response = decorated_func(self, *args, **kwargs)
return response
return inner
...
#this method requires the user to be logged in
#login_required
def do_something(self, param_1):
print('Doing something important with param_1', param_1)
if (False): #..this fails
raise Exception()
You can then successfully do:
>>> a = Outer_Class()
Logging in on something!
>>> a.do_something("foo")
decorated called
Doing something important with param_1

You have the command of
args[0]._login()
in the except. Since args[0] is a string and it doesn't have a _login method, you get the error message mentioned in the question.

pytest: How to force raising Exceptions during unit-testing?

In my python code, I am expecting exceptions could possibly be raised after calling method requests.Session.request(), for example these:
requests.exceptions.ConnectTimeout
requests.exceptions.ReadTimeout
requests.exceptions.Timeout
When any of these expected exceptions are raised, I handle them appropriately, for example possibly a retry situation.
My question, I am using py.test for unit testing, and I purposely want to inject raising exceptions from specific parts of my code. For example, the function that calls requests.Session.request(), instead of returning a valid requests.Response, it raises a requests.exception.
What I want to make sure that my code successfully handles expected and unexpected exceptions coming from other packages, which include those exceptions from requests.
Maybe... Is there a #decorator that I could add to the aforementioned function to raise exceptions upon request during unit testing?
Suggestions for doing exceptions injections for unit testing? (proper phrasing of my question would be greatly appreciated.)
Thanks for the responses!!!
Here is the entire singleton class that creates requests.Session and calls requests.Session.request():
class MyRequest(metaclass=Singleton):
def __init__(self, retry_tries=3, retry_backoff=0.1, retry_codes=None):
self.session = requests.session()
if retry_codes is None:
retry_codes = set(REQUEST_RETRY_HTTP_STATUS_CODES)
self.session.mount(
'http',
HTTPAdapter(
max_retries=Retry(
total=retry_tries,
backoff_factor=retry_backoff,
status_forcelist=retry_codes,
),
),
)
def request(self, request_method, request_url, **kwargs):
try:
return self.session.request(method=request_method, url=request_url, **kwargs)
except Exception as ex:
log.warning(
"Session Request: Failed: {}".format(get_exception_message(ex)),
extra={
'request_method': request_method,
'request_url': request_url
}
)
raise

You can make use of py.test raises, check it here: http://doc.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions
Taking into account your code you could do something along the lines of the following:
from requests.exceptions import ConnectTimeout, ReadTimeout, Timeout
from unittest.mock import patch
import pytest
class TestRequestService:
#patch('path_to_module.MyRequest')
def test_custom_request(self, my_request_mock):
my_request_mock.request.side_effect = ConnectTimeout
with pytest.raises(ConnectTimeout):
my_request_mock.request(Mock(), Mock())
Moreover, you could make use of pytest.parametrize(http://doc.pytest.org/en/latest/parametrize.html) as well:
from requests.exceptions import ConnectTimeout, ReadTimeout, Timeout
from unittest.mock import patch
import pytest
class TestRequestService:
#pytest.mark.parametrize("expected_exception", [ConnectTimeout, ReadTimeout, Timeout])
#patch('path_to_module.MyRequest')
def test_custom_request(self, my_request_mock, expected_exception):
my_request_mock.request.side_effect = expected_exception
with pytest.raises(expected_exception):
my_request_mock.request(Mock(), Mock())
Here you can find some more examples about parametrize: http://layer0.authentise.com/pytest-and-parametrization.html

In my application I am catching exception requests.exceptions.ConnectionError
and returning message which is in expected variable below.
So the test looks like this:
import pytest
import requests
expected = {'error': 'cant connect to given url'}
class MockConnectionError:
def __init__(self, *args, **kwargs):
raise requests.exceptions.ConnectionError
def test_project_method(monkeypatch):
monkeypatch.setattr("requests.get", MockConnectionError)
response = project_method('http://some.url.com/')
assert response == expected

Patching, mocking and dependecy-injection are techniques to inject fake objects. Patching is sometimes hard to do right, on the other hand dependency injection requires that have to change the code you want to test.
This is just a simple example how to use dependency-injection. First the code we want to test:
import requests
...
def fetch_data(url, get=requests.get):
return get(url).json()
# this is how we use fetch_data in productive code:
answer = fetch_data("www.google.com?" + term)
And this is then the test:
import pytest
def test_fetch():
def get_with_timeout(url):
raise ConnectTimeout("message")
with pytest.raises(ConnectTimeout) as e:
# and now we inject the fake get method:
fetch_data("https://google.com", get=get_with_timeout)
assert e.value == "message"
In your example above, the mocking technique would be as follows:
def test_exception():
class TimeoutSessionMock:
def get(self, *args, **kwargs):
raise ConnectTimeout("message")
mr = MyRequest()
mr.session = TimeoutSessionMock()
with pytest.raises(ConnectTimeout) as e:
mr.request("get", "http://google.com")
assert e.value == "message"

How do I return HTTP error code without default template in Tornado?

I am currently using the following to raise a HTTP bad request:
raise tornado.web.HTTPError(400)
which returns a html output:
<html><title>400: Bad Request</title><body>400: Bad Request</body></html>
Is it possible to return just the HTTP response code with a custom body?

You may simulate RequestHandler.send_error method:
class MyHandler(tornado.web.RequestHandler):
def get(self):
self.clear()
self.set_status(400)
self.finish("<html><body>My custom body</body></html>")

Tornado calls RequestHandler.write_error to output errors, so an alternative to VisioN's approach would be override it as suggested by the Tornado docs. The advantage to this approach is that it will allow you to raise HTTPError as before.
The source for RequestHandler.write_error is here. Below you can see an example of a simple modification of write_error that will change the set the status code and change the output if you provide a reason in kwargs.
def write_error(self, status_code, **kwargs):
if self.settings.get("serve_traceback") and "exc_info" in kwargs:
# in debug mode, try to send a traceback
self.set_header('Content-Type', 'text/plain')
for line in traceback.format_exception(*kwargs["exc_info"]):
self.write(line)
self.finish()
else:
self.set_status(status_code)
if kwargs['reason']:
self.finish(kwargs['reason'])
else:
self.finish("<html><title>%(code)d: %(message)s</title>"
"<body>%(code)d: %(message)s</body></html>" % {
"code": status_code,
"message": self._reason,
})

It's better to use the standard interface and define your custom message on the HTTPError.
raise tornado.web.HTTPError(status_code=code, log_message=custom_msg)
You can then parse the error in your RequestHandler and check for the message:
class CustomHandler(tornado.web.RequestHandler):
def write_error(self, status_code, **kwargs):
err_cls, err, traceback = kwargs['exc_info']
if err.log_message and err.log_message.startswith(custom_msg):
self.write("<html><body><h1>Here be dragons</h1></body></html>")

def write_error(self, status_code, **kwargs):
#Function to display custom error page defined in the handler.
#Over written from base handler.
data = {}
data['code'] = status_code
data['message'] = httplib.responses[status_code]
# your other conditions here to create data dict
self.write(TEMPLATES.load('error.html').generate(data=data))
when ever self.send_error() call is initiated write_error() function is called by the request handler. So you can create your custom error data dict here and render it to your custom error page.
http.responses[status_code] returns the error code text like "page not found" based on the status code.

Also you can override get_error_html method in your handler.
For example:
import tornado.web
class CustomHandler(tornado.web.RequestHandler):
def get_error_html(self, status_code, **kwargs);
self.write("<html><body><h1>404!</h1></body></html>")
...
def get(self):
...

This exchange clarifies some of the approaches suggested here, and discounts the reason keyword (which I was thinking about trying).
Q: (by mrtn)
"I want to use raise tornado.web.HTTPError(400, reason='invalid request') to pass a custom reason to the error response, and I hope to do this by overriding the write_error (self, status_code, **kwargs) method.
"But it seems that I can only access self._reason inside write_error, which is not what I want. I also tried kwargs['reason'] but that does not exist."
A: (by Tornado lead developer #bendarnell)
"The exception that exposed the error is available to write_error as an exc_info triple in the keyword arguments. You can access the reason field with something like this:
if "exc_info" in kwargs:
e = kwargs["exc_info"][1]
if isinstance(e, tornado.web.HTTPError):
reason = e.reason
"But note that the reason field is essentially deprecated (it is not present in HTTP/2), so it's probably not the best way to do whatever you're trying to do here (HTTPError's log_message field is a little better, but still not ideal). Just raise your own exception instead of using HTTPError; your write_error override can use self.set_status(400) when it sees the right kind of exception."

For json error response i use follow template:
Request handler:
import json
from tornado.web import RequestHandler
from src.lib.errors import HTTPBadRequest
class JsonHandler(RequestHandler):
def prepare(self):
content_type = ''
if "Content-Type" in self.request.headers:
content_type = self.request.headers['Content-Type']
if content_type == 'application/json':
try:
self.request.body = json.loads(self.request.body.decode('utf-8'))
except ValueError:
raise HTTPBadRequest
def write_error(self, *args, **kwargs):
err_cls, err, traceback = kwargs['exc_info']
self.set_status(err.status_code)
if err.description:
self.write_json(err.description)
self.finish()
def set_default_headers(self):
self.set_header('Content-Type', 'application/json')
def write_json(self, response):
self.write(json.dumps(response))
Errors handler:
from typing import Any
from tornado import httputil
class BaseHTTPError(Exception):
def __init__(
self, status_code: int = 500, description=None, *args: Any, **kwargs: Any
) -> None:
if description is None:
description = {}
self.status_code = status_code
self.description = description
self.args = args
self.kwargs = kwargs
def __str__(self) -> str:
message = "HTTP %d: %s" % (
self.status_code,
httputil.responses.get(self.status_code, "Unknown"),
)
return message
class HTTPBadRequest(BaseHTTPError):
def __init__(self, *args, **kwargs):
super().__init__(status_code=400, description={"error": "Bad Request"}, *args, **kwargs)