I am writing an application that performs REST operations using Kenneth Reitz's requests library and I'm struggling to find a nice way to unit test these applications, because requests provides its methods via module-level methods.
What I want is the ability to synthesize the conversation between the two sides; provide a series of request assertions and responses.
It is in fact a little strange that the library has a blank page about end-user unit testing, while targeting user-friendliness and ease of use. There's however an easy-to-use library by Dropbox, unsurprisingly called responses. Here is its intro post. It says they've failed to employ httpretty, while stating no reason of the fail, and written a library with similar API.
import unittest
import requests
import responses
class TestCase(unittest.TestCase):
#responses.activate
def testExample(self):
responses.add(**{
'method' : responses.GET,
'url' : 'http://example.com/api/123',
'body' : '{"error": "reason"}',
'status' : 404,
'content_type' : 'application/json',
'adding_headers' : {'X-Foo': 'Bar'}
})
response = requests.get('http://example.com/api/123')
self.assertEqual({'error': 'reason'}, response.json())
self.assertEqual(404, response.status_code)
If you use specifically requests try httmock. It's wonderfully simple and elegant:
from httmock import urlmatch, HTTMock
import requests
# define matcher:
#urlmatch(netloc=r'(.*\.)?google\.com$')
def google_mock(url, request):
return 'Feeling lucky, punk?'
# open context to patch
with HTTMock(google_mock):
# call requests
r = requests.get('http://google.com/')
print r.content # 'Feeling lucky, punk?'
If you want something more generic (e.g. to mock any library making http calls) go for httpretty.
Almost as elegant:
import requests
import httpretty
#httpretty.activate
def test_one():
# define your patch:
httpretty.register_uri(httpretty.GET, "http://yipit.com/",
body="Find the best daily deals")
# use!
response = requests.get('http://yipit.com')
assert response.text == "Find the best daily deals"
HTTPretty is far more feature-rich - it offers also mocking status code, streaming responses, rotating responses, dynamic responses (with a callback).
You could use a mocking library such as Mocker to intercept the calls to the requests library and return specified results.
As a very simple example, consider this class which uses the requests library:
class MyReq(object):
def doSomething(self):
r = requests.get('https://api.github.com', auth=('user', 'pass'))
return r.headers['content-type']
Here's a unit test that intercepts the call to requests.get and returns a specified result for testing:
import unittest
import requests
import myreq
from mocker import Mocker, MockerTestCase
class MyReqTests(MockerTestCase):
def testSomething(self):
# Create a mock result for the requests.get call
result = self.mocker.mock()
result.headers
self.mocker.result({'content-type': 'mytest/pass'})
# Use mocker to intercept the call to requests.get
myget = self.mocker.replace("requests.get")
myget('https://api.github.com', auth=('user', 'pass'))
self.mocker.result(result)
self.mocker.replay()
# Now execute my code
r = myreq.MyReq()
v = r.doSomething()
# and verify the results
self.assertEqual(v, 'mytest/pass')
self.mocker.verify()
if __name__ == '__main__':
unittest.main()
When I run this unit test I get the following result:
.
----------------------------------------------------------------------
Ran 1 test in 0.004s
OK
Missing from these answers is requests-mock.
From their page:
>>> import requests
>>> import requests_mock
As a context manager:
>>> with requests_mock.mock() as m:
... m.get('http://test.com', text='data')
... requests.get('http://test.com').text
...
'data'
Or as a decorator:
>>> #requests_mock.mock()
... def test_func(m):
... m.get('http://test.com', text='data')
... return requests.get('http://test.com').text
...
>>> test_func()
'data'
using mocker like in srgerg's answer:
def replacer(method, endpoint, json_string):
from mocker import Mocker, ANY, CONTAINS
mocker = Mocker()
result = mocker.mock()
result.json()
mocker.count(1, None)
mocker.result(json_string)
replacement = mocker.replace("requests." + method)
replacement(CONTAINS(endpoint), params=ANY)
self.mocker.result(result)
self.mocker.replay()
For the requests library, this would intercept the request by method and endpoint you're hitting and replace the .json() on the response with the json_string passed in.
If you break out your response handler/parser into a separate function, you can work with requests.Response objects directly, without needing to mock the client-server interaction.
Code under test
from xml.dom import minidom
from requests.models import Response
def function_under_test(s3_response: Response):
doc = minidom.parseString(s3_response.text)
return (
s3_response.status_code,
doc.getElementsByTagName('Code').item(0).firstChild.data,
)
Test code
import unittest
from io import BytesIO
class Test(unittest.TestCase):
def test_it(self):
s3_response = Response()
s3_response.status_code = 404
s3_response.raw = BytesIO(b"""<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>NoSuchKey</Code>
<Message>The resource you requested does not exist</Message>
<Resource>/mybucket/myfoto.jpg</Resource>
<RequestId>4442587FB7D0A2F9</RequestId>
</Error>
""")
parsed_response = function_under_test(s3_response)
self.assertEqual(404, parsed_response[0])
self.assertEqual("NoSuchKey", parsed_response[1])
There's a library for this, if you want to write your test server with Flask: requests-flask-adaptor
You just have to be careful with the order of imports when monkeypatching.
Related
I'm trying to do what I feel like should be a pretty simple mock in python.
I have a tree that looks like this:
- src
- auth
- authenticator.py
- http_client
- auth_client.py
- tests
- test_authenticator.py
My authenticator.py looks like:
import sys
sys.path.append("src/http_client")
import auth_client
def authenticate_request(headers="", env="dev"):
# logic...
auth_client.do_auth(headers)
# logic...
Then auth_client.py looks like:
import json
import requests
def do_auth(headers):
url = "https://FAKE_URL.com"
r = requests.get(url, headers)
return r.json()
And my test_authenticator.py has a test that looks like
import mock
import sys
sys.path.append("src/auth")
sys.path.append("src/http_client")
import authenticator as auth
import auth_client
#mock.patch('auth_client.do_auth')
def test_good(self, mock_request):
headers = FAKE_HEADERS
mock_request.return_value = '{\"response:\": 200}'
body, statusCode = auth.authenticate_request(
headers=headers)
self.assertEqual(200, statusCode)
# self.assertEqual(body, "Token must have kid")
def mock_request():
return "{\"response\":200}"
But I can't seem to actually mock the function. When I run my tests with pytest I get in the coverage that it's hitting auth_client even though I don't want it to.
How do I properly mock this function?
Thanks!
So I was following this youtube video:
https://youtu.be/6tNS--WetLI?t=1973
When I replicate what is done in the video, my test fails:
Class:
import requests
class JetpackRebooter:
#staticmethod
def reboot_jetpack(secure_token):
response = requests.post('http://my.jetpack/restarting/reboot/', data={'gSecureToken' : secure_token})
return response.status_code
Unit test:
from src.jetpack_rebooter import JetpackRebooter
from unittest.mock import patch
def test_reboot_jetpack():
secure_token = 'ca26bacf85c6d69d0bdaa3ff07df3cc4118abf45'
with patch("jetpack_rebooter.requests.post") as mocked_post:
mocked_post.return_value.status_code = 200
assert JetpackRebooter.reboot_jetpack(secure_token) == 200
However, changing patch("jetpack_rebooter.requests.post") to patch("requests.post") causes the test to pass. Why is this?
patch("jetpack_rebooter.requests.post") does not work because the module's full import path is not jetpack_rebooter, but src.jetpack_rebooter.
Because of that, patch("src.jetpack_rebooter.requests.post") would work.
However, as src.jetpack_rebooter.requests points to the requests module object, there is no difference between patch("src.jetpack_rebooter.requests.post") and patch("requests.post"); but there would be a difference between patch("src.jetpack_rebooter.requests") and patch("requests").
I have following function that I want to write unit test for
def get_user_data(user, session):
'''Given a github user, gets it data'''
url = f'https://api.github.com/users/{user}'
response = session.get(url)
json_response = response.json()
return json_response["login"]
This is how I am testing the above function
from unittest.mock import Mock, MagicMock
from requests import Session
response_payload = {"login": "agrawalo"}
fake_session = MagicMock(spec=Session)
fake_session.get.return_value.get_json.return_value = response_payload
If you notice the code above, instead of mocking "json" method on a response object I have created a mock for "get_json" method. This ends up creating get_json method on a response object. And now when I call
get_user_data("agrawalo", fake_session)
I get
<MagicMock name='mock.get().json().__getitem__()' id='139783470311800'>
Whereas I expect the above unittest to fail with NoAttributeError.
In python 3.8 there is a way to fix above problem using "seal" from unitest.mock.
# from python 3.8
from unittest.mock import seal
seal(fake_session)
get_user_data("agrawalo", fake_session)
How can I do this in python 2.7?
I am using bottle framework. I have code like
from bottle import request
def abc():
x = request.get_header('x')
...
...
data = request.json()
...
...
I am writing UTs for this function, I want to mock get_header and json of bottle.request, and return my mock data from that.
I tried.
from mock import patch
#patch('bottle.request.headers', return_value={'x': 'x'})
#patch('bottle.request.json', return_value=...)
def test_abc(self, _, __):
...
...
But it gives error for request.headers is read-only. I also have to mock request.json.
Thanks for help in advance :).
Check the source code of bottle, headers and json are like:
#DictProperty('environ', 'bottle.request.headers', read_only=True)
def headers(self):
''' A :class:`WSGIHeaderDict` that provides case-insensitive access to
HTTP request headers. '''
return WSGIHeaderDict(self.environ)
So in my pytest cases, I patched request.environ like below:
def test_xxx(monkeypatch):
monkeypatch.setitem(request.environ, 'bottle.request.json', {'name': 'xxx', 'version': '0.1'})
add_xxx()
assert
An easy alternative, to mock a bottle request, could be to inject it into your function:
from bottle import request
def abc(_request=None):
if _request is not None:
request = _request
x = request.get_header('x')
...
...
data = request.json()
...
...
This should be safe as your test code can call your view with a fake request object directly and your production code will skip the conditional.
I am NOT sure how this works with url routes with named params, as I've never used bottle.
Use Boddle https://github.com/keredson/boddle
def test_abc(self, _, __):
with boddle(headers={'x':'x'}):
# tests
Below is a code snippet of my api.py module
# -*- coding: utf-8 -*-
from urllib2 import urlopen
from urllib2 import Request
class API:
def call_api(self, url, post_data=None, header=None):
is_post_request = True if (post_data and header) else False
response = None
try:
if is_post_request:
url = Request(url = url, data = post_data, headers = header)
# Calling api
api_response = urlopen(url)
response = api_response.read()
except Exception as err:
response = err
return response
I am trying to mock urllib2.urlopen in unittest of above module. I have written
# -*- coding: utf-8 -*-
# test_api.py
from unittest import TestCase
import mock
from api import API
class TestAPI(TestCase):
#mock.patch('urllib2.Request')
#mock.patch('urllib2.urlopen')
def test_call_api(self, urlopen, Request):
urlopen.read.return_value = 'mocked'
Request.get_host.return_value = 'google.com'
Request.type.return_value = 'https'
Request.data = {}
_api = API()
assert _api.call_api('https://google.com') == 'mocked'
After I run the unittest, I get an exception
<urlopen error unknown url type: <MagicMock name='Request().get_type()' id='159846220'>>
What am I missing? Please help me out.
You are patching the wrong things: take a look to Where to patch.
In api.py by
from urllib2 import urlopen
from urllib2 import Request
you create a local reference to urlopen and Request in your file. By mock.patch('urllib2.urlopen') you are patching the original reference and leave the api.py's one untouched.
So, replace your patches by
#mock.patch('api.Request')
#mock.patch('api.urlopen')
should fix your issue.... but is not enough.
In your test case api.Request are not used but urllib2.urlopen() create a Request by use the patched version: that is why Request().get_type() is a MagicMock.
For a complete fix you should change your test at all. First the code:
#mock.patch('api.urlopen', autospec=True)
def test_call_api(self, urlopen):
urlopen.return_value.read.return_value = 'mocked'
_api = API()
self.assertEqual(_api.call_api('https://google.com'), 'mocked')
urlopen.assert_called_with('https://google.com')
Now the clarification... In your test you don't call Request() because you pass just the first parameter, so I've removed useless patch. Moreover you are patching urlopen function and not urlopen object, that means the read() method you want mocked is a method of the object return by urlopen() call.
Finally I add a check on urlopen call and autospec=True that is always a good practice.