Changed the titel to a more common one. I guess the problem is not that class specific.
I want to mock google.cloud.pubsub_v1.SubscriberClient
I want to set a fake return value when calling the pull function of the client.
prod code:
from google.cloud import pubsub_v1
def open_subscription() -> None:
with pubsub_v1.SubscriberClient() as subscriber:
logger.info(f'Listening for messages on {config.SUBSCRIPTION_NAME}', {'operation': {'first': True}})
while True:
# get messages
response = subscriber.pull(
request = {
'subscription': config.SUBSCRIPTION_NAME,
'max_messages': config.MAX_MESSAGES
}
)
from the prod code above I want to set the return value for calling the pull method.
I am creating a pull-response object in the test code.
test code:
import unittest
from unittest.mock import MagicMock, patch
from app.pubsub import pubsub_service
from google import pubsub_v1
import json
class TestPubSubService(unittest.TestCase):
def create_test_message(self):
message_string = '{"testKey": "testValue"}'
message_json = json.dumps(message_string, indent=2)
message_data = message_json.encode('utf-8')
pubsub_message = pubsub_v1.PubsubMessage()
pubsub_message.data = message_data
received_message = pubsub_v1.ReceivedMessage()
received_message.ack_id = "testId"
received_message.message = pubsub_message
return received_message
def create_test_pull_response(self, received_message):
pull_response = pubsub_v1.PullResponse()
pull_response.received_messages = [received_message]
return pull_response
#patch('app.pubsub.pubsub_service.pubsub_v1.SubscriberClient')
def test_open_subscription(self, mock_subscriber):
test_message = self.create_test_message()
pull_response = self.create_test_pull_response(test_message)
mock_subscriber.return_value.pull.return_value = MagicMock(return_value = pull_response)
pubsub_service.open_subscription()
At least the MagicMock is in place (without using the patch the real subscriber is in place).
So basically I would say I mocked the subscriberClient.
But I cannot set the return_value for calls to the pull method.
But there wont be a pull retur value. All I get is another magicMock created.
I do not get it why it is not working.
As most stuff I read we usually have to call 'return_value' on the mock, append the name of either the field or function to be set, append that ones 'return_value' and set a value viea MagicMock.
The format should be:
mockFirst.return_value.second.return_value.third.return_value = Mock(return_value = the_value_to_return)
Hopefully you can explain me what I am doing wrong.
Thanks.
edit: tried also the following ones which where the answers in other posts:
Mocking Method Calls In Python
Mock a method of a mocked object in Python?
mock_subscriber.pull.return_value = pull_response
mock_subscriber.return_value.pull.return_value = pull_response
none seems to work. the return value of the pull method stays to be a magicMock.
And this is how it looks like in debugging (hovering over response):
I faced the same issue. But can get idea from the details inside MagicMock.
Try to set return value (based on your screenshot)
mock_subscriber.__enter__().pull.return_value = pull_response
I am using pytest to test a method that calls requests.post. Is there a easy way and preferably without third party libraries to do this?
class Dispatcher:
def __init__(self, url):
self.url = url
self.session = None
def dispatch(self):
return self.session.post(self.url).json()
def test_dispatch():
d = Dispatcher(url="")
d.session = # ... here, how can I mock the return value of json()?
result = d.dispatch()
So this is a pretty straightforward example. We want to set session to be a MagicMock object.
from unittest.mock import MagicMock
def test_dispatch():
expected = {"fizz": "buzz"}
mock_session = MagicMock()
mock_session.configure_mock(
**{
"post.return_value": mock_session,
"json.return_value": expected
}
)
d = Dispatcher(url="")
d.session = mock_session
result = d.dispatch()
assert result == expected
Since Mock objects return a brand new mock object when methods are called on them (without being configured), we have to configure the object as such. If we didn't configure post to return the original mock we have, then it would return a brand new mock object and our test would fail. Conversely you can configure another Mock object to be the return value of post and configure that object, but I prefer this approach.
Explanation showing the call is listed below.
self.session is our mock_session object we created
self.session.post(arg, **kwargs) returns mock_session
mock_session.json() returns the dictionary we specified
I have a python method like
import external_object
from external_lib1 import ExternalClass1
from external_lib2 import Hook
class MyClass(self):
def my_method(self):
ExternalClass.get('arg1') #should be mocked and return a specific value with this arg1
ExternalClass.get('arg2') #should be mocked and return a specific value with this arg2
def get_hook(self):
return Hook() # return a mock object with mocked method on it
def my_method(self):
object_1 = external_object.instance_type_1('args') # those are two different object instanciate from the same lib.
object_2 = external_object.instance_type_2('args')
object_1.method_1('arg') # should return what I want when object_1 mocked
object_2.method_2 ('arg') # should return what I want when object_2 mocked
In my test I would like to realise what I put in comments.
I could manage to do it, but every time it gets really messy.
I use to call flexmock for some stuff (by example ExternalClass.get('arg1') would be mock with a flexmock(ExternalClass).should_return('arg').with_args('arg') # etc...) but I'm tired of using different test libs to mock.
I would like to use only the mock library but I struggle to find a consistent way of doing it.
I like to use python's unittest lib. Concretely the unittest.mock which is a great lib to customize side effects and return value in unit tested functions.
They can be used as follows:
class Some(object):
"""
You want to test this class
external_lib is an external component we cannot test
"""
def __init__(self, external_lib):
self.lib = external_lib
def create_index(self, unique_index):
"""
Create an index.
"""
try:
self.lib.create(index=unique_index) # mock this
return True
except MyException as e:
self.logger.error(e.__dict__, color="red")
return False
class MockLib():
pass
class TestSome(unittest.TestCase):
def setUp(self):
self.lib = MockLib()
self.some = Some(self.lib)
def test_create_index(self):
# This will test the method returns True if everything went fine
self.some.create_index = MagicMock(return_value={})
self.assertTrue(self.some.create_index("test-index"))
def test_create_index_fail(self):
# This will test the exception is handled and return False
self.some.create_index = MagicMock(side_effect=MyException("error create"))
self.assertFalse(self.some.create_index("test-index"))
Put the TestSome() class file somewhere like your-codebase-path/tests and run:
python -m unittest -v
I hope it's useful.
I'm attempting to create a few unit tests for my class. I want to mock these, so that I don't burn through my API quota running some of these tests. I have multiple test cases that will call the fetch method, and depending on the passed URL I'll get different results back.
My example class looks like this:
import requests
class ExampleAPI(object):
def fetch(self, url, params=None, key=None, token=None, **kwargs):
return requests.get(url).json() # Returns a JSON string
The tutorial I'm looking at shows that I can do something like this:
import unittest
from mock import patch
def fake_fetch_test_one(url):
...
class TestExampleAPI(unittest.TestCase):
#patch('mymodule.ExampleAPI.fetch', fake_fetch_test_one)
def test_fetch(self):
e = ExampleAPI()
self.assertEqual(e.fetch('http://my.api.url.example.com'), """{'result': 'True'}""")
When I do this, though, I get an error that says:
TypeError: fake_fetch_test_one() takes exactly 1 argument (3 given)
What is the proper way to mock a requests.get call that is in a method in my class? I'll need the ability to change the mock'd response per test, because different URLs can provide different response types.
Your fake fetch needs to accept the same arguments as the original:
def fake_fetch(self, url, params=None, key=None, token=None, **kwargs):
Note that it's better to mock just the external interface, which means letting fetch call requests.get (or at least, what it thinks is requests.get):
#patch('mymodule.requests.get')
def test_fetch(self, fake_get):
# It would probably be better to just construct
# a valid fake response object whose `json` method
# would return the right thing, but this is a easier
# for demonstration purposes. I'm assuming nothing else
# is done with the response.
expected = {"result": "True"}
fake_get.return_value.json.return_value = expected
e = ExampleAPI()
self.assertEqual(e.fetch('http://my.api.url.example.com'), expected)
from you test method you can monkeypatch your requests module
import unittest
class Mock:
pass
ExampleAPI.requests = Mock()
def fake_get_test_one(url):
/*returns fake get json */
ExampleAPI.requests.get= Mock()
ExampleAPI.requests.json = fake_get_test_one
class TestExampleAPI(unittest.TestCase):
def test_fetch(self):
e = ExampleAPI()
self.assertEqual(e.fetch('http://my.api.url.example.com'), """{'result': 'True'}""")
you can setup the patch in each setup() and corresponding teardown() methods of your test class if needed
So I am working with an API wrapper in python for vk, Europe's Facebook equivalent. The documentation on the vk site has all the API calls that can be used, and the wrapper is able to call them correctly. For example, to get a user's information, you would call api.users.get(id) to get a user by id. My question is this: how can the wrapper correctly handle such a call when neither users or a corresponding users.get() method is defined inside the api object?
I know it involves the __getattr__() and __call__() methods, but I can't find any good documentation on how to use them in this way.
EDIT
the api object is instantiated via api = vk.API(id, email, password)
Let's walk through this together, shall we?
api
To execute api.users.get(), Python first has to know api. And due to your instantiation, it does know it: It's a local variable holding an instance of APISession.
api.users
Then, it has to know api.users. Python first looks at the members of the api instance, at the members of its class (APISession) and at the members of that class' super-classes (only object in the case of APISession). Failing to find a member called users in any of these places, it looks for a member function called __getattr__ in those same places. It will find it right on the instance, because APISession has an (instance) member function of this name.
Python then calls it with 'users' (the name of the so-far missing member) and uses the function's return value as if it were that member. So
api.users
is equivalent to
api.__getattr__('users')
Let's see what that returns.
def __getattr__(self, method_name):
return APIMethod(self, method_name)
Oh. So
api.users # via api.__getattr__('users')
is equivalent to
APIMethod(api, 'users')
creating a new APIMethod instance.
api and 'users' end up as that instance's _api_session and _method_name members. Makes sense, I guess.
api.users.get
Python still hasn't executed our statement. It needs to know api.users.get() to do so. The same game as before repeats, just in the api.users object instead of the api object this time: No member method get() and no member get is found on the APIMethod instance api.users points to, nor on its class or superclasses, so Python turns to the __getattr__ method, which for this class does something peculiar:
def __getattr__(self, method_name):
return APIMethod(self._api_session, self._method_name + '.' + method_name)
A new instance of the same class! Let's plug in the instance members of api.users, and
api.users.get
becomes equivalent to
APIMethod(api, 'users' + '.' + 'get')
So we will have the api object also in api.user.get's _apisession and the string 'users.get' in its _method_name.
api.users.get() (note the ())
So api.users.get is an object. To call it, Python has to pretend it's a function, or more specifically, a method of api.users. It does so, by instead calling api.users.get's __call__ method, which looks like this:
def __call__(self, **method_kwargs):
return self._api_session(self._method_name, **method_kwargs)
Let's work this out:
api.users.get()
# is equivalent to
api.users.get.__call__() # no arguments, because we passed none to `get()`
# will return
api.users.get._api_session(api.users.get._method_name)
# which is
api('users.get')
So now Python is calling the api object as if it were a function. __call__ to the rescue, once more, this time looking like this:
def __call__(self, method_name, **method_kwargs):
response = self.method_request(method_name, **method_kwargs)
response.raise_for_status()
# there are may be 2 dicts in 1 json
# for example: {'error': ...}{'response': ...}
errors = []
error_codes = []
for data in json_iter_parse(response.text):
if 'error' in data:
error_data = data['error']
if error_data['error_code'] == CAPTCHA_IS_NEEDED:
return self.captcha_is_needed(error_data, method_name, **method_kwargs)
error_codes.append(error_data['error_code'])
errors.append(error_data)
if 'response' in data:
for error in errors:
warnings.warn(str(error))
return data['response']
if AUTHORIZATION_FAILED in error_codes: # invalid access token
self.access_token = None
self.get_access_token()
return self(method_name, **method_kwargs)
else:
raise VkAPIMethodError(errors[0])
Now, that's a lot of error handling. For this analysis, I'm only interested in the happy path. I'm only interested in the happy path's result (and how we got there). So lets start at the result.
return data['response']
Where did data come from? It's the first element of response.text interpreted as JSON that does contain a 'response' object. So it seems that from the response object we got, we're extracting the actual response part.
Where did the response object come from? It was returned by
api.method_request('users.get')
Which, for all we care, is a plain normal method call that doesn't require any fancy fallbacks. (Its implementation of course, on some levels, might.)
Assuming the comments are correct, and api is an instance of APISession as defined in this particular commit, then this is a bit of an interesting maze:
So first you want to access api.user. APISession has no such attribute, so it calls __getattr__('user') instead, which is defined as:
def __getattr__(self, method_name):
return APIMethod(self, method_name)
So this constructs an APIMethod(api,'user'). Now you want to call the method get on the APIMethod(api,'user') that is bound to api.users, but an APIMethod doesn't have a get method, so it calls its own __getattr__('get') to figure out what to do:
def __getattr__(self, method_name):
return APIMethod(self._api_session, self._method_name + '.' + method_name)
This returns a APIMethod(api,'users.get') which is then called, invoking the __call__ method of the APIMethod class, which is:
def __call__(self, **method_kwargs):
return self._api_session(self._method_name, **method_kwargs)
So this tries to return api('users.get'), but api is an APISession object, so it invokes the __call__ method of this class, defined as (stripping out the error handling for simplicity):
def __call__(self, method_name, **method_kwargs):
response = self.method_request(method_name, **method_kwargs)
response.raise_for_status()
for data in json_iter_parse(response.text):
if 'response' in data:
return data['response']
So it then calls a method_request('users.get'), which if you follow that method actually does a POST request, and some data comes back as a response, which is then returned.
The users.get() has nothing to do with the api object. As for the users, you are right, if there is no such member defined, then there is certainly some logic inside the __getattr__. So as you can see in the documentation __getattr__ is...
Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name.
So exactly, as there is no users defined for the api's class, then the __getattr__ is being called with 'users' passed as the name parameter. Then, most probably dynamically, depending on the passed parameter, an object is being constructed for the users component and returned, which will be responsible to define or handle in similar way the get() method.
To get the whole idea, try the following:
class A(object):
def __init__(self):
super(A, self).__init__()
self.defined_one = 'I am defined inside A'
def __getattr__(self, item):
print('getting attribute {}'.format(item))
return 'I am ' + item
a = A()
>>> print(a.some_item) # this will call __getattr__ as some_item is not defined
getting attribute some_item
I am some_item
>>> print(a.and_another_one) # this will call __getattr__ as and_another_one is not defined
getting attribute and_another_one
I am and_another_one
>>> print(a.defined_one) # this will NOT call __getattr__ as defined_one is defined in A
I am defined inside A