Changed the titel to a more common one. I guess the problem is not that class specific.
I want to mock google.cloud.pubsub_v1.SubscriberClient
I want to set a fake return value when calling the pull function of the client.
prod code:
from google.cloud import pubsub_v1
def open_subscription() -> None:
with pubsub_v1.SubscriberClient() as subscriber:
logger.info(f'Listening for messages on {config.SUBSCRIPTION_NAME}', {'operation': {'first': True}})
while True:
# get messages
response = subscriber.pull(
request = {
'subscription': config.SUBSCRIPTION_NAME,
'max_messages': config.MAX_MESSAGES
}
)
from the prod code above I want to set the return value for calling the pull method.
I am creating a pull-response object in the test code.
test code:
import unittest
from unittest.mock import MagicMock, patch
from app.pubsub import pubsub_service
from google import pubsub_v1
import json
class TestPubSubService(unittest.TestCase):
def create_test_message(self):
message_string = '{"testKey": "testValue"}'
message_json = json.dumps(message_string, indent=2)
message_data = message_json.encode('utf-8')
pubsub_message = pubsub_v1.PubsubMessage()
pubsub_message.data = message_data
received_message = pubsub_v1.ReceivedMessage()
received_message.ack_id = "testId"
received_message.message = pubsub_message
return received_message
def create_test_pull_response(self, received_message):
pull_response = pubsub_v1.PullResponse()
pull_response.received_messages = [received_message]
return pull_response
#patch('app.pubsub.pubsub_service.pubsub_v1.SubscriberClient')
def test_open_subscription(self, mock_subscriber):
test_message = self.create_test_message()
pull_response = self.create_test_pull_response(test_message)
mock_subscriber.return_value.pull.return_value = MagicMock(return_value = pull_response)
pubsub_service.open_subscription()
At least the MagicMock is in place (without using the patch the real subscriber is in place).
So basically I would say I mocked the subscriberClient.
But I cannot set the return_value for calls to the pull method.
But there wont be a pull retur value. All I get is another magicMock created.
I do not get it why it is not working.
As most stuff I read we usually have to call 'return_value' on the mock, append the name of either the field or function to be set, append that ones 'return_value' and set a value viea MagicMock.
The format should be:
mockFirst.return_value.second.return_value.third.return_value = Mock(return_value = the_value_to_return)
Hopefully you can explain me what I am doing wrong.
Thanks.
edit: tried also the following ones which where the answers in other posts:
Mocking Method Calls In Python
Mock a method of a mocked object in Python?
mock_subscriber.pull.return_value = pull_response
mock_subscriber.return_value.pull.return_value = pull_response
none seems to work. the return value of the pull method stays to be a magicMock.
And this is how it looks like in debugging (hovering over response):
I faced the same issue. But can get idea from the details inside MagicMock.
Try to set return value (based on your screenshot)
mock_subscriber.__enter__().pull.return_value = pull_response
I am using pytest to test a method that calls requests.post. Is there a easy way and preferably without third party libraries to do this?
class Dispatcher:
def __init__(self, url):
self.url = url
self.session = None
def dispatch(self):
return self.session.post(self.url).json()
def test_dispatch():
d = Dispatcher(url="")
d.session = # ... here, how can I mock the return value of json()?
result = d.dispatch()
So this is a pretty straightforward example. We want to set session to be a MagicMock object.
from unittest.mock import MagicMock
def test_dispatch():
expected = {"fizz": "buzz"}
mock_session = MagicMock()
mock_session.configure_mock(
**{
"post.return_value": mock_session,
"json.return_value": expected
}
)
d = Dispatcher(url="")
d.session = mock_session
result = d.dispatch()
assert result == expected
Since Mock objects return a brand new mock object when methods are called on them (without being configured), we have to configure the object as such. If we didn't configure post to return the original mock we have, then it would return a brand new mock object and our test would fail. Conversely you can configure another Mock object to be the return value of post and configure that object, but I prefer this approach.
Explanation showing the call is listed below.
self.session is our mock_session object we created
self.session.post(arg, **kwargs) returns mock_session
mock_session.json() returns the dictionary we specified
In my object, I have several "Loader" functions for different types of data that might require loading. They're stored as:
import foo
import bar
class MyClass
LOADERS = {
'foo': {
'context_manager': False,
'func': foo.data.loader,
'kwargs': {'data_orientation': 'columns'}
},
'bar': {
'context_manager': True,
'func': bar.load_data
'kwargs': {}
}
}
def load_data(self, load_type, path):
func = self.LOADERS[load_type]['func']
kwargs = self.LOADERS[load_type]['kwargs']
if self.LOADERS[load_type]['context_manager']:
with open(path, 'rb') as f:
self.data=func(f, **kwargs)
else:
self.data = func(path, **kwargs)
This is working very well in practice, but I'm finding it's murder to test.
When I write my test:
from mock import MagicMock, patch
import sys
sys.modules['foo'] = MagicMock()
sys.modules['bar'] = MagicMock()
from mypackage import MyClass
#patch('foo.data.loader')
def test_load_foo(mock_loader):
my_obj = MyClass()
my_obj.load_data('foo', 'path/to/data')
mock_loader.assert_called_once()
it fails. Called 0 times.
I halfway suspect that it's because it's not being called directly. But that shouldn't really matter.
Can anyone offer any suggestions? I'm using pytest as my testing engine, but I've found it plays nicely with mock in the past.
Thanks!
The issue arises because you are trying to patch a function within a class attribute as opposed to an instance attribute. The behavior is not the same. This does however offer us some flexibility as it is a class attribute, thus we can modify the object directly on the class we instantiate in our test.
from unittest.mock import MagicMock
from mypackage import MyClass
def test_load_foo():
key_to_test = "foo"
mock_loader = MagicMock()
my_obj = MyClass()
my_obj.LOADERS[key_to_test]["func"] = mock_loader
my_obj.load_data(key_to_test, 'path/to/data')
mock_loader.assert_called_once()
Instead of trying to patch the function, we modify our objects instance to use the MagicMock.
Using this approach I receive the following message:
collected 1 item
tests/test_mypackage.py . [100%]
======================================================================= 1 passed in 0.01s =======================================================================
I'm trying to make a simple test in python, but I'm not able to figure it out how to accomplish the mocking process.
This is the class and def code:
class FileRemoveOp(...)
#apply_defaults
def __init__(
self,
source_conn_keys,
source_conn_id='conn_default',
*args, **kwargs):
super(v4FileRemoveOperator, self).__init__(*args, **kwargs)
self.source_conn_keys = source_conn_keys
self.source_conn_id = source_conn_id
def execute (self, context)
source_conn = Connection(conn_id)
try:
for source_conn_key in self.source_keys:
if not source_conn.check_for_key(source_conn_key):
logging.info("The source key does not exist")
source_conn.remove_file(source_conn_key,'')
finally:
logging.info("Remove operation successful.")
And this is my test for the execute function:
#mock.patch('main.Connection')
def test_remove_execute(self,MockConn):
mock_coon = MockConn.return_value
mock_coon.value = #I'm not sure what to put here#
remove_operator = FileRemoveOp(...)
remove_operator.execute(self)
Since the execute method try to make a connection, I need to mock that, I don't want to make a real connection, just return something mock. How can I make that? I'm used to do testing in Java but I never did on python..
First it is very important to understand that you always need to Mock where it the thing you are trying to mock out is used as stated in the unittest.mock documentation.
The basic principle is that you patch where an object is looked up,
which is not necessarily the same place as where it is defined.
Next what you would need to do is to return a MagicMock instance as return_value of the patched object. So to summarize this you would need to use the following sequence.
Patch Object
prepare MagicMock to be used
return the MagicMock we've just created as return_value
Here a quick example of a project.
connection.py (Class we would like to Mock)
class Connection(object):
def execute(self):
return "Connection to server made"
file.py (Where the Class is used)
from project.connection import Connection
class FileRemoveOp(object):
def __init__(self, foo):
self.foo = foo
def execute(self):
conn = Connection()
result = conn.execute()
return result
tests/test_file.py
import unittest
from unittest.mock import patch, MagicMock
from project.file import FileRemoveOp
class TestFileRemoveOp(unittest.TestCase):
def setUp(self):
self.fileremoveop = FileRemoveOp('foobar')
#patch('project.file.Connection')
def test_execute(self, connection_mock):
# Create a new MagickMock instance which will be the
# `return_value` of our patched object
connection_instance = MagicMock()
connection_instance.execute.return_value = "testing"
# Return the above created `connection_instance`
connection_mock.return_value = connection_instance
result = self.fileremoveop.execute()
expected = "testing"
self.assertEqual(result, expected)
def test_not_mocked(self):
# No mocking involved will execute the `Connection.execute` method
result = self.fileremoveop.execute()
expected = "Connection to server made"
self.assertEqual(result, expected)
I found that this simple solution works in python3: you can substitute a whole class before it is being imported for the first time. Say I have to mock class 'Manager' from real.manager
class MockManager:
...
import real.manager
real.manager.Manager = MockManager
It is possible to do this substitution in init.py if there is no better place.
It may work in python2 too but I did not check.
I am not talking about the Parameterizing a fixture feature that allows a fixture to be run multiple times for a hard-coded set of parameters.
I have a LOT of tests that follow a pattern like:
httpcode = 401 # this is different per call
message = 'some message' # this is different per call
url = 'some url' # this is different per call
mock_req = mock.MagicMock(spec_set=urllib2.Request)
with mock.patch('package.module.urllib2.urlopen', autospec=True) as mock_urlopen, \
mock.patch('package.module.urllib2.Request', autospec=True) as mock_request:
mock_request.return_value = mock_req
mock_urlopen.side_effect = urllib2.HTTPError(url, httpcode, message, {}, None)
connection = MyClass()
with pytest.raises(MyException):
connection.some_function() # this changes
Essentially, I have a class that's an API client, and includes custom, meaningful exceptions that wrap urllib2 errors in something API-specific. So, I have this one pattern - patching some methods, and setting side effects on one of them. I use it in probably a dozen different tests, and the only differences are the three variables which are used in part of the side_effect, and the method of MyClass() that I call.
Is there any way to make this a pytest fixture and pass in these variables?
You can use indirect fixture parametrization
http://pytest.org/latest/example/parametrize.html#deferring-the-setup-of-parametrized-resources
#pytest.fixture()
def your_fixture(request):
httpcode, message, url = request.param
mock_req = mock.MagicMock(spec_set=urllib2.Request)
with mock.patch('package.module.urllib2.urlopen', autospec=True) as mock_urlopen, \
mock.patch('package.module.urllib2.Request', autospec=True) as mock_request:
mock_request.return_value = mock_req
mock_urlopen.side_effect = urllib2.HTTPError(url, httpcode, message, {}, None)
connection = MyClass()
with pytest.raises(MyException):
connection.some_function() # this changes
#pytest.mark.parametrize('your_fixture', [
(403, 'some message', 'some url')
], indirect=True)
def test(your_fixture):
...
and your_fixture will run before test with desired params
I've done a bunch more research on this since posting my question, and the best I can come up with is:
Fixtures don't work this way. Just use a regular function, i.e.:
def my_fixture(httpcode, message, url):
mock_req = mock.MagicMock(spec_set=urllib2.Request)
with mock.patch('package.module.urllib2.urlopen', autospec=True) as mock_urlopen, \
mock.patch('package.module.urllib2.Request', autospec=True) as mock_request:
mock_request.return_value = mock_req
mock_urlopen.side_effect = urllib2.HTTPError(url, httpcode, message, {}, None)
connection = MyClass()
return (connection, mock_request, mock_urlopen)
def test_something():
connection, mock_req, mock_urlopen = my_fixture(401, 'some message', 'some url')
with pytest.raises(MyException):
connection.some_function() # this changes
How to pass parameters into a fixture?
Unpack that idea for a moment: you're asking for a fixture, which is a function, which reacts to parameters. So, return a function, which reacts to parameters:
#pytest.fixture
def get_named_service():
def _get_named_service(name):
result = do_something_with_name(name)
return result
return _get_named_service
Thus, in the test, you can provide the parameters to the function:
def test_stuff(get_named_service):
awesome_service = get_named_service('awesome')
terrible_service = get_named_service('terrible')
# Now you can do stuff with both services.
This is documented as a factory pattern:
https://docs.pytest.org/en/latest/how-to/fixtures.html#factories-as-fixtures
Which, as the OP found, is just a function, but with the advantage of being inside the conftest where all the other common utils and setup/teardown code resides; plus self-documenting the dependencies of the test.
I know this is old, but maybe it helps someone who stumbles on this again
#pytest.fixture
def data_patcher(request):
def get_output_test_data(filename, as_of_date=None):
# a bunch of stuff to configure output
return output
def teardown():
pass
request.addfinalizer(teardown)
return get_output_test_data
and then, inside the function:
with patch('function to patch', new=data_patcher):
Some trick with pytest.mark and we have a fixture with arguments.
from allure import attach
from pytest import fixture, mark
def get_precondition_params(request_fixture, fixture_function_name: str):
precondition_params = request_fixture.keywords.get("preconditions_params")
result = precondition_params.args[0].pop(fixture_function_name) if precondition_params is not None else None
return result
#fixture(scope="function")
def setup_fixture_1(request):
params = get_precondition_params(request, "setup_fixture_1")
return params
#mark.preconditions_params(
{
"setup_fixture_1": {
"param_1": "param_1 value for setup_fixture_1",
"param_2": "param_2 value for setup_fixture_1"
},
}
)
def test_function(setup_fixture_1):
attach(str(setup_fixture_1), "setup_fixture_1 value")
Now we can use one fixture code, parametrize it with mark, and do anything with params inside fixture. And fixture will be executed as precondition (how it must be), not as step (like it will be if we return function from fixture).