The most common way to patch something in a module seems to be to use something like
from unittest.mock import patch
from mypackage.my_module.my_submodule import function_to_test
#patch('mypackage.my_module.my_submodule.fits.open')
def test_something(self, mock_fits_open)
# ...
mock_fits_open.return_value = some_return_value
function_to_test()
# ...
However, with the value passed to the patch decorator being a string, I don't get lots of the nice benefits from IDE. I can't use parts of the string to jump to definitions. I don't get autocomplete (and an implicit spelling check). Nor full refactoring capabilities. And so on.
Using patch.object I can get much closer to what I'm looking for.
from unittest.mock import patch
import mypackage.my_module.my_submodule
from mypackage.my_module.my_submodule import function_to_test
#patch.object(mypackage.my_module.my_submodule.fits, 'open')
def test_something(self, mock_fits_open)
# ...
mock_fits_open.return_value = some_return_value
function_to_test()
# ...
However, this still requires the final part of the name of the referenced object is just a string. Is there a (nice) way to patch an object purely on the reference to that object? That is, I would like to be able to do something like
from unittest.mock import patch
import mypackage.my_module.my_submodule
from mypackage.my_module.my_submodule import function_to_test
#patch.reference(mypackage.my_module.my_submodule.fits.open)
def test_something(self, mock_fits_open)
# ...
mock_fits_open.return_value = some_return_value
function_to_test()
# ...
Patching works by replacing in the namespace where the name is looked up.
The underlying logic of mock.patch is essentially working with a context-managed name shadowing. You could do the same thing manually with:
save original value associated with name (if any)
try overwriting the name
execute the code under test
finally resetting name back to the original value
Therefore, you fundamentally need to patch on a name, there is no patching a reference directly.
You can use the __name__ attribute of the function object:
from unittest.mock import patch
import my_module
#patch.object(my_module, my_module.my_fun.__name__)
def test_something(self, mocked_fun)
# ...
See also: Name of a Python function in a stack trace
Related
what is the easiest way to solve the following problem in extending/altering the functionality of a third party library?
The library offers a class LibraryClass with a function func_to_be_changed. This function has a local variable internal_variable which is the instance of another class SimpleCalculation of that library. I created a new class BetterCalculation in my own module and now want LibraryClass.func_to_be_changed to use an instance of this new class.
# third party library
from third_party_library.utils import SimpleCalculation
class LibraryClass:
def func_to_be_changed(self, x):
# many complicated things go on
internal_variable = SimpleCalculation(x)
# many more complicated things go on
The easiest solution would be, to just copy the code from the third party library, subclass the LibraryClass and overwrite the function func_to_be_changed:
# my module
from third_party_library import LibraryClass
class BetterLibraryClass(LibraryClass):
def func_to_be_changed(self, x):
"""This is an exact copy of LibraryClass.func_to_be_changed."""
# many complicated things go on
internal_variable = BetterCalculation(x) # Attention: this line has been changed!!!
# many more complicated things go on
However, this involves copying of many lines of code. When a new version of the third party class improves on code that was copied without modification, this modifications need to be incorporated manually by another copying step.
I tried to use unittest.mock.patch as I know that the following two snippets work:
# some script
from unittest.mock import patch
import third_party_library
from my_module import BetterCalculation
with patch('third_party_library.utils.SimpleCalculation', BetterCalculation):
local_ = third_party_library.utils.SimpleCalculation(x) # indeed uses `BetterCalculation`
def foo(x):
return third_party_library.utils.SimpleCalculation(x)
with patch('third_party_library.utils.SimpleCalculation', BetterCalculation):
local_ = foo(x) # indeed uses `BetterCalculation`
However, the following does not work:
# some script
from unittest.mock import patch
from third_party_library.utils import SimpleCalculation
from my_module import BetterCalculation
def foo(x):
return SimpleCalculation(x)
with patch('third_party_library.utils.SimpleCalculation', BetterCalculation):
local_ = foo(x) # does not use `BetterCalculation`
# this works again
with patch('__main__.SimpleCalculation', BetterCalculation):
local_ = foo(x) # indeed uses `BetterCalculation`
Therefore, the following won`t work either:
# my module
from unittest.mock import patch
from third_party_library import LibraryClass
from my_module import BetterCalculation
class BetterLibraryClass(LibraryClass):
def func_to_be_changed(self, x):
with patch(
'third_party_library.utils.SimpleCalculation',
BetterCalculation
):
super().func_to_be_changed(x)
Is there an elegant pythonic way to do this? I guess this boils down to the question: What is the equivaltent of __main__ in the last code snippet that needs to replace third_party_library.utils?
Some context
The first string argument in the patch function can have two different meanings depending on the situation. In the first situation the described object has not been imported and is unavailable to the program which would, therefore, result in a NameError without the mocking. However, in the question, an object needs to be overwritten. Therefore, it is available to the program and not using patch would not result in an error.
Disclaimer
I might have used the complete wrong language in here, as for sure there are precise python terms for all the described notions.
Overwriting an object
As shown in the question, the locally imported SimpleCalculation can be overwritten with __main__.SimpleCalculation. Therefore, it is important to remember that you need to tell patch the path to the local object and not how that same object would be imported in the current script.
Let's assume the following module:
# thirdpartymodule/__init__.py
from .utils import foo
def local_foo():
print("Hello local!")
class Bar:
def __init__(self):
foo()
local_foo()
and
# thirdpartymodule/utils.py
def foo():
print("third party module")
We want to override the functions foo and local_foo. But we don't want to override any functions, we want to override the functions foo and local_foo in the context of the file thirdpartymodule/__init__.py. It is unimportant that the function foo enters the context of the file via an import statement, while local_foo is defined locally. So we want to override the functions in the context of thirdpartymodule.foo and thirdpartymodule.local_foo. The context thirdpartymodule.utils.foo is not important here and won't help us. The following snippet illustrates that:
from unittest.mock import patch
from thirdpartymodule import Bar
bar = Bar()
# third party module
# Hello local!
def myfoo():
print("patched function")
with patch("thirdpartymodule.foo", myfoo):
bar = Bar()
# patched function
# Hello local!
# will not work!
with patch("thirdpartymodule.utils.foo", myfoo):
bar = Bar()
# third party module
# Hello local!
with patch("thirdpartymodule.local_foo", myfoo):
bar = Bar()
# third party module
# patched function
In the hypothetical module of the question we first need to assume that the class LibraryClass is defined in the file third_party_library/library_class.py. Then, we want to override third_party_library.library_class.SimpleCalculation and the correct patch would be:
# my module
from unittest.mock import patch
from third_party_library import LibraryClass
from my_module import BetterCalculation
class BetterLibraryClass(LibraryClass):
def func_to_be_changed(self, x):
with patch(
'third_party_library.library_class.SimpleCalculation',
BetterCalculation
):
super().func_to_be_changed(x)
In python, mocking an object using
#patch('foo.bar')
def test_things(self, bar):
bar.return_value= ...
requires that all tested classes use
import foo
and can not use
from foo import bar
In the second case code under test uses the original object, as mock patches names rather than the function itself. This feels very brittle.
How do we write mocks which will work with both forms of import?
Short answer: No
The principle of a mock is to mock one object. If you import the same object from different ways in you your code (which is somehow weird) you need to create a mock for each object.
Example:
import os
from os.path import isdir
from unittest.mock import patch
>>> with patch('os.path') as mock_os_path:
... mock_os_path.isdir.return_value = "Hello"
... mocked_res = os.path.isdir("./")
... res = path.isdir("./")
... print("mocked_res)
... print(res)
...
Hello
True
According to docs
target should be a string in the form 'package.module.ClassName'. The target is imported and the specified object replaced with the new object, so the target must be importable from the environment you are calling patch() from. The target is imported when the decorated function is executed, not at decoration time.
I am trying to use unittests.mock to mock a void method call of an object.
My package is like below
common
baseupgradehandler.py
baseupgradehandler.py
class BaseUpgradeHandler(object):
def __init__(self, upgrade_config, upgrade_state, system_config, pre_step, main_step, post_step):
...
# Method call to be supressed
def start(self, service_manifest, upgrade_bundle):
# type: (service_version_pb2.ServiceManifest, str) -> ()
...
In my test code I am trying to mock the call to start() like below as explained in the documentation.
from workflow.upgradeworkflow import UpgradeWorkflow
from common.serviceregistry import ServiceRegistry
# The above imports are at the start of the test file
...
with patch('common.baseupgradehandler.BaseUpgradeHandler') as handler_mock: # type: Mock
handler_mock.return_value.start.return_value = ''
wf = UpgradeWorkflow(ServiceRegistry(self.service_bundle, config, sys_config, state),
config,
state,
sys_config)
BaseUpgradeHandler object is returned by get_upgrade_handler() method of ServiceRegistry. When I am executing the above code in test I am seeing the BaseUpgradeHandler.start() is still getting called.
Can someone let me know how can I mock the call to a start() so that the method is not called?
EDIT
If I change my patching code like below it is working as expected and BaseUpgradeHandler is getting mocked and start is not getting called.
with patch('common.baseupgradehandler.BaseUpgradeHandler') as handler_mock: # type: Mock
handler_mock.return_value.start.return_value = ''
with patch('common.serviceregistry.ServiceRegistry') as serviceregistry_mock: # type: Mock
serviceregistry_mock.return_value.get_upgrade_handler.return_value = handler_mock
wf = UpgradeWorkflow(ServiceRegistry(self.service_bundle, config, sys_config, state), config, state, sys_config)
wf.start()
Can someone explain me why do I have to patch ServiceRegistry as well?
The code you provided is not enough to see the part that causes the issue. We'd need to see the module serviceregistry to be sure but I'd take an educated guess:
You have a file a.py (aka baseupgradehandler) like this:
class A:
def method(self):
print("It's real!")
And a file b.py (aka serviceregistry) like this:
from a import A
class B:
def get_A(self):
return A()
In your test files you do this:
import unittest
from unittest.mock import patch
from b import B
from a import A
GAME OVER!
The B module right now has already got its reference to the original A class. When, afterwards, you patch('a.A') only the reference in the a module is changed, but patch has no way to know that B has its own reference to the original A.
You can fix this in three ways:
patch the method: this will modify the existing class so all references to that class will be automatically patched
patch b.A too:
with patch('a.A') as h_a, patch('b.A') as h_b:
h_a.return_value.method.return_value = ''
h_b.return_value.method.return_value = ''
Avoid importing the modules before patching (probably not feasible or a good idea):
import unittest
from unittest.mock import patch
class MyTest(unittest.TestCase):
def test_one(self):
with patch('a.A') as h:
h.return_value.method.return_value = ''
from b import B
B().get_A().method()
I have been using unittest.mocks for a while, and I have been re-inventing the wheel sometimes. I decided to make mockito part of my project and now things look way better. Any kind of mock verification is really simple, if you can, I definitively encourage you to make mockito part of your libraries. This library has a good documentation and so far it has been easier than unittest.mock IMHO.
During a test phase, I would like to modify a part of my source code. For instance, I don't need to plot, so the code responsible for plotting can be commented. Is there a proper tool in unittest module to do so?
Another question is regarding the decorator #patch. Is there a way I can put them into source code in runtime during testing?
Try working with mock,
As it sounds it mocks your code and can be manipulated thru the test.
You may mock an method returned val or an object instance etc etc.
https://www.toptal.com/python/an-introduction-to-mocking-in-python
As Ohad the Lad already said, mocks are at your help.
There are several ways how to mock a function, but in general, you will decorate your test-functions - not the function to mock!
In your case the code could look as follows:
# your_file.py
from ... import ... as plt #depends on the class you use
class MyPlotter(object):
def draw_calling_method(self):
....
plt.draw()
...
return something
# test.py
import mock
from unittest import TestCase
from your_file import MyPlotter
from ... import ... as plt # same import as in the your_file.py
class TestMyPlotter(TestCase):
#mock.patch.object(plt, 'draw')
def test_draw_calling_method_returns_something(self, draw):
plotter = MyPlotter()
plotter_return = plotter.draw_calling_method()
self.assertEqual(plotter_return, something)
This will replace all calls to the plt.draw() with MagicMocks and hinder the draw-execution. If all of your test-methods need to mock the draw-call, the mock decorator could also be applied to the class instead. You only have to make sure then, that all your test-methods accept the mock as second argument (as the test_draw_calling_mehtod_returns_something does). Further you could mock any return values the draw() function may have by setting draw.return_value = ... in the test-method. Attention: This has to happen prior to the call of the function which calls the mock, otherwise the return-values will not be applied.
For more information on mocking refer to the docs python-docs. They are quite comprehensive.
One last thing; As Jonathon Reinhart already mentioned, if you feel it difficult to write your tests it may be good to refactor your code before. This will not only make it more testable, but more readable as well!
I want to fake a package in python. I want to define something so that the code can do
from somefakepackage.morefakestuff import somethingfake
And somefakepackage is defined in code and so is everything below it. Is that possible? The reason for doing this is to trick my unittest that I got a package ( or as I said in the title, a module ) in the python path which actually is just something mocked up for this unittest.
Sure. Define a class, put the stuff you need inside that, assign the class to sys.modules["classname"].
class fakemodule(object):
#staticmethod
def method(a, b):
return a+b
import sys
sys.modules["package.module"] = fakemodule
You could also use a separate module (call it fakemodule.py):
import fakemodule, sys
sys.modules["package.module"] = fakemodule
Yes, you can make a fake module:
from types import ModuleType
m = ModuleType("fake_module")
import sys
sys.modules[m.__name__] = m
# some scripts may expect a file
# even though this file doesn't exist,
# it may be used by Python for in error messages or introspection.
m.__file__ = m.__name__ + ".py"
# Add a function
def my_function():
return 10
m.my_function = my_function
Note, in this example its using an actual module (of ModuleType) since some
Python code may expect modules, (instead of a dummy class).
This can be made into a utility function:
def new_module(name, doc=None):
import sys
from types import ModuleType
m = ModuleType(name, doc)
m.__file__ = name + '.py'
sys.modules[name] = m
return m
print(new_module("fake_module", doc="doc string"))
Now other scripts can run:
import fake_module
I took some of the ideas from the other answers and turned them into a Python decorator #modulize which converts a function into a module. This module can then be imported as usual. Here is an example.
#modulize('my_module')
def my_dummy_function(__name__): # the function takes one parameter __name__
# put module code here
def my_function(s):
print(s, 'bar')
# the function must return locals()
return locals()
# import the module as usual
from my_module import my_function
my_function('foo') # foo bar
The code for the decorator is as follows
import sys
from types import ModuleType
class MockModule(ModuleType):
def __init__(self, module_name, module_doc=None):
ModuleType.__init__(self, module_name, module_doc)
if '.' in module_name:
package, module = module_name.rsplit('.', 1)
get_mock_module(package).__path__ = []
setattr(get_mock_module(package), module, self)
def _initialize_(self, module_code):
self.__dict__.update(module_code(self.__name__))
self.__doc__ = module_code.__doc__
def get_mock_module(module_name):
if module_name not in sys.modules:
sys.modules[module_name] = MockModule(module_name)
return sys.modules[module_name]
def modulize(module_name, dependencies=[]):
for d in dependencies: get_mock_module(d)
return get_mock_module(module_name)._initialize_
The project can be found here on GitHub. In particular, I created this for programming contests which only allow the contestant to submit a single .py file. This allows one to develop a project with multiple .py files and then combine them into one .py file at the end.
You could fake it with a class which behaves like somethingfake:
try:
from somefakepackage.morefakestuff import somethingfake
except ImportError:
class somethingfake(object):
# define what you'd expect of somethingfake, e.g.:
#staticmethod
def somefunc():
...
somefield = ...
TL;DR
Patch sys.modules using unittest.mock:
mock.patch.dict(
sys.modules,
{'somefakepackage': mock.Mock()},
)
Explanation
Other answers correctly recommend to fix sys.modules but a proper way to do it is by patching it using mock.patch. Meaning replacing it temporarily (only for when tests are run) with a fake object that optionally imitates the desired behaviour. And restoring it back once tests are finished to not affect other test cases.
The code in TL;DR section will simply make your missing package not raise ImportError. To provide fake package with contents and imitate desired behaviour, initiate mock.Mock(…) with proper arguments (e.g. add attributes via Mock's **kwargs).
Full code example
The code below temporarily patches sys.modules so that it includes somefakepackage and makes it importable from the dependent modules without ImportError.
import sys
import unittest
from unittest import mock
class SomeTestCase(unittest.TestCase):
def test_smth(self):
# implement your testing logic, for example:
self.assertEqual(
123,
somefakepackage_dependent.some_func(),
)
#classmethod
def setUpClass(cls): # called once before all the tests
# define what to patch sys.modules with
cls._modules_patcher = mock.patch.dict(
sys.modules,
{'somefakepackage': mock.Mock()},
)
# actually patch it
cls._modules_patcher.start()
# make the package globally visible and import it,
# just like if you have imported it in a usual way
# placing import statement at the top of the file,
# but relying on a patched dependency
global somefakepackage_dependent
import somefakepackage_dependent
#classmethod # called once after all tests
def tearDownClass(cls):
# restore initial sys.modules state back
cls._modules_patcher.stop()
To read more about setUpClass/tearDownClass methods, see unittest docs.
unittest's built-in mock subpackage is actually a very powerful tool. Better dive deeper into its documentation to get a better understanding.