When should I use autospec=True when using mock.patch and its variants?
On one hand, this article warns us to always use autospec=True:
... you should always use the create_autospec method and the autospec parameter with the #patch and #patch.object decorators.
On the other hand, autospec has serious drawbacks and limits, as explained in idjaw's answer to this question.
So my question is: when should I use autospec=True or create_autospec, and when should I not use it?
I fear not using autospec may result in tests not breaking when they really should break, as described in the mentioned article. However autospec has its drawbacks. How should I act?
I can understand the motivation to suggest the enforcing of using autospec.
Maybe the following could help provide more clarity on what you get and don't get with autospec.
In short, using autospec ensures that the attributes you use in your mock are in fact part of the class you are mocking.
So, with the example below, I'll illustrate how a test will pass when technically you might not want it to:
Take this simple example we will test:
class Foo:
def __init__(self, x):
self.x = x
class Bar:
def __init__(self):
self.y = 4
self.c = Foo('potato')
And the test code:
class TestAutoSpec(unittest.TestCase):
#patch('some_module.Foo')
def test_autospec(self, mock_foo_class):
mock_foo_obj = mock_foo_class.return_value
bar_obj = some_module.Bar()
self.assertTrue(hasattr(bar_obj.c, 'you_should_fail'))
Now, if you look back at the Foo class, you will clearly see you_should_fail is clearly not an attribute in Foo. However, if you run this test code, it will in fact pass. Which is very misleading.
This is because if an attribute does not exist in a MagicMock, it will still be of type MagicMock. If you print type(bar_obj.c.you_should_fail) in that test, you will end up getting:
<class 'unittest.mock.MagicMock'>
This will certainly cause the hasattr test to pass. If you run the above test again, except change your patch to be: #patch('some_module.Foo', autospec=True), it will fail as it should.
Now, to write a successful test for this and still use autospec=True, you simply create the attribute in your mock testing as needed. Remember, the reason this is needed, is because autospec cannot know about the attributes created dynamically, i.e. in the __init__ when you create an instance.
So, the autospec way to do this, would be:
class TestAutoSpec(unittest.TestCase):
#patch('some_module.Foo', autospec=True)
def test_autospec(self, mock_foo_class):
mock_foo_obj = mock_foo_class.return_value
# create the attribute you need from mocked Foo
mock_foo_obj.x = "potato"
bar_obj = some_module.Bar()
self.assertEqual(bar_obj.c.x, 'potato')
self.assertFalse(hasattr(bar_obj.c, 'poof'))
Now, your test will successfully pass at validating your x attribute, while also validating that you don't have some bogus attribute that does not exist in your real Foo class.
Here is also another explanation by Martijn Pieters, that does not necessarily directly answer your question, but gives a very good example and explanation of using autospec that can help further your understanding:
https://stackoverflow.com/a/31710001/1832539
Related
When you patch a function using mock, you have the option to specify autospec as True:
If you set autospec=True then the mock with be created with a spec
from the object being replaced. All attributes of the mock will also
have the spec of the corresponding attribute of the object being
replaced. Methods and functions being mocked will have their arguments
checked and will raise a TypeError if they are called with the wrong
signature.
(http://www.voidspace.org.uk/python/mock/patch.html)
I'm wondering why this isn't the default behaviour? Surely we would almost always want to catch passing incorrect parameters to any function we patch?
The only clear way to explain this, is to actually quote the documentation on the downside of using auto-speccing and why you should be careful when using it:
This isn’t without caveats and limitations however, which is why it is
not the default behaviour. In order to know what attributes are
available on the spec object, autospec has to introspect (access
attributes) the spec. As you traverse attributes on the mock a
corresponding traversal of the original object is happening under the
hood. If any of your specced objects have properties or descriptors
that can trigger code execution then you may not be able to use
autospec. On the other hand it is much better to design your objects
so that introspection is safe [4].
A more serious problem is that it is common for instance attributes to
be created in the init method and not to exist on the class at
all. autospec can’t know about any dynamically created attributes and
restricts the api to visible attributes.
I think the key takeaway here is to note this line: autospec can’t know about any dynamically created attributes and restricts the api to visible attributes
So, to help being more explicit with an example of where autospeccing breaks, this example taken from the documentation shows this:
>>> class Something:
... def __init__(self):
... self.a = 33
...
>>> with patch('__main__.Something', autospec=True):
... thing = Something()
... thing.a
...
Traceback (most recent call last):
...
AttributeError: Mock object has no attribute 'a'
As you can see, auto-speccing has no idea that there is an attribute a being created when creating your Something object.
There is nothing wrong with assigning a value to your instance attribute.
Observe the below functional example:
import unittest
from mock import patch
def some_external_thing():
pass
def something(x):
return x
class MyRealClass:
def __init__(self):
self.a = some_external_thing()
def test_thing(self):
return something(self.a)
class MyTest(unittest.TestCase):
def setUp(self):
self.my_obj = MyRealClass()
#patch('__main__.some_external_thing')
#patch('__main__.something')
def test_my_things(self, mock_something, mock_some_external_thing):
mock_some_external_thing.return_value = "there be dragons"
self.my_obj.a = mock_some_external_thing.return_value
self.my_obj.test_thing()
mock_something.assert_called_once_with("there be dragons")
if __name__ == '__main__':
unittest.main()
So, I'm just saying for my test case I want to make sure that the some_external_thing() method does not affect the behaviour of my unittest, so I'm just assigning my instance attribute the mock per mock_some_external_thing.return_value = "there be dragons".
Answering my own question many years later - another reason is speed.
Depending on how complex your object is, it can be that using autospec can slow your test down significantly. I've found this particularly when patching Django models.
The action of autospeccing itself can execute code, for example via the invocation of descriptors.
>>> class A:
... #property
... def foo(self):
... print("rm -rf /")
...
>>> a = A()
>>> with mock.patch("__main__.a", autospec=False) as m:
... pass
...
>>> with mock.patch("__main__.a", autospec=True) as m:
... pass
...
rm -rf /
Therefore, this is a problematic feature to enable by default and
is opt-in only.
Suppose we have the following structure:
class A():
class __A():
def __to_be_mocked(self):
#something here
def __init__(self):
with A.lock:
if not A.instance:
A.instance = A.__A()
def __getattr__(self,name):
return getattr(self.instance,name)
Now we want to mock the function __to_be_mocked.How can we mock it as the target accepted by mock.patch.object is package.module.ClassName.I have tried all methods like
target = A.__A
target = A.___A
and many more.
EDIT:
I solved it using
target=A._A__A and attribute as '_A__to_be_mocked`
Now the question is __to_be_mocked is inside __A so shouldn't it be ___A__to_be_mocked .
Is it because of setattribute in A or __init__ in A?
I mocked a lot of things in python and after did it lot of times I can say:
NEVER mock/patch __something attributes (AKA private attributes)
AVOID to mock/patch _something attributes (AKA protected attributes)
Private
If you mock private things you'll tangled production and test code. When you do this kind of mocks there is always a way to obtain the same behavior by patching or mocking public or protected stuffs.
To explain better what I mean by tangling production and test code I can use your example: to patch A.__B.__to_be_mocked() (I replaced __A inner class by __B to make it more clear) you need to write something like
patch('amodule.A._A__B._B__to_be_mocked')
Now by patching __to_be_mocked you are spreading A, B and to_be_mocked names in your test: that is exactly what I mean to tangled code. So if you need to change some name you should go in all your test and change your patches and no refactoring tool can propose to you to change _A__B._B string.
Now if you are a good guy and take your tests clean you can have just a few points where these names come out but if it is a singleton I can bet that it will spot out like mushrooms.
I would like to point out that private and protected have nothing to do with some security concern but are just way to make your code more clear. That point is crystal clear in python where you don't need to be a hacker to change private or protected attributes: these conventions are here just to help you on reading code where you can say Oh great! I don't need to understand what is it ... it just the dirty work. IMHO private attributes in python fails this goal (__ is too long and see it really bother me) and protected are just enough.
Side note: little example to understand python's private naming:
>>> class A():
... class __B():
... def __c(self):
... pass
...
>>> a = A()
>>> dir(a)
['_A__B', '__doc__', '__module__']
>>> dir(a._A__B)
['_B__c', '__doc__', '__module__']
To come back at your case: How your code use __to_be_mocked() method? is it possible to have the same effect by patch/mock something else in A (and not A.__A) class?
Finally, if you are mocking private method to sense something to test you are in the wrong place: never test the dirty work it should/may/can change without change your tests. What you need is to test code behavior and not how it is written.
Protected
If you need test, patch or mock protected stuffs maybe your class hide some collaborators: test it and use your test to refactor your code then clean your tests.
Disclaimer
Indeed: I spread this kind of crap in my tests and then I fight to remove it when I understand that I can do it better.
Class & instance members starting with double underscores have their names rewritten to prevent collisions with same-name members in parent classes, making them behave as if "private". So __B here is actually accessible as A._A__B. (Underscore, class name, double underscored member name). Note that if you use the single-underscore convention (_B), no rewriting happens.
That being said, you'll rarely see anyone actually use this form of access and especially not in prod code as things are made "private" for a reason. For mocking, maybe, if there's no better way.
I'm not sure what's the best way to build independent unit tests on methods of a python class. Here is a trivial example:
class MyClass(object):
def __init__(self, data):
self.data = data
def myMethod(self):
#do something
return True
I'm building unit tests like this:
class TestMyClass(unittest.TestCase):
def test_init(self):
mydata = mock.Mock()
c = MyClass(mydata)
self.assertEqual(c.data, mydata)
def test_myMethod(self):
mydata = mock.Mock()
c = MyClass(mydata)
self.assertTrue(c.myMethod())
OK, this is very trivial... but the point is: test_myMethod() depends on __init__ because I have to instantiate the object itself: it isn't really isolated. The only way that has come to my mind to solve this issue is mocking the object itself... but how can I test a real method on a mocked object?
I know that in real world a perfect isolation could be impossible, but I'm curious to know if there are better ways.
It comes down to what you want to test and in how much granularity. For example, what exactly is the assertion that c.data equals the data you have just assigned in the previous line actually test? It tests a single line of practical code, and it tests a rather unimportant implementation detail. This is probably much too granular to bother.
You want to ensure that your object behaves as it should overall. That means you want to test that you can instantiate it, and then assert that
if you instantiate it with data A and then call method B the result will be C
if you instantiate it with data D and then call method B the result will be E
if you try to instantiate it with incorrect data that an error is thrown
Your tests should focus on object behaviour, not on testing implementation minutiae. Write your tests so you could drop in an alternative object implementing the same behaviour in a different way without breaking your tests. You are writing that class to get some specified tasks done; test that your implementation can get those tasks done, not how it gets them done.
Exempli gratia:
def test_worksWithDataA(self):
c = MyClass({'some': 'data'})
self.assertTrue(c.myMethod())
def test_worksWithDataB(self):
c = MyClass({'other': 'data'})
self.assertFalse(c.myMethod())
def test_rejectsDataC(self):
with self.assertRaises(SomeException):
MyClass(None)
You only need to mock something if you need to pass a dependency, but that dependency is too complex to instantiate for the sake of the test. As long as your data is just a dict, there's no need to mock it. If your data is a complex type on its own, then you may want to mock it in order to test MyClass, without getting tangled up in having to test that other class at the same time. When mocking, again, mock the expected behaviour of that class; as a rule of thumb, all the behaviour you're asserting your class has using unit tests can be mocked elsewhere.
Is there any equivalent of strict mocks in python? Some mechanism to report unintended call of mocked methods (action.step2() in this example), just like this in GoogleMock framework.
class Action:
def step1(self, arg):
return False
def step2(self, arg):
return False
def algorithm(action):
action.step1('111')
action.step2('222')
return True
class TestAlgorithm(unittest.TestCase):
def test_algorithm(self):
actionMock = mock.create_autospec(Action)
self.assertTrue(algorithm(actionMock))
actionMock.step1.assert_called_once_with('111')
Looks like it's not supported out of the box. However there are at least two approaches on how to achieve the same result.
Passing list of allowed members
According to mock documentation
spec: This can be either a list of strings or an existing object (a class or instance) that acts as the specification for the mock object. If you pass in an object then a list of strings is formed by calling dir on the object (excluding unsupported magic attributes and methods). Accessing any attribute not in this list will raise an AttributeError.
So, in order to fail your test example just replace
actionMock = mock.create_autospec(Action)
to
actionMock = mock.Mock(spec=['step1'])
Such an approach have certain drawbacks compared to passing class or instance as spec argument, as you have to pass all the allowed methods and than set up expectations on them, effectively registering them twice. Also, if you need to restrict a subset of methods you have to pass list of all methods execept those. This can be achieved as follows:
all_members = dir(Action) # according to docs this is what's happening behind the scenes
all_members.remove('step2') # remove all unwanted methods
actionMock = mock.Mock(spec=all_members)
Setting exceptions on restricted methods
Alternative approach would be to excplicitly set failures on methods you don't want to be called:
def test_algorithm(self):
actionMock = mock.create_autospec(Action)
actionMock.step2.side_effect = AttributeError("Called step2") # <<< like this
self.assertTrue(algorithm(actionMock))
actionMock.step1.assert_called_once_with('111')
This have some limitations as well: you've got to set errors as well as expectations.
As a final note, one radical solution to the problem would be to patch mock to add strict parameter to Mock constructor and send a pull request. Than either it would be accepted or mock maintainers will point out on how to achieve that. :)
Yes, this is possible using the spec= and autospec= arguments. See the mock documentation on Autospeccing for more information. In your example it would become:
action_mock = mock.Mock(spec=Action)
or:
action_mock = mock.Mock('Action', autospec=True)
Another possibility:
Checking call_count individually on restricted methods
Ensure that call_count is zero on methods that should not be called.
class TestAlgorithm(unittest.TestCase):
def test_algorithm(self):
actionMock = mock.create_autospec(Action)
self.assertTrue(algorithm(actionMock))
actionMock.step1.assert_called_once_with('111')
self.assertEqual(actionMock.step2.call_count, 0) # <<< like this
The drawback is that you have to check all unexpected calls one by one.
I want to add two variables to my subclass which is inherited from unittest.testcase
like I have:
import unittest
class mrp_repair_test_case(unittest.TestCase):
def __init__(self, a=None, b=None, methodName=['runTest']):
unittest.TestCase.__init__(self)
self.a= a
self.b = b
def test1(self):
..........
.......
def runtest()
mrp_repair_test_case(a=10,b=20)
suite = unittest.TestLoader().loadTestsFromTestCase(mrp_repair_test_case)
res = unittest.TextTestRunner(stream=out,verbosity=2).run(suite)
how can I acvhieve this:
I am getting this error:
ValueError: no such test method in ****<class 'mrp_repair.unit_test.test.mrp_repair_test_case'>:**** runTest
thanks
At first glance, it looks like you need to create an instance of mrp_repair_test_case. Your current line:
mrp_repair_test_case(a=10,b=20)
doesn't actually do anything.
Try (not tested):
def runtest():
m = mrp_repair_test_case(a=10, b=20)
suite = unittest.TestLoader().loadsTestsFromTestCase(m)
res = unittest.TextTestRunner(stream=out, verbosity=2).run(suite)
This assumes you've set up 'out' as a stream already.
Edit:
By the way, is there any reason you're not using a setUp method to set these values? That would be normal best practice. Looking at the documentation of loadTestsFromTestCase it looks like it will only accept the Class itself not an instance of it, which would mean you're rather working against the design of the unittest module.
Edit 2:
In response to your further information, I would actually set your uid and cursor values seperately at module level before calling the tests. I'm not a huge fan of globals normally, but if I'm understanding you correctly these values will be A) read-only B) always the same for the same customer which avoids most of the normal pitfalls in using them.
Edit 3:
To answer your edit, if you really want to use __init__ you probably can, but you will have to roll your own loadsTestsFromTestCase alternative, and possibly your own TestSuite (you'll have to check the internals of how it works). As I said above, you'll be working against the existing design of the module - to the extent that if you decide to do your testing that way it might be easier to roll your own solution completely from scratch than use unittest. Amend: just checked, you'd definately have to roll your own version of TestSuite, as the existing one creates a new instance of the TestCaseClass for each test.