We Keep Coding

How do I mock methods of a decorated class in python?

Having trouble with applying mocks to a class with a decorator. If I write the class without a decorator, patches are applied as expected. However, once the class is decorated, the same patch fails to apply.
What's going on here, and what's the best way to approach testing classes that may be decorated?
Here's a minimal reproduction.
# module.py
import functools
def decorator(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
#decorator # comment this out and the test passes
class Something:
def do_external(self):
raise Exception("should be mocked")
def run(self):
self.do_external()
# test_module.py
from unittest import TestCase
from unittest.mock import Mock, patch
from module import Something
class TestModule(TestCase):
#patch('module.Something.do_external', Mock())
def test_module(self):
s = Something()
s.run()
If you prefer, here's an online reproducible example of the issue.

So, as I stated in the comment, your wrapper function replaces Something in the module module namespace. So, putting your code in module.py on my computer, observe:
>>> import module
>>> type(module.Something)
<class 'function'>
Since you used the functools.wraps decorator, the object being wrapped is added to the wrapper function at .__wrapped__:
>>> module.Something.__wrapped__
<class 'module.Something'>
>>> type(module.Something.__wrapped__)
<class 'type'>
So when you patch module.Something, you are patching the function object, not the class object. But instances of your class directly reference the class internally, it doesn't matter what global name refers to it. So, observe some more:
>>> import unittest.mock as mock
>>> with mock.patch('module.Something.do_external', mock.Mock()):
... print(module.Something.do_external)
... print(module.Something.__wrapped__.do_external)
...
<Mock id='140609580169920'>
<function Something.do_external at 0x7fe23822cc10>
This is why we see this particular behavior:
>>> with mock.patch('module.Something.do_external', mock.Mock()):
... module.Something().do_external()
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/Users/jarrivillaga/module.py", line 18, in do_external
raise Exception("should be mocked")
Exception: should be mocked
In this particular case, because the __wrapped__ attribute references the original class, we can patch that:
>>> with mock.patch('module.Something.__wrapped__.do_external', mock.Mock()):
... module.Something().do_external()
...
<Mock name='mock()' id='140608505553680'>
But I highly suggest rethinking your decorator design, if this is meant for external/public use. But just fundamentally, module.Something is not a class, it is a function, so you cannot treat it like a class and expect it to work like a class.
Note, the fact that you used wraps makes it possible for the patch to work at all, although, it just hides the problem because putting those other functions as attributes of the wrapper function don't really provide anything useful. wraps is mostly meant to be used when wrapping other functions, where creating a new function that looks like the old function makes sense, in the case of a class, though, you are making a function look like a class, but only superficially. Just removing the #wraps line, observe:
>>> import module
>>> import unittest.mock as mock
>>> with mock.patch('module.Something.do_external', mock.Mock()):
... pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jarrivillaga/miniconda3/lib/python3.9/unittest/mock.py", line 1404, in __enter__
original, local = self.get_original()
File "/Users/jarrivillaga/miniconda3/lib/python3.9/unittest/mock.py", line 1377, in get_original
raise AttributeError(
AttributeError: <function decorator.<locals>.wrapper at 0x7ff820081160> does not have the attribute 'do_external'
So functools.wraps here was just hiding a fundamental error.

How can I tell if a class has a method `__call__`?

A very simple class isn't a callable type:
>>> class myc:
... pass
...
>>> c=myc()
>>> callable(c)
False
How can I tell if a class has a method __call__? Why do the following two ways give opposite results?
>>> myc.__call__
<method-wrapper '__call__' of type object at 0x1104b18>
>>> __call__ in myc.__dict__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name '__call__' is not defined
Thanks.

myc.__call__ is giving you the __call__ method used to call myc itself, not instances of myc. It's the method invoked when you do
new_myc_instance = myc()
not when you do
new_myc_instance()
__call__ in myc.__dict__ gives you a NameError because you forgot to put quotation marks around __call__, and if you'd remembered to put quotation marks, it would have given you False because myc.__call__ isn't found through myc.__dict__. It's found through type(myc).__dict__ (a.k.a. type.__dict__, because type(myc) is type).
To check if myc has a __call__ implementation for its instances, you could perform a manual MRO search, but collections.abc.Callable already implements that for you:
issubclass(myc, collections.abc.Callable)

The reason why myc.__call__ gives you something back 1 instead of raising an AttributeError is because the metaclass of myc (type) has a __call__ method. And if an attribute isn't found on the instance (in this case your class) it's looked up on the class (in this case the metaclass).
For example if you had a custom metaclass the __call__ lookup would've returned something different:
class mym(type):
def __call__(self, *args, **kwargs):
return super().__call__(*args, **kwargs)
class myc(metaclass=mym):
pass
>>> myc.__call__
<bound method mym.__call__ of <class '__main__.myc'>>
Regarding the __call__ in myc.__dict__ that has been answered in the comments and the other answer already: You just forgot the quotations:
>>> '__call__' in myc.__dict__
False
However it could also be that myc subclasses a callable class, in that case it would also give False:
class mysc:
def __call__(self):
return 10
class myc(mysc):
pass
>>> '__call__' in myc.__dict__
False
So you need something more robust. Like searching all the superclasses:
>>> any('__call__' in cls.__dict__ for cls in myc.__mro__)
True
Or as pointed out by user2357112 use collections.Callable which overrides the issubclass-check so that it looks for __call__:
>>> from collections.abc import Callable
>>> issubclass(myc, Callable)
True
1 That's also the reason why you can't just use hasattr(myc, '__call__') to find out if it has a __call__ method itself.

How to mock a property

I'm asking how to mock a class property in a unit test using Python 3. I've tried the following, which makes sense for me following the docs, but it doesn't work:
foo.py:
class Foo():
#property
def bar(self):
return 'foobar'
def test_foo_bar(mocker):
foo = Foo()
mocker.patch.object(foo, 'bar', new_callable=mocker.PropertyMock)
print(foo.bar)
I've installed pytest and pytest_mock and run the test like this:
pytest foo.py
I got the following error:
> setattr(self.target, self.attribute, new_attr)
E AttributeError: can't set attribute
/usr/lib/python3.5/unittest/mock.py:1312: AttributeError
My expectation would be that the test runs without errors.

The property mechanism relies on the property attribute being defined on the object's class. You can't create a "property like" method or attribute on a single instance of a class (for a better understanding, read about Python's descriptor protocol)
Therefore you have to apply the patch to your class - you can use the with statement so that the class is properly restored after your test:
def test_foo_bar(mock):
foo = Foo()
with mock.patch(__name__ + "Foo.bar", new=mocker.PropertyMock)
print(foo.bar)

You can directly return the value if you do not need extra features
from mock import patch
#patch('foo.Foo.bar', 'mocked_property_value')
def test_foo_bar():
foo = Foo()
print(foo.bar)
Or you can wrap MagicMocks with a call to function property:
from mock import patch, MagicMock
#patch('foo.Foo.bar', property(MagicMock(return_value='mocked_property_value')))
def test_foo_bar():
foo = Foo()
print(foo.bar)

Mock modules and subclasses (TypeError: Error when calling the metaclass bases)

To compile documentation on readthedocs, the module h5py has to be mocked. I get an error which can be reproduced with this simple code:
from __future__ import print_function
import sys
try:
from unittest.mock import MagicMock
except ImportError:
# Python 2
from mock import Mock as MagicMock
class Mock(MagicMock):
#classmethod
def __getattr__(cls, name):
return Mock()
sys.modules.update({'h5py': Mock()})
import h5py
print(h5py.File, type(h5py.File))
class A(h5py.File):
pass
print(A, type(A))
class B(A):
pass
The output of this script is:
<Mock id='140342061004112'> <class 'mock.mock.Mock'>
<Mock spec='str' id='140342061005584'> <class 'mock.mock.Mock'>
Traceback (most recent call last):
File "problem_mock.py", line 32, in <module>
class B(A):
TypeError: Error when calling the metaclass bases
str() takes at most 1 argument (3 given)
What is the correct way to mock h5py and h5py.File?
It seems to me that this issue is quite general for documentation with readthedocs where some modules have to be mocked. It would be useful for the community to have an answer.

You can't really use Mock instances to act as classes; it fails hard on Python 2, and works by Python 3 only by accident (see below).
You'd have to return the Mock class itself instead if you wanted them to work in a class hierarchy:
>>> class A(Mock): # note, not called!
... pass
...
>>> class B(A):
... pass
...
>>> B
<class '__main__.B'>
>>> B()
<B id='4394742480'>
If you can't import h5py at all, that means you'll need to keep a manually updated list of classes where you return the class rather than an instance:
_classnames = {
'File',
# ...
}
class Mock(MagicMock):
#classmethod
def __getattr__(cls, name):
return Mock if name in _classnames else Mock()
This is not foolproof; there is no way to detect the parent instance in a classmethod, so h5py.File().File would result in yet another 'class' being returned even if in the actual implementation that was some other object instead. You could partially work around that by creating a new descriptor to use instead of the classmethod decorator that would bind to either the class or to an instance if one is available; that way you at least would have a context in the form of self._mock_name on instances of your Mock class.
In Python 3, using MagicMock directly without further customisation works when used as a base class:
>>> from unittest.mock import MagicMock
>>> h5py = MagicMock()
>>> class A(h5py.File): pass
...
>>> class B(A): pass
...
but this is not really intentional and supported behaviour; the classes and subclasses are 'specced' from the classname string:
>>> A
<MagicMock spec='str' id='4353980960'>
>>> B
<MagicMock spec='str' id='4354132344'>
and thus have all sorts of issues down the line as instanciation doesn't work:
>>> A()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mjpieters/Development/Library/buildout.python/parts/opt/lib/python3.5/unittest/mock.py", line 917, in __call__
return _mock_self._mock_call(*args, **kwargs)
File "/Users/mjpieters/Development/Library/buildout.python/parts/opt/lib/python3.5/unittest/mock.py", line 976, in _mock_call
result = next(effect)
StopIteration

Polymorphism with callables in Python

I have an interface class called iResource, and a number of subclasses, each of which implement the "request" method. The request functions use socket I/O to other machines, so it makes sense to run them asynchronously, so those other machines can work in parallel.
The problem is that when I start a thread with iResource.request and give it a subclass as the first argument, it'll call the superclass method. If I try to start it with "type(a).request" and "a" as the first argument, I get "" for the value of type(a). Any ideas what that means and how to get the true type of the method? Can I formally declare an abstract method in Python somehow?
EDIT: Including code.
def getSocialResults(self, query=''):
#for a in self.types["social"]: print type(a)
tasks = [type(a).request for a in self.types["social"]]
argss = [(a, query, 0) for a in self.types["social"]]
grabbers = executeChainResults(tasks, argss)
return igrabber.cycleGrabber(grabbers)
"executeChainResults" takes a list "tasks" of callables and a list "argss" of args-tuples, and assumes each returns a list. It then executes each in a separate thread, and concatenates the lists of results. I can post that code if necessary, but I haven't had any problems with it so I'll leave it out for now.
The objects "a" are DEFINITELY not of type iResource, since it has a single constructor that just throws an exception. However, replacing "type(a).request" with "iResource.request" invokes the base class method. Furthermore, calling "self.types["social"][0].request" directly works fine, but the above code gives me: "type object 'instance' has no attribute 'request'".
Uncommenting the commented line prints <type 'instance'> several times.

You can just use the bound method object itself:
tasks = [a.request for a in self.types["social"]]
# ^^^^^^^^^
grabbers = executeChainResults(tasks, [(query, 0)] * len(tasks))
# ^^^^^^^^^^^^^^^^^^^^^^^^^
If you insist on calling your methods through the base class you could also do it like this:
from abc import ABCMeta
from functools import wraps
def virtualmethod(method):
method.__isabstractmethod__ = True
#wraps(method)
def wrapper(self, *args, **kwargs):
return getattr(self, method.__name__)(*args, **kwargs)
return wrapper
class IBase(object):
__metaclass__ = ABCMeta
#virtualmethod
def my_method(self, x, y):
pass
class AddImpl(IBase):
def my_method(self, x, y):
return x + y
class MulImpl(IBase):
def my_method(self, x, y):
return x * y
items = [AddImpl(), MulImpl()]
for each in items:
print IBase.my_method(each, 3, 4)
b = IBase() # <-- crash
Result:
7
12
Traceback (most recent call last):
File "testvirtual.py", line 30, in <module>
b = IBase()
TypeError: Can't instantiate abstract class IBase with abstract methods my_method
Python doesn't support interfaces as e.g. Java does. But with the abc module you can ensure that certain methods must be implemented in subclasses. Normally you would do this with the abc.abstractmethod() decorator, but you still could not call the subclasses method through the base class, like you intend. I had a similar question once and I had the idea of the virtualmethod() decorator. It's quite simple. It essentially does the same thing as abc.abstratmethod(), but also redirects the call to the subclasses method. The specifics of the abc module can be found in the docs and in PEP3119.
BTW: I assume you're using Python >= 2.6.

The reference to "<type "instance" >" you get when you are using an "old style class" in Python - i.e.: classes not derived from the "object" type hierarchy. Old style classes are not supposed to work with several of the newer features of the language, including descriptors and others. AND, among other things, - you can't retrieve an attribute (or method) from the class of an old style class using what you are doing:
>>> class C(object):
... def c(self): pass
...
>>> type (c)
<class '__main__.C'>
>>> c = C()
>>> type(c).c
<unbound method C.c>
>>> class D: #not inheriting from object: old style class
... def d(self): pass
...
>>> d = D()
>>> type(d).d
>>> type(d)
<type 'instance'>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'instance' has no attribute 'd'
>>>
Therefore, just make your base class inherit from "object" instead of "nothing" and check if you still get the error message when requesting the "request" method from type(a) :
As for your other observation:
"The problem is that when I start a thread with iResource.request and give it a subclass as the first argument, it'll call the superclass method."
It seems that the "right" thing for it to do is exactly that:
>>> class A(object):
... def b(self):
... print "super"
...
>>> class B(A):
... def b(self):
... print "child"
...
>>> b = B()
>>> A.b(b)
super
>>>
Here, I call a method in the class "A" giving it an specialized instance of "A" - the method is still the one in class "A".