I need to patch current datetime in tests. I am using this solution:
def _utcnow():
return datetime.datetime.utcnow()
def utcnow():
"""A proxy which can be patched in tests.
"""
# another level of indirection, because some modules import utcnow
return _utcnow()
Then in my tests I do something like:
with mock.patch('***.utils._utcnow', return_value=***):
...
But today an idea came to me, that I could make the implementation simpler by patching __call__ of function utcnow instead of having an additional _utcnow.
This does not work for me:
from ***.utils import utcnow
with mock.patch.object(utcnow, '__call__', return_value=***):
...
How to do this elegantly?
When you patch __call__ of a function, you are setting the __call__ attribute of that instance. Python actually calls the __call__ method defined on the class.
For example:
>>> class A(object):
... def __call__(self):
... print 'a'
...
>>> a = A()
>>> a()
a
>>> def b(): print 'b'
...
>>> b()
b
>>> a.__call__ = b
>>> a()
a
>>> a.__call__ = b.__call__
>>> a()
a
Assigning anything to a.__call__ is pointless.
However:
>>> A.__call__ = b.__call__
>>> a()
b
TLDR;
a() does not call a.__call__. It calls type(a).__call__(a).
Links
There is a good explanation of why that happens in answer to "Why type(x).__enter__(x) instead of x.__enter__() in Python standard contextlib?".
This behaviour is documented in Python documentation on Special method lookup.
[EDIT]
Maybe the most interesting part of this question is Why I cannot patch somefunction.__call__?
Because the function don't use __call__'s code but __call__ (a method-wrapper object) use function's code.
I don't find any well sourced documentation about that, but I can prove it (Python2.7):
>>> def f():
... return "f"
...
>>> def g():
... return "g"
...
>>> f
<function f at 0x7f1576381848>
>>> f.__call__
<method-wrapper '__call__' of function object at 0x7f1576381848>
>>> g
<function g at 0x7f15763817d0>
>>> g.__call__
<method-wrapper '__call__' of function object at 0x7f15763817d0>
Replace f's code by g's code:
>>> f.func_code = g.func_code
>>> f()
'g'
>>> f.__call__()
'g'
Of course f and f.__call__ references are not changed:
>>> f
<function f at 0x7f1576381848>
>>> f.__call__
<method-wrapper '__call__' of function object at 0x7f1576381848>
Recover original implementation and copy __call__ references instead:
>>> def f():
... return "f"
...
>>> f()
'f'
>>> f.__call__ = g.__call__
>>> f()
'f'
>>> f.__call__()
'g'
This don't have any effect on f function. Note: In Python 3 you should use __code__ instead of func_code.
I Hope that somebody can point me to the documentation that explain this behavior.
You have a way to work around that: in utils you can define
class Utcnow(object):
def __call__(self):
return datetime.datetime.utcnow()
utcnow = Utcnow()
And now your patch can work like a charm.
Follow the original answer that I consider even the best way to implement your tests.
I've my own gold rule: never patch protected methods. In this case the things are little bit smoother because protected method was introduced just for testing but I cannot see why.
The real problem here is that you cannot to patch datetime.datetime.utcnow directly (is C extension as you wrote in the comment above). What you can do is to patch datetime by wrap the standard behavior and override utcnow function:
>>> with mock.patch("datetime.datetime", mock.Mock(wraps=datetime.datetime, utcnow=mock.Mock(return_value=3))):
... print(datetime.datetime.utcnow())
...
3
Ok that is not really clear and neat but you can introduce your own function like
def mock_utcnow(return_value):
return mock.Mock(wraps=datetime.datetime,
utcnow=mock.Mock(return_value=return_value)):
and now
mock.patch("datetime.datetime", mock_utcnow(***))
do exactly what you need without any other layer and for every kind of import.
Another solution can be import datetime in utils and to patch ***.utils.datetime; that can give you some freedom to change datetime reference implementation without change your tests (in this case take care to change mock_utcnow() wraps argument too).
As commented on the question, since datetime.datetime is written in C, Mock can't replace attributes on the class (see Mocking datetime.today by Ned Batchelder). Instead you can use freezegun.
$ pip install freezegun
Here's an example:
import datetime
from freezegun import freeze_time
def my_now():
return datetime.datetime.utcnow()
#freeze_time('2000-01-01 12:00:01')
def test_freezegun():
assert my_now() == datetime.datetime(2000, 1, 1, 12, 00, 1)
As you mention, an alternative is to track each module importing datetime and patch them. This is in essence what freezegun does. It takes an object mocking datetime, iterates through sys.modules to find where datetime has been imported and replaces every instance. I guess it's arguable whether you can do this elegantly in one function.
Related
After reading the answers to the question about monkey-patching classes in Python I tried to apply the advised solution to the following case.
Imagine that we have a module a.py
class A(object):
def foo(self):
print(1)
class AA(A):
pass
and let us try to monkey patch it as follows. It works when we monkey patch class A:
>>> import a
>>> class B(object):
... def foo(self):
... print(3)
...
>>> a.A = B
>>> x = a.A()
>>> x.foo()
3
But if we try the inherited class, it turns to be not patched:
>>> y = a.AA()
>>> y.foo()
1
Is there any way to monkey patch the class with all its inherited classes?
EDIT
For now, the best solution for me is as follows:
>>> class AB(B, a.AA):
... pass
...
>>> a.AA = AB
>>> x = a.AA()
>>> x.foo()
3
Any complex structure of a.AA will be inherited and the only difference between AB and a.AA will be the foo() method. In this way, we don't modify any internal class attributes (like __base__ or __dict__). The only remaining drawback is that we need to do that for each of the inherited classes.
Is it the best way to do this?
You need to explicitly overwrite the tuple of base classes in a.AA, though I don't recommend modifying classes like this.
>>> import a
>>> class B:
... def foo(self):
... print(2)
...
>>> a.AA.__bases__ = (B,)
>>> a.AA().foo()
2
This will also be reflected in a.A.__subclasses__() (although I am not entirely sure as to how that works; the fact that it is a method suggests that it computes this somehow at runtime, rather than simply returning a value that was modified by the original definition of AA).
It appears that the bases classes in a class statement are simply remembered, rather than used, until some operation needs them (e.g. during attribute lookup). There may be some other subtle corner cases that aren't handled as smoothly: caveat programmator.
Normally, a MagicMock returns new MagicMocks for any attribute or method called on it:
>>> mm = MagicMock()
>>> mm
<MagicMock id='140094558921616'>
>>> mm()
<MagicMock name='mock()' id='140094551215016'>
>>> mm.foo()
<MagicMock name='mock.foo()' id='140094551605656'>
Lately, I've been writing fixtures that instead return the same mock for all methods and attributes on the mocked class:
>>> mm = MagicMock()
>>> mm.return_value = mm
>>> mm.foo.return_value = mm
>>> mm
<MagicMock id='140094551638984'>
>>> mm()
<MagicMock id='140094551638984'>
>>> mm.foo()
<MagicMock id='140094551638984'>
This has been convenient, as my fixture can simply yield this single mock and multiple assertions can be run against it. I have not seen any downsides: the called,call_count, and call_args attributes continue to work accurately for each of the mocked attributes.
However, the fact that MagicMock doesn't work this way out of the box makes me feel like I might be missing something.
Is this a bad idea?
The reason that MagicMock doesn’t work that way out of the box is that tests could no longer distinguish between calls to the mock itself, and calls to the mock’s return value. Here’s the mock behaving as designed: we can distinguish that the mock itself was called with the argument 5, and then the object it returned was called with the argument 6.
from unittest.mock import MagicMock
m = MagicMock()
r = m(5)
r(6)
print(m.call_args_list)
print(r.call_args_list)
Output:
[call(5)]
[call(6)]
If we instead make the mock its own return value, then the tests can no longer distinguish between the two. The test can see that two calls were made and can see the two arguments, but cannot tell whether the calls were made to m itself or to the first call’s return value r.
from unittest.mock import MagicMock
m = MagicMock()
m.return_value = m
r = m(5)
r(6)
print(m.call_args_list)
print(r.call_args_list)
Output:
[call(5), call(6)]
[call(5), call(6)]
Using the example
def foo(a):
def bar(b):
return a+b
return bar
d = {1:foo(1), 2:foo(2)}
It appears that pickle module will not work with a function not defined at the module scope, so pickling 'd' will not work. Is there another pickling mechanism available that I should consider?
I'm afraid that you can't pickle nested functions.
The pickle module serializes functions by name. That is, if you have a function myfunc in a module mymodule it simply saves the name mymodule.myfunc and looks it up again when unserializing. (This is an important security and compatibility issue, as it guarantees that the unserializing code uses its own definition for the function, rather than the original definition which might be compromised or obsolete.)
Alas, pickle can't do that with nested functions, because there's no way to directly address them by name. Your bar function, for instance, can't be accessed from outside of foo.
If you need a serializable object that works like a function, you can instead make a class with a __call__ method:
class foo(object):
def __init__(self, a):
self.a = a
def __call__(self, b): # the function formerly known as "bar"
return self.a + b
This works just like the nested functions in the question, and should pose no problem to pickle. Do be aware though, that you'll need to have the same class definition available when you unserialize a foo instance.
You can pickle nested functions if you use dill instead of pickle.
>>> import dill
>>>
>>> def foo(a):
... def bar(b):
... return a+b
... return bar
...
>>> d = {1:foo(1), 2:foo(2)}
>>>
>>> _d = dill.dumps(d)
>>> d_ = dill.loads(_d)
>>> d_
{1: <function bar at 0x108cfe848>, 2: <function bar at 0x108cfe8c0>}
>>> d[1](0) + d[2](10)
13
>>>
according to Blckknght's answersing. if nested function is the only extrac serialized type and will use it as decorator, you can just add functools.warps at top of inner function defination to lead other interpret find the correct name:
from functools import warps
def foo(func):
#wraps(func)
def bar(b):
return func(b)
return bar
#foo
def zzz(b):
return b
I have code which contains the following two lines in it:-
instanceMethod = new.instancemethod(testFunc, None, TestCase)
setattr(TestCase, testName, instanceMethod)
How could it be re-written without using the "new" module? Im sure new style classes provide some kind of workaround for this, but I am not sure how.
There is a discussion that suggests that in python 3, this is not required. The same works in Python 2.6
http://mail.python.org/pipermail/python-list/2009-April/531898.html
See:
>>> class C: pass
...
>>> c=C()
>>> def f(self): pass
...
>>> c.f = f.__get__(c, C)
>>> c.f
<bound method C.f of <__main__.C instance at 0x10042efc8>>
>>> c.f
<unbound method C.f>
>>>
Reiterating the question for every one's benefit, including mine.
Is there a replacement in Python3 for new.instancemethod? That is, given an arbitrary instance (not its class) how can I add a new appropriately defined function as a method to it?
So following should suffice:
TestCase.testFunc = testFunc.__get__(None, TestCase)
You can replace "new.instancemethod" by "types.MethodType":
from types import MethodType as instancemethod
class Foo:
def __init__(self):
print 'I am ', id(self)
def bar(self):
print 'hi', id(self)
foo = Foo() # prints 'I am <instance id>'
mm = instancemethod(bar, foo) # automatically uses foo.__class__
mm() # prints 'I have been bound to <same instance id>'
foo.mm # traceback because no 'field' created in foo to hold ref to mm
foo.mm = mm # create ref to bound method in foo
foo.mm() # prints 'I have been bound to <same instance id>'
This will do the same:
>>> Testcase.testName = testFunc
Yeah, it's really that simple.
Your line
>>> instanceMethod = new.instancemethod(testFunc, None, TestCase)
Is in practice (although not in theory) a noop. :) You could just as well do
>>> instanceMethod = testFunc
In fact, in Python 3 I'm pretty sure it would be the same in theory as well, but the new module is gone so I can't test it in practice.
To confirm that it's not needed to use new.instancemthod() at all since Python v2.4, here's an example how to replace an instance method. It's also not needed to use descriptors (even though it works).
class Ham(object):
def spam(self):
pass
h = Ham()
def fake_spam():
h._spam = True
h.spam = fake_spam
h.spam()
# h._spam should be True now.
Handy for unit testing.
I was wondering how to check whether a variable is a class (not an instance!) or not.
I've tried to use the function isinstance(object, class_or_type_or_tuple) to do this, but I don't know what type would a class will have.
For example, in the following code
class Foo: pass
isinstance(Foo, **???**) # i want to make this return True.
I tried to substitute "class" with ???, but I realized that class is a keyword in python.
Even better: use the inspect.isclass function.
>>> import inspect
>>> class X(object):
... pass
...
>>> inspect.isclass(X)
True
>>> x = X()
>>> isinstance(x, X)
True
>>> inspect.isclass(x)
False
>>> class X(object):
... pass
...
>>> type(X)
<type 'type'>
>>> isinstance(X,type)
True
The inspect.isclass is probably the best solution, and it's really easy to see how it's actually implemented
def isclass(obj):
"""Return true if the obj is a class.
Class objects provide these attributes:
__doc__ documentation string
__module__ name of module in which this class was defined"""
return isinstance(obj, (type, types.ClassType))
isinstance(X, type)
Return True if X is class and False if not.
This check is compatible with both Python 2.x and Python 3.x.
import six
isinstance(obj, six.class_types)
This is basically a wrapper function that performs the same check as in andrea_crotti answer.
Example:
>>> import datetime
>>> isinstance(datetime.date, six.class_types)
>>> True
>>> isinstance(datetime.date.min, six.class_types)
>>> False
Benjamin Peterson is correct about the use of inspect.isclass() for this job.
But note that you can test if a Class object is a specific Class, and therefore implicitly a Class, using the built-in function issubclass.
Depending on your use-case this can be more pythonic.
from typing import Type, Any
def isclass(cl: Type[Any]):
try:
return issubclass(cl, cl)
except TypeError:
return False
Can then be used like this:
>>> class X():
... pass
...
>>> isclass(X)
True
>>> isclass(X())
False
class Foo: is called old style class and class X(object): is called new style class.
Check this What is the difference between old style and new style classes in Python? . New style is recommended. Read about "unifying types and classes"
simplest way is to use inspect.isclass as posted in the most-voted answer.
the implementation details could be found at python2 inspect and python3 inspect.
for new-style class: isinstance(object, type)
for old-style class: isinstance(object, types.ClassType)
em, for old-style class, it is using types.ClassType, here is the code from types.py:
class _C:
def _m(self): pass
ClassType = type(_C)
Well, inspect.isclass is not working for me, instead, try this
class foo:
pass
var = foo()
if str(type(var)).split(".")[0] == "<class '__main__":
print("this is a class")
else:
print(str(type(var)).split(".")[0])
So basically, type(var) is <class 'a type'>
Example: <class 'int'
But, when var is a class, it will appear something like <class '__main__.classname'>
So we split the string into <class '__main__ and we compare using if, if the string fit perfectly then it's a class
There is an alternative way to check it:
import inspect
class cls():
print(None)
inspect.isclass(cls)
Reference: https://www.kite.com/python/docs/inspect.isclass
In some cases (depending on your system), a simple test is to see if your variable has a __module__ attribute.
if getattr(my_variable,'__module__', None):
print(my_variable, ".__module__ is ",my_variable.__module__)
else:
print(my_variable,' has no __module__.')
int, float, dict, list, str etc do not have __module__
There are some working solutions here already, but here's another one:
>>> import types
>>> class Dummy: pass
>>> type(Dummy) is types.ClassType
True