We Keep Coding

Python make function object subscriptable [duplicate]

I need to patch current datetime in tests. I am using this solution:
def _utcnow():
return datetime.datetime.utcnow()
def utcnow():
"""A proxy which can be patched in tests.
"""
# another level of indirection, because some modules import utcnow
return _utcnow()
Then in my tests I do something like:
with mock.patch('***.utils._utcnow', return_value=***):
...
But today an idea came to me, that I could make the implementation simpler by patching __call__ of function utcnow instead of having an additional _utcnow.
This does not work for me:
from ***.utils import utcnow
with mock.patch.object(utcnow, '__call__', return_value=***):
...
How to do this elegantly?

When you patch __call__ of a function, you are setting the __call__ attribute of that instance. Python actually calls the __call__ method defined on the class.
For example:
>>> class A(object):
... def __call__(self):
... print 'a'
...
>>> a = A()
>>> a()
a
>>> def b(): print 'b'
...
>>> b()
b
>>> a.__call__ = b
>>> a()
a
>>> a.__call__ = b.__call__
>>> a()
a
Assigning anything to a.__call__ is pointless.
However:
>>> A.__call__ = b.__call__
>>> a()
b
TLDR;
a() does not call a.__call__. It calls type(a).__call__(a).
Links
There is a good explanation of why that happens in answer to "Why type(x).__enter__(x) instead of x.__enter__() in Python standard contextlib?".
This behaviour is documented in Python documentation on Special method lookup.

[EDIT]
Maybe the most interesting part of this question is Why I cannot patch somefunction.__call__?
Because the function don't use __call__'s code but __call__ (a method-wrapper object) use function's code.
I don't find any well sourced documentation about that, but I can prove it (Python2.7):
>>> def f():
... return "f"
...
>>> def g():
... return "g"
...
>>> f
<function f at 0x7f1576381848>
>>> f.__call__
<method-wrapper '__call__' of function object at 0x7f1576381848>
>>> g
<function g at 0x7f15763817d0>
>>> g.__call__
<method-wrapper '__call__' of function object at 0x7f15763817d0>
Replace f's code by g's code:
>>> f.func_code = g.func_code
>>> f()
'g'
>>> f.__call__()
'g'
Of course f and f.__call__ references are not changed:
>>> f
<function f at 0x7f1576381848>
>>> f.__call__
<method-wrapper '__call__' of function object at 0x7f1576381848>
Recover original implementation and copy __call__ references instead:
>>> def f():
... return "f"
...
>>> f()
'f'
>>> f.__call__ = g.__call__
>>> f()
'f'
>>> f.__call__()
'g'
This don't have any effect on f function. Note: In Python 3 you should use __code__ instead of func_code.
I Hope that somebody can point me to the documentation that explain this behavior.
You have a way to work around that: in utils you can define
class Utcnow(object):
def __call__(self):
return datetime.datetime.utcnow()
utcnow = Utcnow()
And now your patch can work like a charm.
Follow the original answer that I consider even the best way to implement your tests.
I've my own gold rule: never patch protected methods. In this case the things are little bit smoother because protected method was introduced just for testing but I cannot see why.
The real problem here is that you cannot to patch datetime.datetime.utcnow directly (is C extension as you wrote in the comment above). What you can do is to patch datetime by wrap the standard behavior and override utcnow function:
>>> with mock.patch("datetime.datetime", mock.Mock(wraps=datetime.datetime, utcnow=mock.Mock(return_value=3))):
... print(datetime.datetime.utcnow())
...
3
Ok that is not really clear and neat but you can introduce your own function like
def mock_utcnow(return_value):
return mock.Mock(wraps=datetime.datetime,
utcnow=mock.Mock(return_value=return_value)):
and now
mock.patch("datetime.datetime", mock_utcnow(***))
do exactly what you need without any other layer and for every kind of import.
Another solution can be import datetime in utils and to patch ***.utils.datetime; that can give you some freedom to change datetime reference implementation without change your tests (in this case take care to change mock_utcnow() wraps argument too).

As commented on the question, since datetime.datetime is written in C, Mock can't replace attributes on the class (see Mocking datetime.today by Ned Batchelder). Instead you can use freezegun.
$ pip install freezegun
Here's an example:
import datetime
from freezegun import freeze_time
def my_now():
return datetime.datetime.utcnow()
#freeze_time('2000-01-01 12:00:01')
def test_freezegun():
assert my_now() == datetime.datetime(2000, 1, 1, 12, 00, 1)
As you mention, an alternative is to track each module importing datetime and patch them. This is in essence what freezegun does. It takes an object mocking datetime, iterates through sys.modules to find where datetime has been imported and replaces every instance. I guess it's arguable whether you can do this elegantly in one function.

Python Import Star Creating Hidden Namespace?

I recently ran into some unusual behavior.
foo.py
a = 0
def bar():
print (a)
Console:
>>> import foo
>>> foo.bar()
0
>>> foo.a = 10
>>> foo.bar()
10
Console:
>>> from foo import *
>>> bar()
0
>>> a
0
>>> a = 10
>>> a
10
>>> bar()
0
I'm inferring that import * is actually creating two copies of a - one in the global namespace and one inside the foo module which cannot be accessed. Is this behavior explained/documented anywhere? I'm having trouble figuring out what to search for.
This seems like a notable and unexpected consequence of import * but for some reason I've never seen it brought up before.

There is no such thing as a hidden namespace in Python and the described behaviour is the normal and expected one.
You should read https://docs.python.org/3/tutorial/modules.html#more-on-modules in order to understand better how the globals do do work.

Why dill dumps external classes by reference, no matter what?

In the example below, I have placed the class Foo inside its own module foo.
Why is the external class dumped by ref? The instance ff is not being dumped with its source code.
I am using Python 3.4.3 and dill-0.2.4.
import dill
import foo
class Foo:
y = 1
def bar( self, x ):
return x + y
f = Foo()
ff = foo.Foo()
print( dill.dumps( f, byref=False, recurse=True ) )
print( '\n' )
print( dill.dumps( ff, byref=False, recurse=True ) )
Well, the code above is actually wrong (should be Foo.y, instead of y). Correcting the code gives me an exception while dumping the f instance.

I'm the dill author. The foo.Foo instance (ff) pickles by reference because it's defined in a file. This is primarily for compactness of the pickled string. So the primary issue I can think of when importing a class by reference is that the class definition is not found on the other resource you might want to unpickle to (i.e. no module foo exists there). I believe that's a current feature request (and if it's not, feel free to submit a ticket on the github page).
Note, however, if you do modify the class dynamically, it does pull in the dynamically modified code to the pickled string.
>>> import dill
>>> import foo
>>>
>>> class Foo:
... y = 1
... def bar( self, x ):
... return x + Foo.y
...
>>> f = Foo()
>>> ff = foo.Foo()
So when Foo is defined in __main__, byref is respected.
>>> dill.dumps(f, byref=False)
b'\x80\x03cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01X\x04\x00\x00\x00typeq\x02\x85q\x03Rq\x04X\x03\x00\x00\x00Fooq\x05h\x01X\x06\x00\x00\x00objectq\x06\x85q\x07Rq\x08\x85q\t}q\n(X\r\x00\x00\x00__slotnames__q\x0b]q\x0cX\x03\x00\x00\x00barq\rcdill.dill\n_create_function\nq\x0e(cdill.dill\n_unmarshal\nq\x0fC]\xe3\x02\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00C\x00\x00\x00s\x0b\x00\x00\x00|\x01\x00t\x00\x00j\x01\x00\x17S)\x01N)\x02\xda\x03Foo\xda\x01y)\x02\xda\x04self\xda\x01x\xa9\x00r\x05\x00\x00\x00\xfa\x07<stdin>\xda\x03bar\x03\x00\x00\x00s\x02\x00\x00\x00\x00\x01q\x10\x85q\x11Rq\x12c__builtin__\n__main__\nh\rNN}q\x13tq\x14Rq\x15X\x07\x00\x00\x00__doc__q\x16NX\n\x00\x00\x00__module__q\x17X\x08\x00\x00\x00__main__q\x18X\x01\x00\x00\x00yq\x19K\x01utq\x1aRq\x1b)\x81q\x1c.'
>>> dill.dumps(f, byref=True)
b'\x80\x03c__main__\nFoo\nq\x00)\x81q\x01.'
>>>
However, when the class is defined in a module, byref is not respected.
>>> dill.dumps(ff, byref=False)
b'\x80\x03cfoo\nFoo\nq\x00)\x81q\x01.'
>>> dill.dumps(ff, byref=True)
b'\x80\x03cfoo\nFoo\nq\x00)\x81q\x01.'
Note, that I wouldn't use the recurse option in this case, as Foo.y will likely infinitely recurse. That's also something that I believe there's current ticket for, but if there isn't, there should be.
Let's dig a little deeper… what if we modify the instance...
>>> ff.zap = lambda x: x + ff.y
>>> _ff = dill.loads(dill.dumps(ff))
>>> _ff.zap(2)
3
>>> dill.dumps(ff, byref=True)
b'\x80\x03cfoo\nFoo\nq\x00)\x81q\x01}q\x02X\x03\x00\x00\x00zapq\x03cdill.dill\n_create_function\nq\x04(cdill.dill\n_unmarshal\nq\x05CY\xe3\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00C\x00\x00\x00s\x0b\x00\x00\x00|\x00\x00t\x00\x00j\x01\x00\x17S)\x01N)\x02\xda\x02ff\xda\x01y)\x01\xda\x01x\xa9\x00r\x04\x00\x00\x00\xfa\x07<stdin>\xda\x08<lambda>\x01\x00\x00\x00s\x00\x00\x00\x00q\x06\x85q\x07Rq\x08c__builtin__\n__main__\nX\x08\x00\x00\x00<lambda>q\tNN}q\ntq\x0bRq\x0csb.'
>>> dill.dumps(ff, byref=False)
b'\x80\x03cfoo\nFoo\nq\x00)\x81q\x01}q\x02X\x03\x00\x00\x00zapq\x03cdill.dill\n_create_function\nq\x04(cdill.dill\n_unmarshal\nq\x05CY\xe3\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00C\x00\x00\x00s\x0b\x00\x00\x00|\x00\x00t\x00\x00j\x01\x00\x17S)\x01N)\x02\xda\x02ff\xda\x01y)\x01\xda\x01x\xa9\x00r\x04\x00\x00\x00\xfa\x07<stdin>\xda\x08<lambda>\x01\x00\x00\x00s\x00\x00\x00\x00q\x06\x85q\x07Rq\x08c__builtin__\n__main__\nX\x08\x00\x00\x00<lambda>q\tNN}q\ntq\x0bRq\x0csb.'
>>>
No biggie, it pulls in the dynamically added code. However, we'd probably like to modify Foo and not the instance.
>>> Foo.zap = lambda self,x: x + Foo.y
>>> dill.dumps(f, byref=True)
b'\x80\x03c__main__\nFoo\nq\x00)\x81q\x01.'
>>> dill.dumps(f, byref=False)
b'\x80\x03cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01X\x04\x00\x00\x00typeq\x02\x85q\x03Rq\x04X\x03\x00\x00\x00Fooq\x05h\x01X\x06\x00\x00\x00objectq\x06\x85q\x07Rq\x08\x85q\t}q\n(X\x03\x00\x00\x00barq\x0bcdill.dill\n_create_function\nq\x0c(cdill.dill\n_unmarshal\nq\rC]\xe3\x02\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00C\x00\x00\x00s\x0b\x00\x00\x00|\x01\x00t\x00\x00j\x01\x00\x17S)\x01N)\x02\xda\x03Foo\xda\x01y)\x02\xda\x04self\xda\x01x\xa9\x00r\x05\x00\x00\x00\xfa\x07<stdin>\xda\x03bar\x03\x00\x00\x00s\x02\x00\x00\x00\x00\x01q\x0e\x85q\x0fRq\x10c__builtin__\n__main__\nh\x0bNN}q\x11tq\x12Rq\x13X\x07\x00\x00\x00__doc__q\x14NX\r\x00\x00\x00__slotnames__q\x15]q\x16X\n\x00\x00\x00__module__q\x17X\x08\x00\x00\x00__main__q\x18X\x01\x00\x00\x00yq\x19K\x01X\x03\x00\x00\x00zapq\x1ah\x0c(h\rC`\xe3\x02\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00C\x00\x00\x00s\x0b\x00\x00\x00|\x01\x00t\x00\x00j\x01\x00\x17S)\x01N)\x02\xda\x03Foo\xda\x01y)\x02\xda\x04self\xda\x01x\xa9\x00r\x05\x00\x00\x00\xfa\x07<stdin>\xda\x08<lambda>\x01\x00\x00\x00s\x00\x00\x00\x00q\x1b\x85q\x1cRq\x1dc__builtin__\n__main__\nX\x08\x00\x00\x00<lambda>q\x1eNN}q\x1ftq Rq!utq"Rq#)\x81q$.'
Ok, that's fine, but what about the Foo in our external module?
>>> ff = foo.Foo()
>>>
>>> foo.Foo.zap = lambda self,x: x + foo.Foo.y
>>> dill.dumps(ff, byref=False)
b'\x80\x03cfoo\nFoo\nq\x00)\x81q\x01.'
>>> dill.dumps(ff, byref=True)
b'\x80\x03cfoo\nFoo\nq\x00)\x81q\x01.'
>>>
Hmmm… not good. So the above is probably a pretty compelling use case to change the behavior dill exhibits for classes defined in modules -- or at least enable one of the settings to provide better behavior.
In sum, the answer is: we didn't have a use case for it, so now that we do… this should be a feature request if it is not already.

Python: assert call to a list that contains a variable

This question follows python 2.7.3 syntax. In unittest framework, suppose I have the following set up:
import mock;
my_mock = mock.Mock();
my_patch = mock.patch("my_method", my_mock);
Now suppose my_method takes on a list argument as input.
How Can I use my_mock.assert_any_call to make sure that a call is made to my_method such that the input list contains a particular value?

You can do that by use both mock_calls and call unpacking as documented here. Now a for cycle can be enough to do the work:
>>> import mock
>>> m = mock.Mock()
>>> m([1,2])
<Mock name='mock()' id='140596484020816'>
>>> m([5,6])
<Mock name='mock()' id='140596484020816'>
>>> m([8,9])
<Mock name='mock()' id='140596484020816'>
>>> for name,args,kwrgs in m.mock_calls:
... if 5 in args[0]:
... print("found")
...
found

Is it possible to list all functions in a module? [duplicate]

This question already has answers here:
How to list all functions in a module?
(20 answers)
Closed 7 years ago.
I defined a .py file in this format:
foo.py
def foo1(): pass
def foo2(): pass
def foo3(): pass
I import it from another file:
main.py
from foo import *
# or
import foo
Is it possible list all functions name, e.g. ["foo1", "foo2", "foo3"]?
Thanks for your help, I made a class for what I want, pls comment if you have suggestion
class GetFuncViaStr(object):
def __init__(self):
d = {}
import foo
for y in [getattr(foo, x) for x in dir(foo)]:
if callable(y):
d[y.__name__] = y
def __getattr__(self, val) :
if not val in self.d :
raise NotImplementedError
else:
return d[val]

The cleanest way to do these things is to use the inspect module. It has a getmembers function that takes a predicate as the second argument. You can use isfunction as the predicate.
import inspect
all_functions = inspect.getmembers(module, inspect.isfunction)
Now, all_functions will be a list of tuples where the first element is the name of the function and the second element is the function itself.

you can use dir to explore a namespace.
import foo
print dir(foo)
Example: loading your foo in shell
>>> import foo
>>> dir(foo)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'foo1', 'foo2', 'foo3']
>>>
>>> getattr(foo, 'foo1')
<function foo1 at 0x100430410>
>>> k = getattr(foo, 'foo1')
>>> k.__name__
'foo1'
>>> callable(k)
True
>>>
You can use getattr to get the associated attribute in foo and find out if it callable.
Check the documentation : http://docs.python.org/tutorial/modules.html#the-dir-function
and if you do - "from foo import *" then the names are included in the namespace where you call this.
>>> from foo import *
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__', 'atexit', 'foo1', 'foo2', 'foo3']
>>>
The following brief on introspection in python might help you :
http://www.ibm.com/developerworks/library/l-pyint.html

Like
aaronasterling said, you can use the getmembers functions from the inspect module to do this.
import inspect
name_func_tuples = inspect.getmembers(module, inspect.isfunction)
functions = dict(name_func_tuples)
However, this will include functions that have been defined elsewhere, but imported into that module's namespace.
If you want to get only the functions that have been defined in that module, use this snippet:
name_func_tuples = inspect.getmembers(module, inspect.isfunction)
name_func_tuples = [t for t in name_func_tuples if inspect.getmodule(t[1]) == module]
functions = dict(name_func_tuples)

Try using inspect module like below for exmaple if module --> temp.py
In [26]: import inspect
In [27]: import temp
In [28]: l1 = [x.__name__ for x in temp.__dict__.values() if inspect.isfunction(x)]
In [29]: print l1
['foo', 'coo']

For a wild import
from foo import *
print dir()
you can use dir() without a parameter to show objects in the current module's namespace. This will most probably include more than just the content of foo.
In case of an absolute import (which you should prefer by the way) you can pass the module to dir():
import foo
print dir(foo)
Also check the documentation of dir. As you only wanted functions, you might want to think about using inspect.isfunction. Hope you don't use that list for non-debugging purposes.

if wanting to list functions of the current module (i.e., not an imported one), you could also do something like this:
import sys
def func1(): pass
def func2(): pass
if __name__ == '__main__':
print dir(sys.modules[__name__])