I have a pytest file called test_util that looks like this:
import pytest
class TestUtil:
def test_split_kwargs(self, mocker):
def testfunction_extra(e='5',f='6'):
return e+f
mocker.patch(...)
I would like to use the assert_has_calls method on my testfunction_extra function, but I'm not sure what to put into my mocker.patch statement. I have already tried
mocker.patch(__name__ + '.TestUtil.test_split_kwargs.testfunction_extra')
but this returns the error AttributeError: <function TestUtil.test_split_kwargs at 0x7fce6612e790> does not have the attribute 'testfunction_extra'
You can't patch local variables, only global names and attributes of objects you have a reference to. But good news! You don't need to patch this, because you have a reference to the function right there to wrap in a Mock object yourself.
import unittest.mock
class TestUtil:
def test_split_kwargs(self):
def _(e='5', f='6'):
return e + f
testfunction_extra = unittest.mock.Mock(wraps=_)
...
You could even use a decorator for this, starting in Python 3.9:
class TestUtil:
def test_split_kwargs(self):
#lambda f: unittest.mock.Mock(wraps=f)
def testfunction_extra(e='5', f='6'):
return e + f
...
I'm trying to bypass importing from a module, so in my __init__.py I can inject code like this:
globals().update(
{
"foo": lambda: print("Hello stackoverflow!")
}
)
so if I do import mymodule I will be able to call mymodule.foo. That is a simple concept, useless for the purpose because you can actually just define foo.
So, the idea is to modify the globals module dictionary, so in case it doesn't find the function foo it will go wherever and I can inject the code, for that I tried:
from importer import load #a load function to search for the code
from functools import wraps
def global_get_wrapper(f):
#wraps(f)
def wrapper(*args):
module_name, default = args
res = f(*args)
if res is None:
return load(module_name)
return res
return wrapper
globals().get = global_get_wrapper(globals().get) # trying to substitute get method
But it gives me an error:
AttributeError: 'dict' object attribute 'get' is read-only
The other idea I had is to preload the available function, class, etc names into the module dictionary and lazily load them later.
I run out of ideas to accomplish this and I don't know if this is even possible.
Should I go for writing my own python importer? or is there any other possibility I could not think about?
Thanks in advance.
Instead of hacking globals() it would be better to define __getattr__ for your module as follows:
module_name.py
foo = 'foo'
def bar():
return 'bar'
my_module.py
import sys
import module_name
class MyModule(object):
def foobar(self):
return 'foobar'
def __getattr__(self, item):
return getattr(module_name, item)
sys.modules[__name__] = MyModule()
and then:
>>> import my_module
>>> my_module.foo
'foo'
>>> my_module.bar()
'bar'
>>> my_module.foobar()
'foobar'
PEP 562, which targets Python 3.7, introduces __getattr__ for modules. In the rationale it also describes workarounds for previous Python versions.
It is sometimes convenient to customize or otherwise have control over access to module attributes. A typical example is managing deprecation warnings. Typical workarounds are assigning __class__ of a module object to a custom subclass of types.ModuleType or replacing the sys.modules item with a custom wrapper instance. It would be convenient to simplify this procedure by recognizing __getattr__ defined directly in a module that would act like a normal __getattr__ method, except that it will be defined on module instances.
So your mymodule can look like:
foo = 'bar'
def __getattr__(name):
print('load you custom module and return it')
Here's how it behaves:
>>> import mymodule
>>> mymodule.foo
'bar'
>>> mymodule.baz
load you custom module and return it
I don't quite understand. Would this work for you?
try:
mymodule.foo()
except:
print("whatever you wanted to do")
Disclaimer:
Before reading this post know that I am trying to do something that is unconventional in python. Since "Don't do x" is not an answer to "how do I do x?" let's assume there is a very good reason to do this, even though in most cases it would not be good practice.
The Question
Given I have a class that is dynamically created by applying a decorator to a function, how would I go about pickling an instance of said class?
For example, to set this up it might look like this:
import inspect
from functools import wraps
class BaseClass:
pass
def _make_method(func):
""" decorator for adding self as first argument to function """
#wraps(func)
def decorator(self, *args, **kwargs):
return func(*args, **kwargs)
# set signature to include self
sig = inspect.signature(decorator)
par = inspect.Parameter('self', 1)
new_params = tuple([par] + list(sig.parameters.values()))
new_sig = sig.replace(parameters=new_params,
return_annotation=sig.return_annotation)
decorator.__signature__ = new_sig
return decorator
def snake2camel(snake_str):
""" convert a snake_string to a CamelString """
return "".join(x.title() for x in snake_str.split('_'))
def make_class(func):
""" dynamically create a class setting the call method to function """
name = snake2camel(func.__name__) # get the name of the new class
method = _make_method(func)
cls = type(name, (BaseClass,), {'__call__': method})
return cls()
#make_class
def something(arg):
return arg
Now something is an instance of the dynamically created class Something.
type(something) # -> __main__.Something
isinstance(something, BaseClass) # -> True
which works fine, but when I try to pickle it (or use the multiprocessing module which uses pickle under the hood):
import pickle
pickle.dumps(something) # -> raises
it throws this error:
# PicklingError: Can't pickle <class '__main__.Something'>: attribute lookup Something on __main__ failed
So I thought I could redefine BaseClass to use a reduce method like so:
class BaseClass:
def __reduce__(self):
return make_class, (self.__call__.__func__,)
but then it throws the dreaded "not the same object" error:
# PicklingError: Can't pickle <function something at 0x7fe124cb2d08>: it's not the same object as __main__.something
How can I make this work without bringing in dependencies? I need to be able to pickle the something object so I can use it with the ProcessPoolExecutor class from the concurrent.futures module in python 3.6, so simply using dill or cloudpickle is probably not an option here.
I am trying to Mock a function (that returns some external content) using the python mock module.
I'm having some trouble mocking functions that are imported into a module.
For example, in util.py I have
def get_content():
return "stuff"
I want to mock util.get_content so that it returns something else.
I am trying this:
util.get_content=Mock(return_value="mocked stuff")
If get_content gets invoked inside another module, it never actually seems to return the mocked object. Am I missing something in terms of how to use Mock?
Note that if I invoke the following, things work correctly:
>>> util.get_content=Mock(return_value="mocked stuff")
>>> util.get_content()
"mocked stuff"
However, if get_content is called from inside another module, it invokes the original function instead of the mocked version:
>>> from mymodule import MyObj
>>> util.get_content=Mock(return_value="mocked stuff")
>>> m=MyObj()
>>> m.func()
"stuff"
Contents of mymodule.py
from util import get_content
class MyObj:
def func():
get_content()
So I guess my question is - how do I get invoke the Mocked version of a function from inside a module that I call?
It appears that the from module import function may be to blame here, in that it doesn't point to the Mocked function.
The general case would be to use patch from mock. Consider the following:
utils.py
def get_content():
return 'stuff'
mymodule.py
from util import get_content
class MyClass(object):
def func(self):
return get_content()
test.py
import unittest
from mock import patch
from mymodule import MyClass
class Test(unittest.TestCase):
#patch('mymodule.get_content')
def test_func(self, get_content_mock):
get_content_mock.return_value = 'mocked stuff'
my_class = MyClass()
self.assertEqual(my_class.func(), 'mocked stuff')
self.assertEqual(get_content_mock.call_count, 1)
get_content_mock.assert_called_once()
Note how get_content is mocked, it is not util.get_content, rather mymodule.get_content since we are using it in mymodule.
Above has been tested with mock v2.0.0, nosetests v1.3.7 and python v2.7.9.
I think I have a workaround, though it's still not quite clear on how to solve the general case
In mymodule, if I replace
from util import get_content
class MyObj:
def func():
get_content()
with
import util
class MyObj:
def func():
util.get_content()
The Mock seems to get invoked. It looks like the namespaces need to match (which makes sense). However, the weird thing is that I would expect
import mymodule
mymodule.get_content = mock.Mock(return_value="mocked stuff")
to do the trick in the original case where I am using the from/import syntax (which now pulls in get_content into mymodule). But this still refers to the unmocked get_content.
Turns out the namespace matters - just need to keep that in mind when writing your code.
You have to patch the function where it is being used. In your case that would be in the mymodule module.
import mymodule
>>> mymodule.get_content = Mock(return_value="mocked stuff")
>>> m = mymodule.MyObj()
>>> m.func()
"mocked stuff"
There is a reference in the docs here: http://docs.python.org/dev/library/unittest.mock.html#where-to-patch
Let's assume you're creating your mock inside module foobar:
import util, mock
util.get_content = mock.Mock(return_value="mocked stuff")
If you import mymodule and call util.get_content without first importing foobar, your mock will not be installed:
import util
def func()
print util.get_content()
func()
"stuff"
Instead:
import util
import foobar # substitutes the mock
def func():
print util.get_content()
func()
"mocked stuff"
Note that foobar can be imported from anywhere (module A imports B which imports foobar) as long as foobar is evaluated before util.get_content is called.
While it doesn't provide an answer to your question directly, another possible alternative is to transform your function to a static method using the #staticmethod.
So you could transform your module utils into a class using something like:
class util(object):
#staticmethod
def get_content():
return "stuff"
Then mock patches it correctly.
How can implement the equivalent of a __getattr__ on a class, on a module?
Example
When calling a function that does not exist in a module's statically defined attributes, I wish to create an instance of a class in that module, and invoke the method on it with the same name as failed in the attribute lookup on the module.
class A(object):
def salutation(self, accusative):
print "hello", accusative
# note this function is intentionally on the module, and not the class above
def __getattr__(mod, name):
return getattr(A(), name)
if __name__ == "__main__":
# i hope here to have my __getattr__ function above invoked, since
# salutation does not exist in the current namespace
salutation("world")
Which gives:
matt#stanley:~/Desktop$ python getattrmod.py
Traceback (most recent call last):
File "getattrmod.py", line 9, in <module>
salutation("world")
NameError: name 'salutation' is not defined
There are two basic problems you are running into here:
__xxx__ methods are only looked up on the class
TypeError: can't set attributes of built-in/extension type 'module'
(1) means any solution would have to also keep track of which module was being examined, otherwise every module would then have the instance-substitution behavior; and (2) means that (1) isn't even possible... at least not directly.
Fortunately, sys.modules is not picky about what goes there so a wrapper will work, but only for module access (i.e. import somemodule; somemodule.salutation('world'); for same-module access you pretty much have to yank the methods from the substitution class and add them to globals() eiher with a custom method on the class (I like using .export()) or with a generic function (such as those already listed as answers). One thing to keep in mind: if the wrapper is creating a new instance each time, and the globals solution is not, you end up with subtly different behavior. Oh, and you don't get to use both at the same time -- it's one or the other.
Update
From Guido van Rossum:
There is actually a hack that is occasionally used and recommended: a
module can define a class with the desired functionality, and then at
the end, replace itself in sys.modules with an instance of that class
(or with the class, if you insist, but that's generally less useful).
E.g.:
# module foo.py
import sys
class Foo:
def funct1(self, <args>): <code>
def funct2(self, <args>): <code>
sys.modules[__name__] = Foo()
This works because the import machinery is actively enabling this
hack, and as its final step pulls the actual module out of
sys.modules, after loading it. (This is no accident. The hack was
proposed long ago and we decided we liked enough to support it in the
import machinery.)
So the established way to accomplish what you want is to create a single class in your module, and as the last act of the module replace sys.modules[__name__] with an instance of your class -- and now you can play with __getattr__/__setattr__/__getattribute__ as needed.
Note 1: If you use this functionality then anything else in the module, such as globals, other functions, etc., will be lost when the sys.modules assignment is made -- so make sure everything needed is inside the replacement class.
Note 2: To support from module import * you must have __all__ defined in the class; for example:
class Foo:
def funct1(self, <args>): <code>
def funct2(self, <args>): <code>
__all__ = list(set(vars().keys()) - {'__module__', '__qualname__'})
Depending on your Python version, there may be other names to omit from __all__. The set() can be omitted if Python 2 compatibility is not needed.
A while ago, Guido declared that all special method lookups on
new-style classes bypass __getattr__ and __getattribute__. Dunder methods had previously worked on modules - you could, for example, use a module as a context manager simply by defining __enter__ and __exit__, before those tricks broke.
Recently some historical features have made a comeback, the module __getattr__ among them, and so the existing hack (a module replacing itself with a class in sys.modules at import time) should be no longer necessary.
In Python 3.7+, you just use the one obvious way. To customize attribute access on a module, define a __getattr__ function at the module level which should accept one argument (name of attribute), and return the computed value or raise an AttributeError:
# my_module.py
def __getattr__(name: str) -> Any:
...
This will also allow hooks into "from" imports, i.e. you can return dynamically generated objects for statements such as from my_module import whatever.
On a related note, along with the module getattr you may also define a __dir__ function at module level to respond to dir(my_module). See PEP 562 for details.
This is a hack, but you can wrap the module with a class:
class Wrapper(object):
def __init__(self, wrapped):
self.wrapped = wrapped
def __getattr__(self, name):
# Perform custom logic here
try:
return getattr(self.wrapped, name)
except AttributeError:
return 'default' # Some sensible default
sys.modules[__name__] = Wrapper(sys.modules[__name__])
We don't usually do it that way.
What we do is this.
class A(object):
....
# The implicit global instance
a= A()
def salutation( *arg, **kw ):
a.salutation( *arg, **kw )
Why? So that the implicit global instance is visible.
For examples, look at the random module, which creates an implicit global instance to slightly simplify the use cases where you want a "simple" random number generator.
Similar to what #Håvard S proposed, in a case where I needed to implement some magic on a module (like __getattr__), I would define a new class that inherits from types.ModuleType and put that in sys.modules (probably replacing the module where my custom ModuleType was defined).
See the main __init__.py file of Werkzeug for a fairly robust implementation of this.
This is hackish, but...
# Python 2.7
import types
class A(object):
def salutation(self, accusative):
print("hello", accusative)
def farewell(self, greeting, accusative):
print(greeting, accusative)
def AddGlobalAttribute(classname, methodname):
print("Adding " + classname + "." + methodname + "()")
def genericFunction(*args):
return globals()[classname]().__getattribute__(methodname)(*args)
globals()[methodname] = genericFunction
# set up the global namespace
x = 0 # X and Y are here to add them implicitly to globals, so
y = 0 # globals does not change as we iterate over it.
toAdd = []
def isCallableMethod(classname, methodname):
someclass = globals()[classname]()
something = someclass.__getattribute__(methodname)
return callable(something)
for x in globals():
print("Looking at", x)
if isinstance(globals()[x], (types.ClassType, type)):
print("Found Class:", x)
for y in dir(globals()[x]):
if y.find("__") == -1: # hack to ignore default methods
if isCallableMethod(x,y):
if y not in globals(): # don't override existing global names
toAdd.append((x,y))
# Returns:
# ('Looking at', 'A')
# ('Found Class:', 'A')
# ('Looking at', 'toAdd')
# ('Looking at', '__builtins__')
# ('Looking at', 'AddGlobalAttribute')
# ('Looking at', 'register')
# ('Looking at', '__package__')
# ('Looking at', 'salutation')
# ('Looking at', 'farewell')
# ('Looking at', 'types')
# ('Looking at', 'x')
# ('Looking at', 'y')
# ('Looking at', '__name__')
# ('Looking at', 'isCallableMethod')
# ('Looking at', '__doc__')
# ('Looking at', 'codecs')
for x in toAdd:
AddGlobalAttribute(*x)
if __name__ == "__main__":
salutation("world")
farewell("goodbye", "world")
# Returns:
# hello world
# goodbye world
This works by iterating over the all the objects in the global namespace. If the item is a class, it iterates over the class attributes. If the attribute is callable it adds it to the global namespace as a function.
It ignore all attributes which contain "__".
I wouldn't use this in production code, but it should get you started.
Here's my own humble contribution -- a slight embellishment of #Håvard S's highly rated answer, but a bit more explicit (so it might be acceptable to #S.Lott, even though probably not good enough for the OP):
import sys
class A(object):
def salutation(self, accusative):
print "hello", accusative
class Wrapper(object):
def __init__(self, wrapped):
self.wrapped = wrapped
def __getattr__(self, name):
try:
return getattr(self.wrapped, name)
except AttributeError:
return getattr(A(), name)
_globals = sys.modules[__name__] = Wrapper(sys.modules[__name__])
if __name__ == "__main__":
_globals.salutation("world")
Create your module file that has your classes. Import the module. Run getattr on the module you just imported. You can do a dynamic import using __import__ and pull the module from sys.modules.
Here's your module some_module.py:
class Foo(object):
pass
class Bar(object):
pass
And in another module:
import some_module
Foo = getattr(some_module, 'Foo')
Doing this dynamically:
import sys
__import__('some_module')
mod = sys.modules['some_module']
Foo = getattr(mod, 'Foo')