We Keep Coding

Python monkey patching: instance creation in method of library/object

what is the easiest way to solve the following problem in extending/altering the functionality of a third party library?
The library offers a class LibraryClass with a function func_to_be_changed. This function has a local variable internal_variable which is the instance of another class SimpleCalculation of that library. I created a new class BetterCalculation in my own module and now want LibraryClass.func_to_be_changed to use an instance of this new class.
# third party library
from third_party_library.utils import SimpleCalculation
class LibraryClass:
def func_to_be_changed(self, x):
# many complicated things go on
internal_variable = SimpleCalculation(x)
# many more complicated things go on
The easiest solution would be, to just copy the code from the third party library, subclass the LibraryClass and overwrite the function func_to_be_changed:
# my module
from third_party_library import LibraryClass
class BetterLibraryClass(LibraryClass):
def func_to_be_changed(self, x):
"""This is an exact copy of LibraryClass.func_to_be_changed."""
# many complicated things go on
internal_variable = BetterCalculation(x) # Attention: this line has been changed!!!
# many more complicated things go on
However, this involves copying of many lines of code. When a new version of the third party class improves on code that was copied without modification, this modifications need to be incorporated manually by another copying step.
I tried to use unittest.mock.patch as I know that the following two snippets work:
# some script
from unittest.mock import patch
import third_party_library
from my_module import BetterCalculation
with patch('third_party_library.utils.SimpleCalculation', BetterCalculation):
local_ = third_party_library.utils.SimpleCalculation(x) # indeed uses `BetterCalculation`
def foo(x):
return third_party_library.utils.SimpleCalculation(x)
with patch('third_party_library.utils.SimpleCalculation', BetterCalculation):
local_ = foo(x) # indeed uses `BetterCalculation`
However, the following does not work:
# some script
from unittest.mock import patch
from third_party_library.utils import SimpleCalculation
from my_module import BetterCalculation
def foo(x):
return SimpleCalculation(x)
with patch('third_party_library.utils.SimpleCalculation', BetterCalculation):
local_ = foo(x) # does not use `BetterCalculation`
# this works again
with patch('__main__.SimpleCalculation', BetterCalculation):
local_ = foo(x) # indeed uses `BetterCalculation`
Therefore, the following won`t work either:
# my module
from unittest.mock import patch
from third_party_library import LibraryClass
from my_module import BetterCalculation
class BetterLibraryClass(LibraryClass):
def func_to_be_changed(self, x):
with patch(
'third_party_library.utils.SimpleCalculation',
BetterCalculation
):
super().func_to_be_changed(x)
Is there an elegant pythonic way to do this? I guess this boils down to the question: What is the equivaltent of __main__ in the last code snippet that needs to replace third_party_library.utils?

Some context
The first string argument in the patch function can have two different meanings depending on the situation. In the first situation the described object has not been imported and is unavailable to the program which would, therefore, result in a NameError without the mocking. However, in the question, an object needs to be overwritten. Therefore, it is available to the program and not using patch would not result in an error.
Disclaimer
I might have used the complete wrong language in here, as for sure there are precise python terms for all the described notions.
Overwriting an object
As shown in the question, the locally imported SimpleCalculation can be overwritten with __main__.SimpleCalculation. Therefore, it is important to remember that you need to tell patch the path to the local object and not how that same object would be imported in the current script.
Let's assume the following module:
# thirdpartymodule/__init__.py
from .utils import foo
def local_foo():
print("Hello local!")
class Bar:
def __init__(self):
foo()
local_foo()
and
# thirdpartymodule/utils.py
def foo():
print("third party module")
We want to override the functions foo and local_foo. But we don't want to override any functions, we want to override the functions foo and local_foo in the context of the file thirdpartymodule/__init__.py. It is unimportant that the function foo enters the context of the file via an import statement, while local_foo is defined locally. So we want to override the functions in the context of thirdpartymodule.foo and thirdpartymodule.local_foo. The context thirdpartymodule.utils.foo is not important here and won't help us. The following snippet illustrates that:
from unittest.mock import patch
from thirdpartymodule import Bar
bar = Bar()
# third party module
# Hello local!
def myfoo():
print("patched function")
with patch("thirdpartymodule.foo", myfoo):
bar = Bar()
# patched function
# Hello local!
# will not work!
with patch("thirdpartymodule.utils.foo", myfoo):
bar = Bar()
# third party module
# Hello local!
with patch("thirdpartymodule.local_foo", myfoo):
bar = Bar()
# third party module
# patched function
In the hypothetical module of the question we first need to assume that the class LibraryClass is defined in the file third_party_library/library_class.py. Then, we want to override third_party_library.library_class.SimpleCalculation and the correct patch would be:
# my module
from unittest.mock import patch
from third_party_library import LibraryClass
from my_module import BetterCalculation
class BetterLibraryClass(LibraryClass):
def func_to_be_changed(self, x):
with patch(
'third_party_library.library_class.SimpleCalculation',
BetterCalculation
):
super().func_to_be_changed(x)

How to trigger a late import of another module only when a certain object is needed from my module?

The Situation
I want to have a module that roughly works like the following:
# my_module.py
my_number = 17
from other_module import foo
my_object = foo(23)
However, there is a problem: Installing other_module causes problems for some users and is only required for those who want to use my_object – which in turn is only a small fraction of users. I want to spare those users who do not need my_object from installing other_module.
I therefore want the import of other_module to happen only if my_object is imported from my_module. With other words, the user should be able to run the following without having installed other_module:
from my_module import my_number
My best solution so far
I could provide my_object via a function that contains the import:
# in my_module.py
def get_my_object():
from other_module import foo
my_object = foo(23)
return my_object
The user would then have to do something like:
from my_module import get_my_object
my_object = get_my_object()
Question
Is there a better way to conditionally trigger the import of other_module? I am mostly interested in keeping things as simple as possible for the users.

I would prefer the get_my_object() approach, but as of Python 3.7, what you ask is possible by defining a module-level __getattr__ function:
# my_module.py
my_number = 17
def __getattr__(name):
if name != 'my_object':
raise AttributeError
global my_object
from other_module import foo
my_object = foo(23)
return my_object
This will attempt to import other_module and call foo only once my_object is accessed. A few caveats:
It will not trigger for attempts to access my_object by global variable lookup within my_module. It will only trigger on my_module.my_object attribute access, or a from my_module import my_object import (which performs attribute access under the hood).
If you forget to assign to the global my_object name in __getattr__, my_object will be recomputed on every access.
Module-level __getattr__ does nothing before Python 3.7, so you may want to perform a version check and do something else for Python 3.6 and below:
import sys
if sys.version_info >= (3, 7):
def __getattr__(name):
...
else:
# Do something appropriate. Maybe raise an error. Maybe unconditionally
# import other_module and compute my_object up front.

Approach A – clean solution
Create a new separate module and have the user import the object from the other module. For example
from my_module import my_number # regular use
from my_module.extras import my_object # for a small part of the user base
This means in your code you create a module folder my_module with an __init__.py where you import the usual stuff and don't import the extras submodule.
If you don't want to put extras in my_module (simpler), just create my_object in an individual extras.py module.
Approach B – signals bad architecture 99% of times
You can use importlib.import_module to dynamically import a module inside get_my_object without polluting the global space and that is cleaner than an import inside a function which creates side effects such as overriding your global variable with that import name (see this question and answers), however this is usually a sign of bad coding patterns on other part of the code.
Approach C – simple and effective
I usually tend to favour this simple pattern when there are users that might not have a library, as Python 3 discourages imports that are not at top level:
try:
import other_module
_HAS_OTHER_MODULE_ = True
except:
_HAS_OTHER_MODULE_ = False
def get_my_object():
assert _HAS_OTHER_MODULE_, "Please install other module"
return other_module.foo(23)

This is a common hack to be done:
import sys
class MockModule:
def __init__(self, module):
self.module = module
def __getattr__(self, attr):
if attr == 'dependency_required_var':
try:
import foo
return self.module.dependency_required_var
except ImportError:
raise Exception('foo library is required to use dependency_required_var')
else:
return getattr(self.module, attr)
MockModule.__name__ = __name__
sys.modules[__name__] = MockModule(sys.modules[__name__])
dependency_required_var = 0
With this PEP, we can simply do (we should be able to but I couldn't get it to work) the following in Python 3.7 and higher:
def __getattr__(attr):
if attr == 'dependency_required_var':
try:
import foo
return dependency_required_var
except ImportError:
raise Exception('foo library is required to use dependency_required_var')
else:
return globals()[attr]
The PEP seems to be accepted, but the relevant pull request from the PEP seems to be closed, I'm actually not sure if it has been implemented or not.

Python - Importing module gets global variables

I have two Python scripts, one testclass.py:
import numpy
zz = numpy
class Something(object):
def __init__(self):
self.xp = zz
and one testscript.py:
from testclass import Something
x = Something()
print(x.xp)
I expected testscript.py to throw an error because I thought that testscript only imports the class Something (with its __init__ method), and not the global variable zz. So, given this bevahiour, my question is, when importing from a module, does Python "run" everything in the module file?

Yes. When you execute:
from testclass import Something
It has the same effect as:
import testclass
Something = testclass.Something
More generally, the Python interpreter can't know beforehand what objects your module exposes (unless you explicitly name them in __all__). For an extreme case, consider the following:
a.py:
import random
if random.random() > 0.5:
class Foo(object):
pass
else:
class Bar(object):
pass
Running from a import Foo has a 50% chance of failing because the a module object may or may not have a Foo attribute.

Python using function like variable

I've got a Python module which has several variables with hard-coded values which are used throughout the project. I'd like to bind the variables somehow to a function to redirect to a config file. Is this possible?
# hardcoded_values.py
foo = 'abc'
bar = 1234
# usage somewhere else in another module
from hardcoded_values import *
print foo
print bar
What I want to do is change only hardcoded_values.py, so that print foo transparently calls a function.
# hardcoded_values.py
import config
foo = SomeWrapper(config.get_value, 'foo') # or whatever you can think of to call config.get_value('foo')
...
config.get_value would be a function that is called with parameter 'foo' when using variable foo (as in print foo).

I'm pretty sure that you can't do what you want to do if you import like from hardcoded_values import *.
What you want to do is to set foo to some function, and then apply the property decorator (or equivalent) so that you can call foo as foo rather than foo(). You cannot apply the property decorator to modules for reasons detailed here: Why Is The property Decorator Only Defined For Classes?
Now, if you were to import hardcoded_values then I think there is a way to do what you want to hardcoded_values.foo. I have a pretty good feeling that what I am about to describe is a BAD IDEA that should never be used, but I think it is interesting.
BAD IDEA???
So say you wanted to replace a constant like os.EX_USAGE which on my system is 64 with some function, and then call it as os.EX_USAGE rather than os.EX_USAGE(). We need to be able to use the property decorator, but for that we need a type other than module.
So what can be done is to create a new type on the fly and dump in the __dict__ of a module with a type factory function that takes a module as an argument:
def module_class_factory(module):
ModuleClass = type('ModuleClass' + module.__name__,
(object,), module.__dict__)
return ModuleClass
Now I will import os:
>>> import os
>>> os.EX_USAGE
64
>>> os.getcwd()
'/Users/Eric'
Now I will make a class OsClass, and bind the name os to an instance of this class:
>>> OsClass = module_class_factory(os)
>>> os = OsClass()
>>> os.EX_USAGE
64
>>> os.getcwd()
'/Users/Eric'
Everything still seems to work. Now define a function to replace os.EX_USAGE, noting that it will need to take a dummy self argument:
>>> def foo(self):
... return 42
...
...and bind the class attribute OsClass.EX_USAGE to the function:
>>> OsClass.EX_USAGE = foo
>>> os.EX_USAGE()
42
It works when called by the os object! Now just apply the property decorator:
>>> OsClass.EX_USAGE = property(OsClass.EX_USAGE)
>>> os.EX_USAGE
42
now the constant defined in the module has been transparently replaced by a function call.

You definitely cannot do this if the client code of your module uses from hardcoded_variables import *. That makes references to the contents of hardcoded_variables in the other module's namespace, and you can't do anything with them after that.
If the client code can be changed to just import the module (with e.g. import hardcoded_variables) and then access its attributes (with hardcoded_variables.foo) you do have a chance, but it's a bit awkward.
Python caches modules that have been imported in sys.modules (which is a dictionary). You can replace a module in that dictionary with some other object, such as an instance of a custom class, and use property objects or other descriptors to implement special behavior when you access the object's attributes.
Try making your new hardcoded_variables.py look like this (and consider renaming it, too!):
import sys
class DummyModule(object):
def __init__(self):
self._bar = 1233
#property
def foo(self):
print("foo called!")
return "abc"
#property
def bar(self):
self._bar += 1
return self._bar
if __name__ != "__main__": # Note, this is the opposite of the usual boilerplate
sys.modules[__name__] = DummyModule()

If I understand correctly, you want your hardcoded_variables module evaluated every time you try to access a variable.
I would probably have hardcoded_variables in a document (e.g. json?) and a custom wrapper function like:
import json
def getSettings(var):
with open('path/to/variables.json') as infl:
data = json.load(infl)
return infl[var]

Can I "fake" a package (or at least a module) in python for testing purposes?

I want to fake a package in python. I want to define something so that the code can do
from somefakepackage.morefakestuff import somethingfake
And somefakepackage is defined in code and so is everything below it. Is that possible? The reason for doing this is to trick my unittest that I got a package ( or as I said in the title, a module ) in the python path which actually is just something mocked up for this unittest.

Sure. Define a class, put the stuff you need inside that, assign the class to sys.modules["classname"].
class fakemodule(object):
#staticmethod
def method(a, b):
return a+b
import sys
sys.modules["package.module"] = fakemodule
You could also use a separate module (call it fakemodule.py):
import fakemodule, sys
sys.modules["package.module"] = fakemodule

Yes, you can make a fake module:
from types import ModuleType
m = ModuleType("fake_module")
import sys
sys.modules[m.__name__] = m
# some scripts may expect a file
# even though this file doesn't exist,
# it may be used by Python for in error messages or introspection.
m.__file__ = m.__name__ + ".py"
# Add a function
def my_function():
return 10
m.my_function = my_function
Note, in this example its using an actual module (of ModuleType) since some
Python code may expect modules, (instead of a dummy class).
This can be made into a utility function:
def new_module(name, doc=None):
import sys
from types import ModuleType
m = ModuleType(name, doc)
m.__file__ = name + '.py'
sys.modules[name] = m
return m
print(new_module("fake_module", doc="doc string"))
Now other scripts can run:
import fake_module

I took some of the ideas from the other answers and turned them into a Python decorator #modulize which converts a function into a module. This module can then be imported as usual. Here is an example.
#modulize('my_module')
def my_dummy_function(__name__): # the function takes one parameter __name__
# put module code here
def my_function(s):
print(s, 'bar')
# the function must return locals()
return locals()
# import the module as usual
from my_module import my_function
my_function('foo') # foo bar
The code for the decorator is as follows
import sys
from types import ModuleType
class MockModule(ModuleType):
def __init__(self, module_name, module_doc=None):
ModuleType.__init__(self, module_name, module_doc)
if '.' in module_name:
package, module = module_name.rsplit('.', 1)
get_mock_module(package).__path__ = []
setattr(get_mock_module(package), module, self)
def _initialize_(self, module_code):
self.__dict__.update(module_code(self.__name__))
self.__doc__ = module_code.__doc__
def get_mock_module(module_name):
if module_name not in sys.modules:
sys.modules[module_name] = MockModule(module_name)
return sys.modules[module_name]
def modulize(module_name, dependencies=[]):
for d in dependencies: get_mock_module(d)
return get_mock_module(module_name)._initialize_
The project can be found here on GitHub. In particular, I created this for programming contests which only allow the contestant to submit a single .py file. This allows one to develop a project with multiple .py files and then combine them into one .py file at the end.

You could fake it with a class which behaves like somethingfake:
try:
from somefakepackage.morefakestuff import somethingfake
except ImportError:
class somethingfake(object):
# define what you'd expect of somethingfake, e.g.:
#staticmethod
def somefunc():
...
somefield = ...

TL;DR
Patch sys.modules using unittest.mock:
mock.patch.dict(
sys.modules,
{'somefakepackage': mock.Mock()},
)
Explanation
Other answers correctly recommend to fix sys.modules but a proper way to do it is by patching it using mock.patch. Meaning replacing it temporarily (only for when tests are run) with a fake object that optionally imitates the desired behaviour. And restoring it back once tests are finished to not affect other test cases.
The code in TL;DR section will simply make your missing package not raise ImportError. To provide fake package with contents and imitate desired behaviour, initiate mock.Mock(…) with proper arguments (e.g. add attributes via Mock's **kwargs).
Full code example
The code below temporarily patches sys.modules so that it includes somefakepackage and makes it importable from the dependent modules without ImportError.
import sys
import unittest
from unittest import mock
class SomeTestCase(unittest.TestCase):
def test_smth(self):
# implement your testing logic, for example:
self.assertEqual(
123,
somefakepackage_dependent.some_func(),
)
#classmethod
def setUpClass(cls): # called once before all the tests
# define what to patch sys.modules with
cls._modules_patcher = mock.patch.dict(
sys.modules,
{'somefakepackage': mock.Mock()},
)
# actually patch it
cls._modules_patcher.start()
# make the package globally visible and import it,
# just like if you have imported it in a usual way
# placing import statement at the top of the file,
# but relying on a patched dependency
global somefakepackage_dependent
import somefakepackage_dependent
#classmethod # called once after all tests
def tearDownClass(cls):
# restore initial sys.modules state back
cls._modules_patcher.stop()
To read more about setUpClass/tearDownClass methods, see unittest docs.
unittest's built-in mock subpackage is actually a very powerful tool. Better dive deeper into its documentation to get a better understanding.