We Keep Coding

Is there a way to monkey patch a class inside a module before the module body is executed in python?

I have file main.py which contains the following code:
class A:
def __init__(self, a):
self.a = a
def run(self):
return self.a+10
a = A(4)
print(a.run())
In file test.py, I tried to monkey patch class A in main.py as follows:
import main
class A:
def __init__(self, a):
self.a = a
def run(self):
return self.a+5
main.A = A
Unfortunately, when I run import test from a python interpreter, the module still prints out 14 as opposed to my expected output which is 9.
Is there a way to monkey patch a class inside a module before the module body is executed?

The problem here is that when you imported the main.py file, it executed the code a = A(4) using the real implementation of the class A. Then the rest of your test.py was executed and you replaced the A reference, but it was too late.
You can check that by adding in your test :
print(__name__) # __main__
print(main.__name__) # so70731368_main
print(A.__module__) # __main__
print(main.A.__module__) # __main__
print(main.a.__class__.__module__) # so70731368_main
Here, __main__ is a bit confusing but that's how Python call the first file you run (in your case test.py). The a instance is declared in the so70731368_main module, and it used the A class from the same module, you just changed A after the fact with the definition from the test file (__main__).
The fact that you need to patch two definitions (A and a) defined in the same file is very tricky. unittest.mock.patch is not powerful enough to patch inside an import (it patches after the import).
You can not, in a clean and simple way, prevent a to be instantiated as a main.A (real) class and get printed. What you can do is patch it after, for later uses, that is what you showed.
To answer directly your question : "patching" means replacing one reference by another, so the reference has to already be defined. In your example, it would require to patch between the class definition and the class instantiation (for the print to not use the real a), which is not supported.
There is no simple solution to this problem. If you have control over the code of the main.py file, then try to change it so that it does not instantiate a at import time.

Inherit from a class not imported yet?

I have a modules.py file defining several base classes.
One of these base classe inherit itself from a class defined in another module:
class MyClass(torch.nn.Module):
...
However, I'd prefer not to import torch in that modules.py file yet, because it's quite a big library, and this modules.py file is used by multiple processes, which would mean all of these processes would also have to import torch, leading quickly to an overflow (been there, done that).
Is there a way to define MyClass and specify it's based on torch.nn.Module without having to import torch just yet, and let the processes who really need the torch module do the import themselves ? So the other processes not needing MyClass could just ignore it's there and not try to resolve it, for instance ?

There is a rather advanced way to simulate dynamic import by building a dummy class and importing the real module and overwriting its attributes on first call of its __new__ special method.
Demo:
module A.py:
class Attempt:
_changed = False
def __new__(cls, i):
if not cls._changed:
cls._changed = True
import B
for name, member in B.B.__dict__.items():
if name not in ('__dict__', '__module__'):
setattr(cls, name, member)
return object.__new__(cls)
module B.py:
print('importing module B')
class B:
a = 5
def __init__(self, i):
self.i = i
def __repr__(self):
return f'B({self.i})'
You can then use it that way:
>>> import A
>>> a = A.Attempt(2)
importing module B
>>> b = A.Attempt(3)
>>> print(a,b)
B(2) B(3)
Which proves that the module B is only loaded at creation of the first object.

Python 2.7 mock a void method of a class

I am trying to use unittests.mock to mock a void method call of an object.
My package is like below
common
baseupgradehandler.py
baseupgradehandler.py
class BaseUpgradeHandler(object):
def __init__(self, upgrade_config, upgrade_state, system_config, pre_step, main_step, post_step):
...
# Method call to be supressed
def start(self, service_manifest, upgrade_bundle):
# type: (service_version_pb2.ServiceManifest, str) -> ()
...
In my test code I am trying to mock the call to start() like below as explained in the documentation.
from workflow.upgradeworkflow import UpgradeWorkflow
from common.serviceregistry import ServiceRegistry
# The above imports are at the start of the test file
...
with patch('common.baseupgradehandler.BaseUpgradeHandler') as handler_mock: # type: Mock
handler_mock.return_value.start.return_value = ''
wf = UpgradeWorkflow(ServiceRegistry(self.service_bundle, config, sys_config, state),
config,
state,
sys_config)
BaseUpgradeHandler object is returned by get_upgrade_handler() method of ServiceRegistry. When I am executing the above code in test I am seeing the BaseUpgradeHandler.start() is still getting called.
Can someone let me know how can I mock the call to a start() so that the method is not called?
EDIT
If I change my patching code like below it is working as expected and BaseUpgradeHandler is getting mocked and start is not getting called.
with patch('common.baseupgradehandler.BaseUpgradeHandler') as handler_mock: # type: Mock
handler_mock.return_value.start.return_value = ''
with patch('common.serviceregistry.ServiceRegistry') as serviceregistry_mock: # type: Mock
serviceregistry_mock.return_value.get_upgrade_handler.return_value = handler_mock
wf = UpgradeWorkflow(ServiceRegistry(self.service_bundle, config, sys_config, state), config, state, sys_config)
wf.start()
Can someone explain me why do I have to patch ServiceRegistry as well?

The code you provided is not enough to see the part that causes the issue. We'd need to see the module serviceregistry to be sure but I'd take an educated guess:
You have a file a.py (aka baseupgradehandler) like this:
class A:
def method(self):
print("It's real!")
And a file b.py (aka serviceregistry) like this:
from a import A
class B:
def get_A(self):
return A()
In your test files you do this:
import unittest
from unittest.mock import patch
from b import B
from a import A
GAME OVER!
The B module right now has already got its reference to the original A class. When, afterwards, you patch('a.A') only the reference in the a module is changed, but patch has no way to know that B has its own reference to the original A.
You can fix this in three ways:
patch the method: this will modify the existing class so all references to that class will be automatically patched
patch b.A too:
with patch('a.A') as h_a, patch('b.A') as h_b:
h_a.return_value.method.return_value = ''
h_b.return_value.method.return_value = ''
Avoid importing the modules before patching (probably not feasible or a good idea):
import unittest
from unittest.mock import patch
class MyTest(unittest.TestCase):
def test_one(self):
with patch('a.A') as h:
h.return_value.method.return_value = ''
from b import B
B().get_A().method()

I have been using unittest.mocks for a while, and I have been re-inventing the wheel sometimes. I decided to make mockito part of my project and now things look way better. Any kind of mock verification is really simple, if you can, I definitively encourage you to make mockito part of your libraries. This library has a good documentation and so far it has been easier than unittest.mock IMHO.

Best way to import modules inside a class?

I have a Python module file which is actually a class with its associated public and private functions. I want to use the logging module inside the class and inside all the functions. But the issue is if I import only once inside __init__() the import is not visible in the rest of the functions.Do I have to import it in every function of the class?
What is the best way to import such modules inside a class. I searched various discussions about the same in stackoveflow forums but could not come with something concrete.
Below is my sample class:
class Add:
def __init__(self,no1,no2):
import logging
logging.basicConfig(filename='example2.log',level=logging.INFO)
logging.info("The two parameters passed are {0} and {1}:".format(no1,no2))
self.num1 = no1
self.num2 = no2
def add(self):
logging.debug("Inside add function of Add class")
logging.info("Adding the two class properties")
self.result = self.num1 + self.num2
logging.debug("Exiting add function")
def display(self):
logging.debug("Inside display class")
return self.result
if __name__ == '__main__':
a = Add(10,11)
a.add()
print (a.display)
When I try to run this:
C:\Users\bhatsubh\Desktop\Everything\Codes\Python>python Test.py
Traceback (most recent call last):
File "Test.py", line 21, in <module>
a.add()
File "Test.py", line 10, in add
logging.debug("Inside add function of Add class")
NameError: name 'logging' is not defined

So the problem here is one of scope. When you import inside a function (which occasionaly, see below) then the scope of that import is limited to inside that function:
def foo():
import a
a.something()
def bar():
a.something() # won't work
instead you should try to always import at the top:
import a
def foo():
a.something()
def bar():
a.something()
you should also come up with a good, maintainable way of ordering your imports.
Now, you can do it inside a function, and you can do funky things like dynamic imports and reflection and all kinds of cool stuff, but 99.9% of the time you don't need to, so adding that kind of complexity detracts from maintainability (which should be your goal).
The good times you do want to do it is to resolve complex cyclic imports, or if you're trying to do something funky with dynamic loading for performance. But when it comes to optimising performance, you should always build it first then see where it is breaking.

You could simply assign it to a variable to it:
class Foo:
def __init__(self):
import logging
self._logging = logging
But I would advise to just import it on the module level. It feels very wrong to attach a module to a member variable of a class.
Another option would be to only store the Logger you are going to use:
class Foo:
def __init__(self):
import logging
self.logger = logging.getLogger(__name__ + "." + self.__class__.__name__)
PS: This is probably not the proper way to create a logger for a class, I've never done it and only improvised it here. Read up on how to do it.

Inheriting math methods into a class with only static methods

I need to define a class which extends python's standard math module, without instantiating it (no need for that, all methods in the class are static):
import math
class more_math(math):
#staticmethod
def add_func(x):
return math.sqrt(x)+1
The code above doesn't run properly (script exits), with the error:
TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)
When the class declaration above is set to class more_math:, more_math.add_func(x) is called without error. However, more_math.sqrt(x) [sqrt is a method of math] can't be called, as more_math doesn't have math as its base class.
Ideas on how could this be setup properly?

Think long and hard about whether you actually need to provide the functions math implements. You almost certainly don't; you probably just need to provide your extras. That said, if you do need to provide a more_math module that implements all the standard math functions too, the easiest way to do this is to do a from math import *. This will bring every function the math module defines into your module. This is considered very bad practice, because it pollutes the namespace of your module and can make it difficult to tell what is actually being used. However, in this case, pollution of your module's namespace is exactly what you want.

As #user2357112 commented math is a module and not a class. You can create a more_math module simply by creating a more_math.py file with:
from math import *
def add_func(x):
return sqrt(x)+1
this module can be imported with import more_math or from more_math import add_func.

math isn't a class, it's an instance of class types.ModuleType. You can verify this with isinstance(math, types.ModuleType) which will return True. Normally you can't define a subclass that inherits from an instance of another class. However, it is possible with a bit of hackery.
(I got the idea from a recipe for inheriting from an instances on the ActiveState website.)
Since it is a hack, one might not want to use it in production code. However I thought you (and other readers) might find it a least interesting, if not useful.
Script more_math.py:
from copy import deepcopy
import math
import sys
def class_from_instance(instance):
copy = deepcopy(instance.__dict__)
def __init__(self, *args, **kwargs):
super(InstanceFactory, self).__init__(*args, **kwargs)
self.__dict__.update(copy)
InstanceFactory = type('InstanceFactory',
(instance.__class__,),
{'__init__': __init__})
return InstanceFactory
class MoreMathModule(class_from_instance(math)):
#staticmethod
def added_func(x):
return math.sqrt(x)+1
# Replace this module with an instance of the class above.
ref, sys.modules[__name__] = sys.modules[__name__], MoreMathModule('more_math')
if __name__ == '__main__':
import more_math
x = 42
print('more_math.sqrt({}) -> {:.6f}'.format(x, more_math.sqrt(x)))
print('more_math.added_func({}) -> {:.6f}'.format(x, more_math.added_func(x)))
Output:
more_math.sqrt(42) -> 6.480741
more_math.added_func(42) -> 7.480741