I have a modules.py file defining several base classes.
One of these base classe inherit itself from a class defined in another module:
class MyClass(torch.nn.Module):
...
However, I'd prefer not to import torch in that modules.py file yet, because it's quite a big library, and this modules.py file is used by multiple processes, which would mean all of these processes would also have to import torch, leading quickly to an overflow (been there, done that).
Is there a way to define MyClass and specify it's based on torch.nn.Module without having to import torch just yet, and let the processes who really need the torch module do the import themselves ? So the other processes not needing MyClass could just ignore it's there and not try to resolve it, for instance ?
There is a rather advanced way to simulate dynamic import by building a dummy class and importing the real module and overwriting its attributes on first call of its __new__ special method.
Demo:
module A.py:
class Attempt:
_changed = False
def __new__(cls, i):
if not cls._changed:
cls._changed = True
import B
for name, member in B.B.__dict__.items():
if name not in ('__dict__', '__module__'):
setattr(cls, name, member)
return object.__new__(cls)
module B.py:
print('importing module B')
class B:
a = 5
def __init__(self, i):
self.i = i
def __repr__(self):
return f'B({self.i})'
You can then use it that way:
>>> import A
>>> a = A.Attempt(2)
importing module B
>>> b = A.Attempt(3)
>>> print(a,b)
B(2) B(3)
Which proves that the module B is only loaded at creation of the first object.
Related
I have file main.py which contains the following code:
class A:
def __init__(self, a):
self.a = a
def run(self):
return self.a+10
a = A(4)
print(a.run())
In file test.py, I tried to monkey patch class A in main.py as follows:
import main
class A:
def __init__(self, a):
self.a = a
def run(self):
return self.a+5
main.A = A
Unfortunately, when I run import test from a python interpreter, the module still prints out 14 as opposed to my expected output which is 9.
Is there a way to monkey patch a class inside a module before the module body is executed?
The problem here is that when you imported the main.py file, it executed the code a = A(4) using the real implementation of the class A. Then the rest of your test.py was executed and you replaced the A reference, but it was too late.
You can check that by adding in your test :
print(__name__) # __main__
print(main.__name__) # so70731368_main
print(A.__module__) # __main__
print(main.A.__module__) # __main__
print(main.a.__class__.__module__) # so70731368_main
Here, __main__ is a bit confusing but that's how Python call the first file you run (in your case test.py). The a instance is declared in the so70731368_main module, and it used the A class from the same module, you just changed A after the fact with the definition from the test file (__main__).
The fact that you need to patch two definitions (A and a) defined in the same file is very tricky. unittest.mock.patch is not powerful enough to patch inside an import (it patches after the import).
You can not, in a clean and simple way, prevent a to be instantiated as a main.A (real) class and get printed. What you can do is patch it after, for later uses, that is what you showed.
To answer directly your question : "patching" means replacing one reference by another, so the reference has to already be defined. In your example, it would require to patch between the class definition and the class instantiation (for the print to not use the real a), which is not supported.
There is no simple solution to this problem. If you have control over the code of the main.py file, then try to change it so that it does not instantiate a at import time.
Using the OOP 'State' pattern in python leaded me to this dependency problem:
StateA, StateBand StateC are 3 states implementing the same method event1.
StateB inherits its behavior from StateA.
file a.py:
#from b import StateB
from c import StateC
class StateA(object):
def event1(self):
return StateC()
print type(StateA().event1())
file b.py:
import a
class StateB(a.StateA):
def event1(self):
return self
file c.py:
class StateC(object):
def event1(self):
return self
As long as I don't need StateB in a.py, this works. But what if I want to use type StateB in StateA?
Importing StateB (see the first commented line in a.py) leads to this cyclic dependency error:
ImportError: cannot import name StateB
Circle dependencies is a problem connected with code design. In practice, you could meet that probably only in badly organized code. Try to refactor your code to resolve this issue.
Another approach (which I not suggest you to do, just FYI) is an import outside of module-level import, but you should avoid such an approach.
Since your classes are tightly coupled, I would just put them into the same module:
file my_state.py:
class StateA(object):
def event1(self):
return StateC()
class StateB(StateA):
def event1(self):
return self
class StateC(object):
def event1(self):
return self
print(type(StateA().event1()))
I have two Python scripts, one testclass.py:
import numpy
zz = numpy
class Something(object):
def __init__(self):
self.xp = zz
and one testscript.py:
from testclass import Something
x = Something()
print(x.xp)
I expected testscript.py to throw an error because I thought that testscript only imports the class Something (with its __init__ method), and not the global variable zz. So, given this bevahiour, my question is, when importing from a module, does Python "run" everything in the module file?
Yes. When you execute:
from testclass import Something
It has the same effect as:
import testclass
Something = testclass.Something
More generally, the Python interpreter can't know beforehand what objects your module exposes (unless you explicitly name them in __all__). For an extreme case, consider the following:
a.py:
import random
if random.random() > 0.5:
class Foo(object):
pass
else:
class Bar(object):
pass
Running from a import Foo has a 50% chance of failing because the a module object may or may not have a Foo attribute.
I suspect this is one of those questions that will result in "Why do you want to do that?" but here goes:
I need to find a way to fake the module of a class defined in __main__; ie:
Lets say we have a.py:
class A(object):
name = "a"
pass
And b.py:
import inspect
import a
class B(object):
__name__ = a.__name__
name = "b"
pass
print inspect.getmodule(B)
Calling b.py results in:
<module '__main__' from 'b.py'>
So, how do I get the module of class A taken to be b, without moving it into b ?
You can set the class's __module__ attribute, but there are caveats:
class A(object):
__module__ = 'b'
The main caveat is that this will cause getmodule to return None if the module b has not imported at the time you call getmodule(A). I think it's also possible for problems to arise if you import b in some nonstandard way, like importing it via an implicit relative import instead of as part of a package (if it's part of a package).
Conceptually, though, what you're doing is somewhat dubious. It's risky to set A's module to be some other module, because that module might not be known (i.e., imported) when the real module is, and in fact it's possible the "disguise" module might not even exist.
I want to fake a package in python. I want to define something so that the code can do
from somefakepackage.morefakestuff import somethingfake
And somefakepackage is defined in code and so is everything below it. Is that possible? The reason for doing this is to trick my unittest that I got a package ( or as I said in the title, a module ) in the python path which actually is just something mocked up for this unittest.
Sure. Define a class, put the stuff you need inside that, assign the class to sys.modules["classname"].
class fakemodule(object):
#staticmethod
def method(a, b):
return a+b
import sys
sys.modules["package.module"] = fakemodule
You could also use a separate module (call it fakemodule.py):
import fakemodule, sys
sys.modules["package.module"] = fakemodule
Yes, you can make a fake module:
from types import ModuleType
m = ModuleType("fake_module")
import sys
sys.modules[m.__name__] = m
# some scripts may expect a file
# even though this file doesn't exist,
# it may be used by Python for in error messages or introspection.
m.__file__ = m.__name__ + ".py"
# Add a function
def my_function():
return 10
m.my_function = my_function
Note, in this example its using an actual module (of ModuleType) since some
Python code may expect modules, (instead of a dummy class).
This can be made into a utility function:
def new_module(name, doc=None):
import sys
from types import ModuleType
m = ModuleType(name, doc)
m.__file__ = name + '.py'
sys.modules[name] = m
return m
print(new_module("fake_module", doc="doc string"))
Now other scripts can run:
import fake_module
I took some of the ideas from the other answers and turned them into a Python decorator #modulize which converts a function into a module. This module can then be imported as usual. Here is an example.
#modulize('my_module')
def my_dummy_function(__name__): # the function takes one parameter __name__
# put module code here
def my_function(s):
print(s, 'bar')
# the function must return locals()
return locals()
# import the module as usual
from my_module import my_function
my_function('foo') # foo bar
The code for the decorator is as follows
import sys
from types import ModuleType
class MockModule(ModuleType):
def __init__(self, module_name, module_doc=None):
ModuleType.__init__(self, module_name, module_doc)
if '.' in module_name:
package, module = module_name.rsplit('.', 1)
get_mock_module(package).__path__ = []
setattr(get_mock_module(package), module, self)
def _initialize_(self, module_code):
self.__dict__.update(module_code(self.__name__))
self.__doc__ = module_code.__doc__
def get_mock_module(module_name):
if module_name not in sys.modules:
sys.modules[module_name] = MockModule(module_name)
return sys.modules[module_name]
def modulize(module_name, dependencies=[]):
for d in dependencies: get_mock_module(d)
return get_mock_module(module_name)._initialize_
The project can be found here on GitHub. In particular, I created this for programming contests which only allow the contestant to submit a single .py file. This allows one to develop a project with multiple .py files and then combine them into one .py file at the end.
You could fake it with a class which behaves like somethingfake:
try:
from somefakepackage.morefakestuff import somethingfake
except ImportError:
class somethingfake(object):
# define what you'd expect of somethingfake, e.g.:
#staticmethod
def somefunc():
...
somefield = ...
TL;DR
Patch sys.modules using unittest.mock:
mock.patch.dict(
sys.modules,
{'somefakepackage': mock.Mock()},
)
Explanation
Other answers correctly recommend to fix sys.modules but a proper way to do it is by patching it using mock.patch. Meaning replacing it temporarily (only for when tests are run) with a fake object that optionally imitates the desired behaviour. And restoring it back once tests are finished to not affect other test cases.
The code in TL;DR section will simply make your missing package not raise ImportError. To provide fake package with contents and imitate desired behaviour, initiate mock.Mock(…) with proper arguments (e.g. add attributes via Mock's **kwargs).
Full code example
The code below temporarily patches sys.modules so that it includes somefakepackage and makes it importable from the dependent modules without ImportError.
import sys
import unittest
from unittest import mock
class SomeTestCase(unittest.TestCase):
def test_smth(self):
# implement your testing logic, for example:
self.assertEqual(
123,
somefakepackage_dependent.some_func(),
)
#classmethod
def setUpClass(cls): # called once before all the tests
# define what to patch sys.modules with
cls._modules_patcher = mock.patch.dict(
sys.modules,
{'somefakepackage': mock.Mock()},
)
# actually patch it
cls._modules_patcher.start()
# make the package globally visible and import it,
# just like if you have imported it in a usual way
# placing import statement at the top of the file,
# but relying on a patched dependency
global somefakepackage_dependent
import somefakepackage_dependent
#classmethod # called once after all tests
def tearDownClass(cls):
# restore initial sys.modules state back
cls._modules_patcher.stop()
To read more about setUpClass/tearDownClass methods, see unittest docs.
unittest's built-in mock subpackage is actually a very powerful tool. Better dive deeper into its documentation to get a better understanding.