We Keep Coding

Registering classes to factory with classes in different files

I have a factory as shown in the following code:
class ClassFactory:
registry = {}
#classmethod
def register(cls, name):
def inner_wrapper(wrapped_class):
if name in cls.registry:
print(f'Class {name} already exists. Will replace it')
cls.registry[name] = wrapped_class
return wrapped_class
return inner_wrapper
#classmethod
def create_type(cls, name):
exec_class = cls.registry[name]
type = exec_class()
return type
#ClassFactory.register('Class 1')
class M1():
def __init__(self):
print ("Starting Class 1")
#ClassFactory.register('Class 2')
class M2():
def __init__(self):
print("Starting Class 2")
This works fine and when I do
if __name__ == '__main__':
print(ClassFactory.registry.keys())
foo = ClassFactory.create_type("Class 2")
I get the expected result of dict_keys(['Class 1', 'Class 2']) Starting Class 2
Now the problem is that I want to isolate classes M1 and M2 to their own files m1.py and m2.py, and in the future add other classes using their own files in a plugin manner.
However, simply placing it in their own file
m2.py
from test_ import ClassFactory
#MethodFactory.register('Class 2')
class M2():
def __init__(self):
print("Starting Class 2")
gives the result dict_keys(['Class 1']) since it never gets to register the class.
So my question is: How can I ensure that the class is registered when placed in a file different from the factory, without making changes to the factory file whenever I want to add a new class? How to self register in this way? Also, is this decorator way a good way to do this kind of thing, or are there better practices?
Thanks

How can I ensure that the class is registered when placed in a file different from the factory, without making changes to the factory file whenever I want to add a new class?
I'm playing around with a similar problem, and I've found a possible solution. It seems too much of a 'hack' though, so set your critical thinking levels to 'high' when reading my suggestion below :)
As you've mentioned in one of your comments above, the trick is to force the loading of the individual *.py files that contain individual class definitions.
Applying this to your example, this would involve:
Keeping all class implementations in a specific folders, e.g., structuring the files as follows:
.
└- factory.py # file with the ClassFactory class
└─ classes/
└- __init__.py
└- m1.py # file with M1 class
└- m2.py # file with M2 class
Adding the following statement to the end of your factory.py file, which will take care of loading and registering each individual class:
from classes import *
Add a piece of code like the snippet below to your __init__.py within the classes/ foder, so that to dynamically load all classes [1]:
from inspect import isclass
from pkgutil import iter_modules
from pathlib import Path
from importlib import import_module
# iterate through the modules in the current package
package_dir = Path(__file__).resolve().parent
for (_, module_name, _) in iter_modules([package_dir]):
# import the module and iterate through its attributes
module = import_module(f"{__name__}.{module_name}")
for attribute_name in dir(module):
attribute = getattr(module, attribute_name)
if isclass(attribute):
# Add the class to this package's variables
globals()[attribute_name] = attribute
If I then run your test code, I get the desired result:
# test.py
from factory import ClassFactory
if __name__ == "__main__":
print(ClassFactory.registry.keys())
foo = ClassFactory.create_type("Class 2")
$ python test.py
dict_keys(['Class 1', 'Class 2'])
Starting Class 2
Also, is this decorator way a good way to do this kind of thing, or are there better practices?
Unfortunately, I'm not experienced enough to answer this question. However, when searching for answers to this problem, I've came across the following sources that may be helpful to you:
[2] : this presents a method for registering class existence based on Python Metaclasses. As far as I understand, it relies on the registering of subclasses, so I don't know how well it applies to your case. I did not follow this approach, as I've noticed that the new edition of the book suggests the use of another technique (see bullet below).
[3], item 49 : this is the 'current' suggestion for subclass registering, which relies on the definition of the __init_subclass__() function in a base class.
If I had to apply the __init_subclass__() approach to your case, I'd do the following:
Add a Registrable base class to your factory.py (and slightly re-factor ClassFactory), like this:
class Registrable:
def __init_subclass__(cls, name:str):
ClassFactory.register(name, cls)
class ClassFactory:
registry = {}
#classmethod
def register(cls, name:str, sub_class:Registrable):
if name in cls.registry:
print(f'Class {name} already exists. Will replace it')
cls.registry[name] = sub_class
#classmethod
def create_type(cls, name):
exec_class = cls.registry[name]
type = exec_class()
return type
from classes import *
Slightly modify your concrete classes to inherit from the Registrable base class, e.g.:
from factory import Registrable
class M2(Registrable, name='Class 2'):
def __init__(self):
print ("Starting Class 2")

How does Python support this common problem related to run time polymorphism?

I am trying to exectute the below code but I get errors.
class base:
def callme(data):
print(data)
class A(base):
def callstream(self):
B.stream(self)
def callme(data):
print("child ", data)
class B:
def stream(data):
# below statement doesn't work but I want this to run to achieve run time
# polymorphism where method call is not hardcoded to a certain class reference.
(base)data.callme("streaming data")
# below statement works but it won't call child class overridden method. I
# can use A.callme() to call child class method but then it has to be
# hardcoded to A. which kills the purpose. Any class A or B or XYZ which
# inherits base call should be able to read stream data from stream class.
# How to achive this in Python? SO any class should read the stream data as
# long as it inherits from the base class. This will give my stream class a
# generic ability to be used by any client class as long as they inherit
# base class.
#base.callme("streaming data")
def main():
ob = A()
ob.callstream()
if __name__=="__main__":
main()

I got the output you say you're looking for (in a comment rather than the question -- tsk, tsk) with the following code, based on the code in your question:
class base:
def callme(self, data):
print(data)
class A(base):
def callstream(self):
B.stream(self)
def callme(self, data):
print("child", data)
class B:
#classmethod
def stream(cls, data):
data.callme("streaming data")
def main():
ob = A()
ob.callstream()
if __name__=="__main__":
main()
Basically, I just made sure the instance methods had self parameters, and since you seem to be using B.stream() as a class method, I declared it as such.

Python mock a method when specific argument

I have a python method like
import external_object
from external_lib1 import ExternalClass1
from external_lib2 import Hook
class MyClass(self):
def my_method(self):
ExternalClass.get('arg1') #should be mocked and return a specific value with this arg1
ExternalClass.get('arg2') #should be mocked and return a specific value with this arg2
def get_hook(self):
return Hook() # return a mock object with mocked method on it
def my_method(self):
object_1 = external_object.instance_type_1('args') # those are two different object instanciate from the same lib.
object_2 = external_object.instance_type_2('args')
object_1.method_1('arg') # should return what I want when object_1 mocked
object_2.method_2 ('arg') # should return what I want when object_2 mocked
In my test I would like to realise what I put in comments.
I could manage to do it, but every time it gets really messy.
I use to call flexmock for some stuff (by example ExternalClass.get('arg1') would be mock with a flexmock(ExternalClass).should_return('arg').with_args('arg') # etc...) but I'm tired of using different test libs to mock.
I would like to use only the mock library but I struggle to find a consistent way of doing it.

I like to use python's unittest lib. Concretely the unittest.mock which is a great lib to customize side effects and return value in unit tested functions.
They can be used as follows:
class Some(object):
"""
You want to test this class
external_lib is an external component we cannot test
"""
def __init__(self, external_lib):
self.lib = external_lib
def create_index(self, unique_index):
"""
Create an index.
"""
try:
self.lib.create(index=unique_index) # mock this
return True
except MyException as e:
self.logger.error(e.__dict__, color="red")
return False
class MockLib():
pass
class TestSome(unittest.TestCase):
def setUp(self):
self.lib = MockLib()
self.some = Some(self.lib)
def test_create_index(self):
# This will test the method returns True if everything went fine
self.some.create_index = MagicMock(return_value={})
self.assertTrue(self.some.create_index("test-index"))
def test_create_index_fail(self):
# This will test the exception is handled and return False
self.some.create_index = MagicMock(side_effect=MyException("error create"))
self.assertFalse(self.some.create_index("test-index"))
Put the TestSome() class file somewhere like your-codebase-path/tests and run:
python -m unittest -v
I hope it's useful.

How to implement a strategy pattern with runtime selection of a method?

Context
I'm trying to implement some variant of strategy pattern in Python 2.7.
I want to be able to instantiate a 'my_strategy' base class, but switch between different implementations of a 'score' method at run-time.
I will have many common methods in 'my_strategy' but a bunch of 'score' implementations.
The main illustrates how I want to use it.
Here the scoring implementation is dummy of course.
What I tried (i.e. My code so far)
strategy.py:
from algo_one import *
#from algo_two import *
class my_strategy ( object ):
def __init__(self, candidate = ""):
self.candidate = candidate
self.method = 'default'
self.no = 10
self._algo = algo_one
def set_strategy(self, strategy='default'):
self.strategy = strategy
if self.strategy == 'algo_one':
self._algo = algo_one
elif self.strategy == 'algo_two':
# self._algo = algo_two
pass
else:
self._algo = None
def score(self, *args):
if len(args) > 0:
self.candidate = args[0]
self._algo.score(self.candidate)
if __name__ == "__main__":
s = my_strategy()
s.strategy = 'algo_one'
s.candidate = "hello world"
print s.score()
print s.score("hi")
# s.set_method('algo_two')
# print s.score("hi")
I want to save the selected strategy in some sort of private pointer to the sub-class method.
algo_one.py:
from strategy import my_strategy
class algo_one ( my_strategy ):
def score(self, candidate):
return len(candidate)*self.no
I could have a class-less method, but later I'll need to access public variables of the base class.
algo_two.py:
from strategy import my_strategy
class algo_two ( my_strategy ):
def score(self, candidate):
return len(candidate)*3
I have an empty init.py too.
The errors
1.
in score self._algo.score(self.candidate)
TypeError: unbound method score() must be called with algo_one
instance as first argument (got str instance instead)
2.
If I uncomment the import of the second strategy:
from algo_two import *
I get the following error.
ImportError: cannot import name my_strategy
My guess is that I run into some sort of circular dependency.
3.
from algo_one import *
This is obviously not pretty (unable to detect undefined names), but if I
from algo_one import algo_one
I get
ImportError: cannot import name algo_one
Question
I think the errors are intertwined and that my approach, as a whole, may be flawed. If not just addressing the error, I'm looking for suggestions to improve the design. Or any comment, really. Also I'm open to suggestions regarding the title of this question. Thank you!

You make it much more complicated than it needs to be. Python functions are first class objects so the simplest way to implement the strategy pattern in Python is to pass a 'strategy' function to your "context" object (the one that uses the strategy). The fine part is that any callable object (ie: any object implementing the __call__ method) will work.
def default_score_strategy(scorer):
return len(scorer.candidate) * 3
def universal_answer_score_strategy(scorer):
return 42 # definitly the universal answer <g>
class ComplicatedStrategy(object):
def __init__(self, factor):
self.factor = factor
def __call__(self, scorer):
return len(scorer.candidate) * self.factor
class Scorer(object):
def __init__(self, candidate="", strategy=default_score_strategy):
self.candidate = candidate
self.strategy = strategy
def score(self):
return self.strategy(self)
s1 = Scorer("foo")
s2 = Scorer("bar", strategy=universal_answer_score_strategy)
s3 = Scorer("baaz", strategy=ComplicatedStrategy(365))
Note that your strategies dont have to be in the same module as the Scorer class (well, except the default one of course), and that the module containing the Scorer class doesn't have to import the stratgeies modules - nor know anything about where the strategies are defined:
# main.py
from mylib.scores import Scorer
from myapp.strategies import my_custom_strategy
s = Scorer("yadda", my_custom_strategy)

You don't instantiate your algo object in the __init__ method. Remember, to instantiate a class object, you need to call it:
self._algo = algo_one()
Yes, that's a circular dependency. I don't see however why algo_one and algo_two need to inherit from my_strategy at all. Just make them plain objects, or inherit a base class stored somewhere else. Or, keep them all in the same file - there's no reason to necessarily have classes in separate files in Python.
This is the same problem as 2.

One of your main problems are that your algorithms try to subclass from your base class, which is a huge design flaw (you already noticed that). Use a simple method binding instead, which deals with all the necessary things:
def algo_one(candidate):
# do stuff
return "A fluffy unicorn"
def algo_two(candidate):
# do some other stuff
return "Awesome rabbits"
# not really necessary, just to make it easier to add new algorithms
STRATEGIES = { "one": algo_one, "two": algo_two }
class Strategy(object):
def __init__(self):
...
def set_strategy(self, which):
if which not in STRATEGIES:
raise ValueError("'%s' is an unknown strategy" % which)
# compatibility checks about the entries in STRATEGIES omitted here
self._algo = STRATEGIES[which]
def score(self, *args):
# ...
return self._algo(...)
If you need a more complex approach (this however depends on your requirements), in which everyone knows about each other, split the algorithms and strategy chooser into different classes referencing each other (shortened version below):
class ScoreAlgo(object):
def __init__(self, parent):
self._strategy = parent # if you need a back-reference, just be aware of circular dependencies in the garbage collection
def __del__(self):
self._strategy = None # resolve circular dependency for the GC
def score(self, candidate):
return None
class Strategy(object):
def __init__(self):
...
def set_strategy(self, ...):
...
self._algo = ScoreAlgo(self)
def score(self, ...):
return self._algo.score(...)
(If you need a huge variety of algorithms, you should make ScoreAlgo an abstract base class, for which subclasses have to implement the score() method).
You also could use a mixin pattern (which is a bit more formal than the method binding) or several other ways. This however depends on your overall requirements.
EDIT: I just added a returnto both def score(): stubs to avoid confusion about why those might not return anything.

py.test: how to get the current test's name from the setup method?

I am using py.test and wonder if/how it is possible to retrieve the name of the currently executed test within the setup method that is invoked before running each test. Consider this code:
class TestSomething(object):
def setup(self):
test_name = ...
def teardown(self):
pass
def test_the_power(self):
assert "foo" != "bar"
def test_something_else(self):
assert True
Right before TestSomething.test_the_power becomes executed, I would like to have access to this name in setup as outlined in the code via test_name = ... so that test_name == "TestSomething.test_the_power".
Actually, in setup, I allocate some resource for each test. In the end, looking at the resources that have been created by various unit tests, I would like to be able to see which one was created by which test. Best thing would be to just use the test name upon creation of the resource.

You can also do this using the Request Fixture like this:
def test_name1(request):
testname = request.node.name
assert testname == 'test_name1'

You can also use the PYTEST_CURRENT_TEST environment variable set by pytest for each test case.
PYTEST_CURRENT_TEST environment variable
To get just the test name:
os.environ.get('PYTEST_CURRENT_TEST').split(':')[-1].split(' ')[0]

The setup and teardown methods seem to be legacy methods for supporting tests written for other frameworks, e.g. nose. The native pytest methods are called setup_method as well as teardown_method which receive the currently executed test method as an argument. Hence, what I want to achieve, can be written like so:
class TestSomething(object):
def setup_method(self, method):
print "\n%s:%s" % (type(self).__name__, method.__name__)
def teardown_method(self, method):
pass
def test_the_power(self):
assert "foo" != "bar"
def test_something_else(self):
assert True
The output of py.test -s then is:
============================= test session starts ==============================
platform linux2 -- Python 2.7.3 -- pytest-2.3.3
plugins: cov
collected 2 items
test_pytest.py
TestSomething:test_the_power
.
TestSomething:test_something_else
.
=========================== 2 passed in 0.03 seconds ===========================

Short answer:
Use fixture called request
This fixture has the following interesting attributes:
request.node.originalname = the name of the function/method
request.node.name = name of the function/method and ids of the parameters
request.node.nodeid = relative path to the test file, name of the test class (if in a class), name of the function/method and ids of the parameters
Long answer:
I inspected the content of request.node. Here are the most interesting attributes I found:
class TestClass:
#pytest.mark.parametrize("arg", ["a"])
def test_stuff(self, request, arg):
print("originalname:", request.node.originalname)
print("name:", request.node.name)
print("nodeid:", request.node.nodeid)
Prints the following:
originalname: test_stuff
name: test_stuff[a]
nodeid: relative/path/to/test_things.py::TestClass::test_stuff[a]
NodeID is the most promising if you want to completely identify the test (including the parameters). Note that if the test is as a function (instead of in a class), the class name (::TestClass) is simply missing.
You can parse nodeid as you wish, for example:
components = request.node.nodeid.split("::")
filename = components[0]
test_class = components[1] if len(components) == 3 else None
test_func_with_params = components[-1]
test_func = test_func_with_params.split('[')[0]
test_params = test_func_with_params.split('[')[1][:-1].split('-')
In my example this results to:
filename = 'relative/path/to/test_things.py'
test_class = 'TestClass'
test_func = 'test_stuff'
test_params = ['a']

# content of conftest.py
#pytest.fixture(scope='function', autouse=True)
def test_log(request):
# Here logging is used, you can use whatever you want to use for logs
log.info("STARTED Test '{}'".format(request.node.name))
def fin():
log.info("COMPLETED Test '{}' \n".format(request.node.name))
request.addfinalizer(fin)

Try my little wrapper function which returns the full name of the test, the file and the test name. You can use whichever you like later.
I used it within conftest.py where fixtures do not work as far as I know.
def get_current_test():
full_name = os.environ.get('PYTEST_CURRENT_TEST').split(' ')[0]
test_file = full_name.split("::")[0].split('/')[-1].split('.py')[0]
test_name = full_name.split("::")[1]
return full_name, test_file, test_name

You might have multiple tests, in which case...
test_names = [n for n in dir(self) if n.startswith('test_')]
...will give you all the functions and instance variables that begin with "test_" in self. As long as you don't have any variables named "test_something" this will work.
You can also define a method setup_method(self, method) instead of setup(self) and that will be called before each test method invocation. Using this, you're simply given each method as a parameter. See: http://pytest.org/latest/xunit_setup.html

You could give the inspect module are try.
import inspect
def foo():
print "My name is: ", inspect.stack()[0][3]
foo()
Output: My name is: foo

Try type(self).__name__ perhaps?