I am trying to implement the Strategy design pattern to create an interface for an underlying algorithm to be implemented in a modular fashion.
Currently, as per code below, I have one top-level/parent abstract class (ParentAbstractStrategy) that defines the base interface for the strategy method.
I also have a one-level-down from this abstract class (ChildAbstractStrategy).
The reason I have two abstract classes is because of the attributes they need to hold; see the __init__ methods.
ChildAbstractStrategy is a special case of ParentAbstractStrategy in that it stores an additional attribute: attr2.
Otherwise its interface is identical, as seen by identical strategy method signatures.
Sometimes, I want to be able to directly subclass ParentAbstractStrategy and implement the strategy method (see ConcreteStrategyA), but other times I want to be able to subclass ChildAbstractStrategy, because the extra attribute is required (see ConcreteStrategyB).
An additional complication is that in some subclasses of either abstract class I want to be able to handle additional arguments in the strategy method. This is why I have added **kwargs to all signatures of the strategy method, so that I can pass in whatever additional arguments I want to a subclass, on a case-by-case basis.
This creates the last problem: these extra arguments are not optional in the subclasses. E.g. in the strategy method of ConcreteStrategyB I want to be certain that the caller passed in a third argument.
I'm basically abusing **kwargs to provide what probably should be positional arguments (since I can't give them sane defaults and need their existence to be enforced).
This current solution of using **kwargs for "method overloading" in subclasses feels really messy, and I'm not sure if this means there is a problem with the class inheritance scheme or interface design, or both.
Is there a way that I can achieve these design goals in a cleaner fashion. It feels like I'm missing something big picture here and maybe the class/interface design is bad. Maybe creating two disjoint abstract classes with different signatures for the strategy method?
import abc
class ParentAbstractStrategy(metaclass=abc.ABCMeta):
#abc.abstractmethod
def __init__(self, attr1):
self.attr1 = attr1
#abc.abstractmethod
def strategy(self, arg1, arg2, **kwargs):
raise NotImplementedError
class ChildAbstractStrategy(ParentAbstractStrategy, metaclass=abc.ABCMeta):
#abc.abstractmethod
def __init__(self, attr1, attr2):
super().__init__(attr1)
self.attr2 = attr2
#abc.abstractmethod
def strategy(self, arg1, arg2, **kwargs):
raise NotImplementedError
class ConcreteStrategyA(ParentAbstractStrategy):
def __init__(self, attr1):
super().__init__(attr1)
def strategy(self, arg1, arg2, **kwargs):
print(arg1, arg2)
class ConcreteStrategyB(ChildAbstractStrategy):
def __init__(self, attr1, attr2):
super().__init__(attr1, attr2)
def strategy(self, arg1, arg2, **kwargs):
print(arg1, arg2)
arg3 = kwargs.get("arg3", None)
if arg3 is None:
raise ValueError("Missing arg3")
else:
print(arg3)
Here's an interpreter session demonstrating how it's currently working:
>>> a = ConcreteStrategyA(1)
>>> a.attr1
1
>>> a.strategy("a", "b")
a b
>>> b = ConcreteStrategyB(1, 2)
>>> b.attr1
1
>>> b.attr2
2
>>> b.strategy("a", "b")
a b
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/space/strategy.py", line 42, in strategy
raise ValueError("Missing arg3")
ValueError: Missing arg3
>>> b.strategy("a", "b", arg3="c")
a b
c
Answering my own question.
My usage of **kwargs is 'bad' in this scenario. Why?
As far as I can tell, **kwargs is typically used for:
Wrapper functions, e.g. decorators.
Collecting extra keyword arguments to a function that the function knows about (e.g. see usage in https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html?highlight=plot#matplotlib.pyplot.plot). In this scenario the **kwargs are optional arguments that can be passed into the function and they have sane default values.
Making **kwargs to be required in a function call is defeating their purpose; positional arguments that need to be explicitly supplied should be used instead. That way the interface provided by the function has to be explicitly satisfied by the caller.
There is another problem with using **kwargs in an interface, as I have. It involves the LSP (Liskov Substitution Principle, see https://en.wikipedia.org/wiki/Liskov_substitution_principle). The current implementation is abusing **kwargs in an attempt to define a variable interface for the strategy method among sublcasses. Although syntactically the function signatures for all strategy methods match, semantically the interfaces are different. This is a violation of the LSP, which would require that I could e.g. treat any descendant of ParentAbstractStrategy the same when considering their interfaces, e.g. I should be able to treat the strategy method of ConcreteStrategyA and ConcreteStrategyB the same.
What was my solution?
I have changed the interface of the strategy method to no longer include **kwargs and instead use a mix of positional arguments and keyword arguments with default values.
E.g. if ConcreteStrategyB still needs a third argument arg3 but ConcreteStrategyA does not, I could change the classes to look like this:
class ConcreteStrategyA(ParentAbstractStrategy):
def __init__(self, attr1):
super().__init__(attr1)
def strategy(self, arg1, arg2, arg3=None):
print(arg1, arg2)
class ConcreteStrategyB(ChildAbstractStrategy):
def __init__(self, attr1, attr2):
super().__init__(attr1, attr2)
def strategy(self, arg1, arg2, arg3=None):
print(arg1, arg2)
assert arg3 is not None
print(arg3)
With the interfaces of both parent classes changed to match.
Related
I have to know if this situation is correct
We have an abstract class with one method that requires 2 parameters:
class Base(ABC):
#abstractmethod
def class_method(self, param1, param2):
pass
Then I have to implement 3 classes. Two of them use param1 and param2 in their method, but one of them only uses param1, I don´t need to use it inside!
class ClassA(Base):
def class_method(self, param1, param2):
return param1 + param2
class ClassB(Base):
def class_method(self, param1, param2):
return param1 + param2
class ClassC(Base):
def class_method(self, param1, param2):
return param1
Is this a correct implementation? What´s the best way to manage the unused param2 in the ClassC method?
I tried also to define param2 as optional
class Base(ABC):
#abstractmethod
def class_method(self, param1, param2=None):
pass
But the question is still the same.
Your first example is correct. Whether or not ClassC uses the required parameter is irrelevant, as long as it accepts an argument. Consider this example:
lst: Base = [ClassA(), ClassB(), ClassC()]
for obj in lst:
obj.class_method(x, y)
From a static typing perspective, you don't know what the actual runtime values in lst are, only that they are instances of (subclasses of) Base, and so obj.class_method must accept two arguments.
If you make an argument optional as in your second example, then a subclass cannot turn around and require it, for the same reasons. The following is correct given the static type of lst: no use of class_method should require a second argument.
lst: Base = [ClassA(), ClassB(), ClassC()]
for obj in lst:
obj.class_method(x)
Note this is in the context of conforming to the Liskov substitution principle. For all ABC cares about, the following is "correct":
class ClassD(Base):
class_method = None
You can assign anything you want to class_method, as long as it's not another abstract method, and you'll be able to instantiate ClassD without a problem. Whether that instance behaves properly doesn't matter.
I have following structure for class.
class foo(object):
def __call__(self,param1):
pass
class bar(object):
def __call__(self,param1,param2):
pass
I have many classes of this type. And i am using this callable class as follows.
classes = [foo(), bar()]
for C in classes:
res = C(param1)
'''here i want to put condition if class takes 1 argumnet just pass 1
parameter otherwise pass two.'''
I have think of one pattern like this.
class abc():
def __init__(self):
self.param1 = 'xyz'
self.param2 = 'pqr'
def something(self, classes): # classes = [foo(), bar()]
for C in classes:
if C.__class__.__name__ in ['bar']:
res = C(self.param1, self.param2)
else:
res = C(self.param2)
but in above solution have to maintain list of class which takes two arguments and as i will add more class to file this will become messy.
I dont know whether this is correct(pythonic) way to do it.
On more idea i have in mind is to check how many argument that class is taking. If its 2 then pass an additional argument otherwise pass 1 argument.I have looked at this solution How can I find the number of arguments of a Python function? . But i am not confident enought that this is the best suited solution to my problem.
Few things about this:
There are only two type of classes in my usecase one with 1 argument and one with 2.
Both class takes first argument same so params1 in both case is same argument i am passing. in case of class with two required parameter i am passing additional argument(params2) containing some data.
Ps : Any help or new idea for this problem are appretiated.
UPD : Updated the code.
Basically, you want to use polymorphism on your object's __call__() method, but you have an issue with your callables signature not being the same.
The plain simple answer to this is: you can only use polymorphism on compatible types, which in this case means that your callables MUST have compatible signatures.
Hopefully, there's a quick and easy way to solve this: just modify your methods signatures so they accept varargs and kwargs:
class Foo(object):
def __call__(self,param1, *args, **kw):
pass
class Bar(object):
def __call__(self, param1, param2, *args, **kw):
pass
For the case where you can't change the callable's signature, there's still a workaround: use a lambda as proxy:
def func1(y, z):
pass
def func2(x):
pass
callables = [func1, lambda y, z: func2(y)]
for c in callables:
c(42, 1138)
Note that this last example is actually known as the adapter pattern
Unrelated: this:
if C.__class__.__name__ in ['bar']:
is a inefficient and convoluted way to write:
if C.__class__.__name__ == 'bar':
which is itself an inefficient, convoluted AND brittle way to write:
if type(C) is bar:
which, by itself, is a possible design smell (there are legit use cases for checking the exact type of an object, but most often this is really a design issue).
I have a particular problem, but I will make the example more general.
I have a Parent class with a mandatory constructor parameter and a few optional ones, each with a default value. Then, I inherit Child from it and add a mandatory parameter, and inherit GrandChild from Child and add another mandatory parameter to the constructor. The result is similar to this:
class Parent():
def __init__(self, arg1, opt_arg1='opt_arg1_default_val', opt_arg2='opt_arg2_default_val',
opt_arg3='opt_arg3_default_val', opt_arg4='opt_arg4_default_val'):
self.arg1 = arg1
self.opt_arg1 = opt_arg1
self.opt_arg2 = opt_arg2
self.opt_arg3 = opt_arg3
self.opt_arg4 = opt_arg4
class Child(Parent):
def __init__(self, arg1, arg2, opt_arg1, opt_arg2, opt_arg3, opt_arg4):
super().__init__(arg1, opt_arg1, opt_arg2, opt_arg3, opt_arg4)
self.arg2 = arg2
class GrandChild(Child):
def __init__(self, arg1, arg2, arg3, opt_arg1, opt_arg2, opt_arg3, opt_arg4):
super().__init__(arg1, arg2, opt_arg1, opt_arg2, opt_arg3, opt_arg4)
self.arg3 = arg3
The problem is that this looks rather ugly, especially if I want to inherit more classes from Child, I'd have to copy/paste all the arguments in that new class's constructor.
In search for a solution, I found here that I can solve this problem using **kwargs like so:
class Parent():
def __init__(self, arg1, opt_arg1='opt_arg1_default_val', opt_arg2='opt_arg2_default_val',
opt_arg3='opt_arg3_default_val', opt_arg4='opt_arg4_default_val'):
self.arg1 = arg1
self.opt_arg1 = opt_arg1
self.opt_arg2 = opt_arg2
self.opt_arg3 = opt_arg3
self.opt_arg4 = opt_arg4
class Child(Parent):
def __init__(self, arg1, arg2, **kwargs):
super().__init__(arg1, **kwargs)
self.arg2 = arg2
class GrandChild(Child):
def __init__(self, arg1, arg2, arg3,**kwargs):
super().__init__(arg1, arg2,**kwargs)
self.arg3 = arg3
However, I am not sure if this is the right way.
There is also a slight inconvenience while creating objects of these classes. I am using PyCharm to develop, and in this case the IDE has a useful method of displaying a function/class constructor arguments. For instance, in the first example,
This makes it much easier to develop and can help future developers as well since they can see what other arguments the function has. However, in the second example, the optional arguments are not shown anymore:
And I do not think it is a good practice to use **kwargs in this case, since one would have to dig deeper into the code up to the Parent class to check what optional arguments it has.
I've also looked into using the Builder pattern, but then all I do is move the arguments list from my classes to builder classes, and I have the same problem, builders with lots of arguments that when inherited will create even more arguments on top of the already existing ones. Also in Python, as much as I see, Builder doesn't really make much sense considering all class members are public and can be accessed without needing setters and getters.
Any ideas on how to solve this constructor problem?
The basic idea is to write code that generates the __init__ method for you, with all the parameters specified explicitly rather than via *args and/or **kwargs, and without even needing to repeat yourself with all those self.arg1 = arg1 lines.
And, ideally, it can make it easy to add type annotations that PyCharm can use for popup hints and/or static type checking.1
And, while you're at it, why not build a __repr__ that displays the same values? And maybe even an __eq__, and a __hash__, and maybe lexicographical comparison operators, and conversion to and from a dict whose keys match the attributes for each JSON persistence, and…
Or, even better, use a library that takes care of that for you.
Python 3.7 comes with such a library, dataclasses. Or you can use a third-party library like attrs, that works with Python 3.4 and (with some limitations) 2.7. Or, for simple cases (where your objects are immutable, and you want them to work like a tuple of their attributes in specified order), you can use namedtuple, which works back to 3.0 and 2.6.
Unfortunately, dataclasses doesn't quite work for your use case. If you just write this:
from dataclasses import dataclass
#dataclass
class Parent:
arg1: str
opt_arg1: str = 'opt_arg1_default_val'
opt_arg2: str = 'opt_arg2_default_val'
opt_arg3: str = 'opt_arg3_default_val'
opt_arg4: str = 'opt_arg4_default_val'
#dataclass
class Child(Parent):
arg2: str
… you'll get an error, because it tries to place the mandatory parameter arg2 after the default-values parameters opt_arg1 through opt_arg4.
dataclasses doesn't have any way to reorder parameters (Child(arg1, arg2, opt_arg1=…), or to force them to be keyword-only parameters (Child(*, arg1, opt_arg1=…, arg2)). attrs doesn't have that functionality out of the box, but you can add it.
So, it's not quite as trivial as you'd hope, but it's doable.
But if you wanted to write this yourself, how would you create the __init__ function dynamically?
The simplest option is exec.
You've probably heard that exec is dangerous. But it's only dangerous if you're passing in values that came from your user. Here, you're only passing in values that came from your own source code.
It's still ugly—but sometimes it's the best answer anyway. The standard library's namedtuple used to be one giant exec template., and even the current version uses exec for most of the methods, and so does dataclasses.
Also, notice that all of these modules store the set of fields somewhere in a private class attribute, so subclasses can easily read the parent class's fields. If you didn't do that, you could use the inspect module to get the Signature for your base class's (or base classes', for multiple inheritance) initializer and work it out from there. But just using base._fields is obviously a lot simpler (and allows storing extra metadata that doesn't normally go in signatures).
Here's a dead simple implementation that doesn't handle most of the features of attrs or dataclasses, but does order all mandatory parameters before all optionals.
def makeinit(cls):
fields = ()
optfields = {}
for base in cls.mro():
fields = getattr(base, '_fields', ()) + fields
optfields = {**getattr(base, '_optfields', {}), **optfields}
optparams = [f"{name} = {val!r}" for name, val in optfields.items()]
paramstr = ', '.join(['self', *fields, *optparams])
assignstr = "\n ".join(f"self.{name} = {name}" for name in [*fields, *optfields])
exec(f'def __init__({paramstr}):\n {assignstr}\ncls.__init__ = __init__')
return cls
#makeinit
class Parent:
_fields = ('arg1',)
_optfields = {'opt_arg1': 'opt_arg1_default_val',
'opt_arg2': 'opt_arg2_default_val',
'opt_arg3': 'opt_arg3_default_val',
'opt_arg4': 'opt_arg4_default_val'}
#makeinit
class Child(Parent):
_fields = ('arg2',)
Now, you've got exactly the __init__ methods you wanted on Parent and Child, fully inspectable2 (including help), and without having to repeat yourself.
1. I don't use PyCharm, but I know that well before 3.7 came out, their devs were involved in the discussion of #dataclass and were already working on adding explicit support for it to their IDE, so it doesn't even have to evaluate the class definition to get all that information. I don't know if it's available in the current version, but if not, I assume it will be. Meanwhile, #dataclass already just works for me with IPython auto-completion, emacs flycheck, and so on, which is good enough for me. :)
2. … at least at runtime. PyCharm may not be able to figure things out statically well enough to do popup completion.
When deriving from a builtin type as well as from some other class, it seems that the builtin type's constructor doesn't call the super class constructor. This results in __init__ methods not being called for types that come after the builtin in the MRO.
Example:
class A:
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
print("A().__init__()")
class B(list, A):
def __init__(self, *args, **kwargs):
print("B().__init__() start")
super().__init__(*args, **kwargs)
print("B().__init__() end")
if __name__ == '__main__':
b = B()
In this sample, A.__init__ is never called. When B is defined as class B(A, list) instead -- switching the inheritance order -- it works as intended (i.e. A.__init__ is called).
This very subtle dependence on inheritance order seems rather un-pythonic, is it intended this way? It also means that you must never derive from builtin types in complex class hierarchies, because you can't know where the builtin ends up in the MRO when someone else derives from your classes (maintenance horror). Am I missing something?
Extra info: Python version 3.1
The correct usage of super() is rather subtle and requires some care if the collaborating methods don't all have the same signature. The usual pattern for __init__() methods is the following:
class A(object):
def __init__(self, param_a, **kwargs):
self.param_a = param_a
super(A, self).__init__(**kwargs)
class B(A):
def __init__(self, param_b, **kwargs):
self.param_b = param_b
super(B, self).__init__(**kwargs)
class C(A):
def __init__(self, param_c, **kwargs):
self.param_c = param_c
super(C, self).__init__(**kwargs)
class D(B, C):
def __init__(self, param_d, **kwargs):
self.param_d = param_d
super(D, self).__init__(**kwargs)
d = D(param_a=1, param_b=2, param_c=3, param_d=4)
Note that this requires that all methods collaborate, and that all methods need a somewhat compatible signature to ensure it does not matter at which point the method is called.
The constructors of built-in types don't have constructor signatures that allow participating in such a collaboration. Even if they did call super().__init__() this would be rather useless unless all the constructor signatures were unified. So in the end you are right -- they are not suitable for particpation in collaborative constructor calls.
super() can only be used if either all collaborating methods have the same signature (like e.g. __setattr__()) or if you use the above (or a similar) pattern. Using super() isn't the only pattern to call base class methods, though. If there are no "diamonds" in your multiple inheritance pattern, you can use explicit base class calls, for example B.__init__(self, param_a). Classes with multiple base classes simply call multiple constructors. An even if there are diamonds, you can sometimes use explicit calls, as long as you take care that an __init__() may be called several times without harm.
If you want to use super() for contructors anyway, you indeed shouldn't use subclasses of built-in types (except for object) in multiple inheirtance hierachies. Some further reading:
Raymond Hettinger's Python’s super() considered super!
James Knight's Python's Super is nifty, but you can't use it
I am subclassing an object in order to override a method that I want to add some functionality to. I don't want to completely replace it or add a differently named method but remain compatible to the superclasses method by just adding an optional argument to the method.
Is it possible to work with *args and **kwargs to pass through all arguments to the superclass and still add an optional argument with a default?
I intuitively came up with the following but it doesn't work:
class A(object):
def foo(self, arg1, arg2, argopt1="bar"):
print arg1, arg2, argopt1
class B(A):
def foo(self, *args, argopt2="foo", **kwargs):
print argopt2
A.foo(self, *args, **kwargs)
b = B()
b.foo("a", "b", argopt2="foo")
Of course I can get it to work when I explicitly add all the arguments of the method of the superclass:
class B(A):
def foo(self, arg1, arg2, argopt1="foo", argopt2="bar"):
print argopt2
A.foo(self, arg1, arg2, argopt1=argopt1)
What's the right way to do this, do I have to know and explicitly state all of the overridden methods arguments?
class A(object):
def foo(self, arg1, arg2, argopt1="bar"):
print arg1, arg2, argopt1
class B(A):
def foo(self, *args, **kwargs):
argopt2 = kwargs.get('argopt2', default_for_argopt2)
# remove the extra arg so the base class doesn't complain.
del kwargs['argopt2']
print argopt2
A.foo(self, *args, **kwargs)
b = B()
b.foo("a", "b", argopt2="foo")
What's the right way to do this, do I
have to know and explicitly state all
of the overridden methods arguments?
If you want to cover all cases (rather than just rely on the caller to always do things your way, e.g., always call you only with the extra argument passed by-name, never by position) you do have to code (or dynamically discover) a lot of knowledge about the signature of the method you're overriding -- hardly surprising: inheritance is a strong form of coupling, and overriding methods is one way that coupling presents itself.
You could dynamically discover the superclass's method arguments via inspect.getargspec, in order to make sure you call it properly... but this introspection technique can get tricky if two classes are trying to do exactly the same thing (once you know your superclass's method accepts *a and/or **kw you can do little more than pass all the relevant arguments upwards and hope, with fingers crossed, that the upstream method chain eventually does proper housecleaning before calling a version that's not quite so tolerant).
Such prices may be worth paying when you're designing a wrapper that's meant to be applied dynamically to callables with a wide variety of signatures (especially since in a decorator setting you can arrange to pay the hefty cost of introspection just once per function you're decorating, not every time the resulting wrapper is called). It seems unlikely to be a worthwhile technique in a case such as yours, where you'd better know what you're subclassing (subclassing is strong coupling: doing it blindly is definitely not advisable!), and so you might as well spell out the arguments explicitly.
Yes, if the superclass's code changes drastically (e.g., by altering method signatures), you'll have to revise the subclass as well -- that's (part of) the price of inheritance. The overall price's hefty enough that the new Go programming language does totally without it -- forcing you to apply the Gang of 4's excellent advice to prefer composition over inheritance. In Python complete abstinence from inheritance would just be impractical, but using it soberly and in moderation (and accepting the price you'll pay in terms of coupling when you do) remains advisable.
When subclassing and overriding methods, one must always decide if using super() is a good idea, and this page is good for that.
I'm not saying that super() should be avoided, like the article author may be: I'm saying that super() has some very important prerequisits that must be followed if you don't want super() to come back and bite you.