Is there any way to connect 2 classes (without merging them in 1) and thus avoiding repetition under statement if a: in class Z?
class A:
def __init__(self, a):
self.a = a
self.b = self.a + self.a
class Z:
def __init__(self, z, a=None):
self.z = z
if a: # this part seems like repetition
self.a = a.a
self.b = a.b
a = A('hello')
z = Z('world', a)
assert z.a == a.a # hello
assert z.b == a.b # hellohello
Wondering if python has some tools. I would prefer to avoid loop over instance variables and using setattr.
Something like inheriting from class A to class Z, Z(A) or such.
Here's a trivial example of class inheritance that may help you to understand:
class A:
def __init__(self, a):
self._a = a
self._b = self.a + self.a
class Z(A):
def __init__(self, z, a):
super().__init__(a)
self._z = z
clazz = Z('Hello', 'world')
print(clazz._z, clazz._a)
Conceptually, the standard techniques for associating an A instance with a Z instance are:
Using composition (and delegation)
"Composition" simply means that the A instance itself is an attribute of the Z instance. We call this a "has-a" relationship: every Z has an A that's associated with it.
In normal cases, we can simply pass the A instance to the Z constructor, and have it assign an attribute in __init__. Thus:
class A:
def __init__(self, a):
self.a = a
self.b = self.a + self.a
def action(self): # added for demonstration purposes.
pass
class Z:
def __init__(self, z, a=None):
self.z = z
self._a = a # if not None, this will be an `A` instance
Notice that the attribute for the a instance is specially named to avoid conflicting with the A class attribute names. This is to avoid ambiguity (calling it .a makes one wonder whether my_z.a should get the .a attribute from the A instance, or the entire instance), and to mark it as an implementation detail (normally, outside code won't have a good reason to get the entire A instance out of the Z; the entire point of delegation is to make it so that users of Z don't have to worry about A's interface).
One important limitation is that the composition relationship is one-way by nature: self._a = a gives the Z class access to A contents, but not the other way around. (Of course, it's also possible to build the relationship in both directions, but this will require some planning ahead.)
"Delegation" means that we use some scheme in the code, so that looking something up in a Z instance finds it in the composed A instance when necessary. There are multiple ways to achieve this in Python, at least two of which are worth mentioning:
Explicit delegation per attribute
We define a separate property in the Z class, for each attribute we want to delegate. For example:
# within the `Z` class
#property
def a(self):
return self._a.a
# The setter can also be omitted to make a read-only attribute;
# alternately, additional validation logic can be added to the function.
#a.setter
def a(self, value):
self._a.a = value
For methods, using the same property approach should work, but it may be simpler to make a wrapper function and calling it:
def action(self):
return self._a.action()
Delegation via __getattr__
The __getattr__ magic ("dunder") method allows us to provide fallback logic for looking up an attribute in a class, if it isn't found by the normal means. We can use this for the Z class, so that it will try looking within its _a if all else fails. This looks like:
def __getattr__(self, name):
return getattr(self._a, name)
Here, we use the free function getattr to look up the name dynamically within the A instance.
Using inheritance
This means that each Z instance will, conceptually, be a kind of A instance - classes represent types, and inheriting Z from A means that it will be a subtype of A.
We call this an "is-a" relationship: every Z instance is an A instance. More precisely, a Z instance should be usable anywhere that an A instance could be used, but also Z might contain additional data and/or use different implementations.
This approach looks like:
class A:
def __init__(self, a):
self.a = a
self.b = self.a + self.a
def action(self): # added for demonstration purposes.
return f'{self.z.title()}, {self.a}!'
class Z(A):
def __init__(self, z, a):
# Use `a` to do the `A`-specific initialization.
super().__init__(a)
# Then do `Z`-specific initialization.
self.z = z
The super function is magic that finds the A.__init__ function, and calls it as a method on the Z instance that's currently being initialized. (That is: self will be the same object for both __init__ calls.)
This is clearly more convenient than the delegation and composition approach. Our Z instance actually has a and b attributes as well as z, and also actually has a action method. Thus, code like my_z.action() will use the method from the A class, and accessing the a and b attributes of a Z instance works - because the Z instance actually directly contains that data.
Note in this example that the code for action now tries to use self.z. this won't work for an A instance constructed directly, but it does work when we construct a Z and call action on it:
>>> A('world').action()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in action
AttributeError: 'A' object has no attribute 'z'
>>> Z('hello', 'world').action()
'Hello, world!'
We say that such an A class, which doesn't properly function on its own, is abstract. (There are more tools we can use to prevent accidentally creating an unusable base A; these are outside the scope of this answer.)
This convenience comes with serious implications for design. It can be hard to reason about deep inheritance structures (where the A also inherits from B, which inherits from C...) and especially about multiple inheritance (Z can inherit from B as well as A). Doing these things requires careful planning and design, and a more detailed understanding of how super works - beyond the scope of this answer.
Inheritance is also less flexible. For example, when the Z instance composes an A instance, it's easy to swap that A instance out later for another one. Inheritance doesn't offer that option.
Using mixins
Essentially, using a mixin means using inheritance (generally, multiple inheritance), even though we conceptually want a "has-a" relationship, because the convenient usage patterns are more important than the time spent designing it all up front. It's a complex, but powerful design pattern that essentially lets us build a new class from component parts.
Typically, mixins will be abstract (in the sense described in the previous section). Most examples of mixins also won't contain data attributes, but only methods, because they're generally designed specifically to implement some functionality. (In some programming languages, when using multiple inheritance, only one base class is allowed to contain data. However, this restriction is not necessary and would make no sense in Python, because of how objects are implemented.)
One specific technique common with mixins is that the first base class listed will be an actual "base", while everything else is treated as "just" an abstract mixin. To keep things organized while initializing all the mixins based on the original Z constructor arguments, we use keyword arguments for everything that will be passed to the mixins, and let each mixin use what it needs from the **kwargs.
class Root:
# We use this to swallow up any arguments that were passed "too far"
def __init__(self, *args, **kwargs):
pass
class ZBase(Root):
def __init__(self, z, **kwargs):
# a common pattern is to just accept arbitrary keyword arguments
# that are passed to all the mixins, and let each one sort out
# what it needs.
super().__init__(**kwargs)
self.z = z
class AMixin(Root):
def __init__(self, **kwargs):
# This `super()` call is in case more mixins are used.
super().__init__(**kwargs)
self.a = kwargs['a']
self.b = self.a + self.a
def func(self): # This time, we'll make it do something
return f'{self.z.title()}, {self.a}!'
# We combine the base with the mixins by deriving from both.
# Normally there is no reason to add any more logic here.
class Z(ZBase, AMixin): pass
We can use this like:
>>> # we use keyword arguments for all the mixins' arguments
>>> my_z = Z('hello', a='world')
>>> # now the `Z` instance has everything defined in both base and mixin:
>>> my_z.func()
'Hello, world!'
>>> my_z.z
'hello'
>>> my_z.a
'world'
>>> my_z.b
'worldworld'
The code in AMixin can't stand on its own:
>>> AMixin(a='world').func()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in func
AttributeError: 'AMixin' object has no attribute 'z'
but when the Z instance has both ZBase and AMixin as bases, and is used to call func, the z attribute can be found - because now self is a Z instance, which has that attribute.
The super logic here is a bit tricky. The details are beyond the scope of this post, but suffice to say that with mixin classes that are set up this way, super will forward to the next, sibling base of Z, as long as there is one. It will do this no matter what order the mixins appear in; the Z instance determines the order, and super calls whatever is "next in line". When all the bases have been consulted, next in line is Root, which is just there to intercept the kwargs (since the last mixin doesn't "know" it's last, and passes them on). This is necessary because otherwise, next in line would be object, and object.__init__ raises an exception if there are any arguments.
For more details, see What is a mixin and why is it useful?.
Related
In C++, given a class hierarchy, the most derived class's ctor calls its base class ctor which then initialized the base part of the object, before the derived part is instantiated. In Python I want to understand what's going on in a case where I have the requirement, that Derived subclasses a given class Base which takes a callable in its __init__ method which it then later invokes. The callable features some parameters which I pass in Derived class's __init__, which is where I also define the callable function. My idea then was to pass the Derived class itself to its Base class after having defined the __call__ operator
class Derived(Base):
def __init__(self, a, b):
def _process(c, d):
do_something with a and b
self.__class__.__call__ = _process
super(Derived, self).__init__(self)
Is this a pythonic way of dealing with this problem?
What is the exact order of initialization here? Does one needs to call super as a first instruction in the __init__ method or is it ok to do it the way I did?
I am confused whether it is considered good practice to use super with or without arguments in python > 3.6
What is the exact order of initialization here?
Well, very obviously the one you can see in your code - Base.__init__() is only called when you explicitely ask for it (with the super() call). If Base also has parents and everyone in the chain uses super() calls, the parents initializers will be invoked according to the mro.
Basically, Python is a "runtime language" - except for the bytecode compilation phase, everything happens at runtime - so there's very few "black magic" going on (and much of it is actually documented and fully exposed for those who want to look under the hood or do some metaprogramming).
Does one needs to call super as a first instruction in the init method or is it ok to do it the way I did?
You call the parent's method where you see fit for the concrete use case - you just have to beware of not using instance attributes (directly or - less obvious to spot - indirectly via a method call that depends on those attributes) before they are defined.
I am confused whether it is considered good practice to use super with or without arguments in python > 3.6
If you don't need backward compatibily, use super() without params - unless you want to explicitely skip some class in the MRO, but then chances are there's something debatable with your design (but well - sometimes we can't afford to rewrite a whole code base just to avoid one very special corner case, so that's ok too as long as you understand what you're doing and why).
Now with your core question:
class Derived(Base):
def __init__(self, a, b):
def _process(c, d):
do_something with a and b
self.__class__.__call__ = _process
super(Derived, self).__init__(self)
self.__class__.__call__ is a class attribute and is shared by all instances of the class. This means that you either have to make sure you are only ever using one single instance of the class (which doesn't seem to be the goal here) or are ready to have totally random results, since each new instance will overwrite self.__class__.__call__ with it's own version.
If what you want is to have each instance's __call__ method to call it's own version of process(), then there's a much simpler solution - just make _process an instance attribute and call it from __call__ :
class Derived(Base):
def __init__(self, a, b):
def _process(c, d):
do_something with a and b
self._process = _process
super(Derived, self).__init__(self)
def __call__(self, c, d):
return self._process(c, d)
Or even simpler:
class Derived(Base):
def __init__(self, a, b):
super(Derived, self).__init__(self)
self._a = a
self._b = b
def __call__(self, c, d):
do_something_with(self._a, self._b)
EDIT:
Base requires a callable in ins init method.
This would be better if your example snippet was closer to your real use case.
But when I call super().init() the call method of Derived should not have been instantiated yet or has it?
Now that's a good question... Actually, Python methods are not what you think they are. What you define in a class statement's body using the def statement are still plain functions, as you can see by yourself:
class Foo:
... def bar(self): pass
...
Foo.bar
"Methods" are only instanciated when an attribute lookup resolves to a class attribute that happens to be a function:
Foo().bar
main.Foo object at 0x7f3cef4de908>>
Foo().bar
main.Foo object at 0x7f3cef4de940>>
(if you wonder how this happens, it's documented here)
and they actually are just thin wrappers around a function, instance and class (or function and class for classmethods), which delegate the call to the underlying function, injecting the instance (or class) as first argument. In CS terms, a Python method is the partial application of a function to an instance (or class).
Now as I mentionned upper, Python is a runtime language, and both def and class are executable statements. So by the time you define your Derived class, the class statement creating the Base class object has already been executed (else Base wouldn't exist at all), with all the class statement block being executed first (to define the functions and other class attributes).
So "when you call super().__init()__", the __call__ function of Base HAS been instanciated (assuming it's defined in the class statement for Base of course, but that's by far the most common case).
I have a simple mixin structure in Python. The code should be pretty self-explaining:
class Base:
def __init__(self):
pass
class MixinA:
def __init__(self):
self.x = 0
self.y = 1
def a(self):
print('A: x = ' + str(self.x) + ', y = ' + str(self.y))
class MixinB:
def __init__(self):
self.x = 2
self.z = 3
def b(self):
print('B: x = ' + str(self.x) + ', z = ' + str(self.z))
class MyFirstMix(MixinA, MixinB, Base):
def __init__(self):
Base.__init__(self)
MixinB.__init__(self)
MixinA.__init__(self)
class MySecondMix(MixinA, Base):
def __init__(self):
Base.__init__(self)
MixinA.__init__(self)
I'd like to improve this a bit, so this leads to 3 questions/problems:
MixinA and MixinB both have a member x. Is there a way to make sure, each of the class sees only its own x? As far as I know: No, there isn't.
It's slightly cumbersome to call the constructor for each mixin in the mixed class. Is there a way to automatically call all constructors or to do something with the same effect?
Is there a way to dynamically mixin something in-line, without explicitly creating a class? I'm looking for a syntax like: mix = Base() with MixinA
If my proposed structure is completely wrong, I'm also open for other recommendations on how to handle mixins.
For python class inherent, I believe there are some tricks you need to know:
Class in python2 and python3 are quite different.
Python2 support old-style class, but python3 support new-style class only. Simply speaking: in python3, classes always inherent from a base class object, even though you do not explicitly use it. Check Difference-between-Old-New-Class.
Method Resolution Order (MRO). This determines how derived class search inherent members and functions. See MRO
super function. Combined with MRO, you can easily call parent member or function, without explicitly know the name of parent class. See Super
Now comes to you questions:
MixinA and MixinB both have a member x. Is there a way to make sure, each of the class sees only its own x?
I don't quit understand your meaning. When you refer a class member, you must refer it through its instance or class. So instance_of_MixinA.x and instance_of_MixinB.x are separated. If you are talking about class MyFirstMix(MixinA, MixinB, Base), it depends on how __init__ function is called. In your code, you first populate x by MixinB and then reset its value by MixinA.
It's slightly cumbersome to call the constructor for each mixin in the mixed class. Is there a way to automatically call all constructors or to do something with the same effect.
Your designation make it impossible. You have to call all constructors.
Is there a way to dynamically mixin something in-line, without explicitly creating a class?
I am not sure. But I can give you some suggestions: try outside __init__ members when def class (python3, if you used python2 take care of super):
class Base:
def __init__(self):
pass
class MixinA:
x = 0
y = 1
class MixinB:
x = 2
z = 3
def b(self):
print('B: x = ' + str(self.x) + ', z = ' + str(self.z))
class MyFirstMix(MixinA, MixinB, Base):
def __init__(self):
super().__init__()
class MySecondMix(MixinA, Base):
def __init__(self):
super().__init__()
The variables outside __init__ behaves quit different from inside variables: outside variables belongs to class and will be populated for all instances of this class, while inside variables belongs only to instance (referred by self when you define class), and will be populated only if __init__ is called. That's why you cannot use super to call all the constructors---super only call the priority parent's __init__. See variables-outsite-inside-init
This is a good solution to Mixin class. In above code, MyFirstMix inherents both MixinA and MixinB whose members are all class members (outside __init__). So instances of MyFirstMix will inherent all class members of MixinA and MixinB without call __init__. Here MixinA and MixinB own same class member x, but the MRO determines that when instances of MyFirstMix refer x, x from MixinA should be returned.
Hope this will be helpful. Thanks!
When your inheritance schemes start to suffer from these sorts of issues it's time to consider using a technique called composition instead. A good readable introduction to the topic here. The Wikipedia example is a bit less accessible, but also useful if you can handle the other programming languages. This StackExchange question also offers useful discussion.
At its simplest, rather than a class inheriting from SomeParent and mixing in the Mixin class, you instead have the SomeParent instances each create an instance of Mixin and use that to access the mixin class's functionality.
I'm trying to dynamically add methods to classes at runtime, and am seeing some issues:
#Here we define a set of symbols within an exec statement and put them into the dictionary d
d = {}
exec "def get_%s(self): return self.%s" % (attr_name, attr) in d
#Now, we bind the get method stored in d['get_%s'] to our object (class)
func = d['get_%s' % (attr_name)].__get__(d['get_%s' % (attr_name)], class)
setattr(class_instance, func.__name__, func)
When I try to call the generated get method, I see the below:
Traceback (most recent call last):
File "Textere_AdvancedExample.py", line 77, in <module>
count = fact.get_counter()
File "<string>", line 1, in get_counter
AttributeError: 'function' object has no attribute '_counter'
Edit
Based on some of the exceptional responses given so far, I think I need to clarify why I'm doing things this way.
I'm trying to build an annotation like the below example:
#getters
#singleton
class A() {
def __init__(self):
self._a = "a"
self._b = "b"
}
Based on the names present in the class, the annotation will build getters for the private class variables at runtime and bind them to the singleton instance.
The strategy I've taken is to have an Application Context class with a set of dicts. Then, the context is passed in to the annotation, which adds the instance & class into these dicts.
On startup, the Application Context is then responsible for reading the dictionaries and then building & binding get methods to the respective singleton object.
Edit2
So this development started after some discussions with friends of mine who are Java developers regarding two libraries in particular: Spring & Lombok
I wanted to see if these particular pieces of functionality could be implemented in Python. So the application context came about originally from trying to get a functionality similar to Spring's autowire annotation. I got this working without issue.
Then, I got the generating the getters and setters and realized that I was going to have a fundamental difference in Python from the Java implementation: Lombok does this at compile time and Python is not compiled. This meant that I had to dynamically generate methods based on what's being annotated and bind them to objects manually, all at runtime. Thus, you see this sort of warping of the Java implementation.
For those interested, The full code can be found here
You can easily dynamically add static methods or class methods:
class A:
pass
#staticmethod
def foo0(x):
return x * 2
a = A()
A.foo = foo0
a.foo(3)
return 6
class A:
val = 3
#classmethod
def foo0(cls, x):
return x * cls.val
a = A()
A.foo = foo0
a.foo(2)
return 6
You can also add specific methods to an instance of a class (almost the same way)
class A:
pass
a = A()
a.foo = (lambda x: 2*x)
a.foo(3)
returns 6
You can also add an instance method to a class, through the use of the types module (in fact, this generic way can be used to create also static and class methods, as well as instance only methods):
class A:
def __init__(self, val):
self.val = val
a = A(3)
A.foo = types.MethodType((lambda self, x: self.val * x), None, A)
a.foo(2)
returns 6
But this is really monkey patching, that is a quick and dirty hack that should only be used when you need to pass slightly changed classes and you are not allowed to change the name. The nice and clean way to add functionalities to a class is inheritance
Just to make this answer better, the above is valid for Python 2.
For Python 3, only the first way to create class and static method can be used, because the types module has lost many types.
And you create an instance method as simply as:
class A:
def __init__(self, val):
self.val = val
a = A(3)
A.foo = (lambda self, x: self.val * x)
a.foo(2)
returns 6. No need for special construct here
exec("%s.__dict__['get_%s'] = lambda self: self.%s" % (class_name, attr_name, attr_name))
In the following example, is there any way the a attribute in A can be accessed by the B class or C inner class?
class A:
def __init__(self, a):
self.a = a
def C_test(self):
for i in range(4):
c = self.C()
class C:
print(self.a)
class B(A):
def __init__(self):
print(self.a)
How come I get this error?
Traceback (most recent call last):
File "/Users/home/Desktop/Pygame/test.py", line 1, in <module>
class A:
File "/Users/home/Desktop/Pygame/test.py", line 10, in A
class C:
File "/Users/home/Desktop/Pygame/test.py", line 11, in C
print(self.a)
NameError: name 'self' is not defined
self is not a special variable name in Python - it's just the name that is typically given to the first argument of a method, which is bound to the object calling that method.
Your self.a in class C doesn't appear inside a method definition where self is listed as an argument, so self has no meaning there.
For your B example, this general idea does work, just not automatically. Python does Python does have broadly similar rules to Java when it comes to single inheritance. The reason it doesn't work automatically here is because of Python's data model - member variables are attached to each unique instance, rather than being an intrinsic part of the class. They're usually, as in your examples, attached to the instance when __init__ runs. So, for your B to have an a, it would need to be attached by A.__init__. Python will automatically run that for you if you don't write a B.__init__, but if you do have that, you need to call up explicitly. The easiest way is this:
class B(A):
def __init__(self):
super().__init__()
Again, you might recognise this idea of explicitly chaining up using super from other languages - although the spelling is a bit different here. But the only major technical difference between Python and Java in this respect is that Python needs the parent constructor to be called for the instance variables to even exist (in Java, they will still exist, but might not have the right value, or even be initialised).
For your C example, Python is very different from Java here. Python doesn't usually use the term 'inner class', although you can define a class inside another class (note that you haven't in your code sample, though) - but instances of the inner class won't be associated with an instance of the outer class. So, they behave a bit more like Java inner static classes. You could associate them explicitly by doing something like this:
class A:
def __init__(self, a):
self.a = a
self.my_C = A.C(self)
class C:
def __init__(self, A_self):
print(A_self.a)
But this isn't exactly common, and depending on exactly what your problem is, there is almost always a better way to solve it than trying to shoehorn Java idioms into Python code.
If you derive a class from your own base-class, you have to call the constructor of the base-class with super to inherit the base-classes attributes:
class A(object):
def __init__(self, a):
self.a = a
class B(A):
def __init__(self):
super(B, self).__init__(self)
After running this you can do this for example:
a = A(1)
b = B()
b.a = a.a
b has declared no attribute a, but inherits it from class A by calling the constructor of A via super in class B.
Now b.a evaluates 1 cause it is set to the value of a.a.
Consider this situation:
I get an object of type A which has the function f:
class A:
def f(self):
print 'in f'
def h(self):
print 'in h'
and I get an instance of this class, but I want to override the f function, yet save the rest of the functionality of A. So what I was thinking was something of the sort:
class B(A):
def __init__(self, a):
#something here
....
def f(self):
print 'in B->f'
and the usage would be:
def main(a):
b = B(a)
b.f() #prints "in B->f"
b.h() #print "in h"
What I want is a sort of copy constructor that gets a parent of the current class (A), and returns an instance of this class (B).
How do you do such a thing? How would the __init__ method look?
Note: this post has been edited by the original poster to incorporate changes suggested in the comments, which is why some of the suggestions look redundant or incorrect.
How you construct an object of subclass B "based on" one of class A depends exclusively on how the latter keeps state, if any, and how do you best get to that state and copy it over. In your example, instances of A are stateless, therefore there is absolutely no work you need to do in B's '__init__'. In a more typical example, say:
class A(object):
def __init__(self):
self._x = 23
self._y = 45
def f(self):
print 'in f,', self._x
def h(self):
print 'in h,', self._y
the state would be in the two instance attributes _x and _y, so those are what you need to copy over:
class B(A):
def __init__(self, a):
self._x = a._x
self._y = a._y
def f(self):
print 'in B->f,', self._x
This is the most common and normal approach, where the subclass accepts and directly implements its state-dependence on the superclass -- it's very straightforward and linear.
You normally look for A's instance state aspects in A's '__init__', because most normal, straightforward Python code establishes instance state at initialization (attributes might be added and removed later, or even from code outside of the class's body, but that's not common and generally not advisable).
It is possible to add a little touch of "magic" (introspection-based programming), e.g...:
class B1(A):
def __init__(self, a):
try: s = a.__getstate__()
except AttributeError: s = a.__dict__
try: self.__setstate__(s)
except AttributeError: self.__dict__.update(s)
getstate is a special method that classes may define -- if they do, it's used (e.g. by pickling) to "get the state" of their instances for serialization purpose (otherwise, the instance's __dict__ is deemed to be the instance's "state"). It may return a dict (in which case the .update call updates self's state), but it may also return anything else if the class also defines a __setstate__ that accepts it (so this code tries that route first, before falling back to the update possibility). Note that in this use case either or both of the special methods would be inherited from A -- I wouldn't define / override them in B (unless there are further subtle goals to be achieved that way of course;-).
Is it worth using these four lines of "magic" in lieu of the simple assignments I first suggested? Mostly, no -- simplicity is preferable. But if A does anything special or is subject to external code altering its state, this solution can be more powerful and general (that's what you're buying by accepting its complication). So, you have to know if the latter case applies (and then "go for the big guns" of the special state-related methods), or if A and its instances are "pretty normal vanilla ones", in which case I would strongly recommend choosing simplicity and clarity instead.
Try this:
class A:
def f(self):
print("in f")
def h(self):
print("in h")
class B(A):
def f(self):
print("in B:f")
def test(x):
x.f()
x.h()
test(A())
test(B())
Note, I'm using Python 3, which is the reason for print taking the arguments in parenthesis.
Output:
in f
in h
in B:f
in h
You need to put the self argument into the argument list for instance methods in python.
Once you've done that, it will just work, because all methods are virtual in python.