Related
In C++, given a class hierarchy, the most derived class's ctor calls its base class ctor which then initialized the base part of the object, before the derived part is instantiated. In Python I want to understand what's going on in a case where I have the requirement, that Derived subclasses a given class Base which takes a callable in its __init__ method which it then later invokes. The callable features some parameters which I pass in Derived class's __init__, which is where I also define the callable function. My idea then was to pass the Derived class itself to its Base class after having defined the __call__ operator
class Derived(Base):
def __init__(self, a, b):
def _process(c, d):
do_something with a and b
self.__class__.__call__ = _process
super(Derived, self).__init__(self)
Is this a pythonic way of dealing with this problem?
What is the exact order of initialization here? Does one needs to call super as a first instruction in the __init__ method or is it ok to do it the way I did?
I am confused whether it is considered good practice to use super with or without arguments in python > 3.6
What is the exact order of initialization here?
Well, very obviously the one you can see in your code - Base.__init__() is only called when you explicitely ask for it (with the super() call). If Base also has parents and everyone in the chain uses super() calls, the parents initializers will be invoked according to the mro.
Basically, Python is a "runtime language" - except for the bytecode compilation phase, everything happens at runtime - so there's very few "black magic" going on (and much of it is actually documented and fully exposed for those who want to look under the hood or do some metaprogramming).
Does one needs to call super as a first instruction in the init method or is it ok to do it the way I did?
You call the parent's method where you see fit for the concrete use case - you just have to beware of not using instance attributes (directly or - less obvious to spot - indirectly via a method call that depends on those attributes) before they are defined.
I am confused whether it is considered good practice to use super with or without arguments in python > 3.6
If you don't need backward compatibily, use super() without params - unless you want to explicitely skip some class in the MRO, but then chances are there's something debatable with your design (but well - sometimes we can't afford to rewrite a whole code base just to avoid one very special corner case, so that's ok too as long as you understand what you're doing and why).
Now with your core question:
class Derived(Base):
def __init__(self, a, b):
def _process(c, d):
do_something with a and b
self.__class__.__call__ = _process
super(Derived, self).__init__(self)
self.__class__.__call__ is a class attribute and is shared by all instances of the class. This means that you either have to make sure you are only ever using one single instance of the class (which doesn't seem to be the goal here) or are ready to have totally random results, since each new instance will overwrite self.__class__.__call__ with it's own version.
If what you want is to have each instance's __call__ method to call it's own version of process(), then there's a much simpler solution - just make _process an instance attribute and call it from __call__ :
class Derived(Base):
def __init__(self, a, b):
def _process(c, d):
do_something with a and b
self._process = _process
super(Derived, self).__init__(self)
def __call__(self, c, d):
return self._process(c, d)
Or even simpler:
class Derived(Base):
def __init__(self, a, b):
super(Derived, self).__init__(self)
self._a = a
self._b = b
def __call__(self, c, d):
do_something_with(self._a, self._b)
EDIT:
Base requires a callable in ins init method.
This would be better if your example snippet was closer to your real use case.
But when I call super().init() the call method of Derived should not have been instantiated yet or has it?
Now that's a good question... Actually, Python methods are not what you think they are. What you define in a class statement's body using the def statement are still plain functions, as you can see by yourself:
class Foo:
... def bar(self): pass
...
Foo.bar
"Methods" are only instanciated when an attribute lookup resolves to a class attribute that happens to be a function:
Foo().bar
main.Foo object at 0x7f3cef4de908>>
Foo().bar
main.Foo object at 0x7f3cef4de940>>
(if you wonder how this happens, it's documented here)
and they actually are just thin wrappers around a function, instance and class (or function and class for classmethods), which delegate the call to the underlying function, injecting the instance (or class) as first argument. In CS terms, a Python method is the partial application of a function to an instance (or class).
Now as I mentionned upper, Python is a runtime language, and both def and class are executable statements. So by the time you define your Derived class, the class statement creating the Base class object has already been executed (else Base wouldn't exist at all), with all the class statement block being executed first (to define the functions and other class attributes).
So "when you call super().__init()__", the __call__ function of Base HAS been instanciated (assuming it's defined in the class statement for Base of course, but that's by far the most common case).
I was trying some examples on class inheritance and stumbled, on how to access the methods. In tutorials, the proper form is always Class().method(), however apparently, that's not the only solution.
class A():
def open(self):
print("class A method open")
class B():
def open(self):
print("class B method open")
def close(self):
print("class B method close")
class D(A, B):
def end_reached(self):
print("end reached")
## Why do both of the following lines work & produce the same output?
D().open()
D.open(1)
I expected the last line to give an error, however, the output is the same as from the line above. It did give an error on missing parameter, if the method gets called like D.open(), but giving it just any parameter works.
Is there any difference between the two lines?
The difference lays in the descriptor protocol, specifically in how descriptors provide Python's object-oriented features.
open is a class attribute of various classes; in each case, though, it has the type function. That type implements the descriptor protocol, via the definition of function.__get__, by returning either a function object or a method object, depending on whether open is accessed via the class itself or an instance of the class.
Instance lookup
Given
d = D()
the attribute lookup d.open is equivalent to
type(d).__dict__['open'].__get__(d, D)
which returns a method which wraps the original function. When called, the method passes a reference to d and any of its own arguments to the wrapped function.
f = d.open # The return value of __get__
f() # open(d)
Class lookup
The attribute lookup D.open is equivalent to
D.__dict__['open'].__get__(None, D)
which returns the function open itself.
f = D.open # The return value of __get__
f(1) # open(1)
In order to learn about this, I have to go in depth about the properties of functions and methods.
Accessing a class field of a class via its instances checks if there are __set__ and/or __get__ methods. If they are, the call happens via __get__.
Functions work this way: if a function is a class field of a class (in short, methods), they are called via function.__get__(object)(*args, **kwargs).
And here, we have a difference between Python 2 and Python 3:
Python 2 unconditionally prepends the given object to the argumends and calls the underlying function with function(object, *args, **kwargs). object gets assigned to self then, which by convention is the name of the first parameter of a method.
Python 3 checks if the given object is an instance of the class as it should be. If it isn't, we get an exception. If it is, the call gets processed as above.
Example:
def outer_f(self):
return "f called with", self
class C(object):
f = outer_f
def g(self):
return "g called with", self
c = C()
print(outer_f(c))
print(C.f(c))
print(C.f.__get__(c)())
print(c.f())
print(C.g(c))
print(C.g.__get__(c)())
print(c.g())
class A():
def open(self):
print(self)
print("class A method open")
class B():
def open(self):
print("class B method open")
def close(self):
print("class B method close")
class D(A, B):
def end_reached(self):
print("end reached")
## Why do both of the following lines work & produce the same output?
D().open() # Okay
D.open(1) # BAD
output
<__main__.D object at 0x0045FE90>
class A method open
1
class A method ope
You might want to look at a static method
class A():
#staticmethod
def open():
print("open ran")
class D(A):
def end_reached(self):
print("end reached")
D.open() # Good
output
open ran
That's not an inheritance problem.
As you can see in your method declaration, first argument is self.
D() just creates an object of class D. If you do anything on the object, that object is automatically passed as first argument.
However, when doing D.open(1) you don't invoke the method on the object, so the first argument in the method is first argument passed to it == your code assumes self=1. Because your methods behave like static methods (you don't use self inside them), you don't access self's inside, so the code doesn't break.
But if your open method invoked anything from self (another method or a field),
then you would get an attribute error - because int object probably doesn't have a method like that.
Examples from build-in classes:
"aaa".split()
str.split("aaa")
They both behave the same.
"aaa" is of type str, so interpreted looks into class str for the method and automatically passes "aaa" as self. (Just like your D() is of type D. And d=D(), then d.open() will still know the type.)
The other one shows which class to ask for the method (str) but still needs an argument self. The way we define methods shows that "self" is really an argument as well, so we can just pass it.
Assume I have a class CLASS and I create a methodfnc in it
class CLASS():
def __init__(self,arg):
#initizalize
self.arg = arg
def fnc(self):
print(self.arg)
if I (in my class) wants to call fnc in a method prnt I can do it in two different ways, one using self.fnc and one just using fnc
class CLASS():
def __init__(self,arg):
#initizalize
self.arg = arg
def fnc(self):
print(self.arg)
def prnt(self):
self.fnc() #Method one
fnc() #Method two
which both seems to be working. I do know the self argument and how it works, but I do not understand the difference of the two method/function calls.
This should not work. It works because CLASS in your code is not actually a class since you used keyword def instead of class to define it.
What you actually did is define a function CLASS, which when executed defines some other functions.
To correct this declare your class like this:
class CLASS:
And your second call will raise a NameError probably because fnc does not exist in the scope of your method.
I am a beginner in Python and using Lutz's book to understand OOPS in Python. This question might be basic, but I'd appreciate any help. I researched SO and found answers on "how", but not "why."
As I understand from the book, if Sub inherits Super then one need not call superclass' (Super's) __init__() method.
Example:
class Super:
def __init__(self,name):
self.name=name
print("Name is:",name)
class Sub(Super):
pass
a = Sub("Harry")
a.name
Above code does assign attribute name to the object a. It also prints the name as expected.
However, if I modify the code as:
class Super:
def __init__(self,name):
print("Inside Super __init__")
self.name=name
print("Name is:",name)
class Sub(Super):
def __init__(self,name):
Super(name) #Call __init__ directly
a = Sub("Harry")
a.name
The above code doesn't work fine. By fine, I mean that although Super.__init__() does get called (as seen from the print statements), there is no attribute attached to a. When I run a.name, I get an error, AttributeError: 'Sub' object has no attribute 'name'
I researched this topic on SO, and found the fix on Chain-calling parent constructors in python and Why aren't superclass __init__ methods automatically invoked?
These two threads talk about how to fix it, but they don't provide a reason for why.
Question: Why do I need to call Super's __init__ using Super.__init__(self, name) OR super(Sub, self).__init__(name) instead of a direct call Super(name)?
In Super.__init__(self, name) and Super(name), we see that Super's __init__() gets called, (as seen from print statements), but only in Super.__init__(self, name) we see that the attribute gets attached to Sub class.
Wouldn't Super(name) automatically pass self (child) object to Super? Now, you might ask that how do I know that self is automatically passed? If I modify Super(name) to Super(self,name), I get an error message that TypeError: __init__() takes 2 positional arguments but 3 were given. As I understand from the book, self is automatically passed. So, effectively, we end up passing self twice.
I don't know why Super(name) doesn't attach name attribute to Sub even though Super.__init__() is run. I'd appreciate any help.
For reference, here's the working version of the code based on my research from SO:
class Super:
def __init__(self,name):
print("Inside __init__")
self.name=name
print("Name is:",name)
class Sub(Super):
def __init__(self,name):
#Super.__init__(self, name) #One way to fix this
super(Sub, self).__init__(name) #Another way to fix this
a = Sub("Harry")
a.name
PS: I am using Python-3.6.5 under Anaconda Distribution.
As I understand from the book, if Sub inherits Super then one need not call superclass' (Super's) __init__() method.
This is misleading. It's true that you aren't required to call the superclass's __init__ method—but if you don't, whatever it does in __init__ never happens. And for normal classes, all of that needs to be done. It is occasionally useful, usually when a class wasn't designed to be inherited from, like this:
class Rot13Reader:
def __init__(self, filename):
self.file = open(filename):
def close(self):
self.file.close()
def dostuff(self):
line = next(file)
return codecs.encode(line, 'rot13')
Imagine that you want all the behavior of this class, but with a string rather than a file. The only way to do that is to skip the open:
class LocalRot13Reader(Rot13Reader):
def __init__(self, s):
# don't call super().__init__, because we don't have a filename to open
# instead, set up self.file with something else
self.file = io.StringIO(s)
Here, we wanted to avoid the self.file assignment in the superclass. In your case—as with almost all classes you're ever going to write—you don't want to avoid the self.name assignment in the superclass. That's why, even though Python allows you to not call the superclass's __init__, you almost always call it.
Notice that there's nothing special about __init__ here. For example, we can override dostuff to call the base class's version and then do extra stuff:
def dostuff(self):
result = super().dostuff()
return result.upper()
… or we can override close and intentionally not call the base class:
def close(self):
# do nothing, including no super, because we borrowed our file
The only difference is that good reasons to avoid calling the base class tend to be much more common in normal methods than in __init__.
Question: Why do I need to call Super's __init__ using Super.__init__(self, name) OR super(Sub, self).__init__(name) instead of a direct call Super(name)?
Because these do very different things.
Super(name) constructs a new Super instance, calls __init__(name) on it, and returns it to you. And you then ignore that value.
In particular, Super.__init__ does get called one time either way—but the self it gets called with is that new Super instance, that you're just going to throw away, in the Super(name) case, while it's your own self in the super(Sub, self).__init__(name) case.
So, in the first case, it sets the name attribute on some other object that gets thrown away, and nobody ever sets it on your object, which is why self.name later raises an AttributeError.
It might help you understand this if you add something to both class's __init__ methods to show which instance is involved:
class Super:
def __init__(self,name):
print(f"Inside Super __init__ for {self}")
self.name=name
print("Name is:",name)
class Sub(Super):
def __init__(self,name):
print(f"Inside Sub __init__ for {self}")
# line you want to experiment with goes here.
If that last line is super().__init__(name), super(Sub, self).__init__name), or Super.__init__(self, name), you will see something like this:
Inside Sub __init__ for <__main__.Sub object at 0x10f7a9e80>
Inside Super __init__ for <__main__.Sub object at 0x10f7a9e80>
Notice that it's the same object, the Sub at address 0x10f7a9e80, in both cases.
… but if that last line is Super(name):
Inside Sub __init__ for <__main__.Sub object at 0x10f7a9ea0>
Inside Super __init__ for <__main__.Super object at 0x10f7a9ec0>
Now we have two different objects, at different addresses 0x10f7a9ea0 and 0x10f7a9ec0, and with different types.
If you're curious about what the magic all looks like under the covers, Super(name) does something like this (oversimplifying a bit and skipping over some steps1):
_newobj = Super.__new__(Super)
if isinstance(_newobj, Super):
Super.__init__(_newobj, name)
… while super(Sub, self).__init__(name) does something like this:
_basecls = magically_find_next_class_in_mro(Sub)
_basecls.__init__(self, name)
As a side note, if a book is telling you to use super(Sub, self).__init__(name) or Super.__init__(self, name), it's probably an obsolete book written for Python 2.
In Python 3, you just do this:
super().__init__(name): Calls the correct next superclass by method resolution order. You almost always want this.
super(Sub, self).__init__(name): Calls the correct next superclass—unless you make a mistake and get Sub wrong there. You only need this if you're writing dual-version code that has to run in 2.7 as well as 3.x.
Super.__init__(self, name): Calls Super, whether it's the correct next superclass or not. You only need this if the method resolution order is wrong and you have to work around it.2
If you want to understand more, it's all in the docs, but it can be a bit daunting:
__new__
__init__
super (also see Raymond Hettinger's blog post)
method invocation (also see the HOWTO)
The original introduction to super, __new__, and all the related features was very helpful to me in understanding all of this. I'm not sure if it'll be as helpful to someone who's not coming at this already understanding old-style Python classes, but it's pretty well written, and Guido (obviously) knows what he's talking about, so it might be worth reading.
1. The biggest cheat in this explanation is that super actually returns a proxy object that acts like _baseclass bound to self in the same way methods are bound, which can be used to bind methods, like __init__. This is useful/interesting knowledge if you know how methods work, but probably just extra confusion if you don't.
2. … or if you're working with old-style classes, which don't support super (or proper method-resolution order). This never comes up in Python 3, which doesn't have old-style classes. But, unfortunately, you will see it in lots of tkinter examples, because the best tutorial is still Effbot's, which was written for Python 2.3, when Tkinter was all old-style classes, and has never been updated.
Super(name) is not a "direct call" to the superclass __init__. After all, you called Super, not Super.__init__.
Super.__init__ takes an uninitialized Super instance and initializes it. Super creates and initializes a new, completely separate instance from the one you wanted to initialize (and then you immediately throw the new instance away). The instance you wanted to initialize is untouched.
Super(name) instantiates a new instance of super. Think of this example:
def __init__(self, name):
x1 = Super(name)
x2 = Super("some other name")
assert x1 is not self
assert x2 is not self
In order to explicitly call The Super's constructor on the current instance, you'd have to use the following syntax:
def __init__(self, name):
Super.__init__(self, name)
Now, maybe you don't want read further if you are a beginner.
If you do, you will see that there is a good reason to use super(Sub, self).__init__(name) (or super().__init__(name) in Python 3) instead of Super.__init__(self, name).
Super.__init__(self, name) works fine, as long as you are certain that Super is in fact your superclass. But in fact, you don't know ever that for sure.
You could have the following code:
class Super:
def __init__(self):
print('Super __init__')
class Sub(Super):
def __init__(self):
print('Sub __init__')
Super.__init__(self)
class Sub2(Super):
def __init__(self):
print('Sub2 __init__')
Super.__init__(self)
class SubSub(Sub, Sub2):
pass
You would now expect that SubSub() ends up calling all of the above constructors, but it does not:
>>> x = SubSub()
Sub __init__
Super __init__
>>>
To correct it, you'd have to do:
class Super:
def __init__(self):
print('Super __init__')
class Sub(Super):
def __init__(self):
print('Sub __init__')
super().__init__()
class Sub2(Super):
def __init__(self):
print('Sub2 __init__')
super().__init__()
class SubSub(Sub, Sub2):
pass
Now it works:
>>> x = SubSub()
Sub __init__
Sub2 __init__
Super __init__
>>>
The reason is that the super class of Sub is declared to be Super, in case of multiple inheritance in class SubSub, Python's MRO establishes the inheritance as: SubSub inherits from Sub, which inherits from Sub2, which inherits from Super, which inherits from object.
You can test that:
>>> SubSub.__mro__
(<class '__main__.SubSub'>, <class '__main__.Sub'>, <class '__main__.Sub2'>, <class '__main__.Super'>, <class 'object'>)
Now, the super() call in constructors of each of the classes finds the next class in the MRO so that the constructor of that class can be called.
See https://www.python.org/download/releases/2.3/mro/
I am having some problem using multiple inheritance in Python and can't understand what I am doing wrong.
I have three classes A,B,C defined as follows it does not work.
class A(object):
def __init__(**kwargs):
.
.
class B(object):
def __init__(**kwargs):
# prepare a dictionary "options" with the options used to call A
super(B,self).__init__(**options)
def coolmethod(x):
#some cool stuff
For A and B I don't have any problems.
I want to create a third class C that inherits both from A and B
so that I can the coolmethod defined in B, but would like to use the constructor defined in A.
Trying to define class C(A,B) does not work because the MRO is not defined.
But defining class C(B,A) does not allow me to use A.init rather than B.init.
How can I solve the issue?
You can call A.__init__() directly instead of using super() in C:
class C(B,A):
def __init__(self, **kwargs):
A.__init__(self, **kwargs)
You can use
class A(object):
def __init__(self, **kwargs):
super(A, self).__init__(**kwargs)
if you expect multiple inheritance from A and something else. This way, A.__init__ will always be called.
The order is important because of the way method resolution works in python. If you have C inherit from (A, B), it means that if you invoke a method on C that exists both on A and B, the one on A is selected (it has precedence). If you write super(A, self).method in class A, it means you want to extend functionality provided by method. Therefore, it would be strange to skip over one such extension if both A and B had such extensions and C inherited from both. That's why when you call C.method, it will execute A.method, which will call B.method when it invokes super(A, self).method. In other terms, it's as if A inherited from B for the purpose of method extension. This is different when C inherits from (B, A).
Note that __init__'s first argument should always be self, just like for every method.