We Keep Coding

Why does `type.__new__` call `__init_subclass__`?

I noticed that I couldn't use __init_subclass__ with Django model classes in quite the way I wanted to. It seems that the metaclass hasn't finished creating a child class by the time that a parent class' __init_subclass__ method is run. While I understand what the issue is, and can circumvent it by making a custom metaclass, what I don't understand is why!
In my head, I tend to think that any calls like __new__ should be done before any calls like __init__ happen. But this isn't the case for metaclasses and __init_subclass__, as demonstrated here:
class MetaMeta(type):
print('parsing MetaMeta')
def __call__(cls, *args, **kwargs):
print('entering MetaMeta.__call__')
instance = super().__call__(*args, **kwargs)
print('leaving MetaMeta.__call__')
return instance
class Meta(type, metaclass=MetaMeta):
print('parsing Meta')
def __init__(self, *args, **kwargs):
print(' entering Meta.__init__')
super().__init__(*args, **kwargs)
print(' leaving Meta.__init__')
def __new__(cls, *args, **kwargs):
print(f' entering Meta.__new__')
instance = super().__new__(cls, *args, **kwargs)
print(' leaving Meta.__new__')
return instance
class Parent(object, metaclass=Meta):
print('parsing Parent')
def __init_subclass__(cls, *args, **kwargs):
print(' entering Parent.__init_subclass__')
super().__init_subclass__(*args, **kwargs)
print(' leaving Parent.__init_subclass__')
class Child(Parent):
print('parsing Child')
Which results in:
parsing MetaMeta
parsing Meta
parsing Parent
entering MetaMeta.__call__
entering Meta.__new__
leaving Meta.__new__
entering Meta.__init__
leaving Meta.__init__
leaving MetaMeta.__call__
parsing Child
entering MetaMeta.__call__
entering Meta.__new__
entering Parent.__init_subclass__
leaving Parent.__init_subclass__
leaving Meta.__new__
entering Meta.__init__
leaving Meta.__init__
leaving MetaMeta.__call__
A metaclass can still be setting up the class in Meta.__new__ after __init_subclass__ is called. Which seems odd to me. Why is that the case, and is there any way to provide code in Parent (without a custom metaclass) that is run completely after Meta.__new__ (and probably before Meta.__init__)?
Or am I missing something completely?
FYI, I found some related topics, but not quite what I was looking for:
The call order of python3 metaclass
Arguments of __new__ and __init__ for metaclasses
https://docs.python.org/3/reference/datamodel.html#customizing-class-creation
Perhaps a more concise way to ask this question is "why does Python (v3.9 at least) have Meta.__new__ invoke Parent.__init_subclass__, instead of having MetaMeta.__call__ invoke it immediately after __new__ is complete?
Note that after asking, I did find some python.org discussion around this topic, but I don't think they clarify why:
https://bugs.python.org/issue42775
https://mail.python.org/archives/list/python-dev#python.org/thread/ZMRRNSFSLJZDGGZ66CFCYQBINU62CDNX/

Tricky.
So, it is this way, because it is made so in the language. When adding features for the class creation processes, people did not allow for customization of when __init_subclass__, or the decriptors __set_name__, or the final linearization (mro) is calculated: that is all done at once inside type.__new__ - and any metaclass written will have to call type.__new__ at some point.
However, there is a possible workaround: if type.__new__ won't "see" an __init_subclass__ method, it won't be called.
Therefore, it is possible for a child metaclass to stash away __init_subclass__, call the parent __new__, and then, before leaving its own __new__ to restore and call __init_subclass__.
So, if the problem is specifcially that you need __init_subclass__ to run after Django's metaclass __new__ is completly done, I can think of two options, which both involve inheriting from Django's ORM metaclass, modifying it, and using that as the metaclass for your models.
Then the first option is simply to use another method name than __init_subclass__ in your project. Your custom metaclass calls super().__new__() , Django and Python do their thing, and you call your __init_subclass2__ (or whaterver name you pick).
I think this is the most maintainable and straightforward way to go.
The second option would be what I mentioned earlier: your __new__ method checks all bases for any occurrence of __init_subclass__, delete then temporarily from the classes where they are, storing the original methods, call super().__new__(), then restore the __init_subclass__ methods, and call them afterwards. This has the advantage of working with existing __init_subclass__ in the hyerarchy as they are (as long as the classes themselves are written in Python and not native code: in which case temporarily removing the method won't work). It has the severe disadvantage of ou having to scan the mro by yourself (ou will have to re-do the linearization) searching all existing __init_subclass__ and restoring them afterwards - there might be some hard to spot corner cases.

Why is Python super used in the child's init method?

According to Python docs super()
is useful for accessing inherited methods that have been overridden in
a class.
I understand that super refers to the parent class and it lets you access parent methods. My question is why do people always use super inside the init method of the child class? I have seen it everywhere. For example:
class Person:
def __init__(self, name):
self.name = name
class Employee(Person):
def __init__(self, **kwargs):
super().__init__(name=kwargs['name']) # Here super is being used
def first_letter(self):
return self.name[0]
e = Employee(name="John")
print(e.first_letter())
I can accomplish the same without super and without even an init method:
class Person:
def __init__(self, name):
self.name = name
class Employee(Person):
def first_letter(self):
return self.name[0]
e = Employee(name="John")
print(e.first_letter())
Are there drawbacks with the latter code? It looks so much cleanr to me. I don't even have to use the boilerplate **kwargs and kwargs['argument'] syntax.
I am using Python 3.8.
Edit: Here's another stackoverflow questions which has code from different people who are using super in the child's init method. I don't understand why. My best guess is there's something new in Python 3.8.

The child might want to do something different or more likely additional to what the super class does - in this case the child must have an __init__.
Calling super’s init means that you don’t have to copy/paste (with all the implications for maintenance) that init in the child’s class, which otherwise would be needed if you wanted some additional code in the child init.
But note there are complications about using super’s init if you use multiple inheritance (e.g. which super gets called) and this needs care. Personally I avoid multiple inheritance and keep inheritance to aminimum anyway - it’s easy to get tempted into creating multiple levels of inheritance/class hierarchy but my experience is that a ‘keep it simple’ approach is usually much better.

The potential drawback to the latter code is that there is no __init__ method within the Employee class. Since there is none, the __init__ method of the parent class is called. However, as soon as an __init__ method is added to the Employee class (maybe there's some Employee-specific attribute that needs to be initialized, like an id_number) then the __init__ method of the parent class is overridden and not called (unless super.__init__() is called) and then an Employee will not have a name attribute.

The correct way to use super here is for both methods to use super. You cannot assume that Person is the last (or at least, next-to-last, before object) class in the MRO.
class Person:
def __init__(self, name, **kwargs):
super().__init__(**kwargs)
self.name = name
class Employee(Person):
# Optional, since Employee.__init__ does nothing
# except pass the exact same arguments "upstream"
def __init__(self, **kwargs):
super().__init__(**kwargs)
def first_letter(self):
return self.name[0]
Consider a class definition like
class Bar:
...
class Foo(Person, Bar):
...
The MRO for Foo looks like [Foo, Person, Bar, object]; the call to super().__init__ inside Person.__init__ would call Bar.__init__, not object.__init__, and Person has no way of knowing if values in **kwargs are meant for Bar, so it must pass them on.

Is __init__ a class method?

I was looking into Python's super method and multiple inheritance. I read along something like when we use super to call a base method which has implementation in all base classes, only one class' method will be called even with variety of arguments. For example,
class Base1(object):
def __init__(self, a):
print "In Base 1"
class Base2(object):
def __init__(self):
print "In Base 2"
class Child(Base1, Base2):
def __init__(self):
super(Child, self).__init__('Intended for base 1')
super(Child, self).__init__()# Intended for base 2
This produces TyepError for the first super method. super would call whichever method implementation it first recognizes and gives TypeError instead of checking for other classes down the road. However, this will be much more clear and work fine when we do the following:
class Child(Base1, Base2):
def __init__(self):
Base1.__init__(self, 'Intended for base 1')
Base2.__init__(self) # Intended for base 2
This leads to two questions:
Is __init__ method a static method or a class method?
Why use super, which implicitly choose the method on it's own rather than explicit call to the method like the latter example? It looks lot more cleaner than using super to me. So what is the advantage of using super over the second way(other than writing the base class name with the method call)

super() in the face of multiple inheritance, especially on methods that are present on object can get a bit tricky. The general rule is that if you use super, then every class in the hierarchy should use super. A good way to handle this for __init__ is to make every method take **kwargs, and always use keyword arguments everywhere. By the time the call to object.__init__ occurs, all arguments should have been popped out!
class Base1(object):
def __init__(self, a, **kwargs):
print "In Base 1", a
super(Base1, self).__init__()
class Base2(object):
def __init__(self, **kwargs):
print "In Base 2"
super(Base2, self).__init__()
class Child(Base1, Base2):
def __init__(self, **kwargs):
super(Child, self).__init__(a="Something for Base1")
See the linked article for way more explanation of how this works and how to make it work for you!
Edit: At the risk of answering two questions, "Why use super at all?"
We have super() for many of the same reasons we have classes and inheritance, as a tool for modularizing and abstracting our code. When operating on an instance of a class, you don't need to know all of the gritty details of how that class was implemented, you only need to know about its methods and attributes, and how you're meant to use that public interface for the class. In particular, you can be confident that changes in the implementation of a class can't cause you problems as a user of its instances.
The same argument holds when deriving new types from base classes. You don't want or need to worry about how those base classes were implemented. Here's a concrete example of how not using super might go wrong. suppose you've got:
class Foo(object):
def frob(self):
print "frobbign as a foo"
class Bar(object):
def frob(self):
print "frobbign as a bar"
and you make a subclass:
class FooBar(Foo, Bar):
def frob(self):
Foo.frob(self)
Bar.frob(self)
Everything's fine, but then you realize that when you get down to it,
Foo really is a kind of Bar, so you change it
class Foo(Bar):
def frob(self):
print "frobbign as a foo"
Bar.frob(self)
Which is all fine, except that in your derived class, FooBar.frob() calls Bar.frob() twice.
This is the exact problem super() solves, it protects you from calling superclass implementations more than once (when used as directed...)

As for your first question, __init__ is neither a staticmethod nor a classmethod; it is an ordinary instance method. (That is, it receives the instance as its first argument.)
As for your second question, if you want to explicitly call multiple base class implementations, then doing it explicitly as you did is indeed the only way. However, you seem to be misunderstanding how super works. When you call super, it does not "know" if you have already called it. Both of your calls to super(Child, self).__init__ call the Base1 implementation, because that is the "nearest parent" (the most immediate superclass of Child).
You would use super if you want to call just this immediate superclass implementation. You would do this if that superclass was also set up to call its superclass, and so on. The way to use super is to have each class call only the next implementation "up" in the class hierarchy, so that the sequence of super calls overall calls everything that needs to be called, in the right order. This type of setup is often called "cooperative inheritance", and you can find various articles about it online, including here and here.

Python Inheritance and __init__

I'm learning Python and I've found something about how Python constructs a sub class which confuses me.
I have a class that inherits from the list class as follows.
class foo(list):
def __init__(self, a_bar):
list.__init__([])
self.bar = a_bar
I know that list.__init__([]) needs to be there but I'm confused about it. It seems to me that this line will just create a new list object and then assign it to nothing, so I would suspect that it would just get garbage collected. How does Python know that this list is part of my object? I suspect that there is something happening behind the scenes and I'd like to know what it is.

The multiple-inheritance-safe way of doing it is:
class foo(list):
def __init__(self, a_bar):
super(foo, self).__init__()
...
which, perhaps, makes it clearer that you're calling the baseclass ctor.

You usually do this when subclassing and overriding the __init__() function:
list.__init__(self)
If you're using Python 3, you can make use of super():
super().__init__()

The actual object is not created with __init__ but with __new__. __init__ is not for creating the object itself but for initializing it --- that is, adding attributes, etc. By the time __init__ is called, __new__ has already been called, so in your example the list was already created before your code even runs. __init__ shouldn't return anything because it's supposed to initialize the object "in-place" (by mutating it), so it works by side-effects. (See a previous question and the documentation.)

You're partly right:
list.__init__([])
"creates a new list object." But this code is wrong. The correct code _should_be:
list.__init__(self)
The reason you need it to be there is because you're inheriting from a list that has it's own __init__() method where it (presumably) does important to initialize itself. When you define your own __init__() method, you're effectively overriding the inherited method of the same name. In order to make sure that the parent class's __init__() code is executed as well, you need to call that parent class's __init__().
There are several ways of doing this:
#explicitly calling the __init__() of a specific class
#"list"--in this case
list.__init__(self, *args, **kwargs)
#a little more flexible. If you change the parent class, this doesn't need to change
super(foo, self).__init__(*args, **kwargs)
For more on super() see this question, for guidance on the pitfalls of super, see this article.

understanding python self and init

As might be familiar to most of you, this is from Mark Pilgrim's book DIP, chapter 5
class FileInfo(UserDict):
"store file metadata"
def __init__(self, filename=None):
UserDict.__init__(self)
self["name"] = filename
Well I am new to python, coming from basic C background and having confusion understanding it. Stating what I understand, before what I don't understand.
Statement 0: FileInfo is inheriting from class UserDict
Statement 1: __init__ is not a constructor, however after the class instantiates, this is the first method that is defined.
Statement2: self is almost like this
Now the trouble:
as per St1 init is defined as the first function.
UserDict.__init__(self)
Now within the same function __init__ why is the function being referenced, there is no inherent recursion I guess. Or is it trying to override the __init__ method of the class UserDict which the class FileInfo has inherited and put an extra parameter(key value pair) of filename and reference it to the filename being passed to __init__ method.
I am partly sure, I have answered my question, however as you can sense there is confusion, would be great if someone can explain me how to rule this confusion out with some more advanced use case and detailed example of how generally code is written.

You're correct, the __init__ method is not a constructor, it's an initializer called after the object is instantiated.
In the code you've presented, the __init__ method on the FileInfo class is extending the functionality of the __init__ method of the base class, UserDict. By calling the base __init__ method, it executes any code in the base class's initialization, and then adds its own. Without a call to the base class's __init__ method, only the code explicitly added to FileInfo's __init__ method would be called.
The conventional way to do this is by using the super method.
class FileInfo(UserDict):
"store file metadata"
def __init__(self, filename=None):
super(UserDict, self).__init__()
self["name"] = filename
A common use case is returning extra values or adding additional functionality. In Django's class based views, the method get_context_data is used to get the data dictionary for rendering templates. So in an extended method, you'd get whatever values are returned from the base method, and then add your own.
class MyView(TemplateView):
def get_context_data(self, **kwargs):
context = super(MyClass, self).get_context_data(**kwargs)
context['new_key'] = self.some_custom_method()
return kwargs
This way you do not need to reimplement the functionality of the base method when you want to extend it.

Creating an object in Python is a two-step process:
__new__(self, ...) # constructor
__init__(self, ...) # initializer
__new__ has the responsibility of creating the object, and is used primarily when the object is supposed to be immutable.
__init__ is called after __new__, and does any further configuration needed. Since most objects in Python are mutable, __new__ is usually skipped.
self refers to the object in question. For example, if you have d = dict(); d.keys() then in the keys method self would refer to d, not to dict.
When a subclass has a method of the same name as its parent class, Python calls the subclass' method and ignores the parent's; so if the parent's method needs to be called, the subclass method must call it.

"Or is it trying to override the init method of the class UserDict which the class FileInfo has inherited and put an extra parameter(key value pair) of filename and reference it to the filename being passed to init method."
It's exactly that. UserDict.__init__(self) calls the superclass init method.
Since you come from C, maybe you're not well experienced with OOP, so you could read this article : http://en.wikipedia.org/wiki/Inheritance_(object-oriented_programming) to understand the inheritance principle better (and the "superclass" term I used).

.. the self variable represents the instance of the object itself. In python this is not a hidden parameter as in other languages. You have to declare it explicitly. When you create an instance of the FileInfo class and call its methods, it will be passed automatically,
The __init__ method is roughly what represents a constructor in Python.

The __init__ method of FileInfo is overriding the __init__ method of UserDict.
Then FileInfo.__init__ calls UserDict.__init__ on the newly created FileInfo instance (self). This way all properties and magic available to UserDict are now available to that FileInfo instance (ie. they are inherited from UserDict).
The last line is the reason for overriding UserDict.__init__ : UserDict does not create the wanted property self.filename.

When you call __init__ method for a class that is inheriting from a base class, you generally modify the ancestor class and as a part of customization, you extend the ancestor's init method with proper arguements.

__init__ is not a constructor, however after the class instantiates, this is the first method that is defined.
This method is called when an instance is being initialized, after __new__ (i.e. when you call ClassName()). I'm not sure what difference there is as opposed to a constructor.
Statement2: self is almost like this
Yes but it is not a language construct. The name self is just convention. The first parameter passed to an instance method is always a reference to the class instance itself, so writing self there is just to name it (assign it to variable).
UserDict.__init__(self)
Here you are calling the UserDict's __init__ method and passing it a reference to the new instance (because you are not calling it with self.method_name, it is not passed automatically. You cannot call an inherited class's constructor without referencing its name, or using super). So what you are doing is initializing your object the same way any UserDict object would be initialized.