Here is a basic Metaclass intending to provide a custom __init__:
class D(type):
def __init__(cls, name, bases, attributes):
super(D, cls).__init__(name, bases, attributes)
for hook in R.registered_hooks:
setattr(cls, hook.__qualname__, hook)
The for loop iterates over a list of functions, calling setattr(cls, hook.__qualname__, hook) for each of it.
Here is a Class that use the above Metaclass:
class E(metaclass=D):
def __init__(self):
pass
The weird thing:
E().__dict__ prints {}
But when I call
`E.__dict__ prints {'f1': <function f1 at 0x001>, 'f2': <function f2 at 0x002>}`
I was expecting to add those function as attributes to the instance attributes, since the __init__ provides a custom initialization for a Class but it seems like the attributes were added to Class attributes.
What is the cause of this and how can I add attributes to the instance, in this scenario? Thanks!
They are added as instance attributes. The instance, though, is the class, as an instance of the metaclass. You aren't defining a __init__ method for E; you are defining the method that type uses to initialize E itself.
If you just want attributes added to the instance of E, you're working at the wrong level; you should define a mixin and use multiple inheritance for that.
For __init__ to be called on the instantiation of an object obj, you need to have defined type(obj).__init__. By using a metaclass, you defined type(type(obj)).__init__. You are working at the wrong level.
In this case, you only need inheritance, not a metaclass.
class D():
def __init__(self, *args):
print('D.__init__ was called')
class E(D):
pass
e = E() # prints: D.__init__ was called
Note that you will still find that e.__dict__ is empty.
print(e.__dict__) # {}
This is because you instances have no attributes, methods are stored and looked up in the class.
If you're using a metaclass, that means you're changing the definition of a class, which indirectly affects the attributes of instances of that class. Assuming you actually want to do this:
You're creating a custom __init__ method. As with normal instantiation, __init__ is called on an object that has been already created. Generally when you're doing something meaningful with metaclasses, you'll want to add to the class's dict before instantiation, which is done in the __new__ method to avoid issues that may come up in subclassing. It should look something like this:
class D(type):
def __new__(cls, name, bases, dct):
for hook in R.registered_hooks:
dct[hook.__qualname__] = hook
return super(D, cls).__new__(cls, name, bases, dct)
This will not modify the __dict__ of instances of that class, but attribute resolution for instances also looks for class attributes. So if you add foo as an attribute for E in the metaclass, E().foo will return the expected value even though it's not visible in E().__dict__.
Related
I'm trying to learn how metaclasses work in python 3. Things I want to know are: which functions are called, in what order, and their signatures and returns.
As an example, I know __prepare__ gets called when a class with a metaclass is instantiated with arguments metaclass, name_of_subclass, bases and returns a dictionary representing the future namespace of the instantiated object.
I feel like I understand __prepare__'s step in the process well. What I don't, though, are __init__, __new__, and __call__. What are their arguments? What do they return? How do they all call each other, or in general how does the process go? Currently, I'm stuck on understanding when __init__ is called.
Here is some code I've been messing around with to answer my questions:
#!/usr/bin/env python3
class Logged(type):
#classmethod
def __prepare__(cls, name, bases):
print('In meta __prepare__')
return {}
def __call__(subclass):
print('In meta __call__')
print('Creating {}.'.format(subclass))
return subclass.__new__(subclass)
def __new__(subclass, name, superclasses, attributes, **keyword_arguments):
print('In meta __new__')
return type.__new__(subclass, name, superclasses, attributes)
def __init__(subclass, name, superclasses, attributes, **keyword_arguments):
print('In meta __init__')
class Thing(metaclass = Logged):
def __new__(this, *arguments, **keyword_arguments):
print('In sub __new__')
return super(Thing, this).__new__(this)
def __init__(self, *arguments, **keyword_arguments):
print('In sub __init__')
def hello(self):
print('hello')
def main():
thing = Thing()
thing.hello()
if __name__ == '__main__':
main()
From this and some googling, I know that __new__ is really a static method that returns an instance of some object (usually the object where __new__ is defined, but not always), and that __init__ is called of an instance when it is made. By that logic, I'm confused as to why Thing.__init__() isn't being called. Could someone illuminate?
The output of this code prints 'hello', so an instance of Thing is being created, which further confuses me about init. Here's the output:
In meta __prepare__
In meta __new__
In meta __init__
In meta __call__
Creating <class '__main__.Thing'>
In sub __new__
hello
Any help understanding metaclasses would be appreciated. I've read quite a few tutorials, but I've missed some of these details.
First of all: __prepare__ is optional, you don't need to supply an implementation if all you are doing is return a default {} empty dictionary.
Metaclasses work exactly like classes, in that when you call them, then they produce an object. Both classes and metaclasses are factories. The difference is that a metaclass produces a class object when called, a class produces an instance when called.
Both classes and metaclasses define a default __call__ implementation, which basically does:
Call self.__new__ to produce a new object.
if that new object is an instance of self / a class with this
metaclass, then also call __init__ on that object.
You produced your own __call__ implementation, which doesn't implement that second step, which is why Thing.__init__ is never called.
You may ask: but the __call__ method is defined on the metaclass. That's correct, so it is exactly that method that is called when you call the class with Thing(). All special methods (starting and ending with __) are called on the type (e.g. type(instance) is the class, and type(class) is the metaclass) precisely because Python has this multi-level hierarchy of instances from classes from metaclasses; a __call__ method on the class itself is used to make instances callable. For metaclass() calls, it is the type object itself that provides the __call__ implementation. That's right, metaclasses are both subclasses and instances of type, at the same time.
When writing a metaclass, you should only implement __call__ if you want to customise what happens when you call the class. Leave it at the default implementation otherwise.
If I remove the __call__ method from your metaclass (and ignore the __prepare__ method), then Thing.__init__ is once again called:
>>> class Logged(type):
... def __new__(subclass, name, superclasses, attributes, **keyword_arguments):
... print('In meta __new__')
... return type.__new__(subclass, name, superclasses, attributes)
... def __init__(subclass, name, superclasses, attributes, **keyword_arguments):
... print('In meta __init__')
...
>>> class Thing(metaclass = Logged):
... def __new__(this, *arguments, **keyword_arguments):
... print('In sub __new__')
... return super(Thing, this).__new__(this)
... def __init__(self, *arguments, **keyword_arguments):
... print('In sub __init__')
... def hello(self):
... print('hello')
...
In meta __new__
In meta __init__
>>> thing = Thing()
In sub __new__
In sub __init__
In the metaclass's __call__ method, you're calling Thing's __new__ only, but not __init__. It seems that the default behaviour of __call__ is to invoke both of them, as seen when we call the metaclass's inherited __call__:
def __call__(subclass):
print('In meta __call__')
print('Creating {}.'.format(subclass))
return super().__call__(subclass)
This prints:
Creating <class '__main__.Thing'>.
In sub __new__
In sub __init__
I'm working on a decorator to implement some behaviors for an immutable class. I'd like a class to inherit from namedtuple (to have attribute immutability) and also want to add some new methods. Like this ... but correctly preventing new attributes being assigned to the new class.
When inheriting from namedtuple, you should define __new__ and set __slots__ to be an empty tuple (to maintain immutability):
def define_new(clz):
def __new(cls, *args, **kwargs):
return super(clz, cls).__new__(cls, *args, **kwargs)
clz.__new__ = staticmethod(__new) # delegate namedtuple.__new__ to namedtuple
return clz
#define_new
class C(namedtuple('Foo', "a b c")):
__slots__ = () # Prevent assignment of new vars
def foo(self): return "foo"
C(1,2,3).x = 123 # Fails, correctly
... great. But now I'd like to move the __slots__ assignment into the decorator:
def define_new(clz):
def __new(cls, *args, **kwargs):
return super(clz, cls).__new__(cls, *args, **kwargs)
#clz.__slots__ = ()
clz.__slots__ = (123) # just for testing
clz.__new__ = staticmethod(__new)
return clz
#define_new
class C(namedtuple('Foo', "a b c")):
def foo(self): return "foo"
c = C(1,2,3)
print c.__slots__ # Is the (123) I assigned!
c.x = 456 # Assignment succeeds! Not immutable.
print c.__slots__ # Is still (123)
Which is a little surprising.
Why has moving the assignment of __slots__ into the decorator caused a change in behavior?
If I print C.__slots__, I get the object I assigned. What do the x get stored?
The code doesn't work because __slots__ is not a normal class property consulted at run-time. It is a fundamental property of the class that affects the memory layout of each of its instances, and as such must be known when the class is created and remain static throughout the its lifetime. While Python (presumably for backward compatibility) allows assigning to __slots__ later, the assignment has no effect on the behavior of existing or future instances.
How __slots__ is set
The value of __slots__ determined by the class author is passed to the class constructor when the class object is being created. This is done when the class statement is executed; for example:
class X:
__slots__ = ()
The above statement is equivalent1 to creating a class object and assigning it to X:
X = type('X', (), {'__slots__': ()})
The type object is the metaclass, the factory that creates and returns a class when called. The metaclass invocation accepts the name of the type, its superclasses, and its definition dict. Most of the contents of the definition dict can also be assigned later The definition dict contains directives that affect low-level layour of the class instances. As you discovered, later assignment to __slots__ simply has no effect.
Setting __slots__ from the outside
To modify __slots__ so that it is picked up by Python, one must specify it when the class is being created. This can be accomplished with a metaclass, the type responsible for instantiating types. The metaclass drives the creation of the class object and it can make sure __slots__ makes its way into the class definition dict before the constructor is invoked:
class DefineNew(type):
def __new__(metacls, name, bases, dct):
def __new__(cls, *new_args, **new_kwargs):
return super(defcls, cls).__new__(cls, *new_args, **new_kwargs)
dct['__slots__'] = ()
dct['__new__'] = __new__
defcls = super().__new__(metacls, name, bases, dct)
return defcls
class C(namedtuple('Foo', "a b c"), metaclass=DefineNew):
def foo(self):
return "foo"
Testing results in the expected:
>>> c = C(1, 2, 3)
>>> c.foo()
'foo'
>>> c.bar = 1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'C' object has no attribute 'bar'
Metaclass mixing pitfall
Note that the C type object will itself be an instance of DefineMeta - which is not surprising, since that follows from the definition of a metaclass. But this might cause an error if you ever inherit from both C and a type that specifies a metaclass other than type or DefineMeta. Since we only need the metaclass to hook into class creation, but are not using it later, it is not strictly necessary for C to be created as an instance of DefineMeta - we can instead make it an instance of type, just like any other class. This is achieved by changing the line:
defcls = super().__new__(metacls, name, bases, dct)
to:
defcls = type.__new__(type, name, bases, dct)
The injection of __new__ and __slots__ will remain, but C will be a most ordinary type with the default metaclass.
In conclusion...
Defining a __new__ which simply calls the superclass __new__ is always superfluous - presumably the real code will also do something different in the injected __new__, e.g. provide the default values for the namedtuple.
1
In the actual class definition the compiler adds a couple of additional items to the class dict, such as the module name. Those are useful, but they do not affect the class definition in the fundamental way that __slots__ does. If X had methods, their function objects would also be included in the dict keyed by function name - automatically inserted as a side effect of executing the def statement in the class definition namespace dict.
__slots__ has to be present during class creation. It affects the memory layout of a class's instances, which isn't something you can just change at will. (Imagine if you already had instances of the class and you tried to reassign the class's __slots__ at that point; instances would all break.) The processing that bases the memory layout on __slots__ only happens during class creation.
Assigning __slots__ in a decorator is too late to do anything. It has to happen before class creation, in the class body or a metaclass __new__.
Also, your define_new is pointless; namedtuple.__new__ already does what you need your __new__ to do.
How safe is it to create an instance of a class in the metaclass constructors (__new__ and __init__)? I'm specifically interested for Python 2.7, but what Python 3 does is also welcome.
The Python data model docs sound like they're written for the normal class instance creation case, and I'm not really sure how the rules might be subtly different when its occurring in a metaclass.
For example, lets say I have code like this:
class Meta(type):
NEWED = []
INITED_BEFORE = []
INITED_AFTER = []
def __new__(meta, name, bases, dict):
cls = super(Meta, meta).__new__(meta, name, bases, dict)
instance = cls()
Meta.NEWED.append(instance)
return cls
def __init__(cls, name, bases, dict):
Meta.INITED_BEFORE.append(cls())
super(Meta, cls).__init__(name, bases, dict)
Meta.INITED_AFTER.append(cls())
class Foo(object):
__metaclass__ = Meta
At which points, if any, is it safe to construct an instance of Foo while the metaclass is constructing it, and what sort of caveats are there?
One suspicion I have is that, if Foo inherited other classes, or was subclassed, and those other classes had their own metaclass, then calling cls() in any of the metaclass methods would be calling it on a not-yet-finished class object. Is that true?
One suspicion I have is that, if Foo inherited other classes, or was subclassed, and those other classes had their own metaclass, then calling cls() in any of the metaclass methods would be calling it on a not-yet-finished class object. Is that true?
That is true, to the extent that in the metaclass's __new__, its __init__ won't have been called yet. Your metaclass has an __init__, as might any subclass, so you should make sure that gets called first. So I wouldn't try to make an instance of a class that hasn't been fully instantiated yet.
One thing you might do is manually call the metaclass's __init__ from within __new__. But you would have to set a flag or something in __init__ to make sure that the class doesn't get __init__'d twice. There might be a more clever way too that I'm not thinking of at the moment.
I want to have an instance of class registered when the class is defined. Ideally the code below would do the trick.
registry = {}
def register( cls ):
registry[cls.__name__] = cls() #problem here
return cls
#register
class MyClass( Base ):
def __init__(self):
super( MyClass, self ).__init__()
Unfortunately, this code generates the error NameError: global name 'MyClass' is not defined.
What's going on is at the #problem here line I'm trying to instantiate a MyClass but the decorator hasn't returned yet so it doesn't exist.
Is the someway around this using metaclasses or something?
Yes, meta classes can do this. A meta class' __new__ method returns the class, so just register that class before returning it.
class MetaClass(type):
def __new__(cls, clsname, bases, attrs):
newclass = super(MetaClass, cls).__new__(cls, clsname, bases, attrs)
register(newclass) # here is your register function
return newclass
class MyClass(object):
__metaclass__ = MetaClass
The previous example works in Python 2.x. In Python 3.x, the definition of MyClass is slightly different (while MetaClass is not shown because it is unchanged - except that super(MetaClass, cls) can become super() if you want):
#Python 3.x
class MyClass(metaclass=MetaClass):
pass
As of Python 3.6 there is also a new __init_subclass__ method (see PEP 487) that can be used instead of a meta class (thanks to #matusko for his answer below):
class ParentClass:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
register(cls)
class MyClass(ParentClass):
pass
[edit: fixed missing cls argument to super().__new__()]
[edit: added Python 3.x example]
[edit: corrected order of args to super(), and improved description of 3.x differences]
[edit: add Python 3.6 __init_subclass__ example]
Since python 3.6 you don't need metaclasses to solve this
In python 3.6 simpler customization of class creation was introduced (PEP 487).
An __init_subclass__ hook that initializes all subclasses of a given class.
Proposal includes following example of subclass registration
class PluginBase:
subclasses = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.subclasses.append(cls)
In this example, PluginBase.subclasses will contain a plain list of
all subclasses in the entire inheritance tree. One should note that
this also works nicely as a mixin class.
The problem isn't actually caused by the line you've indicated, but by the super call in the __init__ method. The problem remains if you use a metaclass as suggested by dappawit; the reason the example from that answer works is simply that dappawit has simplified your example by omitting the Base class and therefore the super call. In the following example, neither ClassWithMeta nor DecoratedClass work:
registry = {}
def register(cls):
registry[cls.__name__] = cls()
return cls
class MetaClass(type):
def __new__(cls, clsname, bases, attrs):
newclass = super(cls, MetaClass).__new__(cls, clsname, bases, attrs)
register(newclass) # here is your register function
return newclass
class Base(object):
pass
class ClassWithMeta(Base):
__metaclass__ = MetaClass
def __init__(self):
super(ClassWithMeta, self).__init__()
#register
class DecoratedClass(Base):
def __init__(self):
super(DecoratedClass, self).__init__()
The problem is the same in both cases; the register function is called (either by the metaclass or directly as a decorator) after the class object is created, but before it has been bound to a name. This is where super gets gnarly (in Python 2.x), because it requires you to refer to the class in the super call, which you can only reasonably do by using the global name and trusting that it will have been bound to that name by the time the super call is invoked. In this case, that trust is misplaced.
I think a metaclass is the wrong solution here. Metaclasses are for making a family of classes that have some custom behaviour in common, exactly as classes are for making a family of instances that have some custom behavior in common. All you're doing is calling a function on a class. You wouldn't define a class to call a function on a string, neither should you define a metaclass to call a function on a class.
So, the problem is a fundamental incompatibility between: (1) using hooks in the class creation process to create instances of the class, and (2) using super.
One way to resolve this is to not use super. super solves a hard problem, but it introduces others (this is one of them). If you're using a complex multiple inheritance scheme, super's problems are better than the problems of not using super, and if you're inheriting from third-party classes that use super then you have to use super. If neither of those conditions are true, then just replacing your super calls with direct base class calls may actually be a reasonable solution.
Another way is to not hook register into class creation. Adding register(MyClass) after each of your class definitions is pretty equivalent to adding #register before them or __metaclass__ = Registered (or whatever you call the metaclass) into them. A line down the bottom is much less self-documenting than a nice declaration up the top of the class though, so this doesn't feel great, but again it may actually be a reasonable solution.
Finally, you can turn to hacks that are unpleasant, but will probably work. The problem is that a name is being looked up in a module's global scope just before it's been bound there. So you could cheat, as follows:
def register(cls):
name = cls.__name__
force_bound = False
if '__init__' in cls.__dict__:
cls.__init__.func_globals[name] = cls
force_bound = True
try:
registry[name] = cls()
finally:
if force_bound:
del cls.__init__.func_globals[name]
return cls
Here's how this works:
We first check to see whether __init__ is in cls.__dict__ (as opposed to whether it has an __init__ attribute, which will always be true). If it's inherited an __init__ method from another class we're probably fine (because the superclass will already be bound to its name in the usual way), and the magic we're about to do doesn't work on object.__init__ so we want to avoid trying that if the class is using a default __init__.
We lookup the __init__ method and grab it's func_globals dictionary, which is where global lookups (such as to find the class referred to in a super call) will go. This is normally the global dictionary of the module where the __init__ method was originally defined. Such a dictionary is about to have the cls.__name__ inserted into it as soon as register returns, so we just insert it ourselves early.
We finally create an instance and insert it into the registry. This is in a try/finally block to make sure we remove the binding we created whether or not creating an instance throws an exception; this is very unlikely to be necessary (since 99.999% of the time the name is about to be rebound anyway), but it's best to keep weird magic like this as insulated as possible to minimise the chance that someday some other weird magic interacts badly with it.
This version of register will work whether it's invoked as a decorator or by the metaclass (which I still think is not a good use of a metaclass). There are some obscure cases where it will fail though:
I can imagine a weird class that doesn't have an __init__ method but inherits one that calls self.someMethod, and someMethod is overridden in the class being defined and makes a super call. Probably unlikely.
The __init__ method might have been defined in another module originally and then used in the class by doing __init__ = externally_defined_function in the class block. The func_globals attribute of the other module though, which means our temporary binding would clobber any definition of this class' name in that module (oops). Again, unlikely.
Probably other weird cases I haven't thought of.
You could try to add more hacks to make it a little more robust in these situations, but the nature of Python is both that these kind of hacks are possible and that it's impossible to make them absolutely bullet proof.
The answers here didn't work for me in python3, because __metaclass__ didn't work.
Here's my code registering all subclasses of a class at their definition time:
registered_models = set()
class RegisteredModel(type):
def __new__(cls, clsname, superclasses, attributedict):
newclass = type.__new__(cls, clsname, superclasses, attributedict)
# condition to prevent base class registration
if superclasses:
registered_models.add(newclass)
return newclass
class CustomDBModel(metaclass=RegisteredModel):
pass
class BlogpostModel(CustomDBModel):
pass
class CommentModel(CustomDBModel):
pass
# prints out {<class '__main__.BlogpostModel'>, <class '__main__.CommentModel'>}
print(registered_models)
Calling the Base class directly should work (instead of using super()):
def __init__(self):
Base.__init__(self)
It can be also done with something like this (without a registry function)
_registry = {}
class MetaClass(type):
def __init__(cls, clsname, bases, methods):
super().__init__(clsname, bases, methods)
_registry[cls.__name__] = cls
class MyClass1(metaclass=MetaClass): pass
class MyClass2(metaclass=MetaClass): pass
print(_registry)
# {'MyClass1': <class '__main__.MyClass1'>, 'MyClass2': <class '__main__.MyClass2'>}
Additionally, if we need to use a base abstract class (e.g. Base() class), we can do it this way (notice the metacalss inherits from ABCMeta instead of type)
from abc import ABCMeta
_registry = {}
class MetaClass(ABCMeta):
def __init__(cls, clsname, bases, methods):
super().__init__(clsname, bases, methods)
_registry[cls.__name__] = cls
class Base(metaclass=MetaClass): pass
class MyClass1(Base): pass
class MyClass2(Base): pass
print(_registry)
# {'Base': <class '__main__.Base'>, 'MyClass1': <class '__main__.MyClass1'>, 'MyClass2': <class '__main__.MyClass2'>}
i think you can defined either '__init__' or '__new__' in a class,but why all defined in django.utils.datastructures.py.
my code:
class a(object):
def __init__(self):
print 'aaa'
def __new__(self):
print 'sss'
a()#print 'sss'
class b:
def __init__(self):
print 'aaa'
def __new__(self):
print 'sss'
b()#print 'aaa'
datastructures.py:
class SortedDict(dict):
"""
A dictionary that keeps its keys in the order in which they're inserted.
"""
def __new__(cls, *args, **kwargs):
instance = super(SortedDict, cls).__new__(cls, *args, **kwargs)
instance.keyOrder = []
return instance
def __init__(self, data=None):
if data is None:
data = {}
super(SortedDict, self).__init__(data)
if isinstance(data, dict):
self.keyOrder = data.keys()
else:
self.keyOrder = []
for key, value in data:
if key not in self.keyOrder:
self.keyOrder.append(key)
and what circumstances the SortedDict.__init__ will be call.
thanks
You can define either or both of __new__ and __init__.
__new__ must return an object -- which can be a new one (typically that task is delegated to type.__new__), an existing one (to implement singletons, "recycle" instances from a pool, and so on), or even one that's not an instance of the class. If __new__ returns an instance of the class (new or existing), __init__ then gets called on it; if __new__ returns an object that's not an instance of the class, then __init__ is not called.
__init__ is passed a class instance as its first item (in the same state __new__ returned it, i.e., typically "empty") and must alter it as needed to make it ready for use (most often by adding attributes).
In general it's best to use __init__ for all it can do -- and __new__, if something is left that __init__ can't do, for that "extra something".
So you'll typically define both if there's something useful you can do in __init__, but not everything you want to happen when the class gets instantiated.
For example, consider a class that subclasses int but also has a foo slot -- and you want it to be instantiated with an initializer for the int and one for the .foo. As int is immutable, that part has to happen in __new__, so pedantically one could code:
>>> class x(int):
... def __new__(cls, i, foo):
... self = int.__new__(cls, i)
... return self
... def __init__(self, i, foo):
... self.foo = foo
... __slots__ = 'foo',
...
>>> a = x(23, 'bah')
>>> print a
23
>>> print a.foo
bah
>>>
In practice, for a case this simple, nobody would mind if you lost the __init__ and just moved the self.foo = foo to __new__. But if initialization is rich and complex enough to be best placed in __init__, this idea is worth keeping in mind.
__new__ and __init__ do completely different things. The method __init__ initiates a new instance of a class --- it is a constructor. __new__ is a far more subtle thing --- it can change arguments and, in fact, the class of the initiated object. For example, the following code:
class Meters(object):
def __new__(cls, value):
return int(value / 3.28083)
If you call Meters(6) you will not actually create an instance of Meters, but an instance of int. You might wonder why this is useful; it is actually crucial to metaclasses, an admittedly obscure (but powerful) feature.
You'll note that in Python 2.x, only classes inheriting from object can take advantage of __new__, as you code above shows.
The use of __new__ you showed in django seems to be an attempt to keep a sane method resolution order on SortedDict objects. I will admit, though, that it is often hard to tell why __new__ is necessary. Standard Python style suggests that it not be used unless necessary (as always, better class design is the tool you turn to first).
My only guess is that in this case, they (author of this class) want the keyOrder list to exist on the class even before SortedDict.__init__ is called.
Note that SortedDict calls super() in its __init__, this would ordinarily go to dict.__init__, which would probably call __setitem__ and the like to start adding items. SortedDict.__setitem__ expects the .keyOrder property to exist, and therein lies the problem (since .keyOrder isn't normally created until after the call to super().) It's possible this is just an issue with subclassing dict because my normal gut instinct would be to just initialize .keyOrder before the call to super().
The code in __new__ might also be used to allow SortedDict to be subclassed in a diamond inheritance structure where it is possible SortedDict.__init__ is not called before the first __setitem__ and the like are called. Django has to contend with various issues in supporting a wide range of python versions from 2.3 up; it's possible this code is completely un-neccesary in some versions and needed in others.
There is a common use for defining both __new__ and __init__: accessing class properties which may be eclipsed by their instance versions without having to do type(self) or self.__class__ (which, in the existence of metaclasses, may not even be the right thing).
For example:
class MyClass(object):
creation_counter = 0
def __new__(cls, *args, **kwargs):
cls.creation_counter += 1
return super(MyClass, cls).__new__(cls)
def __init__(self):
print "I am the %dth myclass to be created!" % self.creation_counter
Finally, __new__ can actually return an instance of a wrapper or a completely different class from what you thought you were instantiating. This is used to provide metaclass-like features without actually needing a metaclass.
In my opinion, there was no need of overriding __new__ in the example you described.
Creation of an instance and actual memory allocation happens in __new__, __init__ is called after __new__ and is meant for initialization of instance serving the job of constructor in classical OOP terms. So, if all you want to do is initialize variables, then you should go for overriding __init__.
The real role of __new__ comes into place when you are using Metaclasses. There if you want to do something like changing attributes or adding attributes, that must happen before the creation of class, you should go for overriding __new__.
Consider, a completely hypothetical case where you want to make some attributes of class private, even though they are not defined so (I'm not saying one should ever do that).
class PrivateMetaClass(type):
def __new__(metaclass, classname, bases, attrs):
private_attributes = ['name', 'age']
for private_attribute in private_attributes:
if attrs.get(private_attribute):
attrs['_' + private_attribute] = attrs[private_attribute]
attrs.pop(private_attribute)
return super(PrivateMetaClass, metaclass).__new__(metaclass, classname, bases, attrs)
class Person(object):
__metaclass__ = PrivateMetaClass
name = 'Someone'
age = 19
person = Person()
>>> hasattr(person, 'name')
False
>>> person._name
'Someone'
Again, It's just for instructional purposes I'm not suggesting one should do anything like this.