I have read posts like these:
What is a metaclass in Python?
What are your (concrete) use-cases for metaclasses in Python?
Python's Super is nifty, but you can't use it
But somehow I got confused. Many confusions like:
When and why would I have to do something like the following?
# Refer link1
return super(MyType, cls).__new__(cls, name, bases, newattrs)
or
# Refer link2
return super(MetaSingleton, cls).__call__(*args, **kw)
or
# Refer link2
return type(self.__name__ + other.__name__, (self, other), {})
How does super work exactly?
What is class registry and unregistry in link1 and how exactly does it work? (I thought it has something to do with singleton. I may be wrong, being from C background. My coding style is still a mix of functional and OO).
What is the flow of class instantiation (subclass, metaclass, super, type) and method invocation (
metaclass->__new__, metaclass->__init__, super->__new__, subclass->__init__ inherited from metaclass
) with well-commented working code (though the first link is quite close, but it does not talk about cls keyword and super(..) and registry). Preferably an example with multiple inheritance.
P.S.: I made the last part as code because Stack Overflow formatting was converting the text metaclass->__new__
to metaclass->new
OK, you've thrown quite a few concepts into the mix here! I'm going to pull out a few of the specific questions you have.
In general, understanding super, the MRO and metclasses is made much more complicated because there have been lots of changes in this tricky area over the last few versions of Python.
Python's own documentation is a very good reference, and completely up to date. There is an IBM developerWorks article which is fine as an introduction and takes a more tutorial-based approach, but note that it's five years old, and spends a lot of time talking about the older-style approaches to meta-classes.
super is how you access an object's super-classes. It's more complex than (for example) Java's super keyword, mainly because of multiple inheritance in Python. As Super Considered Harmful explains, using super() can result in you implicitly using a chain of super-classes, the order of which is defined by the Method Resolution Order (MRO).
You can see the MRO for a class easily by invoking mro() on the class (not on an instance). Note that meta-classes are not in an object's super-class hierarchy.
Thomas' description of meta-classes here is excellent:
A metaclass is the class of a class.
Like a class defines how an instance
of the class behaves, a metaclass
defines how a class behaves. A class
is an instance of a metaclass.
In the examples you give, here's what's going on:
The call to __new__ is being
bubbled up to the next thing in the
MRO. In this case, super(MyType, cls) would resolve to type;
calling type.__new__ lets Python
complete it's normal instance
creation steps.
This example is using meta-classes
to enforce a singleton. He's
overriding __call__ in the
metaclass so that whenever a class
instance is created, he intercepts
that, and can bypass instance
creation if there already is one
(stored in cls.instance). Note
that overriding __new__ in the
metaclass won't be good enough,
because that's only called when
creating the class. Overriding
__new__ on the class would work,
however.
This shows a way to dynamically
create a class. Here's he's
appending the supplied class's name
to the created class name, and
adding it to the class hierarchy
too.
I'm not exactly sure what sort of code example you're looking for, but here's a brief one showing meta-classes, inheritance and method resolution:
print('>>> # Defining classes:')
class MyMeta(type):
def __new__(cls, name, bases, dct):
print("meta: creating %s %s" % (name, bases))
return type.__new__(cls, name, bases, dct)
def meta_meth(cls):
print("MyMeta.meta_meth")
__repr__ = lambda c: c.__name__
class A(metaclass=MyMeta):
def __init__(self):
super(A, self).__init__()
print("A init")
def meth(self):
print("A.meth")
class B(metaclass=MyMeta):
def __init__(self):
super(B, self).__init__()
print("B init")
def meth(self):
print("B.meth")
class C(A, B, metaclass=MyMeta):
def __init__(self):
super(C, self).__init__()
print("C init")
print('>>> c_obj = C()')
c_obj = C()
print('>>> c_obj.meth()')
c_obj.meth()
print('>>> C.meta_meth()')
C.meta_meth()
print('>>> c_obj.meta_meth()')
c_obj.meta_meth()
Example output (using Python >= 3.6):
>>> # Defining classes:
meta: creating A ()
meta: creating B ()
meta: creating C (A, B)
>>> c_obj = C()
B init
A init
C init
>>> c_obj.meth()
A.meth
>>> C.meta_meth()
MyMeta.meta_meth
>>> c_obj.meta_meth()
Traceback (most recent call last):
File "metatest.py", line 41, in <module>
c_obj.meta_meth()
AttributeError: 'C' object has no attribute 'meta_meth'
Here's the more pragmatic answer.
It rarely matters
"What is a metaclass in Python". Bottom line, type is the metaclass of all classes. You have almost no practical use for this.
class X(object):
pass
type(X) == type
"What are your (concrete) use cases for metaclasses in Python?". Bottom line. None.
"Python's Super is nifty, but you can't use it". Interesting note, but little practical value. You'll never have a need for resolving complex multiple inheritance networks. It's easy to prevent this problem from arising by using an explicity Strategy design instead of multiple inheritance.
Here's my experience over the last 7 years of Python programming.
A class has 1 or more superclasses forming a simple chain from my class to object.
The concept of "class" is defined by a metaclass named type. I might want to extend the concept of "class", but so far, it's never come up in practice. Not once. type always does the right thing.
Using super works out really well in practice. It allows a subclass to defer to it's superclass. It happens to show up in these metaclass examples because they're extending the built-in metaclass, type.
However, in all subclass situations, you'll make use of super to extend a superclass.
Metaclasses
The metaclass issue is this:
Every object has a reference to it's type definition, or "class".
A class is, itself, also an object.
Therefore a object of type class has a reference to it's type or "class". The "class" of a "class" is a metaclass.
Since a "class" isn't a C++ run-time object, this doesn't happen in C++. It does happen in Java, Smalltalk and Python.
A metaclass defines the behavior of a class object.
90% of your interaction with a class is to ask the class to create a new object.
10% of the time, you'll be using class methods or class variables ("static" in C++ or Java parlance.)
I have found a few use cases for class-level methods. I have almost no use cases for class variables. I've never had a situation to change the way object construction works.
Related
In C++, given a class hierarchy, the most derived class's ctor calls its base class ctor which then initialized the base part of the object, before the derived part is instantiated. In Python I want to understand what's going on in a case where I have the requirement, that Derived subclasses a given class Base which takes a callable in its __init__ method which it then later invokes. The callable features some parameters which I pass in Derived class's __init__, which is where I also define the callable function. My idea then was to pass the Derived class itself to its Base class after having defined the __call__ operator
class Derived(Base):
def __init__(self, a, b):
def _process(c, d):
do_something with a and b
self.__class__.__call__ = _process
super(Derived, self).__init__(self)
Is this a pythonic way of dealing with this problem?
What is the exact order of initialization here? Does one needs to call super as a first instruction in the __init__ method or is it ok to do it the way I did?
I am confused whether it is considered good practice to use super with or without arguments in python > 3.6
What is the exact order of initialization here?
Well, very obviously the one you can see in your code - Base.__init__() is only called when you explicitely ask for it (with the super() call). If Base also has parents and everyone in the chain uses super() calls, the parents initializers will be invoked according to the mro.
Basically, Python is a "runtime language" - except for the bytecode compilation phase, everything happens at runtime - so there's very few "black magic" going on (and much of it is actually documented and fully exposed for those who want to look under the hood or do some metaprogramming).
Does one needs to call super as a first instruction in the init method or is it ok to do it the way I did?
You call the parent's method where you see fit for the concrete use case - you just have to beware of not using instance attributes (directly or - less obvious to spot - indirectly via a method call that depends on those attributes) before they are defined.
I am confused whether it is considered good practice to use super with or without arguments in python > 3.6
If you don't need backward compatibily, use super() without params - unless you want to explicitely skip some class in the MRO, but then chances are there's something debatable with your design (but well - sometimes we can't afford to rewrite a whole code base just to avoid one very special corner case, so that's ok too as long as you understand what you're doing and why).
Now with your core question:
class Derived(Base):
def __init__(self, a, b):
def _process(c, d):
do_something with a and b
self.__class__.__call__ = _process
super(Derived, self).__init__(self)
self.__class__.__call__ is a class attribute and is shared by all instances of the class. This means that you either have to make sure you are only ever using one single instance of the class (which doesn't seem to be the goal here) or are ready to have totally random results, since each new instance will overwrite self.__class__.__call__ with it's own version.
If what you want is to have each instance's __call__ method to call it's own version of process(), then there's a much simpler solution - just make _process an instance attribute and call it from __call__ :
class Derived(Base):
def __init__(self, a, b):
def _process(c, d):
do_something with a and b
self._process = _process
super(Derived, self).__init__(self)
def __call__(self, c, d):
return self._process(c, d)
Or even simpler:
class Derived(Base):
def __init__(self, a, b):
super(Derived, self).__init__(self)
self._a = a
self._b = b
def __call__(self, c, d):
do_something_with(self._a, self._b)
EDIT:
Base requires a callable in ins init method.
This would be better if your example snippet was closer to your real use case.
But when I call super().init() the call method of Derived should not have been instantiated yet or has it?
Now that's a good question... Actually, Python methods are not what you think they are. What you define in a class statement's body using the def statement are still plain functions, as you can see by yourself:
class Foo:
... def bar(self): pass
...
Foo.bar
"Methods" are only instanciated when an attribute lookup resolves to a class attribute that happens to be a function:
Foo().bar
main.Foo object at 0x7f3cef4de908>>
Foo().bar
main.Foo object at 0x7f3cef4de940>>
(if you wonder how this happens, it's documented here)
and they actually are just thin wrappers around a function, instance and class (or function and class for classmethods), which delegate the call to the underlying function, injecting the instance (or class) as first argument. In CS terms, a Python method is the partial application of a function to an instance (or class).
Now as I mentionned upper, Python is a runtime language, and both def and class are executable statements. So by the time you define your Derived class, the class statement creating the Base class object has already been executed (else Base wouldn't exist at all), with all the class statement block being executed first (to define the functions and other class attributes).
So "when you call super().__init()__", the __call__ function of Base HAS been instanciated (assuming it's defined in the class statement for Base of course, but that's by far the most common case).
I was looking into Python's super method and multiple inheritance. I read along something like when we use super to call a base method which has implementation in all base classes, only one class' method will be called even with variety of arguments. For example,
class Base1(object):
def __init__(self, a):
print "In Base 1"
class Base2(object):
def __init__(self):
print "In Base 2"
class Child(Base1, Base2):
def __init__(self):
super(Child, self).__init__('Intended for base 1')
super(Child, self).__init__()# Intended for base 2
This produces TyepError for the first super method. super would call whichever method implementation it first recognizes and gives TypeError instead of checking for other classes down the road. However, this will be much more clear and work fine when we do the following:
class Child(Base1, Base2):
def __init__(self):
Base1.__init__(self, 'Intended for base 1')
Base2.__init__(self) # Intended for base 2
This leads to two questions:
Is __init__ method a static method or a class method?
Why use super, which implicitly choose the method on it's own rather than explicit call to the method like the latter example? It looks lot more cleaner than using super to me. So what is the advantage of using super over the second way(other than writing the base class name with the method call)
super() in the face of multiple inheritance, especially on methods that are present on object can get a bit tricky. The general rule is that if you use super, then every class in the hierarchy should use super. A good way to handle this for __init__ is to make every method take **kwargs, and always use keyword arguments everywhere. By the time the call to object.__init__ occurs, all arguments should have been popped out!
class Base1(object):
def __init__(self, a, **kwargs):
print "In Base 1", a
super(Base1, self).__init__()
class Base2(object):
def __init__(self, **kwargs):
print "In Base 2"
super(Base2, self).__init__()
class Child(Base1, Base2):
def __init__(self, **kwargs):
super(Child, self).__init__(a="Something for Base1")
See the linked article for way more explanation of how this works and how to make it work for you!
Edit: At the risk of answering two questions, "Why use super at all?"
We have super() for many of the same reasons we have classes and inheritance, as a tool for modularizing and abstracting our code. When operating on an instance of a class, you don't need to know all of the gritty details of how that class was implemented, you only need to know about its methods and attributes, and how you're meant to use that public interface for the class. In particular, you can be confident that changes in the implementation of a class can't cause you problems as a user of its instances.
The same argument holds when deriving new types from base classes. You don't want or need to worry about how those base classes were implemented. Here's a concrete example of how not using super might go wrong. suppose you've got:
class Foo(object):
def frob(self):
print "frobbign as a foo"
class Bar(object):
def frob(self):
print "frobbign as a bar"
and you make a subclass:
class FooBar(Foo, Bar):
def frob(self):
Foo.frob(self)
Bar.frob(self)
Everything's fine, but then you realize that when you get down to it,
Foo really is a kind of Bar, so you change it
class Foo(Bar):
def frob(self):
print "frobbign as a foo"
Bar.frob(self)
Which is all fine, except that in your derived class, FooBar.frob() calls Bar.frob() twice.
This is the exact problem super() solves, it protects you from calling superclass implementations more than once (when used as directed...)
As for your first question, __init__ is neither a staticmethod nor a classmethod; it is an ordinary instance method. (That is, it receives the instance as its first argument.)
As for your second question, if you want to explicitly call multiple base class implementations, then doing it explicitly as you did is indeed the only way. However, you seem to be misunderstanding how super works. When you call super, it does not "know" if you have already called it. Both of your calls to super(Child, self).__init__ call the Base1 implementation, because that is the "nearest parent" (the most immediate superclass of Child).
You would use super if you want to call just this immediate superclass implementation. You would do this if that superclass was also set up to call its superclass, and so on. The way to use super is to have each class call only the next implementation "up" in the class hierarchy, so that the sequence of super calls overall calls everything that needs to be called, in the right order. This type of setup is often called "cooperative inheritance", and you can find various articles about it online, including here and here.
Following this answer it seems that a class' metaclass may be changed after the class has been defined by using the following*:
class MyMetaClass(type):
# Metaclass magic...
class A(object):
pass
A = MyMetaClass(A.__name__, A.__bases__, dict(A.__dict__))
Defining a function
def metaclass_wrapper(cls):
return MyMetaClass(cls.__name__, cls.__bases__, dict(cls.__dict__))
allows me to apply a decorator to a class definition like so,
#metaclass_wrapper
class B(object):
pass
It seems that the metaclass magic is applied to B, however B has no __metaclass__ attribute. Is the above method a sensible way to apply metaclasses to class definitions, even though I am definiting and re-definiting a class, or would I be better off simply writing
class B(object):
__metaclass__ = MyMetaClass
pass
I presume there are some differences between the two methods.
*Note, the original answer in the linked question, MyMetaClass(A.__name__, A.__bases__, A.__dict__), returns a TypeError:
TypeError: type() argument 3 must be a dict, not dict_proxy
It seems that the __dict__ attribute of A (the class definition) has a type dict_proxy, whereas the type of the __dict__ attribute of an instance of A has a type dict. Why is this? Is this a Python 2.x vs. 3.x difference?
Admittedly, I am a bit late to the party. However, I fell this was worth adding.
This is completely doable. That being said, there are plenty of other ways to accomplish the same goal. However, the decoration solution, in particular, allows for delayed evaluation ( obj = dec(obj) ), which using __metaclass__ inside the class does not. In typical decorator style, my solution is below.
There is a tricky thing that you may run into if you just construct the class without changing the dictionary or copying its attributes. Any attributes that the class had previously (before decorating) will appear to be missing. So, it is absolutely essential to copy these over and then tweak them as I have in my solution.
Personally, I like to be able to keep track of how an object was wrapped. So, I added the __wrapped__ attribute, which is not strictly necessary. It also makes it more like functools.wraps in Python 3 for classes. However, it can be helpful with introspection. Also, __metaclass__ is added to act more like the normal metaclass use case.
def metaclass(meta):
def metaclass_wrapper(cls):
__name = str(cls.__name__)
__bases = tuple(cls.__bases__)
__dict = dict(cls.__dict__)
for each_slot in __dict.get("__slots__", tuple()):
__dict.pop(each_slot, None)
__dict["__metaclass__"] = meta
__dict["__wrapped__"] = cls
return(meta(__name, __bases, __dict))
return(metaclass_wrapper)
For a trivial example, take the following.
class MetaStaticVariablePassed(type):
def __new__(meta, name, bases, dct):
dct["passed"] = True
return(super(MetaStaticVariablePassed, meta).__new__(meta, name, bases, dct))
#metaclass(MetaStaticVariablePassed)
class Test(object):
pass
This yields the nice result...
|1> Test.passed
|.> True
Using the decorator in the less usual, but identical way...
class Test(object):
pass
Test = metaclass_wrapper(Test)
...yields, as expected, the same nice result.
|1> Test.passed
|.> True
The class has no __metaclass__ attribute set... because you never set it!
Which metaclass to use is normally determined by a name __metaclass__ set in a class block. The __metaclass__ attribute isn't set by the metaclass. So if you invoke a metaclass directly rather than setting __metaclass__ and letting Python figure it out, then no __metaclass__ attribute is set.
In fact, normal classes are all instances of the metaclass type, so if the metaclass always set the __metaclass__ attribute on its instances then every class would have a __metaclass__ attribute (most of them set to type).
I would not use your decorator approach. It obscures the fact that a metaclass is involved (and which one), is still one line of boilerplate, and it's just messy to create a class from the 3 defining features of (name, bases, attributes) only to pull those 3 bits back out from the resulting class, throw the class away, and make a new class from those same 3 bits!
When you do this in Python 2.x:
class A(object):
__metaclass__ = MyMeta
def __init__(self):
pass
You'd get roughly the same result if you'd written this:
attrs = {}
attrs['__metaclass__'] = MyMeta
def __init__(self):
pass
attrs['__init__'] = __init__
A = attrs.get('__metaclass__', type)('A', (object,), attrs)
In reality calculating the metaclass is more complicated, as there actually has to be a search through all the bases to determine whether there's a metaclass conflict, and if one of the bases doesn't have type as its metaclass and attrs doesn't contain __metaclass__ then the default metaclass is the ancestor's metaclass rather than type. This is one situation where I expect your decorator "solution" will differ from using __metaclass__ directly. I'm not sure exactly what would happen if you used your decorator in a situation where using __metaclass__ would give you a metaclass conflict error, but I wouldn't expect it to be pleasant.
Also, if there are any other metaclasses involved, your method would result in them running first (possibly modifying what the name, bases, and attributes are!) and then pulling those out of the class and using it to create a new class. This could potentially be quite different than what you'd get using __metaclass__.
As for the __dict__ not giving you a real dictionary, that's just an implementation detail; I would guess for performance reasons. I doubt there is any spec that says the __dict__ of a (non-class) instance has to be the same type as the __dict__ of a class (which is also an instance btw; just an instance of a metaclass). The __dict__ attribute of a class is a "dictproxy", which allows you to look up attribute keys as if it were a dict but still isn't a dict. type is picky about the type of its third argument; it wants a real dict, not just a "dict-like" object (shame on it for spoiling duck-typing). It's not a 2.x vs 3.x thing; Python 3 behaves the same way, although it gives you a nicer string representation of the dictproxy. Python 2.4 (which is the oldest 2.x I have readily available) also has dictproxy objects for class __dict__ objects.
My summary of your question: "I tried a new tricky way to do a thing, and it didn't quite work. Should I use the simple way instead?"
Yes, you should do it the simple way. You haven't said why you're interested in inventing a new way to do it.
I found a lot of links on metaclasses, and most of them mention that they are useful for implementing factory methods. Can you show me an example of using metaclasses to implement the design pattern?
I'd love to hear people's comments on this, but I think this is an example of what you want to do
class FactoryMetaclassObject(type):
def __init__(cls, name, bases, attrs):
"""__init__ will happen when the metaclass is constructed:
the class object itself (not the instance of the class)"""
pass
def __call__(*args, **kw):
"""
__call__ will happen when an instance of the class (NOT metaclass)
is instantiated. For example, We can add instance methods here and they will
be added to the instance of our class and NOT as a class method
(aka: a method applied to our instance of object).
Or, if this metaclass is used as a factory, we can return a whole different
classes' instance
"""
return "hello world!"
class FactorWorker(object):
__metaclass__ = FactoryMetaclassObject
f = FactorWorker()
print f.__class__
The result you will see is: type 'str'
You can find some helpful examples at wikibooks/python, here and here
There's no need. You can override a class's __new__() method in order to return a completely different object type.
What are the main differences between Python metaclasses and class decorators? Is there something I can do with one but not with the other?
Decorators are much, much simpler and more limited -- and therefore should be preferred whenever the desired effect can be achieved with either a metaclass or a class decorator.
Anything you can do with a class decorator, you can of course do with a custom metaclass (just apply the functionality of the "decorator function", i.e., the one that takes a class object and modifies it, in the course of the metaclass's __new__ or __init__ that make the class object!-).
There are many things you can do in a custom metaclass but not in a decorator (unless the decorator internally generates and applies a custom metaclass, of course -- but that's cheating;-)... and even then, in Python 3, there are things you can only do with a custom metaclass, not after the fact... but that's a pretty advanced sub-niche of your question, so let me give simpler examples).
For example, suppose you want to make a class object X such that print X (or in Python 3 print(X) of course;-) displays peekaboo!. You cannot possibly do that without a custom metaclass, because the metaclass's override of __str__ is the crucial actor here, i.e., you need a def __str__(cls): return "peekaboo!" in the custom metaclass of class X.
The same applies to all magic methods, i.e., to all kinds of operations as applied to the class object itself (as opposed to, ones applied to its instances, which use magic methods as defined in the class -- operations on the class object itself use magic methods as defined in the metaclass).
As given in the chapter 21 of the book 'fluent python', one difference is related to inheritance. Please see these two scripts. The python version is 3.5. One point is that the use of metaclass affects its children while the decorator affects only the current class.
The script use class-decorator to replace/overwirte the method 'func1'.
def deco4cls(cls):
cls.func1 = lambda self: 2
return cls
#deco4cls
class Cls1:
pass
class Cls1_1(Cls1):
def func1(self):
return 3
obj1_1 = Cls1_1()
print(obj1_1.func1()) # 3
The script use metaclass to replace/overwrite the method 'func1'.
class Deco4cls(type):
def __init__(cls, name, bases, attr_dict):
# print(cls, name, bases, attr_dict)
super().__init__(name, bases, attr_dict)
cls.func1 = lambda self: 2
class Cls2(metaclass=Deco4cls):
pass
class Cls2_1(Cls2):
def func1(self):
return 3
obj2_1 = Cls2_1()
print(obj2_1.func1()) # 2!! the original Cls2_1.func1 is replaced by metaclass