After crashing with metaclasses i delved into the topic of metaprogramming in Python and I have a couple of questions that are, imho, not clearly anwered in available docs.
When using both __new__ and __init__ in a metaclass, their arguments must be defined the same?
What's most efficient way to define class __init__ in a metaclass?
Is there any way to refer to class instance (normally self) in a metaclass?
When using both __new__ and __init__
in a metaclass, their arguments must
be defined the same?
I think Alex Martelli explains
it most succinctly:
class Name(Base1,Base2): <<body>>
__metaclass__==suitable_metaclass
means
Name = suitable_metaclass('Name', (Base1,Base2), <<dict-built-by-body>>)
So stop thinking about
suitable_metaclass as a metaclass
for a moment and just regard it as a
class. Whenever you see
suitable_metaclass('Name', (Base1,Base2), <<dict-built-by-body>>)
it tells you that
suitable_metaclass's __new__
method must have a signature
something like
def __new__(metacls, name, bases, dct)
and a __init__ method like
def __init__(cls, name, bases, dct)
So the signatures are not exactly the same, but they differ only in the first argument.
What's most efficient way to define
class __init__ in a metaclass?
What do you mean by efficient? It is
not necessary to define the __init__
unless you want to.
Is there any way to refer to class
instance (normally self) in a
metaclass?
No, and you should not need to.
Anything that depends on the class
instance should be dealt with in the
class definition, rather than in the
metaclass.
For 1: The __init__ and __new__ of any class have to accept the same arguments, because they would be called with the same arguments. It's common for __new__ to take more arguments that it ignores (e.g. object.__new__ takes any arguments and it ignores them) so that __new__ doesn't have to be overridden during inheritance, but you usually only do that when you have no __new__ at all.
This isn't a problem here, because as it was stated, metaclasses are always called with the same set of arguments always so you can't run into trouble. With the arguments at least. But if you're modifying the arguments that are passed to the parent class, you need to modify them in both.
For 2: You usually don't define the class __init__ in a metaclass. You can write a wrapper and replace the __init__ of the class in either __new__ or __init__ of the metaclass, or you can redefine the __call__ on the metaclass. The former would act weirdly if you use inheritance.
import functools
class A(type):
def __call__(cls, *args, **kwargs):
r = super(A, cls).__call__(*args, **kwargs)
print "%s was instantiated" % (cls.__name__, )
print "the new instance is %r" % (r, )
return r
class B(type):
def __init__(cls, name, bases, dct):
super(B, cls).__init__(name, bases, dct)
if '__init__' not in dct:
return
old_init = dct['__init__']
#functools.wraps(old_init)
def __init__(self, *args, **kwargs):
old_init(self, *args, **kwargs)
print "%s (%s) was instantiated" % (type(self).__name__, cls.__name__)
print "the new instance is %r" % (self, )
cls.__init__ = __init__
class T1:
__metaclass__ = A
class T2:
__metaclass__ = B
def __init__(self):
pass
class T3(T2):
def __init__(self):
super(T3, self).__init__()
And the result from calling it:
>>> T1()
T1 was instantiated
the new instance is <__main__.T1 object at 0x7f502c104290>
<__main__.T1 object at 0x7f502c104290>
>>> T2()
T2 (T2) was instantiated
the new instance is <__main__.T2 object at 0x7f502c0f7ed0>
<__main__.T2 object at 0x7f502c0f7ed0>
>>> T3()
T3 (T2) was instantiated
the new instance is <__main__.T3 object at 0x7f502c104290>
T3 (T3) was instantiated
the new instance is <__main__.T3 object at 0x7f502c104290>
<__main__.T3 object at 0x7f502c104290>
For 3: Yes, from __call__ as shown above.
Related
I defined a metaclass which add a method named "test" to the created classes:
class FooMeta(type):
def __new__(mcls, name, bases, attrs):
def test(self):
return super().test()
attrs["test"] = test
cls = type.__new__(mcls, name, bases, attrs)
return cls
Then I create two classes using this Metaclass
class A(metaclass=FooMeta):
pass
class B(A):
pass
When I run
a = A()
a.test()
a TypeError is raised at super().test():
super(type, obj): obj must be an instance or subtype of type
Which means super() cannot infer the parent class correctly. If I change the super call into
def __new__(mcls, name, bases, attrs):
def test(self):
return super(cls, self).test()
attrs["test"] = test
cls = type.__new__(mcls, name, bases, attrs)
return cls
then the raised error becomes:
AttributeError: 'super' object has no attribute 'test'
which is expected as the parent of A does not implement test method.
So my question is what is the correct way to call super() in a dynamically added method? Should I always write super(cls, self) in this case? If so, it is too ugly (for python3)!
Parameterless super() is very special in Python because it triggers some behavior during code compilation time itself: Python creates an invisible __class__ variable which is a reference to the "physical" class statement body were the super() call is embedded (it also happens if one makes direct use of the __class__ variable inside a class method).
In this case, the "physical" class where super() is called is the metaclass FooMeta itself, not the class it is creating.
The workaround for that is to use the version of super which takes 2 positional arguments: the class in which it will search the immediate superclass, and the instance itself.
In Python 2 and other occasions one may prefer the parameterized use of super, it is normal to use the class name itself as the first parameter: at runtime, this name will be available as a global variable in the current module. That is, if class A would be statically coded in the source file, with a def test(...): method, you would use super(A, self).test(...) inside its body.
However, although the class name won't be available as a variable in the module defining the metaclass, you really need to pass a reference to the class as the first argument to super. Since the (test) method receives self as a reference to the instance, its class is given by either self.__class__ or type(self).
TL;DR: just change the super call in your dynamic method to read:
class FooMeta(type):
def __new__(mcls, name, bases, attrs):
def test(self):
return super(type(self), self).test()
attrs["test"] = test
cls = type.__new__(mcls, name, bases, attrs)
return cls
I want to decorate all the methods of my class. I have written a sample small decorator for illustration purpose here.
Decorator:
def debug(func):
msg = func.__name__
#wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper
def debugmethods(cls):
for key, val in vars(cls).items():
if callable(val):
setattr(cls, key, debug(val))
return cls
Now I want to decorate all the methods of my class. One simple way is to use #debugmethods annotation on top of my class but I am trying to understand two other different approaches for doing so.
a) Overriding __new__
class Spam:
def __new__(cls, *args, **kwargs):
clsobj = super().__new__(cls)
clsobj = debugmethods(clsobj)
return clsobj
def __init__(self):
pass
def foo(self):
pass
def bar(self):
pass
spam = Spam()
spam.foo()
b) Writing metaclass
class debugmeta(type):
def __new__(cls, clsname, bases, clsdict):
clsobj = super().__new__(cls, clsname, bases, clsdict)
clsobj = debugmethods(clsobj)
return clsobj
class Spam(metaclass = debugmeta):
def foo(self):
pass
def bar(self):
pass
spam = Spam()
spam.foo()
I am not sure
Why " a) overriding __new__ " doesn't work ?
Why signature of method __new__ is different in metaclass?
Can someone help me understand what am I missing here.
You appear to be confused between __new__ and metaclasses. __new__ is called to create a new object (an instance from a class, a class from a metaclass), it is not a 'class created' hook.
The normal pattern is:
Foo(...) is translated to type(Foo).__call__(Foo, ...), see special method lookups for why that is. The type() of a class is it's metaclass.
The standard type.__call__ implementation used when Foo is a custom Python class will call __new__ to create a new instance, then call the __init__ method on that instance if the result is indeed an instance of the Foo class:
def __call__(cls, *args, **kwargs): # Foo(...) -> cls=Foo
instance = cls.__new__(cls, *args, **kwargs) # so Foo.__new__(Foo, ...)
if isinstance(instance, cls):
instance.__init__(*args, **kwargs) # Foo.__init__(instance, ...)
return instance
So Foo.__new__ is not called when the Foo class itself is created, only when instances of Foo are created.
You don't usually need to use __new__ in classes, because __init__ suffices to initialise the attributes of instances. But for immutable types, like int or tuple, you can only use __new__ to prepare the new instance state, as you can't alter the attributes of an immutable object once it is created. __new__ is also helpful when you want change what kinds of instances ClassObj() produce (such as creating singletons or producing specialised subclasses instead).
The same __call__ -> __new__ and maybe __init__ process applies to metaclasses. A class Foo: ... statement is implemented by calling the metaclass to create a class object, passing in 3 arguments: the class name, the class bases, and the class body, as a dictionary usually. With class Spam(metaclass = debugmeta): ..., that means debugmeta('Spam', (), {...}) is called, which means debugmeta.__new__(debugmeta, 'Spam', (), {...}) is called.
Your first attempt a, setting Spam.__new__ doesn't work, because you are not creating a class object there. Instead, super().__new__(cls) creates an empty Spam() instance with no attributes, so vars() returns an empty dictionary and debugmethods() ends up doing nothing.
If you want to hook into class creation, then you want a metaclass.
I am trying to get my head around metaclass but i still can't really get the concept of it.
For all i know:
Any class is itself an instance of type "type" - therefore "calling" a class just calls the method __call__ on its class - which happens to be type's __call__. The effect of type.__call__ is exactly: on code like:
class A:
pass
b = A()
The sequence of steps i know here is:
1.type.__call__ receives the class A itself as its first parameter.
It calls the A.__new__ - in pseudocode we could write instance = A.__new__(cls) as what runs.
3.That returns an instance of the "A" class
4.Then it calls __init__ on the instance(instance.__init__())
...and returns that instance return instance
But now consider the below code:
class MetaOne(type):
def __new__(meta, classname, supers, classdict):
print('In MetaOne.new:', meta, classname, supers, classdict, sep='\n...')
return type.__new__(meta, classname, supers, classdict)
class Eggs:
pass
print('making class')
class Spam(Eggs, metaclass=MetaOne):
data = 1
def meth(self, arg):
return self.data + arg
print('making instance')
X = Spam()
print('data:', X.data, X.meth(2))
The output from this script is as follows:
making class
In MetaOne.new:
...<class '__main__.MetaOne'>
...Spam
...(<class '__main__.Eggs'>,)
...{'__qualname__': 'Spam', '__module__': '__main__', 'meth': <function Spam.met
h at 0x00000000010C1D08>, 'data': 1}
making instance
data: 1 3
So as per my understanding this is the sequence of steps:
Since Spam is an instance of MetaOne, calling X = Spam() would try to call the __call__ method of MetaOne class which is not there .
Since MetaOne inherits from type it would call the __call__ method of type class with Spam as the first argument.
After that the call lands up in the __new__ method of MetaOne class but it should contain Spam as the first param.
From where does meta argument of MetaOne class come into picture.
Please help me in my understanding.
Since Spam is an instance of MetaOne, calling X = Spam() would try to
call the __call__ method of MetaOne class which is not there .
That is the soruce of your confusion - the __call__ (or __new__ and __init__) of the metaclass is not called when you create ordinary instances of the class.
Also, since there is no __call__ method for MetaOne, the usual inheritance rules apply: the __call__ method on MetaOne's superclass is used (and it is type.__call__)
The metaclass' __new__ and __init__ methods are invoked when the class body itself is executed (as you can see in your example, the "print" in the metaclass' __new__ shows up before the "making instance" text).
When creating an instance of Span itself, the metaclass methods __new__ and __init__ are not called - the metaclass __call__ is called - and it is that that executes the class's(Span's) __new__ and __init__. In other words: the metaclass __call__ is responsible for calling the "ordinary" class's __new__ and __init__.
Since MetaOne inherits from type it would call the call method of
type class with Spam as the first argument.
And it does, but you made no print statements to "view" this happening:
class MyMeta(type):
def __new__(metacls, name, bases, namespace):
print("At meta __new__")
return super().__new__(metacls, name, bases, namespace)
def __call__(cls, *args, **kwd):
print ("at meta __call__")
return super().__call__(*args, **kwd)
def Egg(metaclass=MyMeta):
def __new__(cls):
print("at class __new__")
If I paste this at the ineractive console, at this point it prints:
At meta __new__
Then, going on the interactive session:
In [4]: fried = Egg()
at meta __call__
at class __new__
And the extra mind-twisting thing is that: "type is type's own metaclass": meaning that type's __call__ is also responsible for running the __new__ and __init__ methods on the metaclass itself, when a new (non-meta) class body is executed.
i had a class called CacheObject,and many class extend from it.
now i need to add something common on all classes from this class so i write this
class CacheObject(object):
def __init__(self):
self.updatedict = dict()
but the child class didn't obtain the updatedict attribute.i know calling super init function was optional in python,but is there an easy way to force all of them to add the init rather than walk all the classes and modify them one by one?
I was in a situation where I wanted classes to always call their base classes' constructor in order before they call their own. The following is Python3 code that should do what you want:
class meta(type):
def __init__(cls,name,bases,dct):
def auto__call__init__(self, *a, **kw):
for base in cls.__bases__:
base.__init__(self, *a, **kw)
cls.__init__child_(self, *a, **kw)
cls.__init__child_ = cls.__init__
cls.__init__ = auto__call__init__
class A(metaclass=meta):
def __init__(self):
print("Parent")
class B(A):
def __init__(self):
print("Child")
To illustrate, it will behave as follows:
>>> B()
Parent
Child
<__main__.B object at 0x000001F8EF251F28>
>>> A()
Parent
<__main__.A object at 0x000001F8EF2BB2B0>
I suggest a non-code fix:
Document that super().__init__() should be called by your subclasses before they use any other methods defined in it.
This is not an uncommon restriction. See, for instance, the documentation for threading.Thread in the standard library, which says:
If the subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread.
There are probably many other examples, I just happened to have that doc page open.
You can override __new__. As long as your base classes doesn't override __new__ without calling super().__new__, then you'll be fine.
class CacheObject(object):
def __new__(cls, *args, **kwargs):
instance = super().__new__(cls, *args, **kwargs)
instance.updatedict = {}
return instance
class Foo(CacheObject):
def __init__(self):
pass
However, as some commenters said, the motivation for this seems a little shady. You should perhaps just add the super calls instead.
This isn't what you asked for, but how about making updatedict a property, so that it doesn't need to be set in __init__:
class CacheObject(object):
#property
def updatedict(self):
try:
return self._updatedict
except AttributeError:
self._updatedict = dict()
return self._updatedict
Hopefully this achieves the real goal, that you don't want to have to touch every subclass (other than to make sure none uses an attribute called updatedict for something else, of course).
There are some odd gotchas, though, because it is different from setting updatedict in __init__ as in your question. For example, the content of CacheObject().__dict__ is different. It has no key updatedict because I've put that key in the class, not in each instance.
Regardless of motivation, another option is to use __init_subclass__() (Python 3.6+) to get this kind of behavior. (For example, I'm using it because I want users not familiar with the intricacies of Python to be able to inherit from a class to create specific engineering models, and I'm trying to keep the structure of the class they have to define very basic.)
In the case of your example,
class CacheObject:
def __init__(self) -> None:
self.updatedict = dict()
def __init_subclass__(cls) -> None:
orig_init = cls.__init__
#wraps(orig_init)
def __init__(self, *args, **kwargs):
orig_init(self, *args, **kwargs)
super(self.__class__, self).__init__()
cls.__init__ = __init__
What this does is any class that subclasses CacheObject will now, when created, have its __init__ function wrapped by the parent class—we're replacing it with a new function that calls the original, and then calls super() (the parent's) __init__ function. So now, even if the child class overrides the parent __init__, at the instance's creation time, its __init__ is then wrapped by a function that calls it and then calls its parent.
You can add a decorator to your classes :
def my_decorator(cls):
old_init = cls.__init__
def new_init(self):
self.updatedict = dict()
old_init(self)
cls.__init__ = new_init
return cls
#my_decorator
class SubClass(CacheObject):
pass
if you want to add the decorators to all the subclasses automatically, use a metaclass:
class myMeta(type):
def __new__(cls, name, parents, dct):
return my_decorator(super().__new__(cls, name, parents, dct))
class CacheObject(object, metaclass=myMeta):
pass
Say I've got a metaclass and a class using it:
class Meta(type):
def __call__(cls, *args):
print "Meta: __call__ with", args
class ProductClass(object):
__metaclass__ = Meta
def __init__(self, *args):
print "ProductClass: __init__ with", args
p = ProductClass(1)
Output as follows:
Meta: __call__ with (1,)
Question:
Why isn't ProductClass.__init__ triggered...just because of Meta.__call__?
UPDATE:
Now, I add __new__ for ProductClass:
class ProductClass(object):
__metaclass__ = Meta
def __new__(cls, *args):
print "ProductClass: __new__ with", args
return super(ProductClass, cls).__new__(cls, *args)
def __init__(self, *args):
print "ProductClass: __init__ with", args
p = ProductClass(1)
Is it Meta.__call__'s responsibility to call ProductClass's __new__ and __init__?
There is a difference in OOP between extending a method and overriding it, what you just did in your metaclass Meta is called overriding because you defined your __call__ method and you didn't call the parent __call__. to have the behavior that you want you have to extend __call__ method by calling the parent method:
class Meta(type):
def __call__(cls, *args):
print "Meta: __call__ with", args
return super(Meta, cls).__call__(*args)
Yes - it's up to Meta.__call__ to call ProductClass.__init__ (or not, as the case may be).
To quote the documentation:
for example defining a custom __call__() method in the metaclass
allows custom behavior when the class is called, e.g. not always
creating a new instance.
That page also mentions a scenario where the metaclass's __call__ may return an instance of a different class (i.e. not ProductClass in your example). In this scenario it would clearly be inappropriate to call ProductClass.__init__ automatically.