I define a python class in python interpreter
class A:
pass
I get base class of A using A.__bases__, it shows
(object,)
but when I enter dir(A), the output don't contain __bases__ attribute, then I try dir(object), __bases__ is not found either, where does the __bases__ come from?
The __bases__ attribute in a class is implemented by a descriptor in the metaclass, type. You have to be a little careful though, since type, as one of the building blocks of the Python object model, is an instance of itself, and so type.__bases__ doesn't do what you would want for introspection.
Try this:
descriptor = type.__dict__['__bases__']
print(descriptor, type(descriptor))
You can reproduce the same kind of thing with your own descriptors:
class MyMeta(type):
#property # a property is a descriptor
def foo(cls):
return "foo"
class MyClass(metaclass=MyMeta):
pass
Now if you access MyClass.foo you'll get the string foo. But you won't see foo in the variables defined in MyClass (if you check with vars or dir). Nor can you access it through an instance of MyClass (my_obj = MyClass(); my_obj.foo raises an AttributeError).
it is a special attribute akin to __name__ or __dict__. While the result of function dir actually depends on the implementation of __dir__ function.
You might want to look it on the doc here https://docs.python.org/3/reference/datamodel.html
Related
A method defined on a metaclass is accessible by classes that use the metaclass. However, the method will not be accessible on the instances of these classes.
My first guess was that metaclass methods would not be accessible on either classes or instances.
My second guess was that metaclass methods would be accessible on both classes and instances.
I find it surprising that metaclass methods are instead accessible on classes, but not on instances.
What is the purpose of this behavior? Is there any case where I can use this to an advantage? If there is no intended purpose, how does the implementation work such that this is the resulting behavior?
class Meta(type):
def __new__(mcs, name, bases, dct):
mcs.handle_foo(dct)
return type.__new__(mcs, name, bases, dct)
#classmethod
def handle_foo(mcs, dct):
"""
The sole purpose of this method is to encapsulate some logic
instead of writing code directly in __new__,
and also that subclasses of the metaclass can override this
"""
dct['foo'] = 1
class Meta2(Meta):
#classmethod
def handle_foo(mcs, dct):
"""Example of Metaclass' subclass overriding"""
dct['foo'] = 10000
class A(metaclass=Meta):
pass
class B(metaclass=Meta2):
pass
assert A.foo == 1
assert B.foo == 10000
assert hasattr(A, 'handle_foo')
assert hasattr(B, 'handle_foo')
# What is the purpose or reason of this method being accessible on A and B?
# If there is no purpose, what about the implementation explains why it is accessible here?
instance = A()
assert not hasattr(instance, 'handle_foo')
# Why is this method not accessible on the instance, when it is on the class?
# What is the purpose or reason for this method not being accessible on the instance?
What is the purpose of this behavior? What use case is this behavior intended to support? I am interested in a direct quote from the documentation, if one exists.
If there is no purpose, and this is simply a byproduct of the implementation, why does the implementation result in this behavior? I.e., how are metaclasses implemented such that the methods defined on the metaclass are also defined accessible on classes that use the metaclass, but not the instantiated objects of these classes?
There is only one practical implication of this that I have found is the following: Pycharm will include these metaclass functions in the code completion box when you start typing A. (i.e., the class). I don't want users of my framework to see this. One way to mitigate this as by renaming these methods as private methods (e.g. _handle_foo), but still I would rather these methods not show up in code completion at all. Using a dunder naming convention (__) won't work, as subclasses of the metaclass will not be able to override the methods.
(I've edited this post extensively due to the thoughtful feedback from Miyagi and Serge, in order to make it more clear as to why I am defining methods on the metaclass in the first place: simply in order to encapsulate some behavior instead of putting all the code in __new__, and to allow those methods to be overridden by subclasses of the metaclass)
Let us first look at this in a non-meta situation: We define a function inside a class and access it via the instance.
>>> class Foo:
... def bar(self): ...
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
>>> foo = Foo()
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10dc75790>>
Of note is that the two "attributes" are not the same kind: The class' attribute is the very thing we put into it, but the instance's "attribute" is a dynamically created thing.
Likewise, methods defined on a metaclass are not inherited by the class, they are (dynamically) bound to its classes.
>>> Meta.meta_method # direct access to "class" attribute
<function __main__.Meta.meta_method(cls)>
>>> Foo.meta_method # instance access to "class" attribute
<bound method Meta.meta_method of <class '__main__.Foo'>>
This is the exact same mechanism – because a class is "just" a metaclass' instance.
It should be obvious at this point that the attributes defined on the metaclass and dynamically bound to the class are not the same thing, and there is no reason for them to behave the same. Whether lookup of attributes on an instance picks up metaclass-methods from their dynamic form on the class, directly from the metaclass or not at all is a judgement call.
Python's data model defines that default lookup only takes into account the instance and the instance's type. The instance's type's type is explicitly excluded.
Invoking Descriptors
[…]
The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance, a.x has a lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'], and continuing through the base classes of type(a) excluding metaclasses.
There is no rationale given for this approach. However, it is sufficient to replicate common instantiation+inheritance behaviour of other languages. At the same time, it avoids arbitrarily deep lookups and the issue that type is a recursive metaclass.
Notably, since a metaclass is in full control of how a class behaves, it can directly define methods in the class or redefine attribute access to circumvent the default behaviour.
I have a class with a private constant _BAR = object().
In a child class, outside of a method (no access to self), I want to refer to _BAR.
Here is a contrived example:
class Foo:
_BAR = object()
def __init__(self, bar: object = _BAR):
...
class DFoo(Foo):
"""Child class where I want to access private class variable from parent."""
def __init__(self, baz: object = super()._BAR):
super().__init__(baz)
Unfortunately, this doesn't work. One gets an error: RuntimeError: super(): no arguments
Is there a way to use super outside of a method to get a parent class attribute?
The workaround is to use Foo._BAR, I am wondering though if one can use super to solve this problem.
Inside of DFoo, you cannot refer to Foo._BAR without referring to Foo. Python variables are searched in the local, enclosing, global and built-in scopes (and in this order, it is the so called LEGB rule) and _BAR is not present in any of them.
Let's ignore an explicit Foo._BAR.
Further, it gets inherited: DFoo._BAR will be looked up first in DFoo, and when not found, in Foo.
What other means are there to get the Foo reference? Foo is a base class of DFoo. Can we use this relationship? Yes and no. Yes at execution time and no at definition time.
The problem is when the DFoo is being defined, it does not exist yet. We have no start point to start following the inheritance chain. This rules out an indirect reference (DFoo -> Foo) in a def method(self, ....): line and in a class attribute _DBAR = _BAR.
It is possible to work around this limitation using a class decorator. Define the class and then modify it:
def deco(cls):
cls._BAR = cls.__mro__[1]._BAR * 2 # __mro__[0] is the class itself
return cls
class Foo:
_BAR = 10
#deco
class DFoo(Foo):
pass
print(Foo._BAR, DFoo._BAR) # 10 20
Similar effect can be achieved with a metaclass.
The last option to get a reference to Foo is at execution time. We have the object self, its type is DFoo, and its parent type is Foo and there exists the _BAR. The well known super() is a shortcut to get the parent.
I have assumed only one base class for simplicity. If there were several base classes, super() returns only one of them. The example class decorator does the same. To understand how several bases are sorted to a sequence, see how the MRO works (Method Resolution Order).
My final thought is that I could not think up a use-case where such access as in the question would be required.
Short answer: you can't !
I'm not going into much details about super class itself here. (I've written a pure Python implementation in this gist if you like to read.)
But now let's see how we can call super:
1- Without arguments:
From PEP 3135:
This PEP proposes syntactic sugar for use of the super type to
automatically construct instances of the super type binding to the
class that a method was defined in, and the instance (or class object
for classmethods) that the method is currently acting upon.
The new syntax:
super()
is equivalent to:
super(__class__, <firstarg>)
...and <firstarg> is the first parameter of the method
So this is not an option because you don't have access to the "instance".
(Body of the function/methods is not executed unless it gets called, so no problem if DFoo doesn't exist yet inside the method definition)
2- super(type, instance)
From documentation:
The zero argument form only works inside a class definition, as the
compiler fills in the necessary details to correctly retrieve the
class being defined, as well as accessing the current instance for
ordinary methods.
What were those necessary details mentioned above? A "type" and A "instance":
We can't pass neither "instance" nor "type" which is DFoo here. The first one is because it's not inside the method so we don't have access to instance(self). Second one is DFoo itself. By the time the body of the DFoo class is being executed there is no reference to DFoo, it doesn't exist yet. The body of the class is executed inside a namespace which is a dictionary. After that a new instance of type type which is here named DFoo is created using that populated dictionary and added to the global namespaces. That's what class keyword roughly does in its simple form.
3- super(type, type):
If the second argument is a type, issubclass(type2, type) must be
true
Same reason mentioned in above about accessing the DFoo.
4- super(type):
If the second argument is omitted, the super object returned is
unbound.
If you have an unbound super object you can't do lookup(unless for the super object's attributes itself). Remember super() object is a descriptor. You can turn an unbound object to a bound object by calling __get__ and passing the instance:
class A:
a = 1
class B(A):
pass
class C(B):
sup = super(B)
try:
sup.a
except AttributeError as e:
print(e) # 'super' object has no attribute 'a'
obj = C()
print(obj.sup.a) # 1
obj.sup automatically calls the __get__.
And again same reason about accessing DFoo type mentioned above, nothing changed. Just added for records. These are the ways how we can call super.
Suppose I define a class A with a meta class like this:
class Meta(type):
pass
class A(metaclass=Meta):
pass
Then, when I try to access the name of class A I get the name of the meta class:
A.__class__.__name__
# 'Meta'
However, shouldn't it give me A, my defined class?
Note: I tried to use A.__mro__[0].__name__ and it does give me A, but I am still confused why A.__class__ gives me the meta class name. Does anyone has an explanation of this?
The __class__ dunder reports:
the class to which a class instance belongs.
Quote from instance.__class__
The class A belongs to the class of it's metaclass - only instances of A belong to the class A itself.
a = A()
print(a.__class__.__name__) # 'A'
print(A.__class__.__name__) # 'Meta'
class P: pass
print(P.__class__.__name__) # type
print(P().__class__.__name__) # P
To get the name of the class itself simply use
A.__name__
if you really need it.
I am still a way to groke all of answer to What are metaclasses in Python? - maybe it helps you out.
A is already the class - its name is under A.__name__.
If you try A.__class__.__name__ you will get to the class of which A is instance (that is, its metaclass), name.
A.__mro__[0].__name__ will follow the "method resolution order" for the class A - the __mro__ object is a tuple with all the class hyerarchy that starts in the defined class itself and ends in object. So, A.__mro__[0] will always be A itself - and A.__mro__[0].__name__ is the same as A.__name__.
The __name__ and __qualname__ attributes are writable attributes: changing { __qualname__ after the class is created will change the default __repr__ for instances of that class, for example. Although they are in the language definition and "live" in slots in the class (not on it's dictionary), it is possible to create a __name__ property (I mean, the built-in property object, or any other descriptor) on the metaclass that will dynamically change the __name__ attribute of a class (but not __qualname__ - this must be an attribute of the class, and must be a string)
When inheriting from a class, the child class is accessible on the parent via the .__subclasses__() method.
class BaseClass:
pass
class SubClass(BaseClass):
pass
BaseClass.__subclasses__()
# [<class '__main__.SubClass'>]
However, deleting the child class doesn't seem to remove it from the parent.
del SubClass
BaseClass.__subclasses__()
# [<class '__main__.SubClass'>]
Where does __subclasses__ get its information from? And can I manipulate it?
Or
Is there a proper way to remove a class and have its parent lose reference to it (e.g. BaseClass.remove_subclass(SubClass)?
The subclass contains references to itself internally, so it continues to exist until it is garbage collected. If you force a garbage collection cycle it will disappear from the __subclasses__():
import gc
gc.collect()
and then it has gone.
However make sure you have deleted all other references to the class before you force the garbage collection. For example, if you do it interactively and the last output was the subclass list there will still be a reference to the class in _.
class BaseClass:
pass
class SubClass(BaseClass):
pass
print(BaseClass.__subclasses__())
# [<class '__main__.SubClass'>]
del SubClass
import gc
gc.collect()
print(BaseClass.__subclasses__())
# []
Output with python 3.7 is:
[<class '__main__.SubClass'>]
[]
I should probably also add that while garbage collection works for this simple case you probably shouldn't depend on it in real life: it would be far too easy to accidentally keep a reference to the subclass somewhere in your code and then wonder why the class never goes away.
What you are trying to do here is keep a registry of subclasses so that the factory can return an object of the appropriate class. If you want to be able to add and remove classes from the registry then I think you have to be explicit. You could still use __subclasses__ to find candidate classes, but keep a flag on each class to show whether it is enabled. Then instead of just deleting the subclass set the flag to show the class is no longer in use and then (if you want) delete it.
Where does __subclasses__ get its information from?
For the CPython implementation of Python, the type object keeps a list of weak references under PyTypeObject.tp_subclasses. This is marked as "Not inherited. Internal use only" in the docs, so can be treated as an implementation detail of CPython. See also: How is __subclasses__ method implemented in CPython?.
And can I manipulate it?
Any class has a .__bases__ descriptor which, if changed, updates the references in PyTypeObject.tp_subclasses.
.__bases__ can only be manipulated when the class doesn't directly inherit from object. So while:
class BaseClass: pass
class OtherClass(BaseClass): pass
and
class BaseClass: pass
class OtherClass: pass
BaseClass.__bases__ = (OtherClass, )
# TypeError: __bases__ assignment: 'BaseClass' deallocator differs from 'object'
should be equivalent *. You will get an error. See: https://bugs.python.org/issue672115
You also can't use this to change a class to inherit from object.
class BaseClass: pass
class SubClass(BaseClass): pass
SubClass.__bases__ = (object,)
# TypeError: __bases__ assignment: 'type' object layout differs from 'BaseClass'
You can, however, change the bases of a class to be another class.
class BaseClass: pass
class SubClass(BaseClass): pass
class OtherClass: pass
SubClass.__bases__ = (OtherClass, )
# Or don't define it.
SubClass.__bases__ = (type("OtherClass", (object, ), {}), )
This all updates the parent class:
>>> BaseClass.__subclasses__()
[]
'Every thing in python is an object'
So, should all objects have to have attributes and methods ?
I read below statemts in tutorial site, could you give example of pre-defined object in python that has neither attributes nor methods ?
Some objects have neither attributes nor methods
Everything in Python is indeed an object, this is true. Even classes themselves are considered to be objects and they're indeed the product of the builtin class typewhich is not surprisingly an object too. But objects will almost certainly inherit attributes including methods or data in most circumstances.
So, should all objects have to have attributes and methods ?
Not necessarily all objects have their own attributes. For instance, an object can inherit the attribute from its class and its class's superclasses, but that attribute or method doesn't necessarily live within the instance's namespace dictionary. In fact, instances' namespaces could be empty like the following:
class Foo:
pass
a = A()
print(a.__dict__)
a here doesn't have any attributes aside from those inherited from its class so if you check its namespace through the builtin attribute __dict__ you'll find the namespace to be an empty dictionary. But you might wonder isn't a.__dict__ an attribute of a? Make a distinction between class-level attributes--attributes inherited from the class or its superclasses and instance-attributes--attributes that belong to the instance and usually live in its namespace __dict__.
Could you give example of pre-defined object in python that has neither attributes nor methods ?
If you meant by predefined object, a builtin object, I couldn't imagine such scenario. Again, even if there are no attributes at the object itself, there would be attributes inherited from its class or the class's superclasses if there's any superclass in most cases. Probably and I'm guessing here, the tutorial is asking you to create class that assigns no attributes to its objects, just like the code I included above.
And this already answers your question better: Is everything an object in python like ruby?
There's a hackish way to emulate a Python object with no attributes.
class NoAttr(object):
def __getattribute__(self, attr):
raise AttributeError("no attribute: %s" % attr)
def __setattr__(self, attr, value):
raise AttributeError("can't set attribute: %s" % attr)
def __delattr__(self, attr):
raise AttributeError("no attribute: %s" % attr)
a = NoAttr()
This instance a, for all intents and purposes, in pure Python, behaves like an object with no attributes (you can try hasattr on it).
There may be a low-level way to do this in a C extension by implementing a type in C that pathologically stops Python's object implementation from working. Anyway the margin here is too small for writing one.
A pre-defined object with no attributes would defeat the purpose of pre-defining it.