I understand that in python, whenever you access a class/instance variable, it will call __getattribute__ method to get the result. However I can also use obj.__dict__['x'] directly, and get what I want.
I am a little confused about what is the difference?
Also when I use getattr(obj, name), is it calling __getattribute__ or obj.__dict__[name] internally?
Thanks in advance.
__getattribute__() method is for lower level attribute processing.
Default implementation tries to find the name
in the internal __dict__ (or __slots__). If the attribute is not found, it calls __getattr__().
UPDATE (as in the comment):
They are different ways for finding attributes in the Python data model. They are internal methods designed to fallback properly in any possible situation. A clue: "The machinery is in object.__getattribute__() which transforms b.x into type(b).__dict__['x'].__get__(b, type(b))." from docs.python.org/3/howto/descriptor.html
Attributes in __dict__ are only a subset of all attributes that an object has.
Consider this class:
class C:
ac = "+AC+"
def __init__(self):
self.ab = "+AB+"
def show(self):
pass
An instance ic = C() of this class will have attributes 'ab', 'ac' and 'show' (and few others). The __gettattribute__ will find them all, but only the 'ab' is stored in the ic.__dict__. The other two can be found in C.__dict__.
Not every python object has a dictionary where it's attributes are stored, there are slots, properties and attributes, that are calculated, wenn needed. You can also overwrite __getattribute__ and __getattr__. Attribute access is more complicated, than a simple dictionary lookup. So the normal way to access attribute is by
obj.x
wenn you have a variable with attribute name:
getattr(obj, name)
Normally you shouldn't use the internal attributes __xxx__.
Related
Consider:
class Parent():
def __init__(self, last_name, eye_color):
self.last_name = last_name
self.eye_color = eye_color
def show_info(self):
print("Last Name - "+self.last_name)
print("Eye Color - "+self.eye_color)
billy_cyrus = Parent("Cyrus", "blue")
The above is from the Udacity Python course. I discovered I'm able to call show_info for instance billy_cyrus using either of the following:
billy_cyrus.show_info()
Parent.show_info(billy_cyrus)
I'm curious as to why. Is there a difference between the two methods? If so when would one be used vs. the other? I'm using Python 3.6 if that matters.
In terms of just calling the method, there is no difference most of the time. In terms of how the underlying machinery, works, there is a bit of a difference.
Since show_info is a method, it is a descriptor in the class. That means that when you access it through an instance in which it is not shadowed by another attribute, the . operator calls __get__ on the descriptor to create a bound method for that instance. A bound method is basically a closure that passes in the self parameter for you before any of the other arguments you supply. You can see the binding happen like this:
>>> billy_cyrus.show_info
<bound method Parent.show_info of <__main__.Parent object at 0x7f7598b14be0>>
A different closure is created every time you use the . operator on a class method.
If you access the method through the class object, on the other hand, it does not get bound. The method is a descriptor, which is just a regular attribute of the class:
>>> Parent.show_info
<function __main__.Parent.show_info>
You can simulate the exact behavior of binding a method before calling it by calling its __get__ yourself:
>>> bound_meth = Parent.show_info.__get__(billy_cyrus, type(billy_cyrus))
>>> bound_meth
<bound method Parent.show_info of <__main__.Parent object at 0x7f7598b14be0>>
Again, this will not make any difference to you in 99.99% of cases, since functionally bound_meth() and Parent.bound_meth(billy_cyrus) end up calling the same underlying function object with the same parameters.
Where it matters
There are a couple of places where it matters how you call a class method. One common use case is when you override a method, but want to use the definition provided in the parent class. For example, say I have a class that I made "immutable" by overriding __setattr__. I can still set attributes on the instance, as in the __init__ method shown below:
class Test:
def __init__(self, a):
object.__setattr__(self, 'a', a)
def __setattr__(self, name, value):
raise ValueError('I am immutable!')
If I tried to do a normal call to __setattr__ in __init__ by doing self.a = a, a ValueError would be raised every time. But by using object.__setattr__, I can bypass this limitation. Alternatively, I could do super().__setattr__('a', a) for the same effect, or self.__dict__['a'] = a for a very similar one.
#Silvio Mayolo's answer has another good example, where you would deliberately want to use the class method as a function that could be applied to many objects.
Another place it matters (although not in terms of calling methods), is when you use other common descriptors like property. Unlike methods, properties are data-descriptors. This means that they define a __set__ method (and optionally __delete__) in addition to __get__. A property creates a virtual attribute whose getter and setter are arbitrarily complex functions instead of just simple assignments. To properly use a property, you have to do it through the instance. For example:
class PropDemo:
def __init__(self, x=0):
self.x = x
#property
def x(self):
return self.__dict__['x']
#x.setter
def x(self, value):
if value < 0:
raise ValueError('Not negatives, please!')
self.__dict__['x'] = value
Now you can do something like
>>> inst = PropDemo()
>>> inst.x
0
>>> inst.x = 3
>>> inst.x
3
If you try to access the property through the class, you can get the underlying descriptor object since it will be an unbound attribute:
>>> PropDemo.x
<property at 0x7f7598af00e8>
On a side note, hiding attributes with the same name as a property in __dict__ is a neat trick that works because data descriptors in a class __dict__ trump entries in the instance __dict__, even though instance __dict__ entries trump non-data-descriptors in a class.
Where it can Get Weird
You can override a class method with an instance method in Python. That would mean that type(foo).bar(foo) and foo.bar() don't call the same underlying function at all. This is irrelevant for magic methods because they always use the former invocation, but it can make a big difference for normal method calls.
There are a few ways to override a method on an instance. The one I find most intuitive is to set the instance attribute to a bound method. Here is an example of a modified billy_cyrus, assuming the definition of Parent in the original question:
def alt_show_info(self):
print('Another version of', self)
billy_cyrus.show_info = alt_show_info.__get__(billy_cyrus, Parent)
In this case, calling the method on the instance vs the class would have completely different results. This only works because methods are non-data descriptors by the way. If they were data descriptors (with a __set__ method), the assignment billy_cyrus.show_info = alt_show_info.__get__(billy_cyrus, Parent) would not override anything but would instead just redirect to __set__, and manually setting it in b
billy_cyrus's __dict__ would just get it ignored, as happens with a property.
Additional Resources
Here are a couple of resources on descriptors:
Python Reference - Descriptor Protocol: http://python-reference.readthedocs.io/en/latest/docs/dunderdsc/
(Official?) Descriptor HowTo Guide: https://docs.python.org/3/howto/descriptor.html
There is no semantic difference between the two. It's entirely a matter of style. You would generally use billy_cyrus.show_info() in normal use, but the fact that the second approach is allowed permits you to use Parent.show_info to get the method as a first-class object itself. If that was not allowed, then it would not be possible (or at least, it would be fairly difficult) to do something like this.
function = Parent.show_info
so_many_billy_cyrus = [billy_cyrus, billy_cyrus, billy_cyrus]
map(function, so_many_billy_cyrus)
'Every thing in python is an object'
So, should all objects have to have attributes and methods ?
I read below statemts in tutorial site, could you give example of pre-defined object in python that has neither attributes nor methods ?
Some objects have neither attributes nor methods
Everything in Python is indeed an object, this is true. Even classes themselves are considered to be objects and they're indeed the product of the builtin class typewhich is not surprisingly an object too. But objects will almost certainly inherit attributes including methods or data in most circumstances.
So, should all objects have to have attributes and methods ?
Not necessarily all objects have their own attributes. For instance, an object can inherit the attribute from its class and its class's superclasses, but that attribute or method doesn't necessarily live within the instance's namespace dictionary. In fact, instances' namespaces could be empty like the following:
class Foo:
pass
a = A()
print(a.__dict__)
a here doesn't have any attributes aside from those inherited from its class so if you check its namespace through the builtin attribute __dict__ you'll find the namespace to be an empty dictionary. But you might wonder isn't a.__dict__ an attribute of a? Make a distinction between class-level attributes--attributes inherited from the class or its superclasses and instance-attributes--attributes that belong to the instance and usually live in its namespace __dict__.
Could you give example of pre-defined object in python that has neither attributes nor methods ?
If you meant by predefined object, a builtin object, I couldn't imagine such scenario. Again, even if there are no attributes at the object itself, there would be attributes inherited from its class or the class's superclasses if there's any superclass in most cases. Probably and I'm guessing here, the tutorial is asking you to create class that assigns no attributes to its objects, just like the code I included above.
And this already answers your question better: Is everything an object in python like ruby?
There's a hackish way to emulate a Python object with no attributes.
class NoAttr(object):
def __getattribute__(self, attr):
raise AttributeError("no attribute: %s" % attr)
def __setattr__(self, attr, value):
raise AttributeError("can't set attribute: %s" % attr)
def __delattr__(self, attr):
raise AttributeError("no attribute: %s" % attr)
a = NoAttr()
This instance a, for all intents and purposes, in pure Python, behaves like an object with no attributes (you can try hasattr on it).
There may be a low-level way to do this in a C extension by implementing a type in C that pathologically stops Python's object implementation from working. Anyway the margin here is too small for writing one.
A pre-defined object with no attributes would defeat the purpose of pre-defining it.
Just out of curiosity I'm playing with the __dict__ attribute on Python classes.
I read somewhere that a "class" in python is kind of a dict, and calling __dict__ on a class instance translate the instance into a dict... and I thought "cool! I can use this!"
But this lead me to doubts about the correctness and security of these actions.
For example, if I do:
class fooclass(object):
def __init__(self, arg):
self.arg1 = arg
p = fooclass('value1')
print(p.__dict__)
p.__dict__['arg2'] = 'value2'
print(p.__dict__)
print(p.arg2)
I have this:
>>>{'arg1': 'value1'}
>>>{'arg1': 'value1', 'arg2': 'value2'}
>>>value2
and that's fine, but:
Does the fooclass still have 1 attribute? How can I be sure?
Is it secure to add attributes that way?
Have you ever had cases where this came in handy?
I see that I can't do fooclass.__dict__['arg2'] = 'value2'.. so why this difference between a class and an instance?
You are altering the attributes of the instance. Adding and removing from the __dict__ is exactly what happens for most custom class instances.
The exception is when you have a class that uses __slots__; instances of such a class do not have a __dict__ attribute as attributes are stored in a different way.
Python is a language for consenting adults; there is no protection against adding attributes to instances with a __dict__ in any case, so adding them to the dictionary or by using setattr() makes no difference.
Accessing the __dict__ is helpful when you want to use existing dict methods to access or alter attributes; you can use dict.update() for example:
def Namespace(object):
def __init__(self, **kwargs):
self.__dict__.update(kwargs)
It is also helpful when trying to read an attribute on the instance for which there is a corresponding data descriptor on the class; to bypass the latter you can access the __dict__ on the instance to get to the attribute. You can also use this to test for attributes on the instance and ignore anything the class or the base classes might define.
As for fooclass.__dict__; you are confusing the class with the instances of a class. They are separate objects. A class is a different type of object, and ClassObj.__dict__ is just a proxy object; you can still use setattr() on the class to add arbitrary attributes:
setattr(fooclass, 'arg2', 'value2')
or set attributes directly:
fooclass.arg2 = value2
A class needs to have a __dict__ proxy, because the actual __dict__ attribute is a descriptor object to provide that same attribute on instances. See What is the __dict__.__dict__ attribute of a Python class?
Details about how Python objects work are documented in the Datamodel reference documentation.
With Class attributes and methods, to reference them within the class, should we use Classname. or self.
I think both ways work, but it is a bit wierd compared to Java.
self.attr refers to the instance's attribute, and falls back to the class if there is no attribute of the specified name defined on the instance. Usually this is reasonable behavior. Use self.__class__.attr if you specifically want to get the class's attribute.
It is rare that anyone will override a method on an instance, this requiring the construction of a special "instance method" object, so self.method() is generally correct (even for methods declared using #classmethod or #staticmethod).
This one seems a bit tricky to me. Sometime ago I already managed to overwrite an instance's method with something like:
def my_method(self, attr):
pass
instancemethod = type(self.method_to_overwrite)
self.method_to_overwrite = instancemethod(my_method, self, self.__class__)
which worked very well for me; but now I'm trying to overwrite an instance's __getattribute__() function, which doesn't work for me for the reason the method seems to be
<type 'method-wrapper'>
Is it possible to do anything about that? I couldn't find any decent Python documentation on method-wrapper.
You want to override the attribute lookup algorithm on an per instance basis? Without knowing why you are trying to do this, I would hazard a guess that there is a cleaner less convoluted way of doing what you need to do. If you really need to then as Aaron said, you'll need to install a redirecting __getattribute__ handler on the class because Python looks up special methods only on the class, ignoring anything defined on the instance.
You also have to be extra careful about not getting into infinite recursion:
class FunkyAttributeLookup(object):
def __getattribute__(self, key):
try:
# Lookup the per instance function via objects attribute lookup
# to avoid infinite recursion.
getter = object.__getattribute__(self, 'instance_getattribute')
return getter(key)
except AttributeError:
return object.__getattribute__(self, key)
f = FunkyAttributeLookup()
f.instance_getattribute = lambda attr: attr.upper()
print(f.foo) # FOO
Also, if you are overriding methods on your instance, you don't need to instanciate the method object yourself, you can either use the descriptor protocol on functions that generates the methods or just curry the self argument.
#descriptor protocol
self.method_to_overwrite = my_method.__get__(self, type(self))
# or curry
from functools import partial
self.method_to_overwrite = partial(my_method, self)
You can't overwrite special methods at instance level. For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.
There are a couple of methods which you can't overwrite and __getattribute__() is one of them.
I believe method-wrapper is a wrapper around a method written in C.