Consider a trivial example:
class C:
#staticmethod
def my_static_method():
print("static")
def my_instance_method(self):
print("self")
When I call C().my_static_method(), python doesn't pass the instance of C into my_static_method, and the descriptor that my_static_method references doesn't expect an instance of C, either.
This makes sense.
But then when I call C().my_instance_method(), how does python know to pass the instance of C that I'm calling my_instance_method from in as an argument, without me specifying anything?
As the link explains, function objects are descriptors! Just like staticmethod objects.
They have a __get__ method which returns a bound-method object, which essentially just partially applies the instance itself as the first positional argument. Consider:
>>> def foo(self):
... return self.bar
...
>>> class Baz:
... bar = 42
...
>>> baz = Baz()
>>> bound_method = foo.__get__(baz, Baz)
>>> bound_method
<bound method foo of <__main__.Baz object at 0x7ffcd001c7f0>>
>>> method()
42
By adding the #staticmethod decorator to my_static_method, you told python not to pass the calling instance of C into the function. So you can call this function as C.my_static_method().
By calling C() you created an instance of C. Then you called the non static function my_instance_method() which Python happily passed your new instance of C as the first parameter.
What happens when you call C.my_instance_method() ?
Rhetorical: You'll get a "missing one required arg self" exception -- since my_instance_method only works when calling from an instance unless you decorate it as static.
Of course you can still call the static member from an instance C().my_static_method() but you don't have a self param so no access to the instance.
The key point here is that methods are just functions that happen to be attributes of a class. The actual magic, in Python, happens in the attribute lookup process. The link you give explains earlier just how much happens every time x.y happens in Python. (Remember, everything is an object; that includes functions, classes, modules, type (which is an instance of itself)...)
This process is why descriptors can work at all; why we need explicit self; and why we can do fun things like calling a method with normal function call syntax (as long as we look it up from the class rather than an instance), alias it, mimic the method binding process with functools.partial....
Suppose we have c = C(). When you do c.my_instance_method (never mind calling it for now), Python looks for my_instance_method in type(c) (i.e., in the C class), and also checks if it's a descriptor, and also if it's specifically a data descriptor. Functions are non-data descriptors; even outside of a class, you can write
>>> def x(spam): return spam
...
>>> x.__get__
<method-wrapper '__get__' of function object at 0x...>
Because of the priority rules, as long as c doesn't directly have an attribute attached with the same name, the function will be found in C and its __get__ will be used. Note that the __get__ in question comes from the class - but it isn't using the same process as x.__get__ above. That code looks in the class because that's one of the places checked for an attribute lookup; but when c.my_instance_method redirects to C.my_instance_method.__get__, it's looking there directly - attaching a __get__ attribute directly to the function wouldn't change anything (which is why staticmethod is implemented as a class instead).
That __get__ implements the actual method binding. Let's pretend we found x as a method in the str class:
>>> x.__get__('spam', str)
<bound method x of 'spam'>
>>> x.__get__('spam', str)()
'spam'
Remember, although the function in question takes three arguments, we're calling __get__, itself, as a method - so x gets bound to it in the same way. Equivalently, and more faithful to the actual process:
>>> type(x).__get__(x, 'spam', str)
<bound method x of 'spam'>
>>> type(x).__get__(x, 'spam', str)()
'spam'
So what exactly is that "bound method", anyway?
>>> bound = type(x).__get__(x, 'spam', str)
>>> type(bound)
<class 'method'>
>>> bound.__call__
<method-wrapper '__call__' of method object at 0x...>
>>> bound.__func__
<function x at 0x...>
>>> bound.__self__
'spam'
>>> type(bound)(x, 'eggs')
<bound method x of 'eggs'>
Pretty much what you'd expect: it's a callable object that stores and uses the original function and self value, and does the obvious thing in __call__.
Related
I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.
I just stumbled over this strange behavior when the type of a method changes during subclassing:
class A:
def f(self, x):
return x**2
class B(A):
#classmethod
def f(cls, x):
return x**2
If I now ask for the type of B.f, I'll get the (supposedly) wrong answer:
In [37]: type(B.f)
Out[37]: method
Whereas this works as expected:
In [39]: type(B.__dict__["f"])
Out[39]: classmethod
(Seen in Python 3.4 and 3.6.)
Is this just a bug or is there a specific reason for this?
What's the difference between the attribute f and the .__dict__["f"] item? I thought they were the same.
In a testing suite, I was trying to support both types of methods inside a class to be tested. To be able to do that, I need to know the type in order to pass the correct number of arguments. If it's a normal method (i.e. self is the first argument), I'd just pass None explicitly, which by design shouldn't be used inside the method anyway, since it's not instance-dependent.
Maybe there's a better way to do this, like duck typing the call to the method. But there might be cases where this is not so easy to do, like if the method had *args and **kwargs... Therefore I went with the explicit type check, but got stuck at this point.
No, this is not a bug, this is normal behaviour. A classmethod produces a bound method when accessed on a class. That's exactly the point of a classmethod, to bind a function to the class you access it on or the class of an instance you access it on.
Like function and property objects, classmethod is a descriptor object, it implements a __get__ method. Accessing attributes on an instance or a class is delegated to the __getattribute__ method, and the default implementation of that hook will not just return what it found in object.__dict__[attributename]; it will also bind descriptors, by calling the descriptor.__get__() method. This is a hugely important aspect of Python, it is this mechanism that makes methods and attributes and loads of other things work.
classmethod objects, when bound by the descriptor protocol, return a method object. Method objects are wrappers that record the object bound to, and the function to call when they are called; calling a method really calls the underlying method with the bound object as first argument:
>>> class Foo:
... pass
...
>>> def bar(*args): print(args)
...
>>> classmethod(bar).__get__(None, Foo) # decorate with classmethod and bind
<bound method bar of <class '__main__.Foo'>>
>>> method = classmethod(bar).__get__(None, Foo)
>>> method.__self__
<class '__main__.Foo'>
>>> method.__func__
<function bar at 0x1056f0e18>
>>> method()
(<class '__main__.Foo'>,)
>>> method('additional arguments')
(<class '__main__.Foo'>, 'additional arguments')
So the method object returned for a classmethod object references the class (the second argument to __get__, the owner), and the original function. If you use a class method on an instance, the first argument is still ignored:
>>> classmethod(bar).__get__(Foo(), Foo).__self__ # called on an instance
<class '__main__.Foo'>
Functions, on the other hand, want to bind only to instances; so if the first argument to __get__ is set to None, they simply return self:
>>> bar.__get__(None, Foo) # access on a class
<function bar at 0x1056f0e18>
>>> bar.__get__(Foo(), Foo) # access on an instance
<bound method bar of <__main__.Foo object at 0x105833a90>>
>>> bar.__get__(Foo(), Foo).__self__
<__main__.Foo object at 0x105833160>
If accessing ClassObject.classmethod_object would return the classmethod object itself, like a function object would, then you could never actually use the class method on a class. That'd be rather pointless.
So no, object.attribute is not always the same thing as object.__dict__['attribute']. If object.__dict__['attribute'] supports the descriptor protocol, it'll be invoked.
I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.
I am new to python, and I don't quite understand the __func__ in python 2.7.
I know when I define a class like this:
class Foo:
def f(self, arg):
print arg
I can use either Foo().f('a') or Foo.f(Foo(), 'a') to call this method. However, I can't call this method by Foo.f(Foo, 'a'). But I accidently found that I can use Foo.f.__func__(Foo, 'a') or even Foo.f.__func__(1, 'a') to get the same result.
I print out the values of Foo.f, Foo().f and Foo.f.__func__, and they are all different. However, I have only one piece of code in definition. Who can help to explain how above code actually works, especially the __func__? I get really confused now.
When you access Foo.f or Foo().f a method is returned; it's unbound in the first case and bound in the second. A python method is essentially a wrapper around a function that also holds a reference to the class it is a method of. When bound, it also holds a reference to the instance.
When you call an method, it'll do a type-check on the first argument passed in to make sure it is an instance (it has to be an instance of the referenced class, or a subclass of that class). When the method is bound, it'll provide that first argument, on an unbound method you provide it yourself.
It's this method object that has the __func__ attribute, which is just a reference to the wrapped function. By accessing the underlying function instead of calling the method, you remove the typecheck, and you can pass in anything you want as the first argument. Functions don't care about their argument types, but methods do.
Note that in Python 3, this has changed; Foo.f just returns the function, not an unbound method. Foo().f returns a method still, still bound, but there is no way to create an unbound method any more.
Under the hood, each function object has a __get__ method, this is what returns the method object:
>>> class Foo(object):
... def f(self): pass
...
>>> Foo.f
<unbound method Foo.f>
>>> Foo().f
<bound method Foo.f of <__main__.Foo object at 0x11046bc10>>
>>> Foo.__dict__['f']
<function f at 0x110450230>
>>> Foo.f.__func__
<function f at 0x110450230>
>>> Foo.f.__func__.__get__(Foo(), Foo)
<bound method Foo.f of <__main__.Foo object at 0x11046bc50>>
>>> Foo.f.__func__.__get__(None, Foo)
<unbound method Foo.f>
This isn't the most efficient codepath, so, Python 3.7 adds a new LOAD_METHOD - CALL_METHOD opcode pair that replaces the current LOAD_ATTRIBUTE - CALL_FUNCTION opcode pair precisely to avoid creating a new method object each time. This optimisation transforms the executon path for instance.foo() from type(instance).__dict__['foo'].__get__(instance, type(instance))() with type(instance).__dict__['foo'](instance), so 'manually' passing in the instance directly to the function object. This saves about 20% time on existing microbenchmarks.
I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.