Python 2.7 docs for weakref module say this:
Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
methods (both bound and unbound), ...
And Python 3.3 docs for weakref module say this:
Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
instance methods, ...
To me, these indicate that weakrefs to bound methods (in all versions Python 2.7 - 3.3) should be good, and that weakrefs to unbound methods should be good in Python 2.7.
Yet in Python 2.7, creating a weakref to a method (bound or unbound) results in a dead weakref:
>>> def isDead(wr): print 'dead!'
...
>>> class Foo:
... def bar(self): pass
...
>>> wr=weakref.ref(Foo.bar, isDead)
dead!
>>> wr() is None
True
>>> foo=Foo()
>>> wr=weakref.ref(foo.bar, isDead)
dead!
>>> wr() is None
True
Not what I would have expected based on the docs.
Similarly, in Python 3.3, a weakref to a bound method dies on creation:
>>> wr=weakref.ref(Foo.bar, isDead)
>>> wr() is None
False
>>> foo=Foo()
>>> wr=weakref.ref(foo.bar, isDead)
dead!
>>> wr() is None
True
Again not what I would have expected based on the docs.
Since this wording has been around since 2.7, it's surely not an oversight. Can anyone explain how the statements and the observed behavior are in fact not in contradiction?
Edit/Clarification: In other words, the statement for 3.3 says "instance methods can be weak referenced"; doesn't this mean that it is reasonable to expect that weakref.ref(an instance method)() is not None? and if it None, then "instance methods" should not be listed among the types of objects that can be weak referenced?
Foo.bar produces a new unbound method object every time you access it, due to some gory details about descriptors and how methods happen to be implemented in Python.
The class doesn't own unbound methods; it owns functions. (Check out Foo.__dict__['bar'].) Those functions just happen to have a __get__ which returns an unbound-method object. Since nothing else holds a reference, it vanishes as soon as you're done creating the weakref. (In Python 3, the rather unnecessary extra layer goes away, and an "unbound method" is just the underlying function.)
Bound methods work pretty much the same way: the function's __get__ returns a bound-method object, which is really just partial(function, self). You get a new one every time, so you see the same phenomenon.
You can store a method object and keep a reference to that, of course:
>>> def is_dead(wr): print "blech"
...
>>> class Foo(object):
... def bar(self): pass
...
>>> method = Foo.bar
>>> wr = weakref.ref(method, is_dead)
>>> 1 + 1
2
>>> method = None
blech
This all seems of dubious use, though :)
Note that if Python didn't spit out a new method instance on every attribute access, that'd mean that classes refer to their methods and methods refer to their classes. Having such cycles for every single method on every single instance in the entire program would make garbage collection way more expensive—and before 2.1, Python didn't even have cycle collection, so they would've stuck around forever.
#Eevee's answer is correct but there is a subtlety that is important.
The Python docs state that instance methods (py3k) and un/bound methods (py2.4+) can be weak referenced. You'd expect (naively, as I did) that weakref.ref(foo.bar)() would therefore be non-None, yet it is None, making the weak ref "dead on arrival" (DOA). This lead to my question, if the weakref to an instance method is DOA, why do the docs say you can weak ref a method?
So as #Eevee showed, you can create a non-dead weak reference to an instance method, by creating a strong reference to the method object which you give to weakref:
m = foo.bar # creates a *new* instance method "Foo.bar" and strong refs it
wr = weakref.ref(m)
assert wr() is not None # success
The subtlety (to me, anyways) is that a new instance method object is created every time you use Foo.bar, so even after the above code is run, the following will fail:
wr = weakref.ref(foo.bar)
assert wr() is not None # fails
because foo.bar is new instance of the "Foo instance" foo's "bar" method, different from m, and there is no strong ref to this new instance, so it is immediately gc'd, even if you have created a strong reference to it earlier (it is not the same strong ref). To be clear,
>>> d1 = foo.bla # assume bla is a data member
>>> d2 = foo.bla # assume bla is a data member
>>> d1 is d2
True # which is what you expect
>>> m1 = foo.bar # assume bar is an instance method
>>> m2 = foo.bar
>>> m1 is m2
False # !!! counter-intuitive
This takes many people by surprise since no one expects access to an instance member to be creating a new instance of anything. For example, if foo.bla is a data member of foo, then using foo.bla in your code does not create a new instance of the object referenced by foo.bla. Now if bla is a "function", foo.bla does create a new instance of type "instance method" representing the bound function.
Why the weakref docs (since python 2.4!) don't point that out is very strange, but that's a separate issue.
While I see that there's an accepted answer as to why this should be so, from a simple use-case situation wherein one would like an object that acts as a weakref to a bound method, I believe that one might be able to sneak by with an object as such. It's kind of a runt compared to some of the 'codier' things out there, but it works.
from weakref import proxy
class WeakMethod(object):
"""A callable object. Takes one argument to init: 'object.method'.
Once created, call this object -- MyWeakMethod() --
and pass args/kwargs as you normally would.
"""
def __init__(self, object_dot_method):
self.target = proxy(object_dot_method.__self__)
self.method = proxy(object_dot_method.__func__)
###Older versions of Python can use 'im_self' and 'im_func' in place of '__self__' and '__func__' respectively
def __call__(self, *args, **kwargs):
"""Call the method with args and kwargs as needed."""
return self.method(self.target, *args, **kwargs)
As an example of its ease of use:
class A(object):
def __init__(self, name):
self.name = name
def foo(self):
return "My name is {}".format(self.name)
>>> Stick = A("Stick")
>>> WeakFoo = WeakMethod(Stick.foo)
>>> WeakFoo()
'My name is Stick'
>>> Stick.name = "Dave"
>>> WeakFoo()
'My name is Dave'
Note that evil trickery will cause this to blow up, so depending on how you'd prefer it to work this may not be the best solution.
>>> A.foo = lambda self: "My eyes, aww my eyes! {}".format(self.name)
>>> Stick.foo()
'My eyes, aww my eyes! Dave'
>>> WeakFoo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in __call__
ReferenceError: weakly-referenced object no longer exists
>>>
If you were going to be replacing methods on-the-fly you might need to use a getattr(weakref.proxy(object), 'name_of_attribute_as_string') approach instead. getattr is a fairly fast look-up so that isn't the literal worst thing in the world, but depending on what you're doing, YMMV.
Related
I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.
I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.
When investigating for another question, I found the following:
>>> class A:
... def m(self): return 42
...
>>> a = A()
This was expected:
>>> A.m == A.m
True
>>> a.m == a.m
True
But this I did not expect:
>>> a.m is a.m
False
And especially not this:
>>> A.m is A.m
False
Python seems to create new objects for each method access. Why am I seeing this behavior? I.e. what is the reason why it can't reuse one object per class and one per instance?
Yes, Python creates new method objects for each access, because it builds a wrapper object to pass in self. This is called a bound method.
Python uses descriptors to do this; function objects have a __get__ method that is called when accessed on a class:
>>> A.__dict__['m'].__get__(A(), A)
<bound method A.m of <__main__.A object at 0x10c29bc10>>
>>> A().m
<bound method A.m of <__main__.A object at 0x10c3af450>>
Note that Python cannot reuse A().m; Python is a highly dynamic language and the very act of accessing .m could trigger more code, which could alter behaviour of what A().m would return next time when accessed.
The #classmethod and #staticmethod decorators make use of this mechanism to return a method object bound to the class instead, and a plain unbound function, respectively:
>>> class Foo:
... #classmethod
... def bar(cls): pass
... #staticmethod
... def baz(): pass
...
>>> Foo.__dict__['bar'].__get__(Foo(), Foo)
<bound method type.bar of <class '__main__.Foo'>>
>>> Foo.__dict__['baz'].__get__(Foo(), Foo)
<function Foo.baz at 0x10c2a1f80>
>>> Foo().bar
<bound method type.bar of <class '__main__.Foo'>>
>>> Foo().baz
<function Foo.baz at 0x10c2a1f80>
See the Python descriptor howto for more detail.
However, Python 3.7 adds a new LOAD_METHOD - CALL_METHOD opcode pair that replaces the current LOAD_ATTRIBUTE - CALL_FUNCTION opcode pair precisely to avoid creating a new method object each time. This optimisation transforms the executon path for instance.foo() from type(instance).__dict__['foo'].__get__(instance, type(instance))() with type(instance).__dict__['foo'](instance), so 'manually' passing in the instance directly to the function object. The optimisation falls back to the normal attribute access path (including binding descriptors) if the attribute found is not a pure-python function object.
Because that's the most convenient, least magical and most space efficient way of implementing bound methods.
In case you're not aware, bound methods refers to being able to do something like this:
f = obj.m
# ... in another place, at another time
f(args, but, not, self)
Functions are descriptors. Descriptors are general objects which can behave differently when accessed as attribute of a class or object. They are used to implement property, classmethod, staticmethod, and several other things. The specific operation of function descriptors is that they return themselves for class access, and return a fresh bound method object for instance access. (Actually, this is only true for Python 3; Python 2 is more complicated in this regard, it has "unbound methods" which are basically functions but not quite).
The reason a new object is created on each access is one of simplicity and efficency: Creating a bound method up-front for every method of every instance takes time and space. Creating them on demand and never freeing them is a potential memory leak (although CPython does something similar for other built-in types) and slightly slower in some cases. Complicated weakref-based caching schemes method objects aren't free either and significantly more complicated (historically, bound methods predate weakrefs by far).
I find the following example mildly surprising:
>>> class Foo:
def blah(self):
pass
>>> f = Foo()
>>> def bar(self):
pass
>>> Foo.bar = bar
>>> f.bar
<bound method Foo.bar of <__main__.Foo object at 0x02D18FB0>>
I expected the bound method to be associated with each particular instance, and to be placed in it at construction. It seems logical that the bound method would have to be different for each instance, so that it knows which instance to pass in to the underlying function - and, indeed:
>>> g = Foo()
>>> g.blah is f.blah
False
But my understanding of the process is clearly flawed, since I would not expect assigning a function into a class attribute would put it in instances that had already been created by then.
So, my question is two fold -
Why does assigning a function into a class apply retroactively to instances? What are the actual lookup rules and processes that make this so?
Is this something guaranteed by the language, or just something that happens to happen?
You want to blow your mind, try this:
f.blah is f.blah
That's right, the instance method wrapper is different each time you access it.
In fact an instance method is a descriptor. In other words, f.blah is actually:
Foo.blah.__get__(f, type(f))
Methods are not actually stored on the instance; they are stored on the class, and a method wrapper is generated on the fly to bind the method to the instance.
The instances do not "contain" the method. The lookup process happens dynamically at the time you access foo.bar. It checks to see if the instance has an attribute of that name. Since it doesn't, it looks on the class, whereupon it finds whatever attribute the class has at that time. Note that methods are not special in this regard. You'll see the same effect if you set Foo.bar = 2; after that, foo.bar will evalute to 2.
What is guaranteed by the language is that attribute lookup proceeds in this fashion: first the instance, then the class if the attribute is not found on the instance. (Lookup rules are different for special methods implicitly invoked via operator overloading, etc..)
Only if you directly assign an attribute to the instance will it mask the class attribute.
>>> foo = Foo()
>>> foo.bar
Traceback (most recent call last):
File "<pyshell#79>", line 1, in <module>
foo.bar
AttributeError: 'Foo' object has no attribute 'bar'
>>> foo.bar = 2
>>> Foo.bar = 88
>>> foo.bar
2
All of the above is a separate matter from bound/unbound methods. The class machinery in Python uses the descriptor protocol so that when you access foo.bar, a new bound method instance is created on the fly. That's why you're seeing different bound method instances on your different objects. But note that underlyingly these bound methods rely on the same code object, as defined by the method you wrote in the class:
>>> foo = Foo()
>>> foo2 = Foo()
>>> foo.blah.im_func.__code__ is foo2.blah.im_func.__code__
True
I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.