Related
I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.
I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.
What arguments do types.MethodType expect and what does it return?
https://docs.python.org/3.6/library/types.html doesn't say more about it:
types.MethodType
The type of methods of user-defined class instances.
For an example, from https://docs.python.org/3.6/howto/descriptor.html
To support method calls, functions include the __get__() method for
binding methods during attribute access. This means that all functions
are non-data descriptors which return bound or unbound methods
depending whether they are invoked from an object or a class. In pure
python, it works like this:
class Function(object):
. . .
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
if obj is None:
return self
return types.MethodType(self, obj)
Must the first argument self of types.MethodType be a callable object? In other words, must the class Function be a callable type, i.e. must Function have a method __call__?
If self is a callable object, does it take at least one argument?
Does types.MethodType(self, obj) mean giving obj as the first argument to the callable object self, i.e. currying self with obj?
How does types.MethodType(self, obj) create and return an instance of types.MethodType?
Thanks.
Usually you don't need to create instance of types.MethodType yourself. Instead, you'll get one automatically when you access a method on an instance of a class.
For example, if we make a class, create an instance of it, then access a method on the instance (without calling it), we'll get an instance of types.MethodType back:
import types
class Foo:
def bar(self):
pass
foo = Foo()
method = foo.bar
print(type(method) == types.MethodType) # prints True
The code you excerpt in your question is trying to show how this normally happens. It's not something you usually have to do yourself, though you can if you really want to. For instance, to create another instance of types.MethodType equivalent to method above, we could do:
method_manual = types.MethodType(Foo.bar, foo)
The first argument to MethodType is a callable object (normally a function, but it can be something else, like an instance of the Function class in the example you were reading). The second argument what we're binding the function to. When you call the method object (with e.g. method()), the bound object will be passed into the function as the first argument.
Usually the object the method gets bound to is an instance, though it can be something else. For instance, a classmethod decorated function will bind to the class it is called on, rather than an instance. Here's an example of that (both getting a method bound to a class automatically, and doing it manually ourselves):
class Foo2:
#classmethod
def baz(cls):
pass
foo2 = Foo2()
method2 = Foo2.baz
method2_via_an_instance = foo2.baz
method2_manual = types.MethodType(method2.__func__, Foo2)
All three of the method2-prefixed variables work exactly the same way (when you call them, they'll all call baz with Foo2 as the cls argument). The only wonky thing about the manual approach this time is that it's hard to get at the original baz function without getting a bound method instead, so I fished it out of one of the other bound method objects.
A final note: The name types.MethodType is an alias for the internal type used for bound methods, which doesn't otherwise have an accessible name. Unlike many classes, the repr of an instance is not an expression to recreate it (it will be something like "<bound method Foo.bar of <__main__.Foo object at 0x0000...>>"). Nor is the repr of the type a valid name to access the type by (the repr is "method").
Short Answer:
Must the first argument self of types.MethodType be a callable object?
In other words, must the class Function be a callable type, i.e. must
Function have a method __call__?
Yes
If self is a callable object, does it take at least one argument?
Depends
Does types.MethodType(self, obj) mean giving obj as the first argument
to the callable object self, i.e. currying self with obj?
Yes
How does types.MethodType(self, obj) create and return an instance of
types.MethodType?
It doesn't work like that.
Long Answer:
the code
class Function(object):
. . .
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
if obj is None:
return self
return types.MethodType(self, obj)
As Daniel explained is mainly to demonstrate for
To support method calls, functions include the __get__() method for
binding methods during attribute access. This means that all functions
are non-data descriptors which return bound or unbound methods
depending whether they are invoked from an object or a class. In pure
python, it works like this:
The types.MethodType() works when the Function has an object.
if obj is None would be False
Then it's a method of some object aka. bound method.
It explains how Python grammar work. As a function, it could be called in the
following two ways.
some_func_() or some_class.some_func()
The former part https://docs.python.org/3.6/howto/descriptor.html#invoking-descriptors explained.
For objects, the machinery is in object.__getattribute__() which
transforms b.x into type(b).__dict__['x'].__get__(b, type(b)). The
implementation works through a precedence chain that gives data
descriptors priority over instance variables, instance variables
priority over non-data descriptors, and assigns lowest priority to
__getattr__() if provided.
Here it's some demonstrate code
>>> import types
>>> types.MethodType
<type 'instancemethod'>
>>> def a(self):
... print(1)
...
>>> class B:
... pass
...
>>> types.MethodType(a,B)
<bound method ?.a of <class __main__.B at 0x7f4d3d5aa598>>
>>> B.t = types.MethodType(a,B)
>>> B.t()
1
>>> def s():
... print(3)
...
>>> B.r = types.MethodType(s,B)
>>> B.r
<bound method ?.s of <class __main__.B at 0x7f4d3d5aa598>>
>>> B.r()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: s() takes no arguments (1 given)
See also dynamically adding callable to class as instance "method"
Documentation doesn't say much, but you can always check its source code. The signature of MethodType constructor is:
def __init__(self, func: Callable[..., Any], obj: object) -> None: ...
It accepts a callable and object, and returns None.
MethodType can be used to add an instance method to an object, instead of a function; here's an example:
from types import MethodType
class MyClass:
language = 'Python'
# a function is bound to obj1
obj1 = MyClass()
obj1.say_hello = lambda: 'Hello World!'
print(type(obj1.say_hello)) # type is class 'function'
obj1.say_hello()
# a method is bound to obj2
obj2 = MyClass()
# this is used to bind a "method" to a specific object obj2, rather than a function
obj2.say_hello = MethodType(lambda self: f'Hello {self.language}!', obj2)
print(type(obj2.say_hello)) # type is class 'method'
obj2.say_hello()
It's not something you would ever call. Like most of the classes in the types module, it's more for comparing with existing objects (for example in isinstance).
Python 2.7 docs for weakref module say this:
Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
methods (both bound and unbound), ...
And Python 3.3 docs for weakref module say this:
Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
instance methods, ...
To me, these indicate that weakrefs to bound methods (in all versions Python 2.7 - 3.3) should be good, and that weakrefs to unbound methods should be good in Python 2.7.
Yet in Python 2.7, creating a weakref to a method (bound or unbound) results in a dead weakref:
>>> def isDead(wr): print 'dead!'
...
>>> class Foo:
... def bar(self): pass
...
>>> wr=weakref.ref(Foo.bar, isDead)
dead!
>>> wr() is None
True
>>> foo=Foo()
>>> wr=weakref.ref(foo.bar, isDead)
dead!
>>> wr() is None
True
Not what I would have expected based on the docs.
Similarly, in Python 3.3, a weakref to a bound method dies on creation:
>>> wr=weakref.ref(Foo.bar, isDead)
>>> wr() is None
False
>>> foo=Foo()
>>> wr=weakref.ref(foo.bar, isDead)
dead!
>>> wr() is None
True
Again not what I would have expected based on the docs.
Since this wording has been around since 2.7, it's surely not an oversight. Can anyone explain how the statements and the observed behavior are in fact not in contradiction?
Edit/Clarification: In other words, the statement for 3.3 says "instance methods can be weak referenced"; doesn't this mean that it is reasonable to expect that weakref.ref(an instance method)() is not None? and if it None, then "instance methods" should not be listed among the types of objects that can be weak referenced?
Foo.bar produces a new unbound method object every time you access it, due to some gory details about descriptors and how methods happen to be implemented in Python.
The class doesn't own unbound methods; it owns functions. (Check out Foo.__dict__['bar'].) Those functions just happen to have a __get__ which returns an unbound-method object. Since nothing else holds a reference, it vanishes as soon as you're done creating the weakref. (In Python 3, the rather unnecessary extra layer goes away, and an "unbound method" is just the underlying function.)
Bound methods work pretty much the same way: the function's __get__ returns a bound-method object, which is really just partial(function, self). You get a new one every time, so you see the same phenomenon.
You can store a method object and keep a reference to that, of course:
>>> def is_dead(wr): print "blech"
...
>>> class Foo(object):
... def bar(self): pass
...
>>> method = Foo.bar
>>> wr = weakref.ref(method, is_dead)
>>> 1 + 1
2
>>> method = None
blech
This all seems of dubious use, though :)
Note that if Python didn't spit out a new method instance on every attribute access, that'd mean that classes refer to their methods and methods refer to their classes. Having such cycles for every single method on every single instance in the entire program would make garbage collection way more expensive—and before 2.1, Python didn't even have cycle collection, so they would've stuck around forever.
#Eevee's answer is correct but there is a subtlety that is important.
The Python docs state that instance methods (py3k) and un/bound methods (py2.4+) can be weak referenced. You'd expect (naively, as I did) that weakref.ref(foo.bar)() would therefore be non-None, yet it is None, making the weak ref "dead on arrival" (DOA). This lead to my question, if the weakref to an instance method is DOA, why do the docs say you can weak ref a method?
So as #Eevee showed, you can create a non-dead weak reference to an instance method, by creating a strong reference to the method object which you give to weakref:
m = foo.bar # creates a *new* instance method "Foo.bar" and strong refs it
wr = weakref.ref(m)
assert wr() is not None # success
The subtlety (to me, anyways) is that a new instance method object is created every time you use Foo.bar, so even after the above code is run, the following will fail:
wr = weakref.ref(foo.bar)
assert wr() is not None # fails
because foo.bar is new instance of the "Foo instance" foo's "bar" method, different from m, and there is no strong ref to this new instance, so it is immediately gc'd, even if you have created a strong reference to it earlier (it is not the same strong ref). To be clear,
>>> d1 = foo.bla # assume bla is a data member
>>> d2 = foo.bla # assume bla is a data member
>>> d1 is d2
True # which is what you expect
>>> m1 = foo.bar # assume bar is an instance method
>>> m2 = foo.bar
>>> m1 is m2
False # !!! counter-intuitive
This takes many people by surprise since no one expects access to an instance member to be creating a new instance of anything. For example, if foo.bla is a data member of foo, then using foo.bla in your code does not create a new instance of the object referenced by foo.bla. Now if bla is a "function", foo.bla does create a new instance of type "instance method" representing the bound function.
Why the weakref docs (since python 2.4!) don't point that out is very strange, but that's a separate issue.
While I see that there's an accepted answer as to why this should be so, from a simple use-case situation wherein one would like an object that acts as a weakref to a bound method, I believe that one might be able to sneak by with an object as such. It's kind of a runt compared to some of the 'codier' things out there, but it works.
from weakref import proxy
class WeakMethod(object):
"""A callable object. Takes one argument to init: 'object.method'.
Once created, call this object -- MyWeakMethod() --
and pass args/kwargs as you normally would.
"""
def __init__(self, object_dot_method):
self.target = proxy(object_dot_method.__self__)
self.method = proxy(object_dot_method.__func__)
###Older versions of Python can use 'im_self' and 'im_func' in place of '__self__' and '__func__' respectively
def __call__(self, *args, **kwargs):
"""Call the method with args and kwargs as needed."""
return self.method(self.target, *args, **kwargs)
As an example of its ease of use:
class A(object):
def __init__(self, name):
self.name = name
def foo(self):
return "My name is {}".format(self.name)
>>> Stick = A("Stick")
>>> WeakFoo = WeakMethod(Stick.foo)
>>> WeakFoo()
'My name is Stick'
>>> Stick.name = "Dave"
>>> WeakFoo()
'My name is Dave'
Note that evil trickery will cause this to blow up, so depending on how you'd prefer it to work this may not be the best solution.
>>> A.foo = lambda self: "My eyes, aww my eyes! {}".format(self.name)
>>> Stick.foo()
'My eyes, aww my eyes! Dave'
>>> WeakFoo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in __call__
ReferenceError: weakly-referenced object no longer exists
>>>
If you were going to be replacing methods on-the-fly you might need to use a getattr(weakref.proxy(object), 'name_of_attribute_as_string') approach instead. getattr is a fairly fast look-up so that isn't the literal worst thing in the world, but depending on what you're doing, YMMV.
I hope someone can answer this that has a good deep understanding of Python :)
Consider the following code:
>>> class A(object):
... pass
...
>>> def __repr__(self):
... return "A"
...
>>> from types import MethodType
>>> a = A()
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>> setattr(a, "__repr__", MethodType(__repr__, a, a.__class__))
>>> a
<__main__.A object at 0x00AC6990>
>>> repr(a)
'<__main__.A object at 0x00AC6990>'
>>>
Notice how repr(a) does not yield the expected result of "A" ?
I want to know why this is the case and if there is a way to achieve this...
I contrast, the following example works however (Maybe because we're not trying to override a special method?):
>>> class A(object):
... def foo(self):
... return "foo"
...
>>> def bar(self):
... return "bar"
...
>>> from types import MethodType
>>> a = A()
>>> a.foo()
'foo'
>>> setattr(a, "foo", MethodType(bar, a, a.__class__))
>>> a.foo()
'bar'
>>>
Python usually doesn't call the special methods (those with name surrounded by __) on the instance, but only on the class. (Although this is an implementation detail, it's characteristic of CPython, the standard interpreter.) So there's no way to override __repr__() directly on an instance and make it work. Instead, you need to do something like so:
class A(object):
def __repr__(self):
return self._repr()
def _repr(self):
return object.__repr__(self)
Now you can override __repr__() on an instance by substituting _repr().
As explained in Special Method Lookup:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary … In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass
(The part I've snipped out explains the rationale behind this, if you're interested in that.)
Python doesn't document exactly when an implementation should or shouldn't look up the method on the type; all it documents is, in effect, that implementations may or may not look at the instance for special method lookups, so you shouldn't count on either.
As you can guess from your test results, in the CPython implementation, __repr__ is one of the functions looked up on the type.
Things are slightly different in 2.x, mostly because of the presence of classic classes, but as long as you're only creating new-style classes you can think of them as the same.
The most common reason people want to do this is to monkey-patch different instances of an object to do different things. You can't do that with special methods, so… what can you do? There's a clean solution, and a hacky solution.
The clean solution is to implement a special method on the class that just calls a regular method on the instance. Then you can monkey patch that regular method on each instance. For example:
class C(object):
def __repr__(self):
return getattr(self, '_repr')()
def _repr(self):
return 'Boring: {}'.format(object.__repr__(self))
c = C()
def c_repr(self):
return "It's-a me, c_repr: {}".format(object.__repr__(self))
c._repr = c_repr.__get__(c)
The hacky solution is to build a new subclass on the fly and re-class the object. I suspect anyone who really has a situation where this is a good idea will know how to implement it from that sentence, and anyone who doesn't know how to do so shouldn't be trying, so I'll leave it at that.
The reason for this is special methods (__x__()) are defined for the class, not the instance.
This makes sense when you think about __new__() - it would be impossible to call this on an instance as the instance doesn't exist when it's called.
So you can do this on the class as a whole if you want to:
>>> A.__repr__ = __repr__
>>> a
A
Or on an individual instance, as in kindall's answer. (Note there is a lot of similarity here, but I thought my examples added enough to post this as well).
For new style classes, Python uses a special method lookup that bypasses instances. Here an excerpt from the source:
1164 /* Internal routines to do a method lookup in the type
1165 without looking in the instance dictionary
1166 (so we can't use PyObject_GetAttr) but still binding
1167 it to the instance. The arguments are the object,
1168 the method name as a C string, and the address of a
1169 static variable used to cache the interned Python string.
1170
1171 Two variants:
1172
1173 - lookup_maybe() returns NULL without raising an exception
1174 when the _PyType_Lookup() call fails;
1175
1176 - lookup_method() always raises an exception upon errors.
1177
1178 - _PyObject_LookupSpecial() exported for the benefit of other places.
1179 */
You can either change to an old-style class (don't inherit from object) or you can add dispatcher methods to the class (methods that forward lookups back to the instance). For an example of instance dispatcher methods, see the recipe at http://code.activestate.com/recipes/578091
TLDR: It is impossible to define proper, unbound methods on instances; this applies to special methods as well. Since bound methods are first-class objects, in certain circumstances the difference is not noticeable.
However, special methods are always looked up as proper, unbound methods by Python when needed.
You can always manually fall back to a special method that uses the more generic attribute access. Attribute access covers both bound methods stored as attributes as well as unbound methods that are bound as needed. This is similar to how __repr__ or other methods would use attributes to define their output.
class A:
def __init__(self, name):
self.name = name
def __repr__(self):
# call attribute to derive __repr__
return self.__representation__()
def __representation__(self):
return f'{self.__class__.__name__}({self.name})'
def __str__(self):
# return attribute to derive __str__
return self.name
Unbound versus Bound Methods
There are two meanings to a method in Python: unbound methods of a class and bound methods of an instance of that class.
An unbound method is a regular function on a class or one of its base classes. It can be defined either during class definition, or added later on.
>>> class Foo:
... def bar(self): print('bar on', self)
...
>>> Foo.bar
<function __main__.Foo.bar(self)>
An unbound method exists only once on the class - it is the same for all instances.
A bound method is an unbound method which has been bound to a specific instance. This usually means the method was looked up through the instance, which invokes the function's __get__ method.
>>> foo = Foo()
>>> # lookup through instance
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> # explicit descriptor invokation
>>> type(foo).bar.__get__(foo, type(Foo))
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, "a method" generally means an unbound method that is bound to its instance as required. When Python needs a special method, it directly invokes the descriptor protocol for the unbound method. In consequence, the method is looked up on the class; an attribute on the instance is ignored.
Bound Methods on Objects
A bound method is created anew every time it is fetched from its instance. The result is a first-class object that has identity, can be stored and passed around, and be called later on.
>>> foo.bar is foo.bar # binding happens on every lookup
False
>>> foo_bar = foo.bar # bound methods can be stored
>>> foo_bar() # stored bound methods can be called later
bar on <__main__.Foo object at 0x10c3b6390>
>>> foo_bar()
bar on <__main__.Foo object at 0x10c3b6390>
Being able to store bound methods means they can also be stored as attributes. Storing a bound method on its bound instance makes it appear similar to an unbound method. But in fact a stored bound method behaves subtly different and can be stored on any object that allows attributes.
>>> foo.qux = foo.bar
>>> foo.qux
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> foo.qux is foo.qux # binding is not repeated on every lookup!
True
>>> too = Foo()
>>> too.qux = foo.qux # bound methods can be stored on other instances!
>>> too.qux # ...but are still bound to the original instance!
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
>>> import builtins
>>> builtins.qux = foo.qux # bound methods can be stored...
>>> qux # ... *anywhere* that supports attributes
<bound method Foo.bar of <__main__.Foo object at 0x10c3b6390>>
As far as Python is concerned, bound methods are just regular, callable objects. Just as it has no way of knowing whether too.qux is a method of too, it cannot deduce whether too.__repr__ is a method either.