The Python 3.x language reference described two ways to create a method object:
User-defined method objects may be created when getting an attribute of a class (perhaps via an instance of that class), if that attribute is a user-defined function object or a class method object.
When an instance method object is created by retrieving a user-defined function object from a class via one of its instances, its self attribute is the instance, and the method object is said to be bound. The new method’s func attribute is the original function object.
When a user-defined method object is created by retrieving another method object from a class or instance, the behaviour is the same as for a function object, except that the func attribute of the new instance is not the original method object but its func attribute.
When an instance method object is called, the underlying function (func) is called, inserting the class instance (self) in front of the argument list. For instance, when C is a class which contains a definition for a function f(), and x is an instance of C, calling x.f(1) is equivalent to calling C.f(x, 1).
When an instance method object is derived from a class method object, the “class instance” stored in self will actually be the class itself, so that calling either x.f(1) or C.f(1) is equivalent to calling f(C,1) where f is the underlying function.
In different ways, they both have different __func__ and __self__ values, but I'm not very aware of these two different ways, could someone explain it to me?
Python Language Reference | The Standard Type Hierarchy:
https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy
I'm not 100% sure that I completely understand your question, but maybe looking at an example will be helpful. To start, lets create a class which has a function in the definition:
>>> class Foo(object):
... def method(self):
... pass
...
>>> f = Foo()
User-defined method objects may be created when getting an attribute of a class (perhaps via an instance of that class), if that attribute is a user-defined function object or a class method object.
Ok, so we can create a method object by just accessing the attribute on an instance (if the attribute is a function). In our setup, f is an instance of the class Foo:
>>> type(f.method)
<class 'method'>
Compare that with accessing the method attribute on the class:
>>> type(Foo.method)
<class 'function'>
When an instance method object is created by retrieving a user-defined function object from a class via one of its instances, its __self__ attribute is the instance, and the method object is said to be bound. The new method’s __func__ attribute is the original function object.
This is just telling us what attributes exist on instance methods. Let's check it out:
>>> instance_method = f.method
>>> instance_method.__func__ is Foo.method
True
>>> instance_method.__self__ is f
True
So we see that the method object has a __func__ attribute which is just a reference to the actual Foo.method function. It also has a __self__ attribute that is a reference to the instance.
When an instance method object is called, the underlying function (func) is called, inserting the class instance (self) in front of the argument list. For instance, when C is a class which contains a definition for a function f(), and x is an instance of C, calling x.f(1) is equivalent to calling C.f(x, 1).
Basically, in reference to our example above, this is just saying that If:
instance_method = f.method
Then:
instance_method(arg1, arg2)
executes the following:
instance_method.__func__(instance_method.__self__, arg1, arg2)
For completeness and as an addendum to the excellent answer provided by #mgilson, I wanted to explain the remaining 2 paragraphs referenced in the original question.
First let's create a class with a classmethod:
>>> class Foo(object):
... #classmethod
... def cmethod(cls):
... pass
...
>>> f = Foo()
Now for the 3rd paragraph:
When a user-defined method object is created by retrieving another method object from a class or instance, the behaviour is the same as for a function object, except that the func attribute of the new instance is not the original method object but its func attribute.
This means:
>>> class_method = f.cmethod
>>> class_method.__func__ is Foo.cmethod.__func__
True
>>> class_method.__self__ is Foo
True
Note that __self__ is a reference to the Foo class. Finally, the last paragraph:
When an instance method object is derived from a class method object, the “class instance” stored in self will actually be the class itself, so that calling either x.f(1) or C.f(1) is equivalent to calling f(C,1) where f is the underlying function.
This just says that all of the following are equivalent:
>>> f.cmethod(arg1, arg2)
>>> Foo.cmethod(arg1, arg2)
>>> f.cmethod.__func__(Foo, arg1, arg2)
>>> Foo.cmethod.__func__(Foo, arg1, arg2)
>>> f.cmethod.__func__(f.cmethod.__self__, arg1, arg2)
>>> Foo.cmethod.__func__(Foo.cmethod.__self__, arg1, arg2)
Related
I just stumbled over this strange behavior when the type of a method changes during subclassing:
class A:
def f(self, x):
return x**2
class B(A):
#classmethod
def f(cls, x):
return x**2
If I now ask for the type of B.f, I'll get the (supposedly) wrong answer:
In [37]: type(B.f)
Out[37]: method
Whereas this works as expected:
In [39]: type(B.__dict__["f"])
Out[39]: classmethod
(Seen in Python 3.4 and 3.6.)
Is this just a bug or is there a specific reason for this?
What's the difference between the attribute f and the .__dict__["f"] item? I thought they were the same.
In a testing suite, I was trying to support both types of methods inside a class to be tested. To be able to do that, I need to know the type in order to pass the correct number of arguments. If it's a normal method (i.e. self is the first argument), I'd just pass None explicitly, which by design shouldn't be used inside the method anyway, since it's not instance-dependent.
Maybe there's a better way to do this, like duck typing the call to the method. But there might be cases where this is not so easy to do, like if the method had *args and **kwargs... Therefore I went with the explicit type check, but got stuck at this point.
No, this is not a bug, this is normal behaviour. A classmethod produces a bound method when accessed on a class. That's exactly the point of a classmethod, to bind a function to the class you access it on or the class of an instance you access it on.
Like function and property objects, classmethod is a descriptor object, it implements a __get__ method. Accessing attributes on an instance or a class is delegated to the __getattribute__ method, and the default implementation of that hook will not just return what it found in object.__dict__[attributename]; it will also bind descriptors, by calling the descriptor.__get__() method. This is a hugely important aspect of Python, it is this mechanism that makes methods and attributes and loads of other things work.
classmethod objects, when bound by the descriptor protocol, return a method object. Method objects are wrappers that record the object bound to, and the function to call when they are called; calling a method really calls the underlying method with the bound object as first argument:
>>> class Foo:
... pass
...
>>> def bar(*args): print(args)
...
>>> classmethod(bar).__get__(None, Foo) # decorate with classmethod and bind
<bound method bar of <class '__main__.Foo'>>
>>> method = classmethod(bar).__get__(None, Foo)
>>> method.__self__
<class '__main__.Foo'>
>>> method.__func__
<function bar at 0x1056f0e18>
>>> method()
(<class '__main__.Foo'>,)
>>> method('additional arguments')
(<class '__main__.Foo'>, 'additional arguments')
So the method object returned for a classmethod object references the class (the second argument to __get__, the owner), and the original function. If you use a class method on an instance, the first argument is still ignored:
>>> classmethod(bar).__get__(Foo(), Foo).__self__ # called on an instance
<class '__main__.Foo'>
Functions, on the other hand, want to bind only to instances; so if the first argument to __get__ is set to None, they simply return self:
>>> bar.__get__(None, Foo) # access on a class
<function bar at 0x1056f0e18>
>>> bar.__get__(Foo(), Foo) # access on an instance
<bound method bar of <__main__.Foo object at 0x105833a90>>
>>> bar.__get__(Foo(), Foo).__self__
<__main__.Foo object at 0x105833160>
If accessing ClassObject.classmethod_object would return the classmethod object itself, like a function object would, then you could never actually use the class method on a class. That'd be rather pointless.
So no, object.attribute is not always the same thing as object.__dict__['attribute']. If object.__dict__['attribute'] supports the descriptor protocol, it'll be invoked.
What arguments do types.MethodType expect and what does it return?
https://docs.python.org/3.6/library/types.html doesn't say more about it:
types.MethodType
The type of methods of user-defined class instances.
For an example, from https://docs.python.org/3.6/howto/descriptor.html
To support method calls, functions include the __get__() method for
binding methods during attribute access. This means that all functions
are non-data descriptors which return bound or unbound methods
depending whether they are invoked from an object or a class. In pure
python, it works like this:
class Function(object):
. . .
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
if obj is None:
return self
return types.MethodType(self, obj)
Must the first argument self of types.MethodType be a callable object? In other words, must the class Function be a callable type, i.e. must Function have a method __call__?
If self is a callable object, does it take at least one argument?
Does types.MethodType(self, obj) mean giving obj as the first argument to the callable object self, i.e. currying self with obj?
How does types.MethodType(self, obj) create and return an instance of types.MethodType?
Thanks.
Usually you don't need to create instance of types.MethodType yourself. Instead, you'll get one automatically when you access a method on an instance of a class.
For example, if we make a class, create an instance of it, then access a method on the instance (without calling it), we'll get an instance of types.MethodType back:
import types
class Foo:
def bar(self):
pass
foo = Foo()
method = foo.bar
print(type(method) == types.MethodType) # prints True
The code you excerpt in your question is trying to show how this normally happens. It's not something you usually have to do yourself, though you can if you really want to. For instance, to create another instance of types.MethodType equivalent to method above, we could do:
method_manual = types.MethodType(Foo.bar, foo)
The first argument to MethodType is a callable object (normally a function, but it can be something else, like an instance of the Function class in the example you were reading). The second argument what we're binding the function to. When you call the method object (with e.g. method()), the bound object will be passed into the function as the first argument.
Usually the object the method gets bound to is an instance, though it can be something else. For instance, a classmethod decorated function will bind to the class it is called on, rather than an instance. Here's an example of that (both getting a method bound to a class automatically, and doing it manually ourselves):
class Foo2:
#classmethod
def baz(cls):
pass
foo2 = Foo2()
method2 = Foo2.baz
method2_via_an_instance = foo2.baz
method2_manual = types.MethodType(method2.__func__, Foo2)
All three of the method2-prefixed variables work exactly the same way (when you call them, they'll all call baz with Foo2 as the cls argument). The only wonky thing about the manual approach this time is that it's hard to get at the original baz function without getting a bound method instead, so I fished it out of one of the other bound method objects.
A final note: The name types.MethodType is an alias for the internal type used for bound methods, which doesn't otherwise have an accessible name. Unlike many classes, the repr of an instance is not an expression to recreate it (it will be something like "<bound method Foo.bar of <__main__.Foo object at 0x0000...>>"). Nor is the repr of the type a valid name to access the type by (the repr is "method").
Short Answer:
Must the first argument self of types.MethodType be a callable object?
In other words, must the class Function be a callable type, i.e. must
Function have a method __call__?
Yes
If self is a callable object, does it take at least one argument?
Depends
Does types.MethodType(self, obj) mean giving obj as the first argument
to the callable object self, i.e. currying self with obj?
Yes
How does types.MethodType(self, obj) create and return an instance of
types.MethodType?
It doesn't work like that.
Long Answer:
the code
class Function(object):
. . .
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
if obj is None:
return self
return types.MethodType(self, obj)
As Daniel explained is mainly to demonstrate for
To support method calls, functions include the __get__() method for
binding methods during attribute access. This means that all functions
are non-data descriptors which return bound or unbound methods
depending whether they are invoked from an object or a class. In pure
python, it works like this:
The types.MethodType() works when the Function has an object.
if obj is None would be False
Then it's a method of some object aka. bound method.
It explains how Python grammar work. As a function, it could be called in the
following two ways.
some_func_() or some_class.some_func()
The former part https://docs.python.org/3.6/howto/descriptor.html#invoking-descriptors explained.
For objects, the machinery is in object.__getattribute__() which
transforms b.x into type(b).__dict__['x'].__get__(b, type(b)). The
implementation works through a precedence chain that gives data
descriptors priority over instance variables, instance variables
priority over non-data descriptors, and assigns lowest priority to
__getattr__() if provided.
Here it's some demonstrate code
>>> import types
>>> types.MethodType
<type 'instancemethod'>
>>> def a(self):
... print(1)
...
>>> class B:
... pass
...
>>> types.MethodType(a,B)
<bound method ?.a of <class __main__.B at 0x7f4d3d5aa598>>
>>> B.t = types.MethodType(a,B)
>>> B.t()
1
>>> def s():
... print(3)
...
>>> B.r = types.MethodType(s,B)
>>> B.r
<bound method ?.s of <class __main__.B at 0x7f4d3d5aa598>>
>>> B.r()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: s() takes no arguments (1 given)
See also dynamically adding callable to class as instance "method"
Documentation doesn't say much, but you can always check its source code. The signature of MethodType constructor is:
def __init__(self, func: Callable[..., Any], obj: object) -> None: ...
It accepts a callable and object, and returns None.
MethodType can be used to add an instance method to an object, instead of a function; here's an example:
from types import MethodType
class MyClass:
language = 'Python'
# a function is bound to obj1
obj1 = MyClass()
obj1.say_hello = lambda: 'Hello World!'
print(type(obj1.say_hello)) # type is class 'function'
obj1.say_hello()
# a method is bound to obj2
obj2 = MyClass()
# this is used to bind a "method" to a specific object obj2, rather than a function
obj2.say_hello = MethodType(lambda self: f'Hello {self.language}!', obj2)
print(type(obj2.say_hello)) # type is class 'method'
obj2.say_hello()
It's not something you would ever call. Like most of the classes in the types module, it's more for comparing with existing objects (for example in isinstance).
New to Python and having done some reading, I'm making some methods in my custom class class methods rather than instance methods.
So I tested my code but I hadn't changed some of the method calls to call the method in the class rather than the instance, but they still worked:
class myClass:
#classmethod:
def foo(cls):
print 'Class method foo called with %s.'%(cls)
def bar(self):
print 'Instance method bar called with %s.'%(self)
myClass.foo()
thing = myClass()
thing.foo()
thing.bar()
This produces:
class method foo called with __main__.myClass.
class method foo called with __main__.myClass.
instance method bar called with <__main__.myClass instance at 0x389ba4>.
So what I'm wondering is why I can call a class method (foo) on an instance (thing.foo), (although it's the class that gets passed to the method)? It kind of makes sense, as 'thing' is a 'myClass', but I was expecting Python to give an error saying something along the lines of 'foo is a class method and can't be called on an instance'.
Is this just an expected consequence of inheritance with the 'thing' object inheriting the foo method from its superclass?
If I try to call the instance method via the class:
myClass.bar()
then I get:
TypeError: unbound method bar() must be called with myClass instance...
which makes perfect sense.
You can call it on an instance because #classmethod is a decorator (it takes a function as an argument and returns a new function).
Here is some relavent information from the Python documentation
It can be called either on the class (such as C.f()) or on an instance
(such as C().f()). The instance is ignored except for its class. If a
class method is called for a derived class, the derived class object
is passed as the implied first argument.
There's also quite a good SO discussion on #classmethod here.
Let's start with a quick overview of the descriptor protocol. If a class defines a __get__ method, an instance of that class is a descriptor. Accessing a descriptor as the attribute of another object produces the return value of the __get__ method, not the descriptor itself.
A function is a descriptor; this is how instance methods are implemented. Given
class myClass:
#classmethod:
def foo(cls):
print('Class method foo called with %s.' % (cls,))
def bar(self):
print 'Instance method bar called with %s.'%(self)
c = myClass()
the expression c.bar is equivalent to
myClass.__dict__['bar'].__get__(c, myClass)
while the expression myClass.bar is equivalent to the expression
myClass.__dict__['bar'].__get__(None, myClass)
Note the only difference is in the object passed as the first argument to __get__. The former returns a new method object, which when called passes c and its own arguments on to the function bar. This is why c.bar() and C.bar(c) are equivalent. The latter simply returns the function bar itself.
classmethod is a type that provides a different implementation of __get__. This means that c.foo() and myClass.foo() call __get__ as before:
# c.foo
myClass.__dict__['foo'].__get__(c, myClass)
# myClass.foo
myClass.__dict__['foo'].__get__(None, myClass)
Now, however, both calls return the same method object, and this method object, when called, passes myClass as the first argument to the original function object. That is, c.foo() is equivalent to myClass.foo(), which
is equivalent to x(myClass) (where x is the original function defined before the decoration bound the name foo to an instance of classmethod).
How can I extract the attributes of the "SimpleSubscriber object from the following method object?
<bound method SimpleSubscriber.process of <__main__.SimpleSubscriber object at 0x2ede150>>
This object has been passed as a function, but is a function of an instance as we can see above. I would like to reference the above object (saved in a list in different class than its own) from the instance where this function resides.
Nothing in dir() seems to help.
self within the instance returns:
<__main__.SimpleSubscriber object at 0x2ede150>
so this instance cannot reference itself in its own method, from within the other class.
So basically, I am trying to access the attributes of
<__main__.SimpleSubscriber object at 0x2ede150>
from
<bound method SimpleSubscriber.process of <__main__.SimpleSubscriber object at 0x2ede150>>
I'm trying to extract unique identifiers of the instance. Is that possible?
A bound method holds references to the original function and the instance it is bound to; this is how it passes in the instance as the first argument when your method is called.
If you wanted to access the instance it is bound to, use the __self__ attribute:
>>> class Foo:
... def bar(self):
... pass
...
>>> bound_bar = Foo().bar
>>> bound_bar
<bound method Foo.bar of <__main__.Foo object at 0x101e5f6d8>>
>>> bound_bar.__self__
<__main__.Foo object at 0x101e5f6d8>
You can then access any attributes you need on that instance, including the method itself again:
>>> bound_bar.__self__.bar
<bound method Foo.bar of <__main__.Foo object at 0x101e5f6d8>>
Attributes like these are documented in the Datamodel reference, but for a quick overview the inspect module has a handy table.
In Data Model of the reference, writer spent lots of effort explaining how User-defined methods are created and operated:(See http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy and roll down)
User-defined method objects may be created when getting an attribute
of a class (perhaps via an in- stance of that class), if that
attribute is a user-defined function object, an unbound user-defined
method object, or a class method object. When the attribute is a
user-defined method object, a new method object is only created if the
class from which it is being retrieved is the same as, or a derived
class of, the class stored in the original method object; otherwise,
the original method object is used as it is.
So what's the difference between an unbound user-defined method object and a class method object?
From the "user" point of view, a class method in Python is a method that receives its class as its first parameter - unlike "ordinary" methods which receive an instance of the class as its first parameter - which by convention is called self.
If you retrieve an "ordinary" method from a class, instead of from an instace of that class, you get an "unbound method" - i.e. an object that is a wrapper around a function, but that does not automatically adds either the class itself, nor any instance as the first parameter when it is called. Threfore if you are to call the "unbound method" you have to manually pass an instance of its class as its first parameter.
If you manually call a class method, on the other hand, the class is filled in as the first parameter for you:
>>> class A(object):
... def b(self):
... pass
... #classmethod
... def c(cls):
... pass
...
>>> A.b
<unbound method A.b>
>>> A.c
<bound method type.c of <class '__main__.A'>>
>>> A.c()
>>> A.b()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method b() must be called with A instance as first argument (got nothing instead)
>>>
Under the hood what goes is more or less like this - with "new style classes":
When one defines a class body, the methods are just ordinary functions - when the class body is over, Python calls the class's metaclass (which ordinarily is the builtin type ) - and pass to it as parameters the name, base classes, and class body dictionary. This call yields a class - which in Python is, an object which is a class, since everything is an object.
Now, Python has some nifty ways of customizing attribute access - the so called "descriptors". A descriptor is any object that defines a method named __get__ (or __set__ or __del__ but we don't care about those here). When one access an attribute of a class or object in Python, the object referred by that attribute is returned - except if it is a class attribute, and the object is a descriptor. In that case, instead of returning the object itself, Python calls the __get__ method on that object, and returns its results instead. For example, the property built-in is just a class that implements both __set__, __get__ and __del__ as appropriate.
Now, what happens when the attribute is retrieved, is that any function (or class method or unbound method, as the data model states) on its body, does have a __get__ method, which makes it a descriptor. Basically, a descriptor that at each attribute access to retrieve the object named as the function as it is defined on the function body, creates a new object around that function - an object that when called will have the first parameter automatically filled in - which is to say, a method.
Ex.:
>>> class B(object):
... def c(self):
... pass
... print c
...
<function c at 0x1927398>
>>> print B.c
<unbound method B.c>
>>> b = B()
>>> b.c
<bound method B.c of <__main__.B object at 0x1930a10>
If you want to retrieve the function object, without conversion to a method object, you can do so through the class's __dict__ attribute, which does not trigger the descriptor:
>>> B.__dict__["c"]
<function c at 0x1927398>
>>> B.__dict__["c"].__get__
<method-wrapper '__get__' of function object at 0x1927398>
>>> B.__dict__["c"].__get__(b, B)
<bound method B.c of <__main__.B object at 0x1930a10>>
>>> B.__dict__["c"].__get__(None, B)
<unbound method B.c>
As for "class methods", these are just different type of objects, which are explicitly decorated with the builtin classmethod - The object it returns when its __get__ is called is a wrapper around the original function that will fill in the cls as the first parameter on call.