Consider the following snippet.
class A:
def __next__(self):
return 2
a = A()
print(next(a),a.__next__()) # prints "2,2" as expected
a.__next__ = lambda: 4
print(next(a),a.__next__()) # prints "2,4". I expected "4,4"
Clearly, the property __next__ is updated by the patching, but the inbuilt next function does not resolve that.
The python 3 docs docs on the python datamodel that says
For instance, if a class defines a method named __getitem__(), and x is an instance of this class, then x[i] is roughly equivalent to type(x).__getitem__(x, i).
From this, I came up with a hack as below
class A:
def next_(self):
return 2
def __next__(self):
return self.next_()
a = A()
print(next(a),a.__next__()) # 2,2
a.next_ = lambda: 4
print(next(a),a.__next__()) # 4,4
The code works, but at the expense of another layer of indirection via another next_-method.
My question is: What is the proper way to monkey-patch the __next__ instance method? What is the rationale behind this design in python?
You can't. Special methods are special they cannot be overridden at the instance level. Period. If you want to "customize" the instance behaviour the correct way to do it is to simply have a proper implementation instead of a bogus implementation that you swap at runtime. Change the value instead of the method.
The rationale can be found in The History of Python - Adding Support for User-defined Classes at the end of following section:
Special Methods
As briefly mentioned in the last section, one of my main goals was to
keep the implementation of classes simple. In most object oriented
languages, there are a variety of special operators and methods that
only apply to classes. For example, in C++, there is a special syntax
for defining constructors and destructors that is different than the
normal syntax used to define ordinary function and methods.
I really didn't want to introduce additional syntax to handle special
operations for objects. So instead, I handled this by simply mapping
special operators to a predefined set of "special method" names such
as __init__ and __del__. By defining methods with these names, users
could supply code related to the construction and destruction of
objects.
I also used this technique to allow user classes to redefine the
behavior of Python's operators. As previously noted, Python is
implemented in C and uses tables of function pointers to implement
various capabilities of built-in objects (e.g., “get attribute”, “add”
and “call”). To allow these capabilities to be defined in user-defined
classes, I mapped the various function pointers to special method
names such as __getattr__, __add__, and __call__. There is a direct
correspondence between these names and the tables of function pointers
one has to define when implementing new Python objects in C.
In summary: types defined in C have a structure that contains pointers to special methods. Guido wanted to keep consistency with types defined in Python and so their special methods end up being used at the class level.
Could the implementation always follow the lookup order? Yes... at a huge cost in speed, since now even the C code would have to first perform a dictionary lookup on the instance to ensure whether or not a special method is defined and call that. Given that special methods are called often, especially for built-in types, it makes sense to just have a direct pointer to the function in the class. The behaviour of the python side is just consistent with this.
Python was never bright in the performance sector. Your suggested implementation would run extremely slowly, especially 20 years ago when it was design on way less powerful machines and when JITs were extremely rare and not so well understood (compared to the present).
Related
I want to use a method from another class.
Neither inheritance nor delegation is a good choice (to my understanding) because the existing class is too complicated to override and too expensive to instanciate.
Note that modifying the existing class is not allowed (legacy project, you know).
I came up with a way:
class Old:
def a(self):
print('Old.a')
class Mine:
b = Old.a
and it shows
>>> Mine().b()
Old.a
>>> Mine().b
<bound method Old.a of <__main__.Mine object at 0x...>>
It seems fine.
And I tried with some more complicated cases including property modification (like self.foo = 'bar'), everything seems okay.
My question:
What is actually happening when I define methods like that?
Will that safely do the trick for my need mentioned above?
Explanation
What's happening is that you are defining a callable class property of class Mine called b. However, this works:
m = Mine()
m.b()
But this won't:
Mine.b()
Why doesn't the second way work?
When you call a function of a class, python expects the first argument to be the actual object upon which the function was called. When you do this, the self argument is automatically passed into the function behind the scenes. Since we called Mine.b() without an instantiated instance of any object, no self was passed into b().
Will this "do the trick"?
As for whether this will do the trick, that depends.
As long as Mine can behave the same way as Old, python won't complain. This is because the python interpreter does not care about the "type" of self. As long as it walks like a duck and quacks like a duck, it's a duck (see duck typing). However, can you guarantee this? What if someone goes and changes the implementation of Old.a. Most of the time, as a client of another system we have no say when the private implementation of functions change.
A simpler solution might be to pull out the functionality you are missing into a separate module. Yes, there is some code duplication but at least you can be confident the code won't change from under you.
Ultimately, if you can guarantee the behavior of Old and Mine will be similar enough for the purposes of Old.a, python really shouldn't care.
According to this answer, a class object cls can be replicated with
cls_copy = type('cls_copy', cls.__bases__, dict(cls.__dict__))
This works perfectly for most normal cases. It does not work when the metaclass of cls is not type. My initial naive fix was to do
cls_copy = type(cls)('cls_copy', cls.__bases__, dict(cls.__dict__))
However, this is simply pointless. There is no way to know what a metaclass does, as this answer to a related question points out, how it transforms the input dictionary, what additional keywords it requires, etc.
The original use of type is almost good enough with a couple of minor exceptions:
The __dict__ created by metaclasses that do not end up calling type.__new__ may be of a different type than the usual proxy object.
Classes extending the copy will not have the correct metaclass, which may cause unexpected behavior.
Any properties or other data descriptors defined in the original metaclass will no longer be available on the class object.
I am willing to ignore item #1. It is a corner case that I am willing to document away should I find a viable solution to the other items. Items #2 and #3 can be solved if it were possible to change the metaclass of the copy. I tried (again, naively)
cls_copy = type('cls_copy', cls.__bases__, dict(cls.__dict__),
metaclass=type(cls))
This just raised a TypeError, as could be expected:
TypeError: __init_subclass__() takes no keyword arguments
This makes sense in light of the docs:
Like its identity, an object’s type is also unchangeable. 1
However, the footnote states that
It is possible in some cases to change an object’s type, under certain controlled conditions. It generally isn’t a good idea though, since it can lead to some very strange behaviour if it is handled incorrectly.
What are the conditions under which it is possible to change an object's type, specifically that of a class? Is this one of those cases, and if so, how?
Note
I am aware that copy.deepcopy and inheritance are viable alternatives here. For the purpose of this question, I wish to ignore those alternatives and stick with using type-related techniques.
You could use type.__new__(type(cls), cls.__name__, cls.__bases__, dict(cls.__dict__)). This uses the normal type creation process, but creates an instance of type(cls) instead of type.
As for the __metaclass__ issue, I think that is because __metaclass__ is usually what is called, so type can't use it.
The classify_class_attrs function from the inspect module can be used to determine what kind of object each of a class's attributes is, including whether a function is an instance method, a class method, or a static method. Here is an example:
from inspect import classify_class_attrs
class Example(object):
#classmethod
def my_class_method(cls):
pass
#staticmethod
def my_static_method():
pass
def my_instance_method(self):
pass
print classify_class_attrs(Example)
This will output a list of Attribute objects for each attribute on Example, with metadata about the attribute. The relevant ones in these case are:
Attribute(name='my_class_method', kind='class method', defining_class=<class '__main__.Example'>, object=<classmethod object at 0x100535398>)
Attribute(name='my_instance_method', kind='method', defining_class=<class '__main__.Example'>, object=<unbound method Example.my_instance_method>)
Attribute(name='my_static_method', kind='static method', defining_class=<class '__main__.Example'>, object=<staticmethod object at 0x100535558>)
However, it seems that many objects in Python's standard library can't be introspected this way. I'm guessing this has something to do with the fact that many of them are implemented in C. For example, datetime.datetime.now is described with this Attribute object by inspect.classify_class_attrs:
Attribute(name='now', kind='method', defining_class=<type 'datetime.datetime'>, object=<method 'now' of 'datetime.datetime' objects>)
If we compare this to the metadata returned about the attributes on Example, you'd probably draw the conclusion that datetime.datetime.now is an instance method. But it actually behaves as a class method!
from datetime import datetime
print datetime.now() # called from the class: 2014-09-12 16:13:33.890742
print datetime.now().now() # called from a datetime instance: 2014-09-12 16:13:33.891161
Is there a reliable way to determine whether a method on a stdlib class is a static, class, or instance method?
I think you can get much of what you want, distinguishing five kinds, without relying on anything that isn't documented by inspect:
Python instance methods
Python class methods
Python static methods
Builtin instance methods
Builtin class methods or static methods
But you can't distinguish those last two from each other with using CPython-specific implementation details.
(As far as I know, only 3.x has any builtin static methods in the stdlib… but of course even in 2.x, someone could always define one in an extension module.)
The details of what's available in inspect and even what it means are a little different in each version of Python, partly because things have changed between 2.x and 3.x, partly because inspect is basically a bunch of heuristics that have gradually improved over time.
But at least for CPython 2.6 and 2.7 and 3.3-3.5, the simplest way to distinguish builtin instance methods from the other two types is isbuiltin on the method looked up from the class. For a static method or class method, this will be True; for an instance method, False. For example:
>>> inspect.isbuiltin(str.maketrans)
True
>>> inspect.isbuiltin(datetime.datetime.now)
True
>>> inspect.isbuiltin(datetime.datetime.ctime)
False
Why does this work? Well, isbuiltin will:
Return true if the object is a built-in function or a bound built-in method.
When looked up on an instance, either a regular method or a classmethod-like method is bound. But when looked up on the class, a regular method is unbound, while a classmethod-like method is bound (to the class). And of course a staticmethod-like method ends up as a plain-old function when looked up either way. So, it's a bit indirect, but it will always be correct.*
What about class methods vs. static methods?
In CPython 3.x, builtin static and class method descriptors both return the exact same type when looked up on their class, and none of the documented attributes can be used to distinguish them either. And even if this weren't true, I think the way the reference is written, it's guaranteed that no functions in inspect would be able to distinguish them.
What if we turn to the descriptors themselves? Yes, there are ways we can distinguish them… but I don't think it's something guaranteed by the language:
>>> callable(str.__dict__['maketrans'])
False
>>> callable(datetime.datetime.__dict__['now'])
True
Why does this work? Well, static methods just use a staticmethod descriptor, exactly like in Python (but wrapping a builtin function instead of a function). But class and instance methods use a special descriptor type, instead of using classmethod wrapping a (builtin) function and the (builtin) function itself, as Python class and instance methods do. These special descriptor types, classmethod_descriptor and method_descriptor, are unbound (class and instance) methods, as well as being the descriptors that bind them. There are historical/implementation reasons for this to be true, but I don't think there's anything in the language reference that requires it to be true, or even implies it.
And if you're willing to rely on implementation artifacts, isinstance(m, staticmethod) seems a lot simpler…
All that being said, are there any implementations besides CPython that have both builtin staticmethods and classmethods? If not, remember that practicality beats purity…
* What it's really testing for is whether the thing is callable without an extra argument, but that's basically the same thing as the documented "function or bound method"; either way, it's what you want.
I know that you can't call object.__setattr__ on objects not inherited from object, but is there anything else that is different between the two? I'm working in Python 2.6, if this matters.
Reading this question again I misunderstood what #paper.cut was asking about: the difference between classic classes and new-style classes (not an issue in Python 3+). I do not know the answer to that.
Original Answer*
setattr(instance, name, value) is syntactic sugar for instance.__setattr__(name, value)**.
You would only need to call object.__setattr__(...) inside a class definition, and then only if directly subclassing object -- if you were subclassing something else, Spam for example, then you should either use super() to get the next item in the heirarchy, or call Spam.__setattr__(...) -- this way you don't risk missing behavior that super-classes have defined by skipping over them directly to object.
* applies to Python 3.0+ classes and 2.x new-style classes
**There are two instances where setattr(x, ...) and x.__setattr__(...) are not the same:
x itself has a __setattr__ in it's private dictionary (so x.__dict__[__setattr__] = ... (this is almost certainly an error)
x.__class__ has a __getattribute__ method -- because __getattribute__ intercepts every lookup, even when the method/attribute exists
NB
These two caveats apply to every syntactic sugar shortcut:
setattr
getattr
len
bool
hash
etc
When declaring a class that inherits from a specific class:
class C(dict):
added_attribute = 0
the documentation for class C lists all the methods of dict (either through help(C) or pydoc).
Is there a way to hide the inherited methods from the automatically generated documentation (the documentation string can refer to the base class, for non-overwritten methods)? or is it impossible?
This would be useful: pydoc lists the functions defined in a module after its classes. Thus, when the classes have a very long documentation, a lot of less than useful information is printed before the new functions provided by the module are presented, which makes the documentation harder to exploit (you have to skip all the documentation for the inherited methods until you reach something specific to the module being documented).
I had the same problem and solved it on Python 2.7.6 for Windows (x86) by adding 3 lines to pydoc.py. Instructions:
make your own copy of Lib\pydoc.py
find all occurences of the inherited variable (3 times, on my count) in the file and set it to an empty list right after it is defined. For example, I got line 809:
attrs, inherited = _split_list(attrs, lambda t: t[2] is thisclass)
and wrote inherited = [] on a new line below it.
Now it doesn't print inherited methods anymore.
You can give your class a metaclass with the special method __dir__ that returns its own list of attributes. Pydoc will use this list.
Notes:
this will also affect the behaviour of dir().
in rare cases, use of metaclasses has been known to open a portal to a hell dimension.
pydoc and the help builtin don't support this, but there's no reason you couldn't write your own tool (perhaps by modifying pydoc's source) that would have the behavior you desire. Just walk the dict of classes to get the locally-defined attributes, then look for things that have doc as an attribute.