Looking through decimal.py, it uses NotImplemented in many special methods. e.g.
class A(object):
def __lt__(self, a):
return NotImplemented
def __add__(self, a):
return NotImplemented
The Python docs say:
NotImplemented
Special value which can be returned by the “rich comparison”
special methods (__eq__(), __lt__(),
and friends), to indicate that the
comparison is not implemented with
respect to the other type.
It doesn't talk about other special methods and neither does it describe the behavior.
It seems to be a magic object which if returned from other special methods raises TypeError, and in “rich comparison” special methods does nothing.
e.g.
print A() < A()
prints True, but
print A() + 1
raises TypeError, so I am curious as to what's going on and what is the usage/behavior of NotImplemented.
NotImplemented allows you to indicate that a comparison between the two given operands has not been implemented (rather than indicating that the comparison is valid, but yields False, for the two operands).
From the Python Language Reference:
For objects x and y, first x.__op__(y)
is tried. If this is not implemented
or returns NotImplemented,
y.__rop__(x) is tried. If this is also
not implemented or returns
NotImplemented, a TypeError exception
is raised. But see the following
exception:
Exception to the previous
item: if the left operand is an
instance of a built-in type or a
new-style class, and the right operand
is an instance of a proper subclass of
that type or class and overrides the
base's __rop__() method, the right
operand's __rop__() method is tried
before the left operand's __op__()
method. This is done so that a
subclass can completely override
binary operators. Otherwise, the left
operand's __op__() method would always
accept the right operand: when an
instance of a given class is expected,
an instance of a subclass of that
class is always acceptable.
It actually has the same meaning when returned from __add__ as from __lt__, the difference is Python 2.x is trying other ways of comparing the objects before giving up. Python 3.x does raise a TypeError. In fact, Python can try other things for __add__ as well, look at __radd__ and (though I'm fuzzy on it) __coerce__.
# 2.6
>>> class A(object):
... def __lt__(self, other):
... return NotImplemented
>>> A() < A()
True
# 3.1
>>> class A(object):
... def __lt__(self, other):
... return NotImplemented
>>> A() < A()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unorderable types: A() < A()
See Ordering Comparisions (3.0 docs) for more info.
If you return it from __add__ it will behave like the object has no __add__ method, and raise a TypeError.
If you return NotImplemented from a rich comparison function, Python will behave like the method wasn't implemented, that is, it will defer to using __cmp__.
Related
I am building up my understanding of Python, and recently I understood that functions must be classes(?), and that a def func(): just instantiates an object of class function. I was mindblown when I created an attribute of func honestly.
Lurking in dir(func) I noticed that indeed all the special methods such as .__eq__ are inherited, and I wanted to play around with it:
def func(n):
print(n)
def __eq__(self,func2):
print('hello')
However, it does not work:
>>> func.__eq__(print)
NotImplemented
What would it be the proper way to overload the equality operator for a function? I don't see how to overload it without having a proper class definition.
Because python treats all functions as objects, it might be worth thinking of creating functions as creating instances of the class function (which isn't documented) with the __call__ method being the body of the function created with def. The actual C source of the function class is on Github if you want to know implementation details.
With returning NotImplemented:
In Python, if objects do not override __eq__ or __hash__, there is default implementations where __hash__ = builtins.id and __eq__ is like lambda self, other: self is other. When the comparison operators return NotImplemented, this instructs the runtime to search for another method that does the same thing, like trying __ne__ instead of __eq__, or trying operators from the parent.
>>> def test(a):
... return a
...
>>> def test2(a):
... return a
...
>>> test == test2
False
>>> test.__eq__(test2)
NotImplemented
You can also test this by creating a dummy class that doesn't override __eq__ (like how the function class doesn't):
>>> class testcls:
... pass
...
>>> t1 = testcls()
>>> t2 = testcls()
>>> t1.__eq__(t2)
NotImplemented
>>> t1.__eq__(t1)
True
>>> t1 == t2
False
No, functions don't have to be classes (yet they are in Python), actually. But that is another story.
Seems you misunderstand that __eq__ is a method, i.e. you need a class that it belongs to in order to overload it. What you code does is defines some custom function within another function, which is valid Python code but has nothing to do with __eq__ overloading.
Since you can't inherit function, you can't overwrite it's eq. And even if you could do that, it won't work due to some purely theoretical reasons.
I am trying to understand the usefulness of the method register of abc.ABCMeta.
To my understanding, after reading https://docs.python.org/3/library/abc.html:
the following code:
class MyFoo(Complex): ...
MyFoo.register(Real)
will create the class MyFoo which implements the Complex abstract class.
After registering the MyFoo with Real, the isinstance and issubclass will return as if MyFoo was derived from Real.
What I don't understand is why the Real, doesn't get added to the mro.
I am asking since the following code will not behave as I would have expected:
def trunc(inst):
if isinstance(inst, Real):
return inst.__trunc__() #this should generate error since Complex doesn't have the __trunc__ attr
else:
return NotImplemented
trunc(MyFoo())
Shouldn't I have as a given that when isinstance returns true, then the underlying object should have all the characteristics of the class that it's checked against?
Note, if it's not obvious, i quite new to the language so please bear with me.
Using isinstance to examine properties of an object
The idea of Abstract Base Classes (or ABC) is to provide a way to override behaviour of isinstance for types which one cannot control.
For example, you may want to use isinstance to check whether an object supports addition (i.e. it is allowed to do a + b for a and b which satisfy the check).
For that, you could implement a base class and some subclasses:
class TypeWithAddition:
def __add__(self, other):
raise NotImplementedError("Override __add__ in the subclass")
class MyClass(TypeWithAddition):
def __init__(self, value):
self.value = value
def __add__(self, other):
return MyClass(self.value + other.value)
Then, you could check isinstance before attempting to sum any two objects.
For example:
a = MyClass(7)
b = MyClass(8)
if isinstance(a, TypeWithAddition) and isinstance(b, TypeWithAddition):
c = a + b
print(c.value)
else:
print('Cannot calculate a+b')
Using ABC to extend isinstance for existing classes
The problem with this approach is that it does not work for types which already exist and support addition, but are not inherited from your base class:
>>> isinstance(7, TypeWithAddition)
False
This is where ABC kicks in. One can inherit the base ("interface") class from ABC and then register any existing class with it.
class TypeWithAddition(abc.ABC):
def __add__(self, other):
raise NotImplementedError("Override __add__ in the subclass")
TypeWithAddition.register(int)
And now it looks like int is inherited from TypeWithAddition:
>>> isinstance(7, TypeWithAddition)
True
But of course, int does not really inherit from TypeWithAddition! And there is really no check that it supports everything that you would expect from TypeWithAddition. It is your (programmer's) job to make sure it does before writing TypeWithAddition.register(int).
An error:
You can easily do this, and it won't work well, of course:
class Foo:
pass
TypeWithAddition.register(Foo)
It will now seem that Foo supports addition, but it does not:
>>> Foo() + Foo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'Foo' and 'Foo'
It does not work because it was wrong to register Foo as a TypeWithAddition in the first place.
I hope this make things at least a bit clearer.
Let's say you work with a wrapper object:
class IterOrNotIter:
def __init__(self):
self.f = open('/tmp/toto.txt')
def __getattr__(self, item):
try:
return self.__getattribute__(item)
except AttributeError:
return self.f.__getattribute__(item)
This object implements __iter__, because it passes any call to it to its member f, which implements it. Case in point:
>>> x = IterOrNotIter()
>>> x.__iter__().__next__()
'Whatever was in /tmp/toto.txt\n'
According to the documentation (https://docs.python.org/3/library/stdtypes.html#iterator-types), IterOrNotIter should thus be iterable.
However, the Python interpreter does not recognize an IterOrNotIter object as actually being iterable:
>>> x = IterOrNotIter()
>>> for l in x:
... print(l)
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'IterOrNotIter' object is not iterable
Whereas this works:
>>> x = IterOrNotIter()
>>> for l in x.f:
... print(l)
...
Whatever was in /tmp/toto.txt
I don't understand why.
Basically because your class just doesn't have a real __iter__ method:
>>> hasattr(IterOrNotIter, '__iter__')
False
So it doesn't qualify as iterator because the actual check for __iter__ checks for the existence instead of assuming it's implemented. So workarounds with __getattr__ or __getattribute__ (unfortunatly) don't work.
This is actually mentioned in the documentation for __getattribute__:
Note
This method may still be bypassed when looking up special methods as the result of implicit invocation via language syntax or built-in functions. See Special method lookup.
The latter section also explains the why:
Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).
Emphasis mine.
Is it me or python that is confused with the following code ? I would expect __le__ to be called by a <= ab, not __ge__:
#!/usr/bin/env python2
class B(object):
def __ge__(self, other):
print("__ge__ unexpectedly called")
class A(object):
def __le__(self, other):
print("__le__ called")
class AB(A, B):
pass
a = A()
ab = AB()
a <= ab # --> __ge__ unexpectedly called
ab <= a # --> __le__ called
I get the same behavior with python 2.7, 3.2 and pypy 1.9.
What can I do to get __le__ called instead of __ge__ ??
The short answer is that they wanted to allow AB to override the behavior from A. Python can't call AB.__lt__(a, ab), because a may not be a valid self for an AB method, so instead, it calls AB.__gt__(ab, a), which is valid.
The long answer is a bit more complicated.
According to the docs for rich comparison operators:
There are no swapped-argument versions of these methods (to be used when the left argument does not support the operation but the right argument does); rather, __lt__() and __gt__() are each other’s reflection, __le__() and __ge__() are each other’s reflection, and __eq__() and __ne__() are their own reflection.
In other words, x <= y will call y.__ge__(x) in exactly the same cases where x+y would call y.__radd__(x). To compare:
>>> class X(object):
... def __add__(self, other):
... print('X.add')
>>> class Y(object):
... def __radd__(self, other):
... print('Y.radd')
>>> class XY(X, Y):
... pass
>>> x, xy = X(), XY()
>>> x + xy
Y.radd
According to the docs for reflected operators:
These methods are called to implement the binary arithmetic operations… with reflected (swapped) operands. These functions are only called if the left operand does not support the corresponding operation and the operands are of different types…
Note: If the right operand’s type is a subclass of the left operand’s type and that subclass provides the reflected method for the operation, this method will be called before the left operand’s non-reflected method. This behavior allows subclasses to override their ancestors’ operations.
So, because XY is a subclass of X, XY.__radd__ gets preference over X.__add__. And, likewise, because AB is a subclass of A, AB.__ge__ gets preference over A.__le__.
This probably should be documented better. To figure it out, you have to ignore the parenthetical "to be used when the left argument does not support the operation but the right argument does", guess that you need to look up the normal swapped operators (there's no link, or even mention, here), then ignore the wording that says "These functions are only called if the left operand does not support the corresponding operation", and see the "Note", which contradicts what came above… Also notice that the docs explicitly say, "There are no implied relationships among the comparison operators", only a paragraph before describing the swapped cases, which imply exactly such relationships…
Finally, this case seems odd, because AB, rather than overriding __ge__ itself, just inherited it from B, which knows nothing about A and is unrelated to it. Presumably B didn't intend to have its subclasses override A's behavior. But if B were meant to be used as a mixin for A-derived classes, maybe it would intend exactly such an override. And at any rate, the rule is probably already complicated enough without getting into where each method came from in the MRO. Whatever the reasoning, where the __ge__ comes from is irrelevant; if it's there on the subclass, it gets called.
For your added final, question, "What can I do to get __le__ called instead of __ge__ ??"… well, you really can't, any more than you can get X.__add__ called instead of XY.__radd__. Of course you can always implement an AB.__ge__ (or XY.__radd__) that calls A.__le__ (or X.__add__), but it's presumably easier to just implement AB.__ge__ in such a way that it works with an A as its other argument in the first place. Alternatively, you could remove the inheritance and find some other way to model whatever you were modeling that way. Or you could explicitly call a.__le__(ab) instead of a<=ab. But otherwise, if you designed your classes in a way that takes advantage of the "no implied relationships" to do something weird, you were misled by the docs, and will have to redesign them somehow.
The Python docs clearly state that x==y calls x.__eq__(y). However it seems that under many circumstances, the opposite is true. Where is it documented when or why this happens, and how can I work out for sure whether my object's __cmp__ or __eq__ methods are going to get called.
Edit: Just to clarify, I know that __eq__ is called in preferecne to __cmp__, but I'm not clear why y.__eq__(x) is called in preference to x.__eq__(y), when the latter is what the docs state will happen.
>>> class TestCmp(object):
... def __cmp__(self, other):
... print "__cmp__ got called"
... return 0
...
>>> class TestEq(object):
... def __eq__(self, other):
... print "__eq__ got called"
... return True
...
>>> tc = TestCmp()
>>> te = TestEq()
>>>
>>> 1 == tc
__cmp__ got called
True
>>> tc == 1
__cmp__ got called
True
>>>
>>> 1 == te
__eq__ got called
True
>>> te == 1
__eq__ got called
True
>>>
>>> class TestStrCmp(str):
... def __new__(cls, value):
... return str.__new__(cls, value)
...
... def __cmp__(self, other):
... print "__cmp__ got called"
... return 0
...
>>> class TestStrEq(str):
... def __new__(cls, value):
... return str.__new__(cls, value)
...
... def __eq__(self, other):
... print "__eq__ got called"
... return True
...
>>> tsc = TestStrCmp("a")
>>> tse = TestStrEq("a")
>>>
>>> "b" == tsc
False
>>> tsc == "b"
False
>>>
>>> "b" == tse
__eq__ got called
True
>>> tse == "b"
__eq__ got called
True
Edit: From Mark Dickinson's answer and comment it would appear that:
Rich comparison overrides __cmp__
__eq__ is it's own __rop__ to it's __op__ (and similar for __lt__, __ge__, etc)
If the left object is a builtin or new-style class, and the right is a subclass of it, the right object's __rop__ is tried before the left object's __op__
This explains the behaviour in theTestStrCmp examples. TestStrCmp is a subclass of str but doesn't implement its own __eq__ so the __eq__ of str takes precedence in both cases (ie tsc == "b" calls b.__eq__(tsc) as an __rop__ because of rule 1).
In the TestStrEq examples, tse.__eq__ is called in both instances because TestStrEq is a subclass of str and so it is called in preference.
In the TestEq examples, TestEq implements __eq__ and int doesn't so __eq__ gets called both times (rule 1).
But I still don't understand the very first example with TestCmp. tc is not a subclass on int so AFAICT 1.__cmp__(tc) should be called, but isn't.
You're missing a key exception to the usual behaviour: when the right-hand operand is an instance of a subclass of the class of the left-hand operand, the special method for the right-hand operand is called first.
See the documentation at:
http://docs.python.org/reference/datamodel.html#coercion-rules
and in particular, the following two paragraphs:
For objects x and y, first
x.__op__(y) is tried. If this is not
implemented or returns
NotImplemented, y.__rop__(x) is
tried. If this is also not implemented
or returns NotImplemented, a
TypeError exception is raised. But see
the following exception:
Exception to the previous item: if the
left operand is an instance of a
built-in type or a new-style class,
and the right operand is an instance
of a proper subclass of that type or
class and overrides the base’s
__rop__() method, the right
operand’s __rop__() method is tried
before the left operand’s __op__()
method.
Actually, in the docs, it states:
[__cmp__ is c]alled by comparison operations if rich comparison (see above) is not defined.
__eq__ is a rich comparison method and, in the case of TestCmp, is not defined, hence the calling of __cmp__
As I know, __eq__() is a so-called “rich comparison” method, and is called for comparison operators in preference to __cmp__() below. __cmp__() is called if "rich comparison" is not defined.
So in A == B:
If __eq__() is defined in A it will be called
Else __cmp__() will be called
__eq__() defined in 'str' so your __cmp__() function was not called.
The same rule is for __ne__(), __gt__(), __ge__(), __lt__() and __le__() "rich comparison" methods.
Is this not documented in the Language Reference? Just from a quick look there, it looks like __cmp__ is ignored when __eq__, __lt__, etc are defined. I'm understanding that to include the case where __eq__ is defined on a parent class. str.__eq__ is already defined so __cmp__ on its subclasses will be ignored. object.__eq__ etc are not defined so __cmp__ on its subclasses will be honored.
In response to the clarified question:
I know that __eq__ is called in
preferecne to __cmp__, but I'm not
clear why y.__eq__(x) is called in
preference to x.__eq__(y), when the
latter is what the docs state will
happen.
Docs say x.__eq__(y) will be called first, but it has the option to return NotImplemented in which case y.__eq__(x) is called. I'm not sure why you're confident something different is going on here.
Which case are you specifically puzzled about? I'm understanding you just to be puzzled about the "b" == tsc and tsc == "b" cases, correct? In either case, str.__eq__(onething, otherthing) is being called. Since you don't override the __eq__ method in TestStrCmp, eventually you're just relying on the base string method and it's saying the objects aren't equal.
Without knowing the implementation details of str.__eq__, I don't know whether ("b").__eq__(tsc) will return NotImplemented and give tsc a chance to handle the equality test. But even if it did, the way you have TestStrCmp defined, you're still going to get a false result.
So it's not clear what you're seeing here that's unexpected.
Perhaps what's happening is that Python is preferring __eq__ to __cmp__ if it's defined on either of the objects being compared, whereas you were expecting __cmp__ on the leftmost object to have priority over __eq__ on the righthand object. Is that it?