Is it me or python that is confused with the following code ? I would expect __le__ to be called by a <= ab, not __ge__:
#!/usr/bin/env python2
class B(object):
def __ge__(self, other):
print("__ge__ unexpectedly called")
class A(object):
def __le__(self, other):
print("__le__ called")
class AB(A, B):
pass
a = A()
ab = AB()
a <= ab # --> __ge__ unexpectedly called
ab <= a # --> __le__ called
I get the same behavior with python 2.7, 3.2 and pypy 1.9.
What can I do to get __le__ called instead of __ge__ ??
The short answer is that they wanted to allow AB to override the behavior from A. Python can't call AB.__lt__(a, ab), because a may not be a valid self for an AB method, so instead, it calls AB.__gt__(ab, a), which is valid.
The long answer is a bit more complicated.
According to the docs for rich comparison operators:
There are no swapped-argument versions of these methods (to be used when the left argument does not support the operation but the right argument does); rather, __lt__() and __gt__() are each other’s reflection, __le__() and __ge__() are each other’s reflection, and __eq__() and __ne__() are their own reflection.
In other words, x <= y will call y.__ge__(x) in exactly the same cases where x+y would call y.__radd__(x). To compare:
>>> class X(object):
... def __add__(self, other):
... print('X.add')
>>> class Y(object):
... def __radd__(self, other):
... print('Y.radd')
>>> class XY(X, Y):
... pass
>>> x, xy = X(), XY()
>>> x + xy
Y.radd
According to the docs for reflected operators:
These methods are called to implement the binary arithmetic operations… with reflected (swapped) operands. These functions are only called if the left operand does not support the corresponding operation and the operands are of different types…
Note: If the right operand’s type is a subclass of the left operand’s type and that subclass provides the reflected method for the operation, this method will be called before the left operand’s non-reflected method. This behavior allows subclasses to override their ancestors’ operations.
So, because XY is a subclass of X, XY.__radd__ gets preference over X.__add__. And, likewise, because AB is a subclass of A, AB.__ge__ gets preference over A.__le__.
This probably should be documented better. To figure it out, you have to ignore the parenthetical "to be used when the left argument does not support the operation but the right argument does", guess that you need to look up the normal swapped operators (there's no link, or even mention, here), then ignore the wording that says "These functions are only called if the left operand does not support the corresponding operation", and see the "Note", which contradicts what came above… Also notice that the docs explicitly say, "There are no implied relationships among the comparison operators", only a paragraph before describing the swapped cases, which imply exactly such relationships…
Finally, this case seems odd, because AB, rather than overriding __ge__ itself, just inherited it from B, which knows nothing about A and is unrelated to it. Presumably B didn't intend to have its subclasses override A's behavior. But if B were meant to be used as a mixin for A-derived classes, maybe it would intend exactly such an override. And at any rate, the rule is probably already complicated enough without getting into where each method came from in the MRO. Whatever the reasoning, where the __ge__ comes from is irrelevant; if it's there on the subclass, it gets called.
For your added final, question, "What can I do to get __le__ called instead of __ge__ ??"… well, you really can't, any more than you can get X.__add__ called instead of XY.__radd__. Of course you can always implement an AB.__ge__ (or XY.__radd__) that calls A.__le__ (or X.__add__), but it's presumably easier to just implement AB.__ge__ in such a way that it works with an A as its other argument in the first place. Alternatively, you could remove the inheritance and find some other way to model whatever you were modeling that way. Or you could explicitly call a.__le__(ab) instead of a<=ab. But otherwise, if you designed your classes in a way that takes advantage of the "no implied relationships" to do something weird, you were misled by the docs, and will have to redesign them somehow.
Related
Base on my understanding, magic methods such as __str__ , __next__, __setattr__ are built-in features in Python. They will automatically called when a instance object is created. It also plays a role of overridden. What else some important features of magic method do I omit or ignore?
"magic" methods in python do specific things in specific contexts.
For example, to "override" the addition operator (+), you'd define a __add__ method. subtraction is __sub__, etc.
Other methods are called during object creation (__new__, __init__). Other methods are used with specific language constructs (__enter__, __exit__ and you might argue __init__ and __next__).
Really, there's nothing special about magic methods other than they are guaranteed to be called by the language at specific times. As the programmer, you're given the power to hook into structure and change the way an object behaves in those circumstances.
For a near complete summary, have a look at the python data model.
There is a lot you can do with magic methods and since it can be hard finding the right way to get started, I'd like to give you some inspiration on what I'm using a lot.
While you're probably already using some of them (like __init__), I would start learning on the operator specific magic methods, which helped me a lot optimising classes and how I use them. The magic method __mul__ for example allows you to describe what should happen to your class in case it's getting called by the multiplication operator. In the following example you can see, that the interpreter first looks for a multiplicand's __mul__ method and if this doesn't exist (like in the second example) it tries to call the multiplier's __rmul__ method.
Example 1:
class a:
def __mul__(self, other):
print("__mul__ a")
def __rmul__(self, other):
print("__rmul__ a")
class b:
def __mul__(self, other):
print("__mul__ b")
def __rmul__(self, other):
print("__rmul__ b")
ia = a()
ib = b()
ia * ib
# prints __mul__ a
Example 2:
class a:
pass
class b:
def __mul__(self, other):
print("__mul__ b")
def __rmul__(self, other):
print("__rmul__ b")
ia = a()
ib = b()
ia * ib
# prints __rmul__ b
Any other operator works corresponding to this example. I hope that helps you getting started enhancing your magic method skills.
hint:
To make your classes comparable you can use __cmp__.
The method is called with the two classes which are compared.
Return a positive value if the first class is bigger.
Return a nagative value if the second class is bigger.
Return zero if they have the same size.
You don't have to use the magic methods for every possibility.
I'm taking my first computing science course, and we just learned about class implementation and inheritance. In particular, we just covered method overriding and how classes we define inherit from the object superclass by default. As one of my examples trying out this particular case of inheritance, I used the following code:
class A:
def __init__(self, i):
self.i = i
def __str__(self):
return "A"
# Commenting out these two lines to not override __eq__(), just use the
# default from our superclass, object
#def __eq__(self, other):
#return self.i == other.i
x = A(2)
y = A(2)
>>>print(x == y)
False
>>>print(x.__eq__(y))
NotImplemented
I expected the result from (x == y), because as I understand it the default for __eq__() is to check if they're the same objects or not, not worrying about the contents. Which is False, x and y have the same contents but are different objects. The second one surprised me though.
So my questions: I thought (x==y) and x.__eq__(y) were synonymous and made exactly the same call. Why do these produce differing output? And why does the second conditional return NotImplemented?
The == operator is equivalent to the eq function, which will internally call the __eq__ method of the left operand if it exists to try to determine equality. This is not the only thing it will do, and if __eq__ does not exist, as is the case here, it will do other checks, such as checking whether the two are the same object, or __cmp__ pre-Python 3.
So in a nutshell, your confusion arises from this assumption, which is incorrect:
I thought (x==y) and x.__eq__(y) were synonymous and made exactly the same call
In fact, (x==y) and operators.eq(x, y) are synonymous, and x.__eq__(y) is one of the things eq(x, y) will try to check.
The NotImplemented value you're seeing returned from your inherited __eq__ method is a special builtin value used as a sentinel in Python. It can be returned by __magic__ methods that implement mathematical or comparison operators to indicate that the class does not support the operator that was attempted (with the provided arguments).
This can be more useful than raising an exception, as it allows Python to fall back to other options to resolve the operator use. For instance, if you do x + y, Python will first try to run x.__add__(y). If that returns NotImplemented, it will next try the "reverse" version, y.__radd__(x), which may work if y is a more sophisticated type than x is.
In the case you're asking about, x == y, Python first tries x.__eq__(y), then y.__eq__(x), and finally x is y (which will always evaluate to a Boolean value). Since object.__eq__ returns NotImplemented in all cases, your class falls back to the identity comparison when you use the real operator, but shows you the NotImplemented sentinel when you call __eq__ directly.
If you have implemented the __eq__() function for a class, it gets called when you use x == y. Otherwise x == y relies on a default comparison logic. __eq__() does not get implemented automatically when you define a class.
Although the title can be interpreted as three questions, the actual problem is simple to describe. On a Linux system I have python 2.7.3 installed, and want to be warned about python 3 incompatibilities. Therefore, my code snippet (tester.py) looks like:
#!/usr/bin/python -3
class MyClass(object):
def __eq__(self, other):
return False
When I execute this code snippet (thought to be only to show the problem, not an actual piece of code I am using in my project) as
./tester.py
I get the following deprecation warning:
./tester.py:3: DeprecationWarning: Overriding __eq__ blocks inheritance of __hash__ in 3.x
class MyClass(object):
My question: How do I change this code snippet to get rid of the warning, i.e. to make it compatible to version 3? I want to implement the equality operator in the correct way, not just suppressing the warning or something similar.
From the documentation page for Python 3.4:
If a class does not define an __eq__() method it should not define a __hash__() operation either; if it defines __eq__() but not __hash__(), its instances will not be usable as items in hashable collections. If a class defines mutable objects and implements an __eq__() method, it should not implement __hash__(), since the implementation of hashable collections requires that a key’s hash value is immutable (if the object’s hash value changes, it will be in the wrong hash bucket).
Basically, you need to define a __hash()__ function.
The problem is that for user-defined classes, the __eq()__ and __hash()__ functions are automatically defined.
x.__hash__() returns an appropriate value such that x == y implies
both that x is y and hash(x) == hash(y).
If you define just the __eq()__, then __hash()__ is set to return None. So you will hit the wall.
The simpler way out if you don't want to bother about implementing the __hash()__ and you know for certain that your object will never be hashed, you just explicitly declare __hash__ = None which takes care of the warning.
Alex: python's -3 option is warning you about a potential problem; it doesn't know that you aren't using instances of MyClass in sets or as keys in mappings, so it warns that something that you might have been relying on wouldn't work, if you were. If you aren't using MyClass that way, just ignore the warning. It's a dumb tool to help you catch potential problems; in the end, you're expected to be the one with the actual intelligence to work out which warnings actually matter.
If you really care about suppressing the warning - or, indeed, if a class is mutable and you want to make sure it isn't used in sets or as the key in any mapping - the simple assignment __hash__ = None (as Sudipta pointed out) in the class body shall do that for you. Since None isn't callable, this makes instances non-hashable.
class MyClass (object):
def __eq__(self, other): return self is other
__hash__ = None
In Python, the comparison operators -- <, <=, ==, !=, >, >= -- can be implemented to mean whatever is relevant to the implementing class. In Python 2 that was done by overriding __cmp__, and in Python 3 by overriding __lt__ and friends. What is the advantage of having an issubclass() built-in method instead of allowing for expressions such as bool < int (true), int < object (true), int <= int, or int < float (false). In particular, I'll note that classes ordered by issubclass() constitutes a partially ordered set in the mathematical sense.
The Python 3 equivalent of what I'm thinking would look like what's below. This code doesn't replace issubclass() (though looping over the MRO would accomplish that, right?). However, wouldn't this be more intuitive?
#functools.total_ordering
class Type(type):
"Metaclass whose instances (which are classes) can use <= instead issubclass()"
def __lt__(self, other):
try:
return issubclass(self, other) and self != other
except TypeError: # other isn't a type or tuple of types
return NotImplemented
def __eq__(self, other):
if isinstance(other, tuple): # For compatibility with __lt__
for other_type in other:
if type(self) is type(other_type):
return False
return True
else:
return type(self) is type(other)
Actual Question: What is the advantage of having an issubclass() built-in method instead of allowing for expressions such as bool < int (true), int < object (true), int <= int, or int < float (false).
Because it would be against the Zen of Python: http://www.python.org/dev/peps/pep-0020/
Explicit is better than implicit.
If you look at the following line of code in isolation:
issubclass(a, b)
It's perfectly obvious that a and b are variables containing classes and we are checking if a is a subclass of b. And if they happen to not contain classes, you'll know.
But looking at this
a < b
Would not tell you anything. You need to examine the surrounding code to determine they contain classes before you know that we are checking if the class in a is a subclass of b. And if say a=5 and b=6 it will still run "fine".
But Python is flexible, so if you really want this, you can implement a base type with such behaviour as you've shown.
Actually - as an aside - the prevalence of overloading operators in C++ for example is a significant drawback of the language (at least in my eyes) because when you see a + b it might as well launch a nuclear missile for all you know.... until you check types of a/b, look up the class implementation and + operator overload implementation (if any... and if not see if the parent class has any.... and if not see if the parent parent...)
One advantage in bold:
issubclass(class, classinfo)
Return true if class is a subclass (direct, indirect or virtual) of classinfo. A class is considered a subclass of itself. classinfo may be a tuple of class objects, in which case every entry in classinfo will be checked. In any other case, a TypeError exception is raised.
Another is that it's descriptive; not everyone using Python is a mathematician.
I would say the advantage is non-functional. The only technical difference is that < is infix.
But this question isn't about technical stuff. It seems to be about semantics and ease of reading.
Using < would denote order. Although class hierarchy can be interpreted as "orderable", it'll always be an approximation. A non-obvious one, for many people.
Using issubclass is clearer, still simple and doesn't lend itself to no other interpretation other than what it actually does: check if an object/classinfo is a subclass of class.
Plain, simple, unambiguous, effective. Those are the advantages. Maybe you don't/can't take advantage of them. But that's already personal taste.
The Python docs clearly state that x==y calls x.__eq__(y). However it seems that under many circumstances, the opposite is true. Where is it documented when or why this happens, and how can I work out for sure whether my object's __cmp__ or __eq__ methods are going to get called.
Edit: Just to clarify, I know that __eq__ is called in preferecne to __cmp__, but I'm not clear why y.__eq__(x) is called in preference to x.__eq__(y), when the latter is what the docs state will happen.
>>> class TestCmp(object):
... def __cmp__(self, other):
... print "__cmp__ got called"
... return 0
...
>>> class TestEq(object):
... def __eq__(self, other):
... print "__eq__ got called"
... return True
...
>>> tc = TestCmp()
>>> te = TestEq()
>>>
>>> 1 == tc
__cmp__ got called
True
>>> tc == 1
__cmp__ got called
True
>>>
>>> 1 == te
__eq__ got called
True
>>> te == 1
__eq__ got called
True
>>>
>>> class TestStrCmp(str):
... def __new__(cls, value):
... return str.__new__(cls, value)
...
... def __cmp__(self, other):
... print "__cmp__ got called"
... return 0
...
>>> class TestStrEq(str):
... def __new__(cls, value):
... return str.__new__(cls, value)
...
... def __eq__(self, other):
... print "__eq__ got called"
... return True
...
>>> tsc = TestStrCmp("a")
>>> tse = TestStrEq("a")
>>>
>>> "b" == tsc
False
>>> tsc == "b"
False
>>>
>>> "b" == tse
__eq__ got called
True
>>> tse == "b"
__eq__ got called
True
Edit: From Mark Dickinson's answer and comment it would appear that:
Rich comparison overrides __cmp__
__eq__ is it's own __rop__ to it's __op__ (and similar for __lt__, __ge__, etc)
If the left object is a builtin or new-style class, and the right is a subclass of it, the right object's __rop__ is tried before the left object's __op__
This explains the behaviour in theTestStrCmp examples. TestStrCmp is a subclass of str but doesn't implement its own __eq__ so the __eq__ of str takes precedence in both cases (ie tsc == "b" calls b.__eq__(tsc) as an __rop__ because of rule 1).
In the TestStrEq examples, tse.__eq__ is called in both instances because TestStrEq is a subclass of str and so it is called in preference.
In the TestEq examples, TestEq implements __eq__ and int doesn't so __eq__ gets called both times (rule 1).
But I still don't understand the very first example with TestCmp. tc is not a subclass on int so AFAICT 1.__cmp__(tc) should be called, but isn't.
You're missing a key exception to the usual behaviour: when the right-hand operand is an instance of a subclass of the class of the left-hand operand, the special method for the right-hand operand is called first.
See the documentation at:
http://docs.python.org/reference/datamodel.html#coercion-rules
and in particular, the following two paragraphs:
For objects x and y, first
x.__op__(y) is tried. If this is not
implemented or returns
NotImplemented, y.__rop__(x) is
tried. If this is also not implemented
or returns NotImplemented, a
TypeError exception is raised. But see
the following exception:
Exception to the previous item: if the
left operand is an instance of a
built-in type or a new-style class,
and the right operand is an instance
of a proper subclass of that type or
class and overrides the base’s
__rop__() method, the right
operand’s __rop__() method is tried
before the left operand’s __op__()
method.
Actually, in the docs, it states:
[__cmp__ is c]alled by comparison operations if rich comparison (see above) is not defined.
__eq__ is a rich comparison method and, in the case of TestCmp, is not defined, hence the calling of __cmp__
As I know, __eq__() is a so-called “rich comparison” method, and is called for comparison operators in preference to __cmp__() below. __cmp__() is called if "rich comparison" is not defined.
So in A == B:
If __eq__() is defined in A it will be called
Else __cmp__() will be called
__eq__() defined in 'str' so your __cmp__() function was not called.
The same rule is for __ne__(), __gt__(), __ge__(), __lt__() and __le__() "rich comparison" methods.
Is this not documented in the Language Reference? Just from a quick look there, it looks like __cmp__ is ignored when __eq__, __lt__, etc are defined. I'm understanding that to include the case where __eq__ is defined on a parent class. str.__eq__ is already defined so __cmp__ on its subclasses will be ignored. object.__eq__ etc are not defined so __cmp__ on its subclasses will be honored.
In response to the clarified question:
I know that __eq__ is called in
preferecne to __cmp__, but I'm not
clear why y.__eq__(x) is called in
preference to x.__eq__(y), when the
latter is what the docs state will
happen.
Docs say x.__eq__(y) will be called first, but it has the option to return NotImplemented in which case y.__eq__(x) is called. I'm not sure why you're confident something different is going on here.
Which case are you specifically puzzled about? I'm understanding you just to be puzzled about the "b" == tsc and tsc == "b" cases, correct? In either case, str.__eq__(onething, otherthing) is being called. Since you don't override the __eq__ method in TestStrCmp, eventually you're just relying on the base string method and it's saying the objects aren't equal.
Without knowing the implementation details of str.__eq__, I don't know whether ("b").__eq__(tsc) will return NotImplemented and give tsc a chance to handle the equality test. But even if it did, the way you have TestStrCmp defined, you're still going to get a false result.
So it's not clear what you're seeing here that's unexpected.
Perhaps what's happening is that Python is preferring __eq__ to __cmp__ if it's defined on either of the objects being compared, whereas you were expecting __cmp__ on the leftmost object to have priority over __eq__ on the righthand object. Is that it?