Suppose I define an abstract base class like this:
from abc import abstractmethod, ABCMeta
class Quacker(object):
__metaclass__ = ABCMeta
#abstractmethod
def quack(self):
return "Quack!"
This ensures any class deriving from Quacker must implement the quack method. But if I define the following:
class PoliteDuck(Quacker):
def quack(self, name):
return "Quack quack %s!" % name
d = PoliteDuck() # no error
I'm allowed to instantiate the class because I've provided the quack method, but the function signatures don't match. I can see how this might be useful in some situations, but I'm in interested in ensuring I can definitely call the abstract methods. This might fail if the function signature is different!
So: how can I enforce a matching function signature? I would expect an error when creating the object if the signatures don't match, just like if I hadn't defined it at all.
I know that this is not idiomatic, and that Python is the wrong language to be using if I want these sorts of guarantees, but that's beside the point - is it possible?
It's worse than you think. Abstract methods are tracked by name only, so you don't even have to make quack a method in order to instantiate the child class.
class SurrealDuck(Quacker):
quack = 3
d = SurrealDuck()
print d.quack # Shows 3
There is nothing in the system that enforces that quack is even a callable object, let alone one whose arguments match the abstract method's original. At best, you could subclass ABCMeta and add code yourself to compare type signatures in the child to the originals in the parent, but this would be nontrivial to implement.
(Currently, marking something as "abstract" essentially just adds the name to a frozen set attribute in the parent (Quacker.__abstractmethods__). Making a class instantiable is as simple as setting this attribute to an empty iterable, which is useful for testing.)
I recommend you look at pylint. I ran this code through it's static analysis, and on the line where you defined the quack() method, it reported:
Argument number differs from overridden method (arguments-differ)
(https://en.wikipedia.org/wiki/Pylint)
I don't think this has changed in the base python language, but I did find one workaround that might be useful. The mypy package does seem to enforce signature conformity on abstract base classes and their concrete implementation. So basically if you define a signature on the abstract base class, all concrete classes have to follow the same exact signature.
Here is an example that will break in mypy. The code is taken from the mypy website, but I adapted it for this answer.
The first example is code that will pass. Note that the signatures for the eat method are the same and mypy does not complain.
from abc import ABCMeta, abstractmethod
class Animal(metaclass=ABCMeta):
#abstractmethod
def eat(self, food: str) -> None: pass
#property
#abstractmethod
def can_walk(self) -> bool: pass
class Cat(Animal):
def eat(self, food: str) -> None:
pass # Body omitted
#property
def can_walk(self) -> bool:
return True
y = Cat() # OK
But let's adapt this code a bit and now mypy throws an error:
from abc import ABCMeta, abstractmethod
class Animal(metaclass=ABCMeta):
#abstractmethod
def eat(self, food: str) -> None: pass
#property
#abstractmethod
def can_walk(self) -> bool: pass
class Cat(Animal):
def eat(self, food: str, drink: str) -> None:
pass # Body omitted
#property
def can_walk(self) -> bool:
return True
y = Cat() # Error
Mypy is still very much a work-in-progress but it does work in this case. There are some corner cases where some signature variants might not get caught, but otherwise seems to work for most practical applications.
Related
I have a base class that looks something like this:
class myBaseClass:
def __init__(self):
self.name = None # All subclasses must define this
def foo(self): # All subclasses must define this
raise NotImplementedError
def bar(self): # Optional -- not all subclasses will define this
raise NotImplementedError
My API specification stipulates that anyone creating a subclass of myBaseClass must provide a meaningful value for .name, and for the function .foo(). However, .bar() is optional and calling code should be able to handle the case where that results in a NotImplementedError.
When and how should I check that subclasses contributed by third parties meet these requirements?
The options seem to be:
Build subclasses exclusively via metaclasses. However, this approach will be unfamiliar and potentially confusing to most of the contributors to my project, who tend not to be expert developers.
Add an __init_subclass__ method to the base class and use this to infer whether the subclass has overridden everything it is supposed to override. Seems to work, but feels a bit 'kludgy'.
Write build-time tests to instantiate each subclass, call each 'required' method, and verify that they do not raise a NotImplementedError. Seems like an excessive computational effort to answer such a simple question (calling .foo() may be expensive).
Ignore the issue. Deal with it if and when it causes something else to break.
I'm sure I'm not the only person who needs to deal with this issue - is there a 'correct' approach here?
Here's how I would structure it.
First off, what you're looking for here is an abstract base class. Using the built-in modules you can easily define it as such and have methods be forced to have an implementation, otherwise the class will raise an exception when instantiated.
If the name attribute needs to be set always, then you should make it part of the constructor arguments.
Because bar is not always required I wouldn't define it as a method in the base class you have. Instead I would make a child class that is also abstract and define it there as required. When checking to see if the method is available you can use isinstance.
This is what my final code would look like:
from abc import ABC, abstractmethod
class FooBaseClass(ABC):
def __init__(self, name):
self.name = name
#abstractmethod
def foo(self):
"""Some useful docs for foo"""
class FooBarBaseClass(FooBaseClass, ABC):
#abstractmethod
def bar(self):
"""Some useful docs for bar"""
When creating instances you can pick the base class you want and will be forced to define the methods.
class FooClass(FooBaseClass):
def __init__(self):
super().__init__("foo")
def foo(self):
print("Calling foo from FooClass")
class FooBarClass(FooBarBaseClass):
def __init__(self):
super().__init__("foobar")
def foo(self):
print("Calling foo from FooBarClass")
def bar(self):
print("Calling bar from FooBarClass")
Example checking if bar is callable:
def do_operation(obj: FooBaseClass):
obj.foo()
if isinstance(obj, FooBarBaseClass):
obj.bar()
Example:
do_operation(FooClass())
do_operation(FooBarClass())
Calling foo from FooClass
Calling foo from FooBarClass
Calling bar from FooBarClass
An example of invalid code
class InvalidClass(FooBaseClass):
def __init__(self):
super().__init__("foo")
InvalidClass()
Traceback (most recent call last):
File "C:\workspace\so\test.py", line 52, in <module>
InvalidClass()
TypeError: Can't instantiate abstract class InvalidClass with abstract method foo
I'm trying to create a base class with a number of abstract python properties, in python 3.7.
I tried it one way (see 'start' below) using the #property, #abstractmethod, #property.setter annotations. This worked but it doesn't raise an exception if the subclass doesn't implement a setter. That's the point of using #abstract to me, so that's no good.
So I tried doing it another way (see 'end' below) using two #abstractmethod methods and a 'property()', which is not abstract itself but uses those methods. This approach generates an error when instantiating the subclass:
# {TypeError}Can't instantiate abstract class FirstStep with abstract methods end
I'm clearly implementing the abstract methods, so I don't understand what it means. The 'end' property is not marked #abstract, but if I comment it out, it does run (but I don't get my property). I also added that test non-abstract method 'test_elapsed_time' to demonstrate I have the class structure and abstraction right (it works).
Any chance I'm doing something dumb, or is there some special behavior around property() that's causing this?
class ParentTask(Task):
def get_first_step(self):
# {TypeError}Can't instantiate abstract class FirstStep with abstract methods end
return FirstStep(self)
class Step(ABC):
# __metaclass__ = ABCMeta
def __init__(self, task):
self.task = task
# First approach. Works, but no warnings if don't implement setter in subclass
#property
#abstractmethod
def start(self):
pass
#start.setter
#abstractmethod
def start(self, value):
pass
# Second approach. "This method for 'end' may look slight messier, but raises errors if not implemented.
#abstractmethod
def get_end(self):
pass
#abstractmethod
def set_end(self, value):
pass
end = property(get_end, set_end)
def test_elapsed_time(self):
return self.get_end() - self.start
class FirstStep(Step):
#property
def start(self):
return self.task.start_dt
# No warnings if this is commented out.
#start.setter
def start(self, value):
self.task.start_dt = value
def get_end(self):
return self.task.end_dt
def set_end(self, value):
self.task.end_dt = value
I suspect this is a bug in the interaction of abstract methods and properties.
In your base class, the following things happen, in order:
You define an abstract method named start.
You create a new property that uses the abstract method from 1) as its getter. The name start now refers to this property, with the only reference to the original name now held by Self.start.fget.
Python saves a temporary reference to start.setter, because the name start is about to be bound to yet another object.
You create a second abstract method named start
The reference from 3) is given the abstract method from 4) to define a new property to replace the once currently bound to the name start. This property has as its getter the method from 1 and as its setter the method from 4). Now start refers to this property; start.fget refers to the method from 1); start.fset refers to the method from 4).
At this point, you have a property, whose component functions are abstract methods. The property itself was not decorated as abstract, but the definition of property.__isabstractmethod__ marks it as such because all its component methods are abstract. More importantly, you have the following entries in Step.__abstractmethods__:
start, the property
end, the property
set_end, the setter for end
gen_end, the getter for end
Note that the component functions for the start property are missing, because __abstractmethods__ stores names of, not references to, things that need to be overriden. Using property and the resulting property's setter method as decorators repeatedly replace what the name start refers to.
Now, in your child class, you define a new property named start, shadowing the inherited property, which has no setter and a concrete method as its getter. At this point, it doesn't matter if you provide a setter for this property or not, because as far as the abc machinery is concerned, you have provided everything it asked for:
A concrete method for the name start
Concrete methods for the names get_end and set_end
Implicitly a concrete definition for the name end, because all of the underlying functions for the property end have been provided concrete definitions.
#chepner answered and explained it well. Based on that, I came up with a way around it that is... well... you decide. Sneaky at best. But it achieves my 3 main goals:
Raises exceptions for unimplemented setters in subclasses
Supports the python property semantics (vs. functions etc)
Avoids boilerplate re-declaring every property in every subclass which still might not have solved #1 anyway.
Just declare the abstract get/set functions in the base class (not the property). Then add a #classmethod initializer to the base class that creates the actual properties using those abstract methods, but at that point, they'll be concrete methods on the subclass.
It's a one liner after the subclass declaration to init the properties. Nothing enforces that call being made, so it's not ironclad. Not a big savings in this example, but I'll have many properties. The end results doesn't look as dirty as I thought it would. Would like to hear comments or warnings of things I'm overlooking.
from abc import abstractmethod, ABC
class ParentTask(object):
def __init__(self):
self.first_step = FirstStep(self)
self.second_step = SecondStep(self)
print(self.first_step.end)
print(self.second_step.end)
class Step(ABC):
def __init__(self, task):
self.task = task
#classmethod
def init_properties(cls):
cls.end = property(cls.get_end, cls.set_end)
#abstractmethod
def get_end(self):
pass
#abstractmethod
def set_end(self, value):
pass
class FirstStep(Step):
def get_end(self):
return 1
def set_end(self, value):
self.task.end = value
class SecondStep(Step):
def get_end(self):
return 2
def set_end(self, value):
self.task.end = value
FirstStep.init_properties()
SecondStep.init_properties()
ParentTask()
Given these two classes:
class MyClass(ABC):
#abstractmethod
def my_method(self, my_parameter: str):
pass
class MySecondClass(MyClass):
def my_method(self, my_parameter):
pass
Does my_parameter in my_method of MySecondClass have an inferred type of Any or str based on PEP 484? I couldn't find an example detailing the above within the linked document.
It's Any. You can try it out by running mypy:
from abc import ABC, abstractmethod
class MyClass(ABC):
#abstractmethod
def my_method(self, my_parameter: str):
pass
class MySecondClass(MyClass):
def my_method(self, my_parameter):
reveal_type(my_parameter)
pass
MySecondClass().my_method(3)
With or without the --check-untyped-defs flag, this passes type checking, and reveal_type reports Any as the type. (Without the flag, type checking is also skipped for the method body.)
Note that even with annotations, it is permitted for an overriding method to have a different signature than an overridden method, as long as the overriding method's signature is more permissive. For example, a method that takes an Animal can override a method that takes a Dog.
I know virtual methods from PHP or Java.
How can they be implemented in Python?
Or have I to define an empty method in an abstract class and override it?
Sure, and you don't even have to define a method in the base class. In Python methods are better than virtual - they're completely dynamic, as the typing in Python is duck typing.
class Dog:
def say(self):
print "hau"
class Cat:
def say(self):
print "meow"
pet = Dog()
pet.say() # prints "hau"
another_pet = Cat()
another_pet.say() # prints "meow"
my_pets = [pet, another_pet]
for a_pet in my_pets:
a_pet.say()
Cat and Dog in Python don't even have to derive from a common base class to allow this behavior - you gain it for free. That said, some programmers prefer to define their class hierarchies in a more rigid way to document it better and impose some strictness of typing. This is also possible - see for example the abc standard module.
raise NotImplementedError() (dynamic type checking)
This is the recommended exception to raise on "pure virtual methods" of "abstract" base classes that don't implement a method.
https://docs.python.org/3.5/library/exceptions.html#NotImplementedError says:
This exception is derived from RuntimeError. In user defined base classes, abstract methods should raise this exception when they require derived classes to override the method.
As others said, this is mostly a documentation convention and is not required, but this way you get a more meaningful exception than a missing attribute error.
dynamic.py
class Base(object):
def virtualMethod(self):
raise NotImplementedError()
def usesVirtualMethod(self):
return self.virtualMethod() + 1
class Derived(Base):
def virtualMethod(self):
return 1
print Derived().usesVirtualMethod()
Base().usesVirtualMethod()
gives:
2
Traceback (most recent call last):
File "./dynamic.py", line 13, in <module>
Base().usesVirtualMethod()
File "./dynamic.py", line 6, in usesVirtualMethod
return self.virtualMethod() + 1
File "./dynamic.py", line 4, in virtualMethod
raise NotImplementedError()
NotImplementedError
typing.Protocol (static type checking, Python 3.8)
Python 3.8 added typing.Protocol which now allows us to also statically type check that a virtual method is implemented on a subclass.
protocol.py
from typing import Protocol
class CanFly(Protocol):
def fly(self) -> str:
pass
def fly_fast(self) -> str:
return 'CanFly.fly_fast'
class Bird(CanFly):
def fly(self):
return 'Bird.fly'
def fly_fast(self):
return 'Bird.fly_fast'
class FakeBird(CanFly):
pass
assert Bird().fly() == 'Bird.fly'
assert Bird().fly_fast() == 'Bird.fly_fast'
# mypy error
assert FakeBird().fly() is None
# mypy error
assert FakeBird().fly_fast() == 'CanFly.fly_fast'
If we run this file, the asserts pass, as we didn't add any dynamic typechecking:
python protocol.py
but if we typecheck if mypy:
python -m pip install --user mypy
mypy protocol.py
we get an error as expected:
protocol.py:22: error: Cannot instantiate abstract class "FakeBird" with abstract attribute "fly"
protocol.py:24: error: Cannot instantiate abstract class "FakeBird" with abstract attribute "fly"
It is a bit unfortunate however that the error checking only picks up the error on instantiation, and not at class definition.
typing.Protocol counts methods as abstract when their body is "empty"
I'm not sure what they count as empty, but both all of the following count as empty:
pass
... ellipsis object
raise NotImplementedError()
So the best possibility is likely:
protocol_empty.py
from typing import Protocol
class CanFly(Protocol):
def fly(self) -> None:
raise NotImplementedError()
class Bird(CanFly):
def fly(self):
return None
class FakeBird(CanFly):
pass
Bird().fly()
FakeBird().fly()
which fails as desired:
protocol_empty.py:15: error: Cannot instantiate abstract class "FakeBird" with abstract attribute "fly"
protocol_empty.py:15: note: The following method was marked implicitly abstract because it has an empty function body: "fly". If it is not meant to be abstract, explicitly return None.
but if e.g. we replace the:
raise NotImplementedError()
with some random "non-empty" statement such as:
x = 1
then mypy does not count them as virtual and gives no errors.
#abc.abstractmethod: metaclass syntax changed in Python 3
In Python 3 metaclasses are declared as:
class C(metaclass=abc.ABCMeta):
instead of the Python 2:
class C:
__metaclass__=abc.ABCMeta
so now to use #abc.abstractmethod which was previously mentioned at https://stackoverflow.com/a/19316077/895245 you need:
abc_cheat.py
class C(metaclass=abc.ABCMeta):
#abc.abstractmethod
def m(self, i):
pass
try:
c = C()
except TypeError:
pass
else:
assert False
But TODO: what is the advantage of ABCMeta over just raise NotImplementedError? It has a disadvantage that you are forced to define a metaclass, so more work, but I don't see an advantage. https://peps.python.org/pep-0544 does mention both approaches in passing.
Outro
Bibiography:
https://peps.python.org/pep-0544 the typing.Protocol PEP
Is it possible to make abstract classes?
What to use in replacement of an interface/protocol in python
Tested on Python 3.10.7, mypy 0.982, Ubuntu 21.10.
Python methods are always virtual.
Actually, in version 2.6 python provides something called abstract base classes and you can explicitly set virtual methods like this:
from abc import ABCMeta
from abc import abstractmethod
...
class C:
__metaclass__ = ABCMeta
#abstractmethod
def my_abstract_method(self, ...):
It works very well, provided the class does not inherit from classes that already use metaclasses.
source: http://docs.python.org/2/library/abc.html
Python methods are always virtual
like Ignacio said yet
Somehow class inheritance may be a better approach to implement what you want.
class Animal:
def __init__(self,name,legs):
self.name = name
self.legs = legs
def getLegs(self):
return "{0} has {1} legs".format(self.name, self.legs)
def says(self):
return "I am an unknown animal"
class Dog(Animal): # <Dog inherits from Animal here (all methods as well)
def says(self): # <Called instead of Animal says method
return "I am a dog named {0}".format(self.name)
def somethingOnlyADogCanDo(self):
return "be loyal"
formless = Animal("Animal", 0)
rover = Dog("Rover", 4) #<calls initialization method from animal
print(formless.says()) # <calls animal say method
print(rover.says()) #<calls Dog says method
print(rover.getLegs()) #<calls getLegs method from animal class
Results should be:
I am an unknown animal
I am a dog named Rover
Rover has 4 legs
Something like a virtual method in C++ (calling method implementation of a derived class through a reference or pointer to the base class) doesn't make sense in Python, as Python doesn't have typing. (I don't know how virtual methods work in Java and PHP though.)
But if by "virtual" you mean calling the bottom-most implementation in the inheritance hierarchy, then that's what you always get in Python, as several answers point out.
Well, almost always...
As dplamp pointed out, not all methods in Python behave like that. Dunder method don't. And I think that's a not so well known feature.
Consider this artificial example
class A:
def prop_a(self):
return 1
def prop_b(self):
return 10 * self.prop_a()
class B(A):
def prop_a(self):
return 2
Now
>>> B().prop_b()
20
>>> A().prob_b()
10
However, consider this one
class A:
def __prop_a(self):
return 1
def prop_b(self):
return 10 * self.__prop_a()
class B(A):
def __prop_a(self):
return 2
Now
>>> B().prop_b()
10
>>> A().prob_b()
10
The only thing we've changes was making prop_a() a dunder method.
A problem with the first behavior can be that you can't change the behavior of prop_a() in the derived class without impacting the behavior of prop_b(). This very nice talk by Raymond Hettinger gives an example for a use case where this is inconvenient.
Python 3.6 introduced __init_subclass__ and this let you simply do this:
class A:
def method(self):
'''method needs to be overwritten'''
return NotImplemented
def __init_subclass__(cls):
if cls.method is A.method:
raise NotImplementedError(
'Subclass has not overwritten method {method}!')
The benefit of this solution is that you avoid the abc metaclass and give the user a direct imperative how to do it right. In addition to another answer here that raises NotImplementedError when calling the method. This solution is checked on runtime and not only IF the user calls the method.
This is a two-part question, but the second part is dependent on the first part.
For educational purposes, I am trying to implement an abstract base class and test suite for groups (the concept from abstract algebra). Part of the definition of an algebraic group is equivalent to a type constraint, and I want to implement that type constraint on an ABC, and have something complain if the methods on the concrete classes don't conform to that constraint.
I've got a first-pass implementation for this for the group of Boolean values under logical and, but there are at least two things wrong with it, and I'm hoping you can help me fix it.
from __future__ import annotations
from abc import ABC, abstractmethod
class AbsGroup(ABC):
#abstractmethod
def op(self, other: AbsGroup) -> AbsGroup: # <-- Line-of-interest #1
pass
class Bool(AbsGroup):
def __init__(self, val="False"):
if val not in ["True", "False"]:
raise ValueError("Invalid Bool value %s" % val)
self.val = val
def op(self, other):
"""Logical AND"""
if self.val == "True" and other.val == "True": # <-- Line-of-interest #2
return Bool("True")
return Bool("False")
def __eq__(self, other):
return self.val == other.val
def __repr__(self):
return self.val
Firstly: Line-of-interest #1 is what's doing the type-constraint work, but the current implementation is wrong. It only checks that the method receives and returns an AbsGroup instance. This could be any AbsGroup instance. I want it to check that for the concrete class it gets inherited by, it receives and returns an instance of that concrete class (so in the case of Bool it receives and returns an instance of Bool). The point of the exercise is to do this in one location, rather than having to set it specifically on each concrete class. I presume this is done with some type-hinting generics that are a little bit deeper than I've yet to delve with regard to type-hinting. How do I do this?
Secondly: how do I check the concrete method is complying with the abstract type hint? The type inspector in my IDE (PyCharm) complains at Line-of-interest #2, because it's expecting other to be of type AbsGroup, which doesn't have a val attribute. This is expected, and would go away if I could figure out the solution to the first problem, but my IDE is the only thing I can find that notices this discrepancy. mypy is silent on the matter by default, as are flake8 and pylint. It's great that PyCharm is on the ball, but if I wanted to incorporate this into a workflow, what command would I have to run that would fail in the event of my concrete method not complying with the abstract signature?
First tip: If mypy doesnt tell you enough, try mypy --strict.
You correctly realized that the type annotation for op in the base class is not restrictive enough and in fact would be incompatible with the child class.
Take a look at this not-working example.
from __future__ import annotations
from abc import ABC, abstractmethod
class AbsGroup(ABC):
#abstractmethod
def op(self, other: AbsGroup) -> AbsGroup:
pass
class Bool(AbsGroup):
def __init__(self, val: str = "False") -> None:
self.val = val
def op(self, other: Bool) -> Bool:
...
I annotated op in Bool with the correct type but now mypy complains:
file.py:15: error: Argument 1 of "op" is incompatible with supertype
"AbsGroup "; supertype defines the argument type as "AbsGroup"
You have two options: Either make the base annotation even less restrictive (Any) or make your class a Generic one:
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import TypeVar, Generic
T = TypeVar('T')
class AbsGroup(Generic[T], ABC):
#abstractmethod
def op(self, other: T) -> T:
pass
# EDIT: ADDED QUOTES AROUND Bool
class Bool(AbsGroup['Bool']):
def __init__(self, val: str = "False") -> None:
self.val = val
def op(self, other: Bool) -> Bool:
...
This involves several steps:
create a type variable T (looks similar to generic type variables in other languages)
let the base class also inherit from Generic[T] making it a generic class
change the op method to take and return a T
let the child class inherit from AbsGroup[Bool] (in C++ this is known as CRTP)
This silences mypy --strict and PyCharm correctly infers the return type of op.
Edit:
The previous child class definition looked like this class Bool(AbsGroup[Bool]): ... without quotes. But this does not work and will throw a NameError when creating the class:
NameError: name 'Bool' is not defined
This is expected behaviour as written in PEP 563.
[...] However, there are APIs in the typing module that use other syntactic constructs of the language, and those will still require working around forward references with string literals. The list includes:
[...]
base classes:
class C(Tuple['<type>', '<type>']): ...
So the quotes are still required in this case even though we used the future import.
Just a note: why are you using string symbols for the boolean values? There are already two perfectly working instances called True and False. This will make your code much simpler. E.g. the check in the constructor can be simplified to if type(val) is bool (I would not use isinstance here since you dont want val to be a custom type, probably?).