Is it a good practice to have non-abstract methods on abstract base classes? I mean, methods that can, but don't have to, be present on subclasses of particular ABC?
Technically it is possible, as seen in the below example (ConcreteDataProvider does not implement disconnect; it only has to implement connect):
from abc import ABC, abstractmethod
class AbstractDataProvider(ABC):
#abstractmethod
def connect(self):
pass
def disconnect(self):
pass
class ConcreteDataProvider(AbstractDataProvider):
def connect(self):
pass
data_provider = ConcreteDataProvider()
It is fine to have ABC with concrete methods to provide default implementations. The standard library has several such cases:
The example of using abc.ABCMeta provides a non-abstract method.
The get_iterator() method is also part of the MyIterable abstract base class, but it does not have to be overridden in non-abstract derived classes.
The ABCs of the collections.abc module provide default methods (termed "Mixin Methods").
ABC
Inherits from
Abstract Methods
Mixin Methods
...
...
...
...
Iterator
Iterable
__next__
__iter__
Generator
Iterator
send, throw
close, __iter__, __next__
...
...
...
...
When "do nothing" is a sensible default, there is no problem with a concrete method having a default implementation that is just pass. This can especially be useful when you expect that many implementations will need this method: by providing a default, it is safe for client code to always call this method.
Note: When the pattern is specifically connect/disconnect, open/close, or similar pairs of methods to be called before/after usage, the __enter__/__exit__ pair of a context manager is the appropriate interface.
I want to define a typing protocol for a custom mapping class. This class needs to be very similar to a MutableMapping, except that it has a couple of additional methods beyond those those that collections.abc.MutableMapping defines as abstract methods (specifically .copy()), and I also want to specify in advance which types any implementations of the custom mapping class must use as keys and values.
After reading through PEP 544 I thought I should be able to do this:
from typing import Hashable, MutableMapping, Protocol, TypeVar
TVarMapProt = TypeVar("TVarMapProt", bound="VariableMappingProtocol")
class VariableMappingProtocol(MutableMapping[Hashable, int], Protocol):
"""
Protocol for the type behaviour of a class which
acts basically like a dict of int objects.
"""
def copy(self: TVarMapProt) -> TVarMapProt:
# TODO replace return type with Self if PEP 673 goes through
...
The idea being that in my code I can state that a type VariableMappingProtocol is expected, and then the user would have to use their own class that was defined like this in order to avoid a typing error:
TCusVarMap = TypeVar("CusVarMap", bound="CustomVariableMapping")
class CustomVariableMapping
"""Defines all the methods that a MutableMapping does, but mapping Hashables to ints"""
def __getitem__(self, key: Hashable) -> int:
# implementation goes here
...
# etc. for __setitem__, and so on...
def copy(self) -> TCusVarMap:
# implementation goes here
...
The problem is that when I run mypy on the code defining VariableMappingProtocol I get this error:
test.py:7: error: All bases of a protocol must be protocols
Found 1 error in 1 file (checked 1 source file)
If I delete MutableMapping so that VariableMappingProtocol only inherits from Protocol then the error goes away. But then I'm not getting the structural typing I want for all the abstract methods from MutableMapping.
So it seems the problem is that typing.MutableMapping is not considered a Protocol? But that's weird, especially as I can treat some of the other types in typing as Protocols, e.g. this example (from PEP 544)
from typing import Protocol, Sized
class SizedProtocol(Sized, Protocol):
"""
Protocol for the type behaviour of a class which has a __len__ method.
"""
...
which doesn't throw any mypy errors.
How can I inherit from MutableMapping as a Protocol, and thus avoid the need to write out all the methods of MutableMapping in my VariableMappingProtocol definition?
Mypy allows us to write class stubs which can be placed in the same directory as the actual class. This stub is very similar to an interface as known from other languages. Is it possible to have a client use the stub and the implementation strictly follow the stub?
Example I would like to work:
class IDependency:
def do_something(self) -> None: ...
def do_something_else(self) -> None: ...
class Service:
def __init__(self, dependency: IDependency):
dependency.do_something()
dependency.do_something_else() # this fails silently
class DependencyImplementation(IDependency):
def do_something(self) -> None:
print("doing something")
# Note there is no `do_something_else` here.
This works. However, if DependencyImplementation doesn't implement the do_something method, there is no error from Mypy and no error from Python itself. The call just doesn't do anything. Do I have to write raise NotImplementedException() or annotate each method with #abc.abstractmethod for this to work? Is there some special flags in Mypy or the Python interpreter?
Is this a use case for Mypy Protocols? It seems to be coming soon (maybe Python 4?)
This is indeed something you can do using either #abc.abstractmethod or protocols. The former is akin to using Java's abstract classes; the latter will be kin to using Go's interfaces or Rust traits.
Here is an example that uses ABCs:
from abc import abstractmethod
class Parent:
#abstractmethod
def foo(self) -> None: ...
# Missing an implementation for 'foo'!
class Child(Parent): pass
print(Child()) # Error: Cannot instantiate abstract class 'Child' with abstract attribute 'foo'
A few things to note about this example:
You get an error on the instantiation of the Child class, not on the declaration. This is to support the use case where you never instantiate Child, but instead subclass it again and define foo in that second subclass.
We don't need to add the usual abc metaclass to Parent ( e.g. class Parent(metaclass=ABCMeta)): mypy will understand what #abc.abstractmethod means with or without it. Include the metaclass only if you want the Python interpreter to also enforce that you've correctly overridden anything marked as being abstract at runtime.
ABCs are not quite interfaces -- you can still define fields and non-abstract methods. They're more akin to Java-style abstract classes.
You can also use protocols, though for now you'll need to first pip install typing_extensions to use it. Here's an example:
from typing_extensions import Protocol
class CanFoo(Protocol):
def foo(self) -> None: ...
class Child: pass
def expects_fooable(x: CanFoo) -> None: ...
x = Child()
expects_fooable(x) # Error: Argument 1 to "expects_fooable" has incompatible type "Child"; expected "CanFoo"
A few notes:
Here, Child deliberately does not inherit from CanFoo: there is no explicit link between a class and the protocol(s) it implements: protocols are very similar to Go-style interfaces and can be more ad-hoc. Contrast this to languages like Java, where you do need to include a "implements Blah" in the class definition.
Unlike the previous error, we do not get an error on the instantiation of Child: there's nothing inherently wrong with it. Rather, we get an exception when we try using it improperly.
A few final notes:
Stub files may superficially look like interfaces, but they're really not: they're more just a way to bring types to code which we cannot easily modify and add type hints to. You can think of them being vaguely similar to C-style header files: it's a way of storing the signatures of existing objects independently from the source code.
"Typeshed" is the name of a specific project which includes stubs for the standard library and a few popular 3rd party modules. That word is not a synonym for "stub files". Similarly, the term "class stub" is also a bit of a misnomer: there are only stub files, which may or may not contain definitions for classes. (If the original Python or C extension library you're trying to type contains only functions, the corresponding stub file would also likely only contain signatures for those functions.)
I'm using the abc module to define an interface that subclasses must support. There're also some common methods that are present in all subclasses. Is it ok to put them in the abstract class or should that only contain abstract methods (i.e. decorated with #abc.abstractmethod) ?
TL; DR; Yes, it is OK for an abstract class to have non-abstract methods.
Typically what we call an abstract class is just a class that cannot be instantiated.
On the other hand what we call an interface is a class which has only method declarations but no implementations. In particular its an abstract class because it doesn't have a constructor.
Of course in Python there are no real interfaces: every method has to have a body. But we can somewhat emulate interfaces via raise NotImplementedError().
Anyway interfaces form a subset of abstract classes. This obviously suggests that there are abstract classes that are not interfaces. This is exactly the case you are describing. Yes, abstract class may contain implemented, non-abstract methods. And it is not a bad practice. This is especially useful when a given method does not depend on concrete implementation.
For example consider an interface for a generic parser (I'm thinking about json.load and json.loads):
class ILoader(ABC):
#abstractmethod
def load(self, stream):
raise NotImplementedError()
It's completely OK to give loads method which accepts a string instead of stream with a default implementation:
class AbstractLoader(ABC):
#abstractmethod
def load(self, stream):
raise NotImplementedError()
def loads(self, text):
stream = io.StringIO(text)
return self.load(stream)
although I would use Abstract prefix instead of I. ;)
What is the difference between abstract class and interface in Python?
What you'll see sometimes is the following:
class Abstract1:
"""Some description that tells you it's abstract,
often listing the methods you're expected to supply."""
def aMethod(self):
raise NotImplementedError("Should have implemented this")
Because Python doesn't have (and doesn't need) a formal Interface contract, the Java-style distinction between abstraction and interface doesn't exist. If someone goes through the effort to define a formal interface, it will also be an abstract class. The only differences would be in the stated intent in the docstring.
And the difference between abstract and interface is a hairsplitting thing when you have duck typing.
Java uses interfaces because it doesn't have multiple inheritance.
Because Python has multiple inheritance, you may also see something like this
class SomeAbstraction:
pass # lots of stuff - but missing something
class Mixin1:
def something(self):
pass # one implementation
class Mixin2:
def something(self):
pass # another
class Concrete1(SomeAbstraction, Mixin1):
pass
class Concrete2(SomeAbstraction, Mixin2):
pass
This uses a kind of abstract superclass with mixins to create concrete subclasses that are disjoint.
What is the difference between abstract class and interface in Python?
An interface, for an object, is a set of methods and attributes on that object.
In Python, we can use an abstract base class to define and enforce an interface.
Using an Abstract Base Class
For example, say we want to use one of the abstract base classes from the collections module:
import collections
class MySet(collections.Set):
pass
If we try to use it, we get an TypeError because the class we created does not support the expected behavior of sets:
>>> MySet()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MySet with abstract methods
__contains__, __iter__, __len__
So we are required to implement at least __contains__, __iter__, and __len__. Let's use this implementation example from the documentation:
class ListBasedSet(collections.Set):
"""Alternate set implementation favoring space over speed
and not requiring the set elements to be hashable.
"""
def __init__(self, iterable):
self.elements = lst = []
for value in iterable:
if value not in lst:
lst.append(value)
def __iter__(self):
return iter(self.elements)
def __contains__(self, value):
return value in self.elements
def __len__(self):
return len(self.elements)
s1 = ListBasedSet('abcdef')
s2 = ListBasedSet('defghi')
overlap = s1 & s2
Implementation: Creating an Abstract Base Class
We can create our own Abstract Base Class by setting the metaclass to abc.ABCMeta and using the abc.abstractmethod decorator on relevant methods. The metaclass will be add the decorated functions to the __abstractmethods__ attribute, preventing instantiation until those are defined.
import abc
For example, "effable" is defined as something that can be expressed in words. Say we wanted to define an abstract base class that is effable, in Python 2:
class Effable(object):
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def __str__(self):
raise NotImplementedError('users must define __str__ to use this base class')
Or in Python 3, with the slight change in metaclass declaration:
class Effable(object, metaclass=abc.ABCMeta):
#abc.abstractmethod
def __str__(self):
raise NotImplementedError('users must define __str__ to use this base class')
Now if we try to create an effable object without implementing the interface:
class MyEffable(Effable):
pass
and attempt to instantiate it:
>>> MyEffable()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyEffable with abstract methods __str__
We are told that we haven't finished the job.
Now if we comply by providing the expected interface:
class MyEffable(Effable):
def __str__(self):
return 'expressable!'
we are then able to use the concrete version of the class derived from the abstract one:
>>> me = MyEffable()
>>> print(me)
expressable!
There are other things we could do with this, like register virtual subclasses that already implement these interfaces, but I think that is beyond the scope of this question. The other methods demonstrated here would have to adapt this method using the abc module to do so, however.
Conclusion
We have demonstrated that the creation of an Abstract Base Class defines interfaces for custom objects in Python.
Python >= 2.6 has Abstract Base Classes.
Abstract Base Classes (abbreviated
ABCs) complement duck-typing by
providing a way to define interfaces
when other techniques like hasattr()
would be clumsy. Python comes with
many builtin ABCs for data structures
(in the collections module), numbers
(in the numbers module), and streams
(in the io module). You can create
your own ABC with the abc module.
There is also the Zope Interface module, which is used by projects outside of zope, like twisted. I'm not really familiar with it, but there's a wiki page here that might help.
In general, you don't need the concept of abstract classes, or interfaces in python (edited - see S.Lott's answer for details).
In a more basic way to explain:
An interface is sort of like an empty muffin pan.
It's a class file with a set of method definitions that have no code.
An abstract class is the same thing, but not all functions need to be empty. Some can have code. It's not strictly empty.
Why differentiate:
There's not much practical difference in Python, but on the planning level for a large project, it could be more common to talk about interfaces, since there's no code. Especially if you're working with Java programmers who are accustomed to the term.
Python doesn't really have either concept.
It uses duck typing, which removed the need for interfaces (at least for the computer :-))
Python <= 2.5:
Base classes obviously exist, but there is no explicit way to mark a method as 'pure virtual', so the class isn't really abstract.
Python >= 2.6:
Abstract base classes do exist (http://docs.python.org/library/abc.html). And allow you to specify methods that must be implemented in subclasses. I don't much like the syntax, but the feature is there. Most of the time it's probably better to use duck typing from the 'using' client side.
In general, interfaces are used only in languages that use the single-inheritance class model. In these single-inheritance languages, interfaces are typically used if any class could use a particular method or set of methods. Also in these single-inheritance languages, abstract classes are used to either have defined class variables in addition to none or more methods, or to exploit the single-inheritance model to limit the range of classes that could use a set of methods.
Languages that support the multiple-inheritance model tend to use only classes or abstract base classes and not interfaces. Since Python supports multiple inheritance, it does not use interfaces and you would want to use base classes or abstract base classes.
http://docs.python.org/library/abc.html
Abstract classes are classes that contain one or more abstract methods. Along with abstract methods, Abstract classes can have static, class and instance methods.
But in case of interface, it will only have abstract methods not other. Hence it is not compulsory to inherit abstract class but it is compulsory to inherit interface.
For completeness, we should mention PEP3119
where ABC was introduced and compared with interfaces,
and original Talin's comment.
The abstract class is not perfect interface:
belongs to the inheritance hierarchy
is mutable
But if you consider writing it your own way:
def some_function(self):
raise NotImplementedError()
interface = type(
'your_interface', (object,),
{'extra_func': some_function,
'__slots__': ['extra_func', ...]
...
'__instancecheck__': your_instance_checker,
'__subclasscheck__': your_subclass_checker
...
}
)
ok, rather as a class
or as a metaclass
and fighting with python to achieve the immutable object
and doing refactoring
...
you'll quite fast realize that you're inventing the wheel
to eventually achieve
abc.ABCMeta
abc.ABCMeta was proposed as a useful addition of the missing interface functionality,
and that's fair enough in a language like python.
Certainly, it was able to be enhanced better whilst writing version 3, and adding new syntax and immutable interface concept ...
Conclusion:
The abc.ABCMeta IS "pythonic" interface in python