I found multiple (slightly different) ways to define abstract classes in Python. I read the documentation and also could not find an answer here on stackoverflow.
The main difference between the three examples (see code below) is:
A sets a new metaclass abc.ABCMeta explicitly
B inherits from abc.ABC
C inherits from objects but defines #abc.abstractmethod classes
It seems that A and B are not different (i.e. also B has the new metaclass abc.ABCMeta). However, class C remains of type type.
What are the impacts of not defining a metaclass for C? When is it necessary to define the metaclass or is it wrong/bad style to not define the abc.ABCMeta metaclass for an abstract class? Nonetheless, the class C seems to behave as I expect from an ABC.
import abc
class A(metaclass=abc.ABCMeta):
# Alternatively put __metaclass__ = abc.ABCMeta here
#abc.abstractmethod
def foo(self):
raise NotImplementedError
class B(abc.ABC):
#abc.abstractmethod
def foo(self):
raise NotImplementedError
class C(object):
#abc.abstractmethod
def foo(self):
raise NotImplementedError
class Aimpl(A):
def foo(self):
print("Aimpl")
class Bimpl(B):
def foo(self):
print("Bimpl")
class Cimpl(C):
#def foo(self):
# print("Cimpl")
pass
Aimpl().foo() # Aimpl
print(isinstance(Aimpl, A)) # False
print(issubclass(Aimpl, A)) # True
print(isinstance(Aimpl, abc.ABCMeta)) # True
print(type(A)) # <class 'abc.ABCMeta'>
print("---")
Bimpl().foo() # Bimpl
print(isinstance(Bimpl, B)) # False
print(issubclass(Bimpl, B)) # True
print(isinstance(Bimpl, abc.ABCMeta)) # True
print(type(B)) # <class 'abc.ABCMeta'>
print("---")
Cimpl().foo() # Cimpl
print(isinstance(Cimpl, C)) # False
print(issubclass(Cimpl, C)) # True
print(isinstance(Cimpl, abc.ABCMeta)) # False
print(type(C)) # <class 'type'>
print("---")
The abc.ABCMeta class is necessary to actually enforce the abstractmethod behaviour. Its itention is to disallow instantiation of any classes which do not implement the abstract method. The decorator itself cannot enforce that, the metaclass is enforcing the decorator upon instantiation:
class Foo:
#abstractmethod
def bar(self):
pass
Foo() # works
However:
class Foo(metaclass=ABCMeta):
#abstractmethod
def bar(self):
pass
Foo()
Traceback (most recent call last):
File "<input>", line 1, in <module>
TypeError: Can't instantiate abstract class Foo with abstract methods bar
So, without the metaclass, the abstractmethod decorator doesn't do anything.
abc.ABC is merely a shorthand so you can do Foo(ABC) instead of Foo(metaclass=ABCMeta), that is all:
A helper class that has ABCMeta as its metaclass. With this class,
an abstract base class can be created by simply deriving from ABC
avoiding sometimes confusing metaclass usage [..]
https://docs.python.org/3/library/abc.html#abc.ABC
Related
By definition, we cannot instantiate an abstract class:
>>> import abc
>>> class A(abc.ABC):
... #abc.abstractmethod
... def f(self): raise NotImplementedError
...
>>> A()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class A with abstract method f
So isn’t it a contradiction that an instance of a concrete subclass is an instance of the abstract class?
>>> class B(A):
... def f(self): return 'foo'
...
>>> isinstance(B(), A)
True
There's a difference between an object being an instance of a class and the act of instantiating a class. Inheritance means that if B is a subclass of A, then isinstance(B(), A) is true, even though B, not A, is the class being instantiated.
If you could never concretize an abstract class, there would be no point in defining the abstract class in the first place. The purpose of an abstract class is to provide an incomplete template for other classes; you can't simply use the abstract class as-is without doing making some additional definitions.
Put another way, given b = B(), b is an instance of A and B, but only B is the type of b. Is-type-of and is-instance-of are two different relations.
I'm trying to define an inner class in a hierarchy of classes and I can't figure out the right way to make sure that the inner class correctly subclasses the parents' corresponding inner classes without throwing an exception in the case where one or more of the immediate parent classes has not themselves subclassed that inner class.
I've also tried many times to write this question in a more approachable way, so I apologise if it's a bit of a headscratcher!
Hopefully this example will clarify things:
(assume for this question that we don't know which, if any, of B or C define subclasses of A.Inner - obviously in this example neither do, but that's not the point.)
Cheers.
class A:
class Inner:
...
class B(A):
...
class C(A):
...
class D(B, C):
class Inner(B.Inner, C.Inner):
...
>>>
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-163-6f592c573c6f> in <module>
6 class C(A):
7 ...
----> 8 class D(B, C):
9 class Inner(B.Inner, C.Inner):
10 ...
<ipython-input-163-6f592c573c6f> in D()
7 ...
8 class D(B, C):
----> 9 class Inner(B.Inner, C.Inner):
10 ...
TypeError: duplicate base class Inner
You can use the fact that if A.inner is a class, and B or C do not explicitly subclass the inner class, then B.inner and C.inner are the same object -> A.inner:
>>> class A:
... class inner: pass
...
>>> class B(A): pass
...
>>> class C(A): pass
...
>>> B.inner is C.inner
True
>>> C.inner is A.inner
True
We're leveraging a dictionary to ensure uniqueness and order (use collections.OrderedDict or some other ordered_set implementation if you are on a version that does not yet guarantee dict order). We can determine which classes we need to subclass on D.inner like so:
inner_classes = {getattr(klass, 'inner'): True for klass in [C, B] # control your MRO here.
if getattr(klass, 'inner') is not A.inner}
# Ensure that we also extend A.inner, in case neither B nor C subclasses A.inner
inner_classes[A.inner] = True
class D(A):
class inner(*inner_classes.keys()): pass
In this way we get consistent MRO, it doesn't matter which class (if any) subclasses A.inner, and D.inner works.
Alright folks, going on from g.d.d.c's partial answer, I think we have a solution here that works cleanly and simply for arbitrary class hierarchies.
I'll do some testing on it for a couple of days and see if it does the job. I'm still not happy with the requirement to specify the bases manually - I feel like some sort of introspection inside the wrapper could handle this automatically. Please do chime in with more suggestions.
Thanks heaps again to g.d.d.c for kicking me in the right direction!
def inner_class(*bases):
def _inner_class(cls):
bs = []
for b in bases:
try: bs.append(getattr(b, cls.__name__))
except AttributeError: pass
bs = sorted(set(bs), key = bs.index)
return type(cls)(
cls.__name__,
(cls, *bs),
{}
)
return _inner_class
class A:
class Inner:
...
class B(A):
class Inner(A.Inner):
...
class C(A):
...
class D(A):
...
class E(B, C, D):
#inner_class(B, C, D)
class Inner:
...
print(E.Inner.mro())
>>> [<class '__main__.Inner'>, <class '__main__.E.Inner'>, <class '__main__.B.Inner'>, <class '__main__.A.Inner'>, <class 'object'>]
This is the setup I want:
A should be an abstract base class with a static & abstract method f(). B should inherit from A. Requirements:
1. You should not be able to instantiate A
2. You should not be able to instantiate B, unless it implements a static f()
Taking inspiration from this question, I've tried a couple of approaches. With these definitions:
class abstractstatic(staticmethod):
__slots__ = ()
def __init__(self, function):
super(abstractstatic, self).__init__(function)
function.__isabstractmethod__ = True
__isabstractmethod__ = True
class A:
__metaclass__ = abc.ABCMeta
#abstractstatic
def f():
pass
class B(A):
def f(self):
print 'f'
class A2:
__metaclass__ = abc.ABCMeta
#staticmethod
#abc.abstractmethod
def f():
pass
class B2(A2):
def f(self):
print 'f'
Here A2 and B2 are defined using usual Python conventions and A & B are defined using the way suggested in this answer. Following are some operations I tried and the results that were undesired.
With classes A/B:
>>> B().f()
f
#This should have thrown, since B doesn't implement a static f()
With classes A2/B2:
>>> A2()
<__main__.A2 object at 0x105beea90>
#This should have thrown since A2 should be an uninstantiable abstract class
>>> B2().f()
f
#This should have thrown, since B2 doesn't implement a static f()
Since neither of these approaches give me the output I want, how do I achieve what I want?
You can't do what you want with just ABCMeta. ABC enforcement doesn't do any type checking, only the presence of an attribute with the correct name is enforced.
Take for example:
>>> from abc import ABCMeta, abstractmethod, abstractproperty
>>> class Abstract(object):
... __metaclass__ = ABCMeta
... #abstractmethod
... def foo(self): pass
... #abstractproperty
... def bar(self): pass
...
>>> class Concrete(Abstract):
... foo = 'bar'
... bar = 'baz'
...
>>> Concrete()
<__main__.Concrete object at 0x104b4df90>
I was able to construct Concrete() even though both foo and bar are simple attributes.
The ABCMeta metaclass only tracks how many objects are left with the __isabstractmethod__ attribute being true; when creating a class from the metaclass (ABCMeta.__new__ is called) the cls.__abstractmethods__ attribute is then set to a frozenset object with all the names that are still abstract.
type.__new__ then tests for that frozenset and throws a TypeError if you try to create an instance.
You'd have to produce your own __new__ method here; subclass ABCMeta and add type checking in a new __new__ method. That method should look for __abstractmethods__ sets on the base classes, find the corresponding objects with the __isabstractmethod__ attribute in the MRO, then does typechecking on the current class attributes.
This'd mean that you'd throw the exception when defining the class, not an instance, however. For that to work you'd add a __call__ method to your ABCMeta subclass and have that throw the exception based on information gathered by your own __new__ method about what types were wrong; a similar two-stage process as what ABCMeta and type.__new__ do at the moment. Alternatively, update the __abstractmethods__ set on the class to add any names that were implemented but with the wrong type and leave it to type.__new__ to throw the exception.
The following implementation takes that last tack; add names back to __abstractmethods__ if the implemented type doesn't match (using a mapping):
from types import FunctionType
class ABCMetaTypeCheck(ABCMeta):
_typemap = { # map abstract type to expected implementation type
abstractproperty: property,
abstractstatic: staticmethod,
# abstractmethods return function objects
FunctionType: FunctionType,
}
def __new__(mcls, name, bases, namespace):
cls = super(ABCMetaTypeCheck, mcls).__new__(mcls, name, bases, namespace)
wrong_type = set()
seen = set()
abstractmethods = cls.__abstractmethods__
for base in bases:
for name in getattr(base, "__abstractmethods__", set()):
if name in seen or name in abstractmethods:
continue # still abstract or later overridden
value = base.__dict__.get(name) # bypass descriptors
if getattr(value, "__isabstractmethod__", False):
seen.add(name)
expected = mcls._typemap[type(value)]
if not isinstance(namespace[name], expected):
wrong_type.add(name)
if wrong_type:
cls.__abstractmethods__ = abstractmethods | frozenset(wrong_type)
return cls
With this metaclass you get your expected output:
>>> class Abstract(object):
... __metaclass__ = ABCMetaTypeCheck
... #abstractmethod
... def foo(self): pass
... #abstractproperty
... def bar(self): pass
... #abstractstatic
... def baz(): pass
...
>>> class ConcreteWrong(Abstract):
... foo = 'bar'
... bar = 'baz'
... baz = 'spam'
...
>>> ConcreteWrong()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class ConcreteWrong with abstract methods bar, baz, foo
>>>
>>> class ConcreteCorrect(Abstract):
... def foo(self): return 'bar'
... #property
... def bar(self): return 'baz'
... #staticmethod
... def baz(): return 'spam'
...
>>> ConcreteCorrect()
<__main__.ConcreteCorrect object at 0x104ce1d10>
I am looking for ways / best practices on testing methods defined in an abstract base class. One thing I can think of directly is performing the test on all concrete subclasses of the base class, but that seems excessive at some times.
Consider this example:
import abc
class Abstract(object):
__metaclass__ = abc.ABCMeta
#abc.abstractproperty
def id(self):
return
#abc.abstractmethod
def foo(self):
print "foo"
def bar(self):
print "bar"
Is it possible to test bar without doing any subclassing?
In newer versions of Python you can use unittest.mock.patch()
class MyAbcClassTest(unittest.TestCase):
#patch.multiple(MyAbcClass, __abstractmethods__=set())
def test(self):
self.instance = MyAbcClass() # Ha!
Here is what I have found: If you set __abstractmethods__ attribute to be an empty set you'll be able to instantiate abstract class. This behaviour is specified in PEP 3119:
If the resulting __abstractmethods__ set is non-empty, the class is considered abstract, and attempts to instantiate it will raise TypeError.
So you just need to clear this attribute for the duration of tests.
>>> import abc
>>> class A(metaclass = abc.ABCMeta):
... #abc.abstractmethod
... def foo(self): pass
You cant instantiate A:
>>> A()
Traceback (most recent call last):
TypeError: Can't instantiate abstract class A with abstract methods foo
If you override __abstractmethods__ you can:
>>> A.__abstractmethods__=set()
>>> A() #doctest: +ELLIPSIS
<....A object at 0x...>
It works both ways:
>>> class B(object): pass
>>> B() #doctest: +ELLIPSIS
<....B object at 0x...>
>>> B.__abstractmethods__={"foo"}
>>> B()
Traceback (most recent call last):
TypeError: Can't instantiate abstract class B with abstract methods foo
You can also use unittest.mock (from 3.3) to override temporarily ABC behaviour.
>>> class A(metaclass = abc.ABCMeta):
... #abc.abstractmethod
... def foo(self): pass
>>> from unittest.mock import patch
>>> p = patch.multiple(A, __abstractmethods__=set())
>>> p.start()
{}
>>> A() #doctest: +ELLIPSIS
<....A object at 0x...>
>>> p.stop()
>>> A()
Traceback (most recent call last):
TypeError: Can't instantiate abstract class A with abstract methods foo
As properly put by lunaryon, it is not possible. The very purpose of ABCs containing abstract methods is that they are not instantiatable as declared.
However, it is possible to create a utility function that introspects an ABC, and creates a dummy, non abstract class on the fly. This function could be called directly inside your test method/function and spare you of having to wite boiler plate code on the test file just for testing a few methods.
def concreter(abclass):
"""
>>> import abc
>>> class Abstract(metaclass=abc.ABCMeta):
... #abc.abstractmethod
... def bar(self):
... return None
>>> c = concreter(Abstract)
>>> c.__name__
'dummy_concrete_Abstract'
>>> c().bar() # doctest: +ELLIPSIS
(<abc_utils.Abstract object at 0x...>, (), {})
"""
if not "__abstractmethods__" in abclass.__dict__:
return abclass
new_dict = abclass.__dict__.copy()
for abstractmethod in abclass.__abstractmethods__:
#replace each abc method or property with an identity function:
new_dict[abstractmethod] = lambda x, *args, **kw: (x, args, kw)
#creates a new class, with the overriden ABCs:
return type("dummy_concrete_%s" % abclass.__name__, (abclass,), new_dict)
You can use multiple inheritance practice to have access to the implemented methods of the abstract class. Obviously following such design decision depends on the structure of the abstract class since you need to implement abstract methods (at least bring the signature) in your test case.
Here is the example for your case:
class Abstract(object):
__metaclass__ = abc.ABCMeta
#abc.abstractproperty
def id(self):
return
#abc.abstractmethod
def foo(self):
print("foo")
def bar(self):
print("bar")
class AbstractTest(unittest.TestCase, Abstract):
def foo(self):
pass
def test_bar(self):
self.bar()
self.assertTrue(1==1)
No, it's not. The very purpose of abc is to create classes that cannot be instantiated unless all abstract attributes are overridden with concrete implementations. Hence you need to derive from the abstract base class and override all abstract methods and properties.
Perhaps a more compact version of the concreter proposed by #jsbueno could be:
def concreter(abclass):
class concreteCls(abclass):
pass
concreteCls.__abstractmethods__ = frozenset()
return type('DummyConcrete' + abclass.__name__, (concreteCls,), {})
The resulting class still has all original abstract methods (which can be now called, even if this is not likely to be useful...) and can be mocked as needed.
Abstract base classes can still be handy in Python. In writing an abstract base class where I want every subclass to have, say, a spam() method, I want to write something like this:
class Abstract(object):
def spam(self):
raise NotImplementedError
The challenge comes in also wanting to use super(), and to do it properly by including it in the entire chain of subclasses. In this case, it seems I have to wrap every super call like the following:
class Useful(Abstract):
def spam(self):
try:
super(Useful, self).spam()
except NotImplementedError, e:
pass
print("It's okay.")
That's okay for a simple subclass, but when writing a class that has many methods, the try-except thing gets a bit cumbersome, and a bit ugly. Is there a more elegant way of subclassing from abstract base classes? Am I just Doing It Wrong?
You can do this cleanly in python 2.6+ with the abc module:
import abc
class B(object):
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def foo(self):
print 'In B'
class C(B):
def foo(self):
super(C, self).foo()
print 'In C'
C().foo()
The output will be
In B
In C
Do not write all that code. Simple inspection of the abstract class can save you writing all that code.
If the method is abstract, the concrete subclass does not call super.
If the method is concrete, the concrete subclass does call super.
The key point to understand this is super() is for implementing cooperative inheritance. How the classes cooperate is up to you the programmer. super() is not magic and does not know exactly what you want! There is not much point in using super for a flat hierarchy that doesn't need cooperative inheritance, so in that case S. Lott's suggestion is spot on. Subclasses of Useful may or may not want to use super() depending their goals :)
For example: Abstract is A. A <- B, but then you want to support insertion of C like so A <- C <- B.
class A(object):
"""I am an abstract abstraction :)"""
def foo(self):
raise NotImplementedError('I need to be implemented!')
class B(A):
"""I want to implement A"""
def foo(self):
print('B: foo')
# MRO Stops here, unless super is not A
position = self.__class__.__mro__.index
if not position(B) + 1 == position(A):
super().foo()
b = B()
b.foo()
class C(A):
"""I want to modify B and all its siblings (see below)"""
def foo(self):
print('C: foo')
# MRO Stops here, unless super is not A
position = self.__class__.__mro__.index
if not position(C) + 1 == position(A):
super().foo()
print('')
print('B: Old __base__ and __mro__:\n')
print('Base:', B.__bases__)
print('MRO:', B.__mro__)
print('')
# __mro__ change implementation
B.__bases__ = (C,)
print('B: New __base__ and __mro__:\n')
print('Base:', B.__bases__)
print('MRO:', B.__mro__)
print('')
b.foo()
And the output:
B: foo
B: Old __base__ and __mro__:
Base: (<class '__main__.A'>,)
MRO: (<class '__main__.B'>, <class '__main__.A'>, <class 'object'>)
B: New __base__ and __mro__:
Base: (<class '__main__.C'>,)
MRO: (<class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class 'object'>)
B: foo
C: foo