Consider the following code example
import abc
class ABCtest(abc.ABC):
#abc.abstractmethod
def foo(self):
raise RuntimeError("Abstract method was called, this should be impossible")
class ABCtest_B(ABCtest):
pass
test = ABCtest_B()
This correctly raises the error:
Traceback (most recent call last):
File "/.../test.py", line 10, in <module>
test = ABCtest_B()
TypeError: Can't instantiate abstract class ABCtest_B with abstract methods foo
However when the subclass of ABCtest also inherits from a built in type like str or list there is no error and test.foo() calls the abstract method:
class ABCtest_C(ABCtest, str):
pass
>>> test = ABCtest_C()
>>> test.foo()
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
test.foo()
File "/.../test.py", line 5, in foo
raise RuntimeError("Abstract method was called, this should be impossible")
RuntimeError: Abstract method was called, this should be impossible
This seems to happen when inheriting from any class defined in C including itertools.chain and numpy.ndarray but still correctly raises errors with classes defined in python. Why would implementing one of a built in types break the functionality of abstract classes?
Surprisingly, the test that prevents instantiating abstract classes happens in object.__new__, rather than anything defined by the abc module itself:
static PyObject *
object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
...
if (type->tp_flags & Py_TPFLAGS_IS_ABSTRACT) {
...
PyErr_Format(PyExc_TypeError,
"Can't instantiate abstract class %s "
"with abstract methods %U",
type->tp_name,
joined);
(Almost?) all built-in types that aren't object supply a different __new__ that overrides object.__new__ and does not call object.__new__. When you multiple-inherit from a non-object built-in type, you inherit its __new__ method, bypassing the abstract method check.
I don't see anything about __new__ or multiple inheritance from built-in types in the abc documentation. The documentation could use enhancement here.
It seems kind of strange that they'd use a metaclass for the ABC implementation, making it a mess to use other metaclasses with abstract classes, and then put the crucial check in core language code that has nothing to do with abc and runs for both abstract and non-abstract classes.
There's a report for this issue on the issue tracker that's been languishing since 2009.
I asked a similar question and based on user2357112 supports Monicas linked bug report, I came up with this workaround (based on the suggestion from Xiang Zhang):
from abc import ABC, abstractmethod
class Base(ABC):
#abstractmethod
def foo(self):
pass
#abstractmethod
def bar(self):
pass
def __new__(cls, *args, **kwargs):
abstractmethods = getattr(cls, '__abstractmethods__', None)
if abstractmethods:
msg = "Can't instantiate abstract class {name} with abstract method{suffix} {methods}"
suffix = 's' if len(abstractmethods) > 1 else ''
raise TypeError(msg.format(name=cls.__name__, suffix=suffix, methods=', '.join(abstractmethods)))
return super().__new__(cls, *args, **kwargs)
class Derived(Base, tuple):
pass
Derived()
This raises TypeError: Can't instantiate abstract class Derived with abstract methods bar, foo, which is the original behaviour.
Related
I have a decorator in my library which takes a user's class and creates a new version of it, with a new metaclass, it is supposed to completely replace the original class. Everything works; except for super() calls:
class NewMeta(type):
pass
def deco(cls):
cls_dict = dict(cls.__dict__)
if "__dict__" in cls_dict:
del cls_dict["__dict__"]
if "__weakref__" in cls_dict:
del cls_dict["__weakref__"]
return NewMeta(cls.__name__, cls.__bases__, cls_dict)
#deco
class B:
def x(self):
print("Hi there")
#deco
class A(B):
def x(self):
super().x()
Using this code like so, yields an error:
>>> a = A()
>>> a.x()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in x
TypeError: super(type, obj): obj must be an instance or subtype of type
Some terminology:
The source code class A as produced by class A(B):.
The produced class A*, as produced by NewMeta(cls.__name__, cls.__bases__, cls_dict).
A is established by Python to be the type when using super inside of the methods of A*. How can I correct this?
There's some suboptimal solutions like calling super(type(self), self).x, or passing cls.__mro__ instead of cls.__bases__ into the NewMeta call (so that obj=self always inherits from the incorrect type=A). The first is unsustainable for end users, the 2nd pollutes the inheritance chains and is confusing as the class seems to inherit from itself.
Python seems to introspect the source code, or maybe stores some information to automatically establish the type, and in this case, I'd say it is failing to do so;
How could I make sure that inside of the methods of A A* is established as the type argument of argumentless super calls?
The argument-free super uses the __class__ cell, which is a regular function closure.
Data Model: Creating the class object
__class__ is an implicit closure reference created by the compiler if any methods in a class body refer to either __class__ or super.
>>> class E:
... def x(self):
... return __class__ # return the __class__ cell
...
>>> E().x()
__main__.E
>>> # The cell is stored as a __closure__
>>> E.x.__closure__[0].cell_contents is E().x() is E
True
Like any other closure, this is a lexical relation: it refers to class scope in which the method was literally defined. Replacing the class with a decorator still has the methods refer to the original class.
The simplest fix is to explicitly refer to the name of the class, which gets rebound to the newly created class by the decorator.
#deco
class A(B):
def x(self):
super(A, self).x()
Alternatively, one can change the content of the __class__ cell to point to the new class:
def deco(cls):
cls_dict = dict(cls.__dict__)
cls_dict.pop("__dict__", None)
cls_dict.pop("__weakref__", None)
new_cls = NewMeta(cls.__name__, cls.__bases__, cls_dict)
for method in new_cls.__dict__.values():
if getattr(method, "__closure__", None) and method.__closure__[0].cell_contents is cls:
method.__closure__[0].cell_contents = new_cls
return new_cls
I have been experimenting a little with the abc module in python. A la
>>> import abc
In the normal case you expect your ABC class to not be instantiated if it contains an unimplemented abstractmethod. You know like as follows:
>>> class MyClass(metaclass=abc.ABCMeta):
... #abc.abstractmethod
... def mymethod(self):
... return -1
...
>>> MyClass()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyClass with abstract methods mymethod
OR for any derived Class. It all seems to work fine until you inherit from something ... say dict or list as in the following:
>>> class YourClass(list, metaclass=abc.ABCMeta):
... #abc.abstractmethod
... def yourmethod(self):
... return -1
...
>>> YourClass()
[]
This is surprising because type is probably the primary factory or metaclass -ish thing anyway or so I assume from the following.
>>> type(abc.ABCMeta)
<class 'type'>
>>> type(list)
<class 'type'>
From some investigation I found out that it boils down to something as simple as adding an __abstractmethod__ attribute to the class' object and rest happens by itself:
>>> class AbstractClass:
... pass
...
>>> AbstractClass.__abstractmethods__ = {'abstractmethod'}
>>> AbstractClass()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class AbstractClass with abstract methods abstractmethod
So one can simply avoid the check by intentionally overriding the __new__ method and clearing out __abstractmethods__ as in below:
>>> class SupposedlyAbstractClass(metaclass=abc.ABCMeta):
... def __new__(cls):
... cls.__abstractmethods__ = {}
... return super(AbstractClass, cls).__new__(cls)
... #abc.abstractmethod
... def abstractmethod(self):
... return -1
...
>>> SupposedlyAbstractClass()
<__main__.SupposedlyAbstractClass object at 0x000001FA6BF05828>
This behaviour is the same in Python 2.7 and in Python 3.7 as I have personally checked. I am not aware if this is the same for all other python implementations.
Finally, down to the question ... Why has this been made to behave like so? Is it wise we should never make abstract classes out of list, tuple or dict? or should I just go ahead and add a __new__ class method checking for __abstractmethods__ before instantiation?
The problem
If you have the next class:
from abc import ABC, abstractmethod
class Foo(list, ABC):
#abstractmethod
def yourmethod(self):
pass
the problem is that and object of Foo can be created without any error because Foo.__new__(Foo) delegates the call directly to list.__new__(Foo) instead of ABC.__new__(Foo) (which is responsible of checking that all abstract methods are implemented in the class that is going to be instantiated)
We could implement __new__ on Foo and try to call ABC.__new__:
class Foo(list, ABC):
def __new__(cls, *args, **kwargs):
return ABC.__new__(cls)
#abstractmethod
def yourmethod(self):
pass
Foo()
But he next error is raised:
TypeError: object.__new__(Foo) is not safe, use list.__new__()
This is due to ABC.__new__(Foo) invokes object.__new__(Foo) which seems that is not allowed when Foo inherits from list
A possible solution
You can add additional code on Foo.__new__ in order to check that all abstract methods in the class to be instantiated are implemented (basically do the job of ABC.__new__).
Something like this:
class Foo(list, ABC):
def __new__(cls, *args, **kwargs):
if hasattr(cls, '__abstractmethods__') and len(cls.__abstractmethods__) > 0:
raise TypeError(f"Can't instantiate abstract class {cls.__name__} with abstract methods {', '.join(cls.__abstractmethods__)}")
return super(Foo, cls).__new__(cls)
#abstractmethod
def yourmethod(self):
return -1
Now Foo() raises an error. But the next code runs without any issue:
class Bar(Foo):
def yourmethod(self):
pass
Bar()
class F(object):
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def f(self, a, b): print("F")
class FF(F):
def f(self): print("FF")
f = FF()
f.f()
Here, I define an abstract method f with two arguments. I want to restrict subclass so that it has same arguments like superclass.
How to do that?
There isn't any trivial way to check that the method signatures match.
However, assuming:
Your subclass only the derives from the concerned abstract class or the abstract class is the first in its mro
And that the implementation of the abstract class is empty, to eliminate any possibility of unwanted results such as binding new attributes to the subclass,
You could make a super call to the abstract method passing the paramters from the subclass method. The call only serves to enforce a signature match.
class F(object):
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def f(self, a, b): pass
class FF(F):
def f(self, *args, **kwargs):
super(FF, self).f(*args, **kwargs)
...
f = FF()
f.f()
Traceback (most recent call last):
File "python", line 17, in <module>
File "python", line 12, in f
TypeError: f() takes exactly 3 arguments (1 given)
Python abstract methods (unlike others) can have implementations and can be called via super; so maybe this is one use case after all.
You can use abcmeta library: https://github.com/mortymacs/abcmeta
. It gives you more restrictions on derived classes.
I'm trying to figure out how to provide a base class to plugin writers so that they provide definitions for several static methods.
A plugin class is a collection of static methods which will never be instantiated.
I know how to use ABC to prevent instantiation of a class missing method implementations, which will not provide the safety I would like. Is there a pattern to prevent definition?
You can do it by writing your own metaclass similar to ABCMeta, which checks for abstract methods at class-definition time and raises an error if it finds any. Here's an example:
class ClassABC(type):
def __init__(cls, name, bases, attrs):
abstracts = set()
for base in bases:
abstracts.update(getattr(base, '__abstractclassmethods__', set()))
for abstract in abstracts:
if getattr(getattr(cls, abstract), '__isabstractmethod__', False):
raise TypeError("Your class doesn't define {0}".format(abstract))
for attr in attrs:
if getattr(attrs[attr], '__isabstractmethod__', False):
abstracts.add(attr)
cls.__abstractclassmethods__ = abstracts
class BaseClass(object):
__metaclass__ = ClassABC
#abc.abstractmethod
def foo(self):
print("I am base foo")
Then:
>>> class Derived(BaseClass):
... pass
Traceback (most recent call last):
File "<pyshell#10>", line 1, in <module>
class Derived(BaseClass):
File "<pyshell#8>", line 8, in __init__
raise TypeError("Your class doesn't define {0}".format(abstract))
TypeError: Your class doesn't define foo
My example is fairly quick and dirty and only scantily tested, and you might want to refine it to check various sorts of edge cases. What I did is I raised the error if there is an abstract method that wasn't defined in the currently-being-defined class. (Otherwise an error would be raised because the abstract base class doesn't define concrete implementations of its own abstract methods.)
I created the empty abstract class AbstractStorage and inherited the Storage class from it:
import abc
import pymongo as mongo
host = mongo.MongoClient()
print(host.alive()) # True
class AbstractStorage(metaclass=abc.ABCMeta):
pass
class Storage(AbstractStorage):
dbh = host
def __init__(self):
print('__init__')
Storage()
I expected the output to be
True
__init__
however, the one I'm getting is
True
Traceback (most recent call last):
File "/home/vaultah/run.py", line 16, in <module>
Storage()
TypeError: Can't instantiate abstract class Storage with abstract methods dbh
The problem (apparently) goes away if I remove metaclass=abc.ABCMeta (so that AbstractStorage becomes an ordinary class) and/or if I set dbh to some other value.
What's going on here?
This isn't really a problem with ABCs, it's a problem with PyMongo. There is an issue about it here. It seems that pymongo overrides __getattr__ to return some sort of database class. This means that host.__isabstractmethod__ returns a Database object, which is true in a boolean context. This cause ABCMeta to believe that host is an abstract method:
>>> bool(host.__isabstractmethod__)
True
The workaround described in the issue report is to manually set host.__isabstractmethod__ = False on your object. The last comment on the issue suggests a fix has been put in for pymongo 3.0.
Short Version
mongo.MongoClient returns an object that appears to be (is?) an abstract method, which you then assign to the dbh field in Storage. This makes Storage an abstract class, so instantiating it raises a TypeError.
Note that I don't have pymongo, so I can't tell you anything more about MongoClient than how it gets treated by ABCMeta.
Long Version
The ABCMeta.__new__ method looks inside each field of the new class it's creating. Any field that itself has a True (or "true-like") __isabstractmethod__ field is considered an abstract method. If a class has any non-overridden abstract methods, the whole class is considered abstract, so any attempt to instantiate it is an error.
From an earlier version of the standard library's abc.py:
def __new__(mcls, name, bases, namespace):
cls = super().__new__(mcls, name, bases, namespace)
# Compute set of abstract method names
abstracts = {name
for name, value in namespace.items()
if getattr(value, "__isabstractmethod__", False)}
# ...
cls.__abstractmethods__ = frozenset(abstracts)
# ...
This is not mentioned in the abc.ABCMeta class docs, but a bit lower, under the #abc.abstractmethod decorator:
In order to correctly interoperate with the abstract base class machinery, the descriptor must identify itself as abstract using __isabstractmethod__. In general, this attribute should be True if any of the methods used to compose the descriptor are abstract.
Example
I created a bogus "abstract-looking" class with an __isabstractmethod__ attribute, and two supposedly-concrete subclasses of AbstractStorage. You'll see that one produces the exact error you're getting:
#!/usr/bin/env python3
import abc
# I don't have pymongo, so I have to fake it. See CounterfeitAbstractMethod.
#import pymongo as mongo
class CounterfeitAbstractMethod():
"""
This class appears to be an abstract method to the abc.ABCMeta.__new__
method.
Normally, finding an abstract method in a class's namespace means
that class is also abstract, so instantiating that class is an
error.
If a class derived from abc.ABCMeta has an instance of
CounterfeitAbstractMethod as a value anywhere in its namespace
dictionary, any attempt to instantiate that class will raise a
TypeError: Can't instantiate abstract class <classname> with
abstract method <fieldname>.
"""
__isabstractmethod__ = True
class AbstractStorage(metaclass=abc.ABCMeta):
def __init__(self):
"""
Do-nothing initializer that prints the name of the (sub)class
being initialized.
"""
print(self.__class__.__name__ + ".__init__ executing.")
return
class ConcreteStorage(AbstractStorage):
"""
A concrete class that also _appears_ concrete to abc.ABCMeta. This
class can be instantiated normally.
"""
whatever = "Anything that doesn't appear to be an abstract method will do."
class BogusStorage(AbstractStorage):
"""
This is (supposedly) a concrete class, but its whatever field appears
to be an abstract method, making this whole class abstract ---
abc.ABCMeta will refuse to construct any this class.
"""
#whatever = mongo.MongoClient('localhost', 27017)
whatever = CounterfeitAbstractMethod()
def main():
"""
Print details of the ConcreteStorage and BogusStorage classes.
"""
for cls in ConcreteStorage, BogusStorage:
print(cls.__name__ + ":")
print(" whatever field: " + str(cls.whatever))
print(" abstract methods: " + str(cls.__abstractmethods__))
print(" Instantiating...")
print(" ", end="")
# KABOOM! Instantiating BogusStorage will raise a TypeError,
# because it appears to be an _abstract_ class.
instance = cls()
print(" instance: " + str(instance))
print()
return
if "__main__" == __name__:
main()
Running this produces:
$ ./storage.py
ConcreteStorage:
whatever field: Anything that doesn't appear to be an abstract method will do.
abstract methods: frozenset()
Instantiating...
ConcreteStorage.__init__ executing.
instance: <__main__.ConcreteStorage object at 0x253afd0>
BogusStorage:
whatever field: <__main__.CounterfeitAbstractMethod object at 0x253ad50>
abstract methods: frozenset({'whatever'})
Instantiating...
Traceback (most recent call last):
File "./storage.py", line 75, in <module>
main()
File "./storage.py", line 68, in main
instance = cls()
TypeError: Can't instantiate abstract class BogusStorage with abstract methods whatever