I created the empty abstract class AbstractStorage and inherited the Storage class from it:
import abc
import pymongo as mongo
host = mongo.MongoClient()
print(host.alive()) # True
class AbstractStorage(metaclass=abc.ABCMeta):
pass
class Storage(AbstractStorage):
dbh = host
def __init__(self):
print('__init__')
Storage()
I expected the output to be
True
__init__
however, the one I'm getting is
True
Traceback (most recent call last):
File "/home/vaultah/run.py", line 16, in <module>
Storage()
TypeError: Can't instantiate abstract class Storage with abstract methods dbh
The problem (apparently) goes away if I remove metaclass=abc.ABCMeta (so that AbstractStorage becomes an ordinary class) and/or if I set dbh to some other value.
What's going on here?
This isn't really a problem with ABCs, it's a problem with PyMongo. There is an issue about it here. It seems that pymongo overrides __getattr__ to return some sort of database class. This means that host.__isabstractmethod__ returns a Database object, which is true in a boolean context. This cause ABCMeta to believe that host is an abstract method:
>>> bool(host.__isabstractmethod__)
True
The workaround described in the issue report is to manually set host.__isabstractmethod__ = False on your object. The last comment on the issue suggests a fix has been put in for pymongo 3.0.
Short Version
mongo.MongoClient returns an object that appears to be (is?) an abstract method, which you then assign to the dbh field in Storage. This makes Storage an abstract class, so instantiating it raises a TypeError.
Note that I don't have pymongo, so I can't tell you anything more about MongoClient than how it gets treated by ABCMeta.
Long Version
The ABCMeta.__new__ method looks inside each field of the new class it's creating. Any field that itself has a True (or "true-like") __isabstractmethod__ field is considered an abstract method. If a class has any non-overridden abstract methods, the whole class is considered abstract, so any attempt to instantiate it is an error.
From an earlier version of the standard library's abc.py:
def __new__(mcls, name, bases, namespace):
cls = super().__new__(mcls, name, bases, namespace)
# Compute set of abstract method names
abstracts = {name
for name, value in namespace.items()
if getattr(value, "__isabstractmethod__", False)}
# ...
cls.__abstractmethods__ = frozenset(abstracts)
# ...
This is not mentioned in the abc.ABCMeta class docs, but a bit lower, under the #abc.abstractmethod decorator:
In order to correctly interoperate with the abstract base class machinery, the descriptor must identify itself as abstract using __isabstractmethod__. In general, this attribute should be True if any of the methods used to compose the descriptor are abstract.
Example
I created a bogus "abstract-looking" class with an __isabstractmethod__ attribute, and two supposedly-concrete subclasses of AbstractStorage. You'll see that one produces the exact error you're getting:
#!/usr/bin/env python3
import abc
# I don't have pymongo, so I have to fake it. See CounterfeitAbstractMethod.
#import pymongo as mongo
class CounterfeitAbstractMethod():
"""
This class appears to be an abstract method to the abc.ABCMeta.__new__
method.
Normally, finding an abstract method in a class's namespace means
that class is also abstract, so instantiating that class is an
error.
If a class derived from abc.ABCMeta has an instance of
CounterfeitAbstractMethod as a value anywhere in its namespace
dictionary, any attempt to instantiate that class will raise a
TypeError: Can't instantiate abstract class <classname> with
abstract method <fieldname>.
"""
__isabstractmethod__ = True
class AbstractStorage(metaclass=abc.ABCMeta):
def __init__(self):
"""
Do-nothing initializer that prints the name of the (sub)class
being initialized.
"""
print(self.__class__.__name__ + ".__init__ executing.")
return
class ConcreteStorage(AbstractStorage):
"""
A concrete class that also _appears_ concrete to abc.ABCMeta. This
class can be instantiated normally.
"""
whatever = "Anything that doesn't appear to be an abstract method will do."
class BogusStorage(AbstractStorage):
"""
This is (supposedly) a concrete class, but its whatever field appears
to be an abstract method, making this whole class abstract ---
abc.ABCMeta will refuse to construct any this class.
"""
#whatever = mongo.MongoClient('localhost', 27017)
whatever = CounterfeitAbstractMethod()
def main():
"""
Print details of the ConcreteStorage and BogusStorage classes.
"""
for cls in ConcreteStorage, BogusStorage:
print(cls.__name__ + ":")
print(" whatever field: " + str(cls.whatever))
print(" abstract methods: " + str(cls.__abstractmethods__))
print(" Instantiating...")
print(" ", end="")
# KABOOM! Instantiating BogusStorage will raise a TypeError,
# because it appears to be an _abstract_ class.
instance = cls()
print(" instance: " + str(instance))
print()
return
if "__main__" == __name__:
main()
Running this produces:
$ ./storage.py
ConcreteStorage:
whatever field: Anything that doesn't appear to be an abstract method will do.
abstract methods: frozenset()
Instantiating...
ConcreteStorage.__init__ executing.
instance: <__main__.ConcreteStorage object at 0x253afd0>
BogusStorage:
whatever field: <__main__.CounterfeitAbstractMethod object at 0x253ad50>
abstract methods: frozenset({'whatever'})
Instantiating...
Traceback (most recent call last):
File "./storage.py", line 75, in <module>
main()
File "./storage.py", line 68, in main
instance = cls()
TypeError: Can't instantiate abstract class BogusStorage with abstract methods whatever
Related
I understand that when a class inherits from an Abstract Django Model, it will not inherit the meta attribute abstract = True, which makes sense.
However in the below example nothing has inherited from it, but yet it's Meta.abstract is False even though its defined to be True:
from django.db import models
from django.db.models.base import ModelBase
class MyMeta(ModelBase):
def __new__(cls, name, bases, attrs, **kwargs):
"""Check that all implemented (not abstract) classes have a foo attribute"""
Class = super().__new__(cls, name, bases, attrs, **kwargs)
if not Class.Meta.abstract:
print(Class)
print('Class.Meta.ordering:', Class.Meta.ordering) # Sanity check
print('Class.Meta.abstract:', Class.Meta.abstract)
if not hasattr(Class, 'foo'):
raise NotImplementedError('Please add a foo attribute')
return Class
class MyAbstractModel(models.Model, metaclass=MyMeta):
name = models.CharField(max_length=250)
class Meta:
abstract = True
ordering = ('-name',)
Prints:
<class 'myapp.models.base.MyAbstractModel'>
Class.Meta.ordering: -name
Class.Meta.abstract: False
Add raises (even though it should not raise because it is an abstract class):
NotImplementedError: Please add a foo attribute
You are accessing the wrong attribute, namely Class.Meta. Since this attribute will be inherited by any child classes Django actually modifies [GitHub] it for abstract models:
if abstract:
# Abstract base models can't be instantiated and don't appear in
# the list of models for an app. We do the final setup for them a
# little differently from normal models.
attr_meta.abstract = False
new_class.Meta = attr_meta
return new_class
The attribute Django actually uses behind the scenes (All of the attributes are properly copied over to it) is _meta. Hence if you write the following you'll get the expected result:
if not Class._meta.abstract:
...
Abstract is False because children classes don't become abstract classes themselves. You need to explicitly set it True.
There is info about that in the docs.
https://docs.djangoproject.com/en/4.1/topics/db/models/
Consider the following code example
import abc
class ABCtest(abc.ABC):
#abc.abstractmethod
def foo(self):
raise RuntimeError("Abstract method was called, this should be impossible")
class ABCtest_B(ABCtest):
pass
test = ABCtest_B()
This correctly raises the error:
Traceback (most recent call last):
File "/.../test.py", line 10, in <module>
test = ABCtest_B()
TypeError: Can't instantiate abstract class ABCtest_B with abstract methods foo
However when the subclass of ABCtest also inherits from a built in type like str or list there is no error and test.foo() calls the abstract method:
class ABCtest_C(ABCtest, str):
pass
>>> test = ABCtest_C()
>>> test.foo()
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
test.foo()
File "/.../test.py", line 5, in foo
raise RuntimeError("Abstract method was called, this should be impossible")
RuntimeError: Abstract method was called, this should be impossible
This seems to happen when inheriting from any class defined in C including itertools.chain and numpy.ndarray but still correctly raises errors with classes defined in python. Why would implementing one of a built in types break the functionality of abstract classes?
Surprisingly, the test that prevents instantiating abstract classes happens in object.__new__, rather than anything defined by the abc module itself:
static PyObject *
object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
...
if (type->tp_flags & Py_TPFLAGS_IS_ABSTRACT) {
...
PyErr_Format(PyExc_TypeError,
"Can't instantiate abstract class %s "
"with abstract methods %U",
type->tp_name,
joined);
(Almost?) all built-in types that aren't object supply a different __new__ that overrides object.__new__ and does not call object.__new__. When you multiple-inherit from a non-object built-in type, you inherit its __new__ method, bypassing the abstract method check.
I don't see anything about __new__ or multiple inheritance from built-in types in the abc documentation. The documentation could use enhancement here.
It seems kind of strange that they'd use a metaclass for the ABC implementation, making it a mess to use other metaclasses with abstract classes, and then put the crucial check in core language code that has nothing to do with abc and runs for both abstract and non-abstract classes.
There's a report for this issue on the issue tracker that's been languishing since 2009.
I asked a similar question and based on user2357112 supports Monicas linked bug report, I came up with this workaround (based on the suggestion from Xiang Zhang):
from abc import ABC, abstractmethod
class Base(ABC):
#abstractmethod
def foo(self):
pass
#abstractmethod
def bar(self):
pass
def __new__(cls, *args, **kwargs):
abstractmethods = getattr(cls, '__abstractmethods__', None)
if abstractmethods:
msg = "Can't instantiate abstract class {name} with abstract method{suffix} {methods}"
suffix = 's' if len(abstractmethods) > 1 else ''
raise TypeError(msg.format(name=cls.__name__, suffix=suffix, methods=', '.join(abstractmethods)))
return super().__new__(cls, *args, **kwargs)
class Derived(Base, tuple):
pass
Derived()
This raises TypeError: Can't instantiate abstract class Derived with abstract methods bar, foo, which is the original behaviour.
I'm trying to figure out how to provide a base class to plugin writers so that they provide definitions for several static methods.
A plugin class is a collection of static methods which will never be instantiated.
I know how to use ABC to prevent instantiation of a class missing method implementations, which will not provide the safety I would like. Is there a pattern to prevent definition?
You can do it by writing your own metaclass similar to ABCMeta, which checks for abstract methods at class-definition time and raises an error if it finds any. Here's an example:
class ClassABC(type):
def __init__(cls, name, bases, attrs):
abstracts = set()
for base in bases:
abstracts.update(getattr(base, '__abstractclassmethods__', set()))
for abstract in abstracts:
if getattr(getattr(cls, abstract), '__isabstractmethod__', False):
raise TypeError("Your class doesn't define {0}".format(abstract))
for attr in attrs:
if getattr(attrs[attr], '__isabstractmethod__', False):
abstracts.add(attr)
cls.__abstractclassmethods__ = abstracts
class BaseClass(object):
__metaclass__ = ClassABC
#abc.abstractmethod
def foo(self):
print("I am base foo")
Then:
>>> class Derived(BaseClass):
... pass
Traceback (most recent call last):
File "<pyshell#10>", line 1, in <module>
class Derived(BaseClass):
File "<pyshell#8>", line 8, in __init__
raise TypeError("Your class doesn't define {0}".format(abstract))
TypeError: Your class doesn't define foo
My example is fairly quick and dirty and only scantily tested, and you might want to refine it to check various sorts of edge cases. What I did is I raised the error if there is an abstract method that wasn't defined in the currently-being-defined class. (Otherwise an error would be raised because the abstract base class doesn't define concrete implementations of its own abstract methods.)
I'm trying to get a python2 program working in python3, it has the following Meta class definition. Which works just fine on Py2. What's the "best" way to have this be compatible with both py2 and py3?
It's failing in the unit test where it does:
try:
raise Actor.DoesNotExist
except Actor.DoesNotExist:
pass
Failure is:
AttributeError: type object 'Actor' has no attribute 'DoesNotExist'
The base meta class definition is:
class MetaDocument(type):
def __new__(meta,name,bases,dct):
class DoesNotExist(BaseException):
pass
class MultipleDocumentsReturned(BaseException):
pass
dct['DoesNotExist'] = DoesNotExist
dct['MultipleDocumentsReturned'] = MultipleDocumentsReturned
class_type = type.__new__(meta, name, bases, dct)
if not class_type in document_classes:
if name == 'Document' and bases == (object,):
pass
else:
document_classes.append(class_type)
return class_type
class Document(object):
__metaclass__ = MetaDocument
You could use the MetaDocument() metaclass as a factory to produce a class replacing your Document class, re-using the class attributes:
class Document(object):
# various and sundry methods and attributes
body = vars(Document).copy()
body.pop('__dict__', None)
body.pop('__weakref__', None)
Document = MetaDocument(Document.__name__, Document.__bases__, body)
This doesn't require you to build the 3rd argument, the class body, manually.
You can turn this into a class decorator:
def with_metaclass(mcls):
def decorator(cls):
body = vars(cls).copy()
# clean out class body
body.pop('__dict__', None)
body.pop('__weakref__', None)
return mcls(cls.__name__, cls.__bases__, body)
return decorator
then use as:
#with_metaclass(MetaDocument)
class Document(object):
# various and sundry methods and attributes
Alternatively, use the six library for this:
#six.add_metaclass(MetaDocument)
class Document(object):
where the #six.add_metaclass() decorator also takes care of any __slots__ you may have defined; my simpler version above doesn't.
six also has a six.with_metaclass() base-class factory:
class Document(six.with_metaclass(MetaDocument)):
which injects an extra base class into the MRO.
six has a utility for this.
class Document(six.with_metaclass(MetaDocument, object)):
# class definition, without the __metaclass__
The only side effect is that the class hierarchy changes from
>>> Document.__mro__
(<class 'test.Document'>, <type 'object'>)
to
>>> Document.__mro__
(<class 'test.Document'>, <class 'test.NewBase'>, <type 'object'>)
because with_metaclass actually returns a new class with the appropriate metaclass.
I need to create a business query model, in which I need to create a circular dependency, I am using a look a like design of django models to implement it,
#Modeule a.py
import b
class A:
b_obj = B()
a_property_1 = ObjectAttribute(b_obj.b_property_1) # a_property_1 is dependent on b_property_1
a_property_2 = ObjectAttribute(b_obj.b_property_2)
#Module b.py
import a
class B:
a_obj = A()
b_property_1 = ObjectAttribute(a_obj.a_property_1)
b_property_2 = ObjectAttribute(a_obj.a_property_2)
When I execute the above program, it will throw an error, name 'B' is not defined on executing a.py and viceversa.
After that, I did a bit research on this to figure out and findout django models already implemented something like this via ForeignKey
https://docs.djangoproject.com/en/dev/ref/models/fields/#foreignkey
All I need to implement the my ForeignKey module, can some one please help me in understanding the logic and writing the code in below format.
#Modeule a.py
import b
class A:
b_obj = MyForeignKey('B')
a_property_1 = ObjectAttribute(b_obj.b_property_1) # a_property_1 is dependent on b_property_1
a_property_2 = ObjectAttribute(b_obj.b_property_2)
#Module b.py
import a
class B:
a_obj = MyForeignKey('A')
b_property_1 = ObjectAttribute(a_obj.a_property_1)
b_property_2 = ObjectAttribute(a_obj.a_property_2)
There are some ways to do that. One of which would be for your foreign Key to be made as proxy classes to the actuall classes, that on instantiating, just annotate the class model, and forhe next subsequent attribute access instantiate the proxied-to class, and keep its reference, Subsequent attributes would just be redirected to the underlying class.
One mechanism that allows such hooks to be executed on attribute fecth (remebering that in Pyhton a class "method" is just a callable attribute - so it works for methods as well), is to implement the __getattribute__ method.
Let's supose you have a "models" module (or other kind of registry) wher all your models are referenced, after creation -- your code could look more or less like this:
import models
class MyForeignKey(objec):
def __init__(self, model_name, *args, **kw):
self._model_name = model_name
self._args = args
self._kw = kw
def _instantiate(self):
self._object = getattr(models, self._model_name)(*self._args, **self._kw)
def __getattribute__(self, attr):
if attr in ("_model_name", "_args", "_kw", "_object", "_instantiate"):
return object.__getattribute__(self, attr)
if not hasattr(self, "_object"):
self._instantiate()
return getattr(self._object, attr)
def __setattr__(self, attr, value):
if attr in ("_model_name", "_args", "_kw", "_object"):
return object.__setattr__(self, attr, value)
if not hasattr(self, "_object"):
self._instantiate()
return setattr(self._object, attr, value)
Note that (a) your models have to inherit from "object" like I commented in the question and (b) - this is ot complete if you implement "dunder" methods (python double underscore methods) to override behavior on any of the models - in that case, you have to set the appropriate te dunder methods to do the proxying as well.