We Keep Coding

When and how to check Python subclasses meet specification

I have a base class that looks something like this:
class myBaseClass:
def __init__(self):
self.name = None # All subclasses must define this
def foo(self): # All subclasses must define this
raise NotImplementedError
def bar(self): # Optional -- not all subclasses will define this
raise NotImplementedError
My API specification stipulates that anyone creating a subclass of myBaseClass must provide a meaningful value for .name, and for the function .foo(). However, .bar() is optional and calling code should be able to handle the case where that results in a NotImplementedError.
When and how should I check that subclasses contributed by third parties meet these requirements?
The options seem to be:
Build subclasses exclusively via metaclasses. However, this approach will be unfamiliar and potentially confusing to most of the contributors to my project, who tend not to be expert developers.
Add an __init_subclass__ method to the base class and use this to infer whether the subclass has overridden everything it is supposed to override. Seems to work, but feels a bit 'kludgy'.
Write build-time tests to instantiate each subclass, call each 'required' method, and verify that they do not raise a NotImplementedError. Seems like an excessive computational effort to answer such a simple question (calling .foo() may be expensive).
Ignore the issue. Deal with it if and when it causes something else to break.
I'm sure I'm not the only person who needs to deal with this issue - is there a 'correct' approach here?

Here's how I would structure it.
First off, what you're looking for here is an abstract base class. Using the built-in modules you can easily define it as such and have methods be forced to have an implementation, otherwise the class will raise an exception when instantiated.
If the name attribute needs to be set always, then you should make it part of the constructor arguments.
Because bar is not always required I wouldn't define it as a method in the base class you have. Instead I would make a child class that is also abstract and define it there as required. When checking to see if the method is available you can use isinstance.
This is what my final code would look like:
from abc import ABC, abstractmethod
class FooBaseClass(ABC):
def __init__(self, name):
self.name = name
#abstractmethod
def foo(self):
"""Some useful docs for foo"""
class FooBarBaseClass(FooBaseClass, ABC):
#abstractmethod
def bar(self):
"""Some useful docs for bar"""
When creating instances you can pick the base class you want and will be forced to define the methods.
class FooClass(FooBaseClass):
def __init__(self):
super().__init__("foo")
def foo(self):
print("Calling foo from FooClass")
class FooBarClass(FooBarBaseClass):
def __init__(self):
super().__init__("foobar")
def foo(self):
print("Calling foo from FooBarClass")
def bar(self):
print("Calling bar from FooBarClass")
Example checking if bar is callable:
def do_operation(obj: FooBaseClass):
obj.foo()
if isinstance(obj, FooBarBaseClass):
obj.bar()
Example:
do_operation(FooClass())
do_operation(FooBarClass())
Calling foo from FooClass
Calling foo from FooBarClass
Calling bar from FooBarClass
An example of invalid code
class InvalidClass(FooBaseClass):
def __init__(self):
super().__init__("foo")
InvalidClass()
Traceback (most recent call last):
File "C:\workspace\so\test.py", line 52, in <module>
InvalidClass()
TypeError: Can't instantiate abstract class InvalidClass with abstract method foo

How to define an abstract metaclass in python

When defining an abstract metaclass in python and instantiating it like this:
from abc import ABC, abstractmethod
class AbstractMetaClass(type, ABC):
#abstractmethod
def func(self):
pass
class MyClass(metaclass=AbstractMetaClass):
pass
I would have expected my code to fail, since MyClass is an instance of an abstract class. Instead it runs with no problems.
What is happening and how can I do this?

Well, you simply found out it does not work. What you are thinking about makes sense: maybe it should fail. It is just that abstract classes are not designed to work as metaclasses, and work collaboratively with "type". I actually find incredible as most Python object mechanisms happen to "just work" when used with metaclasses - including properties, special dunder methods like __getitem__ and operator methods and so on. You just hit one thing that happened not to work.
If your design really makes sense, you may just want to manually make the check for abstract methods on your "abstract metaclass" __init__ method:
from abc import classmethod
class AbstractMetaClass(type):
def __init__(cls, name, bases, ns, **kwargs):
for meth_name, meth in cls.__class__.__dict__.items():
if getattr(meth, "__isabstractmethod__", False):
raise TypeError(f"Can't create new class {name} with no abstract classmethod {meth_name} redefined in the metaclass")
return super().__init__(name, bases, ns, **kwargs)
#abstractmethod
def func(cls):
pass
note that for clarity, it is better that ordinary methods on a metaclass have "cls" as the first argument rather than "self" (althought that might be a personal taste)

Must a class implement all abstract methods?

Suppose you have the following class:
class Base(object):
def abstract_method(self):
raise NotImplementedError
Can you then implement a inheriting class, which does not implement the abstract method? For example, when it does not need that specific method. Will that give problems or is it just bad practice?

If you're implementing abstract methods the way you show, there's nothing enforcing the abstractness of the class as a whole, only of the methods that don't have a concrete definition. So you can create an instance of Base, not only of its subclasses.
b = Base() # this works following your code, only b.abstract_method() raises
def Derived(Base):
... # no concrete implementation of abstract_method, so this class works the same
However, if you use the abc module from the standard library to designate abstract methods, it will not allow you to instantiate an instance of any class that does not have a concrete implementation of any abstract methods it has inherited. You can leave inherited abstract methods unimplemented in an intermediate abstract base class (e.g. a subclass of the original base, that is itself intended to still be abstract), but you can't make any instances.
Here's what using abc looks like:
from abc import ABCMeta, abstractmethod
class ABCBase(metaclass=ABCMeta):
#abstractmethod
def abstract_method(self, arg):
...
class ABCDerived(ABCBase):
... # we don't implement abstract_method here, so we're also an abstract class
d = ABCDerived() # raises an error

How to extend a class with functions based on data and checked at compile time?

I have some classes with methods that are generated by data. These classes are available to a user, and I want type checking at class instantiation time. So if a user doesn't implement these classes correctly, an error occurs at class construction time. That is, I want to write a higher order function that extends a class' signature and marks those methods as abstract.
So, using regular code, the implementation would look like this:
from abc import abstractmethod
class A():
#abstractmethod
def process_init(arg):
pass
and the user class
class B(A):
def process_init(arg):
print(arg)
Easy. Except in my case 'init' is actually data, and could be a long list of atoms. Now, code generation to a file is ugly in python 3. So I want to do something like:
class A():
def dummy(arg):
pass
for i in ['init', 'start', 'pause', 'stop']:
name = 'process_' + i
setattr(A, name, dummy)
make_abstract_method(A, name)
such that the following will object when B is instantiated because the abstract method has no implementation
from a import A
class B(A):
pass
B()
What is the magic incantation for make_abstract_method? (I want a instantiation time error, not a runtime error or exception, so having dummy execute raise NotImplementedError doesn't work here.)

There is no good way to accomplish exactly what you want using abc. abc is quite limited in Python. And fundamentally, it requires that the abstractmethods should be marked when the class object is instantiated. So, you could rely on the undefined behavior of modifying locals() in the class definition statement:
from abc import abstractmethod, ABCMeta
class A(metaclass=ABCMeta):
def dummy(arg):
pass
for i in ['init', 'start', 'pause', 'stop']:
name = 'process_' + i
locals()[name] = abstractmethod(dummy)
Where, ideally, you'd like to be able to do:
from abc import abstractmethod, ABCMeta
class A(metaclass=ABCMeta):
def dummy(arg):
pass
for i in ['init', 'start', 'pause', 'stop']:
name = 'process_' + i
setattr(A, name, abstractmethod(A.dummy))
But then the abstractmethods will not be enforced.
Likely, you'd be better off calling the the type constructor directly (in this case, of course, of the type subclass ABCMeta):
A = ABCMeta(
'A',
(object,),
{
f"process_{i}": abstractmethod(dummy)
for i in ['init', 'start', 'pause', 'stop']
}
)
And of course, you cannot accomplish any of this at compile time, which for Python just does translation from source code to byte code, with minor optimizations, with no type-checking.
Now, it will throw a runtime error if you try to instantiate a B object:
class B(A):
pass
B() # TypeError: Can't instantiate abstract class B with abstract methods dummy, process_init, process_pause, process_start, process_stop
But abc won't complain at class definition time (which is the best you could hope for anyway).
You could write a custom metaclass to accomplish this at class definition time.

Python: abstract instance variable?

Is there a way to declare an abstract instance variable for a class in python?
For example, we have an abstract base class, Bird, with an abstract method fly implemented using the abc package, and the abstract instance variable feathers (what I'm looking for) implemented as a property.
from abc import ABCMeta, abstractmethod
class Bird(metaclass=ABCMeta):
#property
#abstractmethod
def feathers(self):
"""The bird's feathers."""
#abstractmethod
def fly(self):
"""Take flight."""
The problem is that Eagle, a class derived from Bird, is required to have feathers implemented as a property method. So the following is not an acceptable class, but I'd like it to be
class Eagle(Bird):
def __init__(self):
self.feathers = 'quill'
def fly(self):
print('boy are my arms tired')
There might be a problem since the requirement is on the instance itself, and really after its instantiation, so I don't know if things like the abc package will still work.
Are there some standard ways of handling this?

The abc system doesn't include a way to declare an abstract instance variable. The code that determines whether a class is concrete or abstract has to run before any instances exist; it can inspect a class for methods and properties easily enough, but it has no way to tell whether instances would have any particular instance attribute.
The closest thing is probably a variable annotation:
class Bird(metaclass=ABCMeta):
feathers : str
...
abc won't do anything with that annotation, but it at least expresses to readers and static type checkers that instances of Bird are supposed to have a feathers instance variable. (Whether static type checkers will understand that this instance variable is supposed to come from subclasses, I don't know.)

Something simple like the following can work, using a common property:
class Bird(object):
#property
def feathers(self):
try:
return self._feathers
except AttributeError:
raise WhateverError('No feathers') # maybe obfuscate inner structure
class Eagle(Bird):
def __init__(self):
self._feathers = 'quill'
>>> Bird().feathers
WhateverError: No feathers
>>> Eagle().feathers
'quill'