Say I have a json file look like:
{
"foo": ["hi", "there"],
"bar": ["nothing"]
}
I'd like to create an abstract base class (ABC), where the name of abstract methods are the keys of the json above, i.e.:
from abc import ABCMeta, abstractmethod
class MyABC(metaclass=ABCMeta):
#abstractmethod
def foo(self):
pass
#abstractmethod
def bar(self):
pass
The problem is the json file actually has lots of keys. I wonder if there's any way like:
import json
with open("the_json.json") as f:
the_json = json.load(f)
class MyABC(metaclass=ABCMeta):
# for k in the_json.keys():
# create abstract method k
Thanks for the suggestions from the comments, but somehow it doesn't work as expected. Here is what I tried:
class MyABC(metaclass=ABCMeta):
pass
def f(self):
pass
setattr(MyABC, "foo", abstractmethod(f))
# I also tried
# setattr(MyABC, "foo", abstractmethod(lambda self: ...))
# Try to define another class that inherits MyABC
class MyClass(MyABC):
pass
c = MyClass()
# Now this should trigger TypeError but it doesn't
# I can even call c.foo() without getting any errors
This may work :
from abc import ABCMeta, abstractmethod
with open("the_json.json") as f:
the_json = json.load(f)
class MyABC(metaclass=ABCMeta):
def func(self):
pass
for k in the_json:
locals()[k] = abstractmethod(func)
# Delete attribute "func" is a must
# Otherwise it becomes an additional abstract method in MyABC
delattr(MyABC, "func")
delattr(MyABC, "f")
delattr(MyABC, "k")
class MyClass(MyABC):
pass
MyClass()
# TypeError: Can't instantiate abstract class MyClass with abstract methods bar, foo
It will correctly throw an error if you try to instantiate MyABC, or a subclass of MyABC that doesn't implement the abstract methods.
Related
I have designed a way to inherit from a class programmatically, so basically extending a class. It works pretty nicely. However, I need to pickle the class, and this is not possible as the extended class is defined within a function.
Here is a toy example for the extended class:
import pickle
class A():
pass
def extend_class(base_class):
class B(base_class):
def hello(self):
print('Yo!')
return B
extended_class = extend_class(A)
b = extended_class()
b.hello()
However if I do
pickle.dump(b, open('extended_class.pickle', 'w'))
this returns:
AttributeError: Can't pickle local object 'extend_class.<locals>.B'
Any workaround? I don't necessarely need to use my way to extend a class. Any other way would be acceptable, as long as I can pickle the class in the end.
You could declare a class in the global domain, then, declare it global in the factory function, and overwrite it.
It looks like a bit of a hack, and there might be a better way (maybe by injecting the class directly into the class dict), but nonetheless, here it is:
import pickle
class A:
pass
class _B:
pass
B = _B
def extend_class(base_class):
global B
class B(base_class):
def hello(self):
print('Yo!')
return B
if __name__ == '__main__':
extended_class = extend_class(A)
b = extended_class()
b.hello()
with open('tst.pickle', "wb") as f:
pickle.dump(b, f)
This is the setup I want:
A should be an abstract base class with a static & abstract method f(). B should inherit from A. Requirements:
1. You should not be able to instantiate A
2. You should not be able to instantiate B, unless it implements a static f()
Taking inspiration from this question, I've tried a couple of approaches. With these definitions:
class abstractstatic(staticmethod):
__slots__ = ()
def __init__(self, function):
super(abstractstatic, self).__init__(function)
function.__isabstractmethod__ = True
__isabstractmethod__ = True
class A:
__metaclass__ = abc.ABCMeta
#abstractstatic
def f():
pass
class B(A):
def f(self):
print 'f'
class A2:
__metaclass__ = abc.ABCMeta
#staticmethod
#abc.abstractmethod
def f():
pass
class B2(A2):
def f(self):
print 'f'
Here A2 and B2 are defined using usual Python conventions and A & B are defined using the way suggested in this answer. Following are some operations I tried and the results that were undesired.
With classes A/B:
>>> B().f()
f
#This should have thrown, since B doesn't implement a static f()
With classes A2/B2:
>>> A2()
<__main__.A2 object at 0x105beea90>
#This should have thrown since A2 should be an uninstantiable abstract class
>>> B2().f()
f
#This should have thrown, since B2 doesn't implement a static f()
Since neither of these approaches give me the output I want, how do I achieve what I want?
You can't do what you want with just ABCMeta. ABC enforcement doesn't do any type checking, only the presence of an attribute with the correct name is enforced.
Take for example:
>>> from abc import ABCMeta, abstractmethod, abstractproperty
>>> class Abstract(object):
... __metaclass__ = ABCMeta
... #abstractmethod
... def foo(self): pass
... #abstractproperty
... def bar(self): pass
...
>>> class Concrete(Abstract):
... foo = 'bar'
... bar = 'baz'
...
>>> Concrete()
<__main__.Concrete object at 0x104b4df90>
I was able to construct Concrete() even though both foo and bar are simple attributes.
The ABCMeta metaclass only tracks how many objects are left with the __isabstractmethod__ attribute being true; when creating a class from the metaclass (ABCMeta.__new__ is called) the cls.__abstractmethods__ attribute is then set to a frozenset object with all the names that are still abstract.
type.__new__ then tests for that frozenset and throws a TypeError if you try to create an instance.
You'd have to produce your own __new__ method here; subclass ABCMeta and add type checking in a new __new__ method. That method should look for __abstractmethods__ sets on the base classes, find the corresponding objects with the __isabstractmethod__ attribute in the MRO, then does typechecking on the current class attributes.
This'd mean that you'd throw the exception when defining the class, not an instance, however. For that to work you'd add a __call__ method to your ABCMeta subclass and have that throw the exception based on information gathered by your own __new__ method about what types were wrong; a similar two-stage process as what ABCMeta and type.__new__ do at the moment. Alternatively, update the __abstractmethods__ set on the class to add any names that were implemented but with the wrong type and leave it to type.__new__ to throw the exception.
The following implementation takes that last tack; add names back to __abstractmethods__ if the implemented type doesn't match (using a mapping):
from types import FunctionType
class ABCMetaTypeCheck(ABCMeta):
_typemap = { # map abstract type to expected implementation type
abstractproperty: property,
abstractstatic: staticmethod,
# abstractmethods return function objects
FunctionType: FunctionType,
}
def __new__(mcls, name, bases, namespace):
cls = super(ABCMetaTypeCheck, mcls).__new__(mcls, name, bases, namespace)
wrong_type = set()
seen = set()
abstractmethods = cls.__abstractmethods__
for base in bases:
for name in getattr(base, "__abstractmethods__", set()):
if name in seen or name in abstractmethods:
continue # still abstract or later overridden
value = base.__dict__.get(name) # bypass descriptors
if getattr(value, "__isabstractmethod__", False):
seen.add(name)
expected = mcls._typemap[type(value)]
if not isinstance(namespace[name], expected):
wrong_type.add(name)
if wrong_type:
cls.__abstractmethods__ = abstractmethods | frozenset(wrong_type)
return cls
With this metaclass you get your expected output:
>>> class Abstract(object):
... __metaclass__ = ABCMetaTypeCheck
... #abstractmethod
... def foo(self): pass
... #abstractproperty
... def bar(self): pass
... #abstractstatic
... def baz(): pass
...
>>> class ConcreteWrong(Abstract):
... foo = 'bar'
... bar = 'baz'
... baz = 'spam'
...
>>> ConcreteWrong()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class ConcreteWrong with abstract methods bar, baz, foo
>>>
>>> class ConcreteCorrect(Abstract):
... def foo(self): return 'bar'
... #property
... def bar(self): return 'baz'
... #staticmethod
... def baz(): return 'spam'
...
>>> ConcreteCorrect()
<__main__.ConcreteCorrect object at 0x104ce1d10>
From this answer to "what is a metaclass?" I got this:
You write class Foo(object) first, but the class object Foo is not created in memory yet.
Python will look for metaclass in the class definition. If it finds it, it will use it to create the object class Foo. If it doesn't, it will use type to create the class.
Having tested it, it seems that the attributes of the class are instantiated before the constructor of the class is run. What am I misunderstanding?
Test code:
class meta(type):
def __init__(cls, name, bases, dic):
type.__init__(cls, name, bases, dic)
print hasattr(cls, "a")
cls.a = "1"
class A(object):
a = "a"
__metaclass__ = meta
class B(object):
__metaclass__ = meta
class C(object):
__metaclass__ = meta
a = "a"
print A.a
print B.a
print C.a
Output:
True
False
True
1
1
1
The class body is run before the class is constructed, yes.
The body of the class provides a temporary namespace, and all local names in that namespace are given as a dictionary to construct the class object, together with the base classes and a name for the class.
You can do this with the type() constructor too:
>>> Foo = type('Foo', (), {'a': 1})
>>> Foo.a
1
The class body is basically executed as a function, with the local namespace of that function being used to create the class attributes, the 3rd argument to type() above.
In python 3 you have a little more influence on that process with the __prepare__ hook on a metaclass. __prepare__ should be a class method that returns a initial namespace for the class body; use it to inject extra names into the generated class body before the class body is executed:
class MyMeta(type):
#classmethod
def __prepare__(mcl, name, bases):
return {'a': 1}
I'm writing an plugin framework and I want to be able to write a decorator interface, which will convert user class to ABC class and substitute all methods with abstractmethods. I cannot get it working and I suppose the problem is connected with wrong mro, but I can be wrong.
I basically need to be albe to write:
#interface
class X:
def test(self):
pass
x = X() # should fail, because test will be abstract method.
substituting methods with their abstract versions is straightforward (you have to iterate over func's and replace them with abc.abstractmethod(func)), but I've got problem with creating dynamic type, which will be an ABCmeta metaclass.
Right now I've got something like:
from abc import ABCMeta
class Interface(metaclass=ABCMeta):
pass
def interface(cls):
newcls = type(cls.__name__, (Interface, cls), {})
# substitute all methods with abstract ones
for name, func in inspect.getmembers(newcls, predicate=inspect.isfunction):
setattr(newcls, name, abstractmethod(func))
return newcls
but it doesnot work - Ican initialize class X without errors.
With standard usage of ABC in Python, we can write:
class X(metaclass=ABCMeta):
#abstractmethod
def test(self):
pass
x = X() # it will fail
How can I create dynamic type in Python3, which will behave like it will have metaclass ABCmeta and will substitute all functions with abstract ones?
The trick is not to use setattr to reset each of the attributes, but instead to pass those modified attributes to the type function as a dictionary:
import inspect
from abc import ABCMeta, abstractmethod
class Interface(metaclass=ABCMeta):
pass
def interface(cls):
attrs = {n: abstractmethod(f)
for n, f in inspect.getmembers(cls, predicate=inspect.isfunction)}
return type(cls.__name__, (Interface, cls), attrs)
#interface
class X(metaclass=ABCMeta):
def test(self):
pass
x = X()
# does fail:
# Traceback (most recent call last):
# File "test.py", line 19, in <module>
# x = X() # should fail, because test will be abstract method.
# TypeError: Can't instantiate abstract class X with abstract methods test
I am looking for ways / best practices on testing methods defined in an abstract base class. One thing I can think of directly is performing the test on all concrete subclasses of the base class, but that seems excessive at some times.
Consider this example:
import abc
class Abstract(object):
__metaclass__ = abc.ABCMeta
#abc.abstractproperty
def id(self):
return
#abc.abstractmethod
def foo(self):
print "foo"
def bar(self):
print "bar"
Is it possible to test bar without doing any subclassing?
In newer versions of Python you can use unittest.mock.patch()
class MyAbcClassTest(unittest.TestCase):
#patch.multiple(MyAbcClass, __abstractmethods__=set())
def test(self):
self.instance = MyAbcClass() # Ha!
Here is what I have found: If you set __abstractmethods__ attribute to be an empty set you'll be able to instantiate abstract class. This behaviour is specified in PEP 3119:
If the resulting __abstractmethods__ set is non-empty, the class is considered abstract, and attempts to instantiate it will raise TypeError.
So you just need to clear this attribute for the duration of tests.
>>> import abc
>>> class A(metaclass = abc.ABCMeta):
... #abc.abstractmethod
... def foo(self): pass
You cant instantiate A:
>>> A()
Traceback (most recent call last):
TypeError: Can't instantiate abstract class A with abstract methods foo
If you override __abstractmethods__ you can:
>>> A.__abstractmethods__=set()
>>> A() #doctest: +ELLIPSIS
<....A object at 0x...>
It works both ways:
>>> class B(object): pass
>>> B() #doctest: +ELLIPSIS
<....B object at 0x...>
>>> B.__abstractmethods__={"foo"}
>>> B()
Traceback (most recent call last):
TypeError: Can't instantiate abstract class B with abstract methods foo
You can also use unittest.mock (from 3.3) to override temporarily ABC behaviour.
>>> class A(metaclass = abc.ABCMeta):
... #abc.abstractmethod
... def foo(self): pass
>>> from unittest.mock import patch
>>> p = patch.multiple(A, __abstractmethods__=set())
>>> p.start()
{}
>>> A() #doctest: +ELLIPSIS
<....A object at 0x...>
>>> p.stop()
>>> A()
Traceback (most recent call last):
TypeError: Can't instantiate abstract class A with abstract methods foo
As properly put by lunaryon, it is not possible. The very purpose of ABCs containing abstract methods is that they are not instantiatable as declared.
However, it is possible to create a utility function that introspects an ABC, and creates a dummy, non abstract class on the fly. This function could be called directly inside your test method/function and spare you of having to wite boiler plate code on the test file just for testing a few methods.
def concreter(abclass):
"""
>>> import abc
>>> class Abstract(metaclass=abc.ABCMeta):
... #abc.abstractmethod
... def bar(self):
... return None
>>> c = concreter(Abstract)
>>> c.__name__
'dummy_concrete_Abstract'
>>> c().bar() # doctest: +ELLIPSIS
(<abc_utils.Abstract object at 0x...>, (), {})
"""
if not "__abstractmethods__" in abclass.__dict__:
return abclass
new_dict = abclass.__dict__.copy()
for abstractmethod in abclass.__abstractmethods__:
#replace each abc method or property with an identity function:
new_dict[abstractmethod] = lambda x, *args, **kw: (x, args, kw)
#creates a new class, with the overriden ABCs:
return type("dummy_concrete_%s" % abclass.__name__, (abclass,), new_dict)
You can use multiple inheritance practice to have access to the implemented methods of the abstract class. Obviously following such design decision depends on the structure of the abstract class since you need to implement abstract methods (at least bring the signature) in your test case.
Here is the example for your case:
class Abstract(object):
__metaclass__ = abc.ABCMeta
#abc.abstractproperty
def id(self):
return
#abc.abstractmethod
def foo(self):
print("foo")
def bar(self):
print("bar")
class AbstractTest(unittest.TestCase, Abstract):
def foo(self):
pass
def test_bar(self):
self.bar()
self.assertTrue(1==1)
No, it's not. The very purpose of abc is to create classes that cannot be instantiated unless all abstract attributes are overridden with concrete implementations. Hence you need to derive from the abstract base class and override all abstract methods and properties.
Perhaps a more compact version of the concreter proposed by #jsbueno could be:
def concreter(abclass):
class concreteCls(abclass):
pass
concreteCls.__abstractmethods__ = frozenset()
return type('DummyConcrete' + abclass.__name__, (concreteCls,), {})
The resulting class still has all original abstract methods (which can be now called, even if this is not likely to be useful...) and can be mocked as needed.