I'm currently implementing some unit tests for my company's build scripts. To eliminate bloat and make it a little easier to implement new tests, I'm making all my test classes inherit from a custom subclass called BasicTest that inherits from PyUnit's TestCase.
There are currently two functions that all tests utilize from BasicTest: The constructor (Although it could obviously be overwritten in the future) and the runTest() method that is the default method name that the super's constructor uses if no value is passed in (e.g. BasicTest() would create a test that will execute the runTest() method when called upon, whereas BasicTest('Foo') would use the Foo() method).
I would like to make runTest() simply run all possible tests from the inheriting object it is called on. However, as runTest() is defined only in BasicTest and inherited by the subclasses, I'm looking for a way to dynamically call all of the subclass' methods from the super. I know this violates the rules of OO programming, but from what I can see, Python was never one to follow rules in the first place :)
For clarity, the following illustrates my intentions:
I want runTest() to be called from a subclass object and only handle that object's methods. Let's say SubclassTest() that has methods TestParse() and TestExec(). I want it so that:
sub = SubClassTest()
sub.runTest()
runs TestParse() and TestExec(), but I want the runTest() method to be defined in and inherited from BasicTest without being overriden.
one can create metaclass which will collect all interesting methods of subclasses into class property
class TestMetaclass(type):
def __new__(cls, name, bases, attrs):
own_tests = [v for k,v in attrs.iteritems() if k.startswith('test')]
attrs['test_method_list'] = own_tests
return super(TestMetaclass, cls).__new__(cls, name, bases, attrs)
set this metaclass to base class as __metaclass__
and implement runTests method
class BaseTest():
test_method_list = []
__metaclass__ = TestMetaclass
def runTests(self):
for method in self.test_method_list:
method(self)
And after this all subclasses will be constructed using this metaclass
class TestOne(BaseTest):
def test_foo(self):
pass
In the end one can use collected methods running runTests() method
TestOne().runTests()
Sample code:
load base class .py file as module
and inspect
import inspect
import imp
imp.load_source((name of class by which to want that module), (path base class name of file).py)
module = __import__((name of class by which to want that module))
inspect.getmembers(module) will give you dict of name, cls
Hope this helps
Related
I am trying to use the attr package to simply create a metaclass with attributes and methods to use in a further class definition in Python 3. I want to use the attrs package since I have a lot of simple storage classes that only need a few attributes on initialization. Everything works fine, except for when I am trying to add the metaclass to the main class, the code fails with
TypeError: __init__() takes from 1 to 2 positional arguments but 4 were given
A simple MWE would be:
from attr import attrs, attrib
from abc import ABCMeta
#attrs
class MetaClass(ABCMeta):
my_attribute = attrib()
def my_method(self):
pass
class MyClass(object, metaclass=MetaClass):
pass
For Python 2/3 compatibility I usually use the six package and the add_metaclass decorator therein, but I would be happy if it would work in either Python 2 or 3.
The attrs library does generate an __init__ method for the class it is decorating. However, metaclasses have a well defined signature to __new__ and __init__ methods, the arguments of which are filled in by the Python runtime itself, whenever a class statement is executed (along with its body, that is).
I mean - it is the Python runtime that fills in the arguments "class, name, bases, namespace" in a call for a metaclass __init__, and you can't easily change that so that class, attr1, attr2 are passed instead, as the __init__ created by attrs would require.
In short, without looking further into attrs documentation to see if you can supress its overriding of __init__ (and maybe some of other magic methods it creates), you can't use attrs with metaclasses.
For the record, Python 3.7 "dataclasses" allow one to "turn-off" the creation of an __init__ method, but even so, metaclasses are something that have to be carefully thought of, and it is hard to think of an advanced feature for instrumenting classes that would work out of the box for metaclasses just because the language syntax allow for it.
All in all, this might be a "x,y" problem - I'd suggest not trying to use a 3rd party package with with metaclasses just because you think some of its features will play nicely there, and instead, describe what is that you want to achieve with your custom meta-classes in another question.
A few lines of code in a metaclass __new__ method will probably give you the wanted features, but without unknown side-effects.
In particular, if you simply want to add my_attribute = attrib() to all classes created with your custom metaclass, you should not try to create this in the metaclass. Perhaps, you simply need a Base superclass, and not metaclasses at all:
from abc import ABC
...
#attrs
class Base(ABC):
my_attribute = attrib()
class MyClass(Base):
pass
Again, I don't know about the attr lib, and maybe it does not work with
inheritance (but I doubt - it should do just fine), then you could use a
metaclass to inject the attributes and apply the #attrs decorator on your classes, but not try to that on your metaclass itself:
from attr import attrs, attrib
from abc import ABCMeta
class MetaClass(ABCMeta):
my_attribute = attrib()
def __new__(metacls, name, bases, namespace, **kw):
cls = super().__new__(metacls, name, bases, namespace, **kw)
# if the 'attrs' decorator modifies the type of "cls",
# the original __init__ won't be called automatically.
# since we are inheriting from other superclass, we'd better
# call it manually here, and suppress its automatic execution
# bellow.
super(MetaClass, cls).__init__(cls, name, bases, namespace, **kw)
cls.my_attribute = attrib()
return attrs(cls)
def __init__(cls, name, bases, namespace, **kw):
pass
class MyClass(object, metaclass=MetaClass):
pass
Could someone explain what is the difference between the following class definitions derived from BaseClass and in what cases it would matter how they are defined:
class BaseClass(object):
def __init__(self):
# ...
def as_dict(self):
# ...
class SomeClass(BaseClass):
def as_dict(self):
# Does this somehow change the method compared to 'AnotherClass.as_dict()' below
return super(SomeClass, self).as_dict()
class AnotherClass(BaseClass): pass
SomeOtherClass = BaseClass
Since you add an as_dict method to SomeClass that contains a single super call, there's no difference in the end behavior for that class. There's a difference in the fact that, overall, you've added a couple more function calls that are not necessary.
AnotherClass behaves just like SomeClass since SomeClass doesn't do anything different in as_dict. It inherits the methods of BaseClass as usual.
SomeOtherClass is simply another name for BaseClass, you aren't creating a subclassing relationship there, just attaching another name by which you can refer to that class.
This is using BaseClass with a different name.
class AnotherClass(BaseClass): pass
This is using BaseClass but modifying the method "as_dict". Inside the as_dict method, you can do anything (ie.modify the parameters sent to this method) then run the usual function of the as_dict method with super(SomeClass, self).as_dict()
class SomeClass(BaseClass):
def as_dict(self):
# Does this somehow change the method compared to 'AnotherClass.as_dict()' below
return super(SomeClass, self).as_dict()
This is simply assigning BaseClass to SomeOtherClass, which means they can use BaseClass through both keywords.
SomeOtherClass = BaseClass
Let's have an external module called fields.py with a metaclass FieldModelMetaclass which alters class FieldModel:
# module fields.py
class FieldModelMetaclass(type):
def __new__(cls, name, bases, attrs):
# code to alter class creation
class FieldModel(object):
__metaclass__ = FieldModelMetaclass
print 'run at parse time BEFORE metaclass applies'
# module consumer.py
import fields
def adjust_metaclass_new(cls, name, bases, attrs):
# do an extra work on class `cls' and attributes `attrs'
# Is somehow possible from here alter `FieldModelMetaclass.__new__`
# by injection of adjust_metaclass_new to modify FieldModel
# before it gets instantiated ?
mymodel = fields.FieldModel()
As far as I know, metaclass of a class takes an effect at compile runtime so at the moment when I do import a module where is defined, class is already affected.
How can I intercept effect of metaclass at class creation from an external module ?
In this case, you cannot easily intercept the use of the MetaClass. The FieldModelMetaclass.__new__ method is called right after the class FieldModel body completes, when importing the module (so at runtime, not compile time).
You could in theory use a sys.setrace() hook to inject additional Python code after the metaclass has been created but before the class statement executes, but this way madness lies.
Another theoretical option would be to parse the module source code, rewrite it (using the ast module), but again, this way more madness lies.
You can just add a __call__ method to the metaclass or a __new__ method to each class created with the metaclass, if all you wanted was to intercept creation of new instances:
import fields
def metaclass__call__(cls, *args, **kw):
instance = super(FieldModelMetaclass, cls).__call__(*args, **kw)
# do something to the instance
return instance
FieldModelMetaclass.__call__ = metaclass__call__
If your goal was to alter the way FieldModelMetaclass produces the class objects, then your best option is to just alter the class objects again after import. You can always add more attributes to the class, or replace the class with a different one using your own metaclass.
I want to have an instance of class registered when the class is defined. Ideally the code below would do the trick.
registry = {}
def register( cls ):
registry[cls.__name__] = cls() #problem here
return cls
#register
class MyClass( Base ):
def __init__(self):
super( MyClass, self ).__init__()
Unfortunately, this code generates the error NameError: global name 'MyClass' is not defined.
What's going on is at the #problem here line I'm trying to instantiate a MyClass but the decorator hasn't returned yet so it doesn't exist.
Is the someway around this using metaclasses or something?
Yes, meta classes can do this. A meta class' __new__ method returns the class, so just register that class before returning it.
class MetaClass(type):
def __new__(cls, clsname, bases, attrs):
newclass = super(MetaClass, cls).__new__(cls, clsname, bases, attrs)
register(newclass) # here is your register function
return newclass
class MyClass(object):
__metaclass__ = MetaClass
The previous example works in Python 2.x. In Python 3.x, the definition of MyClass is slightly different (while MetaClass is not shown because it is unchanged - except that super(MetaClass, cls) can become super() if you want):
#Python 3.x
class MyClass(metaclass=MetaClass):
pass
As of Python 3.6 there is also a new __init_subclass__ method (see PEP 487) that can be used instead of a meta class (thanks to #matusko for his answer below):
class ParentClass:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
register(cls)
class MyClass(ParentClass):
pass
[edit: fixed missing cls argument to super().__new__()]
[edit: added Python 3.x example]
[edit: corrected order of args to super(), and improved description of 3.x differences]
[edit: add Python 3.6 __init_subclass__ example]
Since python 3.6 you don't need metaclasses to solve this
In python 3.6 simpler customization of class creation was introduced (PEP 487).
An __init_subclass__ hook that initializes all subclasses of a given class.
Proposal includes following example of subclass registration
class PluginBase:
subclasses = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.subclasses.append(cls)
In this example, PluginBase.subclasses will contain a plain list of
all subclasses in the entire inheritance tree. One should note that
this also works nicely as a mixin class.
The problem isn't actually caused by the line you've indicated, but by the super call in the __init__ method. The problem remains if you use a metaclass as suggested by dappawit; the reason the example from that answer works is simply that dappawit has simplified your example by omitting the Base class and therefore the super call. In the following example, neither ClassWithMeta nor DecoratedClass work:
registry = {}
def register(cls):
registry[cls.__name__] = cls()
return cls
class MetaClass(type):
def __new__(cls, clsname, bases, attrs):
newclass = super(cls, MetaClass).__new__(cls, clsname, bases, attrs)
register(newclass) # here is your register function
return newclass
class Base(object):
pass
class ClassWithMeta(Base):
__metaclass__ = MetaClass
def __init__(self):
super(ClassWithMeta, self).__init__()
#register
class DecoratedClass(Base):
def __init__(self):
super(DecoratedClass, self).__init__()
The problem is the same in both cases; the register function is called (either by the metaclass or directly as a decorator) after the class object is created, but before it has been bound to a name. This is where super gets gnarly (in Python 2.x), because it requires you to refer to the class in the super call, which you can only reasonably do by using the global name and trusting that it will have been bound to that name by the time the super call is invoked. In this case, that trust is misplaced.
I think a metaclass is the wrong solution here. Metaclasses are for making a family of classes that have some custom behaviour in common, exactly as classes are for making a family of instances that have some custom behavior in common. All you're doing is calling a function on a class. You wouldn't define a class to call a function on a string, neither should you define a metaclass to call a function on a class.
So, the problem is a fundamental incompatibility between: (1) using hooks in the class creation process to create instances of the class, and (2) using super.
One way to resolve this is to not use super. super solves a hard problem, but it introduces others (this is one of them). If you're using a complex multiple inheritance scheme, super's problems are better than the problems of not using super, and if you're inheriting from third-party classes that use super then you have to use super. If neither of those conditions are true, then just replacing your super calls with direct base class calls may actually be a reasonable solution.
Another way is to not hook register into class creation. Adding register(MyClass) after each of your class definitions is pretty equivalent to adding #register before them or __metaclass__ = Registered (or whatever you call the metaclass) into them. A line down the bottom is much less self-documenting than a nice declaration up the top of the class though, so this doesn't feel great, but again it may actually be a reasonable solution.
Finally, you can turn to hacks that are unpleasant, but will probably work. The problem is that a name is being looked up in a module's global scope just before it's been bound there. So you could cheat, as follows:
def register(cls):
name = cls.__name__
force_bound = False
if '__init__' in cls.__dict__:
cls.__init__.func_globals[name] = cls
force_bound = True
try:
registry[name] = cls()
finally:
if force_bound:
del cls.__init__.func_globals[name]
return cls
Here's how this works:
We first check to see whether __init__ is in cls.__dict__ (as opposed to whether it has an __init__ attribute, which will always be true). If it's inherited an __init__ method from another class we're probably fine (because the superclass will already be bound to its name in the usual way), and the magic we're about to do doesn't work on object.__init__ so we want to avoid trying that if the class is using a default __init__.
We lookup the __init__ method and grab it's func_globals dictionary, which is where global lookups (such as to find the class referred to in a super call) will go. This is normally the global dictionary of the module where the __init__ method was originally defined. Such a dictionary is about to have the cls.__name__ inserted into it as soon as register returns, so we just insert it ourselves early.
We finally create an instance and insert it into the registry. This is in a try/finally block to make sure we remove the binding we created whether or not creating an instance throws an exception; this is very unlikely to be necessary (since 99.999% of the time the name is about to be rebound anyway), but it's best to keep weird magic like this as insulated as possible to minimise the chance that someday some other weird magic interacts badly with it.
This version of register will work whether it's invoked as a decorator or by the metaclass (which I still think is not a good use of a metaclass). There are some obscure cases where it will fail though:
I can imagine a weird class that doesn't have an __init__ method but inherits one that calls self.someMethod, and someMethod is overridden in the class being defined and makes a super call. Probably unlikely.
The __init__ method might have been defined in another module originally and then used in the class by doing __init__ = externally_defined_function in the class block. The func_globals attribute of the other module though, which means our temporary binding would clobber any definition of this class' name in that module (oops). Again, unlikely.
Probably other weird cases I haven't thought of.
You could try to add more hacks to make it a little more robust in these situations, but the nature of Python is both that these kind of hacks are possible and that it's impossible to make them absolutely bullet proof.
The answers here didn't work for me in python3, because __metaclass__ didn't work.
Here's my code registering all subclasses of a class at their definition time:
registered_models = set()
class RegisteredModel(type):
def __new__(cls, clsname, superclasses, attributedict):
newclass = type.__new__(cls, clsname, superclasses, attributedict)
# condition to prevent base class registration
if superclasses:
registered_models.add(newclass)
return newclass
class CustomDBModel(metaclass=RegisteredModel):
pass
class BlogpostModel(CustomDBModel):
pass
class CommentModel(CustomDBModel):
pass
# prints out {<class '__main__.BlogpostModel'>, <class '__main__.CommentModel'>}
print(registered_models)
Calling the Base class directly should work (instead of using super()):
def __init__(self):
Base.__init__(self)
It can be also done with something like this (without a registry function)
_registry = {}
class MetaClass(type):
def __init__(cls, clsname, bases, methods):
super().__init__(clsname, bases, methods)
_registry[cls.__name__] = cls
class MyClass1(metaclass=MetaClass): pass
class MyClass2(metaclass=MetaClass): pass
print(_registry)
# {'MyClass1': <class '__main__.MyClass1'>, 'MyClass2': <class '__main__.MyClass2'>}
Additionally, if we need to use a base abstract class (e.g. Base() class), we can do it this way (notice the metacalss inherits from ABCMeta instead of type)
from abc import ABCMeta
_registry = {}
class MetaClass(ABCMeta):
def __init__(cls, clsname, bases, methods):
super().__init__(clsname, bases, methods)
_registry[cls.__name__] = cls
class Base(metaclass=MetaClass): pass
class MyClass1(Base): pass
class MyClass2(Base): pass
print(_registry)
# {'Base': <class '__main__.Base'>, 'MyClass1': <class '__main__.MyClass1'>, 'MyClass2': <class '__main__.MyClass2'>}
What are the main differences between Python metaclasses and class decorators? Is there something I can do with one but not with the other?
Decorators are much, much simpler and more limited -- and therefore should be preferred whenever the desired effect can be achieved with either a metaclass or a class decorator.
Anything you can do with a class decorator, you can of course do with a custom metaclass (just apply the functionality of the "decorator function", i.e., the one that takes a class object and modifies it, in the course of the metaclass's __new__ or __init__ that make the class object!-).
There are many things you can do in a custom metaclass but not in a decorator (unless the decorator internally generates and applies a custom metaclass, of course -- but that's cheating;-)... and even then, in Python 3, there are things you can only do with a custom metaclass, not after the fact... but that's a pretty advanced sub-niche of your question, so let me give simpler examples).
For example, suppose you want to make a class object X such that print X (or in Python 3 print(X) of course;-) displays peekaboo!. You cannot possibly do that without a custom metaclass, because the metaclass's override of __str__ is the crucial actor here, i.e., you need a def __str__(cls): return "peekaboo!" in the custom metaclass of class X.
The same applies to all magic methods, i.e., to all kinds of operations as applied to the class object itself (as opposed to, ones applied to its instances, which use magic methods as defined in the class -- operations on the class object itself use magic methods as defined in the metaclass).
As given in the chapter 21 of the book 'fluent python', one difference is related to inheritance. Please see these two scripts. The python version is 3.5. One point is that the use of metaclass affects its children while the decorator affects only the current class.
The script use class-decorator to replace/overwirte the method 'func1'.
def deco4cls(cls):
cls.func1 = lambda self: 2
return cls
#deco4cls
class Cls1:
pass
class Cls1_1(Cls1):
def func1(self):
return 3
obj1_1 = Cls1_1()
print(obj1_1.func1()) # 3
The script use metaclass to replace/overwrite the method 'func1'.
class Deco4cls(type):
def __init__(cls, name, bases, attr_dict):
# print(cls, name, bases, attr_dict)
super().__init__(name, bases, attr_dict)
cls.func1 = lambda self: 2
class Cls2(metaclass=Deco4cls):
pass
class Cls2_1(Cls2):
def func1(self):
return 3
obj2_1 = Cls2_1()
print(obj2_1.func1()) # 2!! the original Cls2_1.func1 is replaced by metaclass