We Keep Coding

Apply a python decorator to all inheriting classes

In my django project (django 1.6 - soon upgrading to 1.9, python 2.7),
I'd like to apply a decorator on all my project's model classes (20-30 classes).
All of these classes already inherit from a parent class called 'LinkableModel', which I wrote for a certain (non-related) purpose.
Now, I'd like to apply a class decorator to all these models.
(specifically I'm referring to decorator 'python_2_unicode_compatible': https://docs.djangoproject.com/en/1.9/ref/utils/#django.utils.encoding.python_2_unicode_compatible).
When I add this decorator to their parent class 'LinkableModel', it's not inherited.
Is there any way to apply a decorator to multiple classes, without adding it to each and every class?
(Theoretically I even don't mind if this decorator will be by default applied to all classes in my project...)
Code snippet:
#python_2_unicode_compatible
class LinkableModel(models.Model):
...
...
...
class MyModel1(LinkableModel):
...
...
...
class MyModel2(LinkableModel):
...
...
...

In Python 3.7 now you can do it this way:
class ParentClass:
def __init_subclass__(cls, **kwargs):
return your_decorator(_cls=cls)
it will apply decorator for each subclass of ParentClass
UPDATED: full example:
def your_decorator(_cls):
print("Hello, I'm decor!")
def wrapper():
return _cls()
return wrapper
class ParentClass:
def __init_subclass__(cls, **kwargs):
return your_decorator(_cls=cls)
class A(ParentClass):
pass
a = A()

There is (as far as I know) no simple way, because you cannot inherit decorators.
The simplest solution I can imagine is:
globals_ = globals()
for name, cls in globals_.items():
if subclass(cls, Base):
globals_[name] = decorator(cls)
It simply iterates over every global variable already defined in current module, and if it happens to be class inheriting from Base (or Base itself), it decorates it with decorator.
Note that subclass will not be decorated if:
it's created after this snippet,
it's created in another module,
it's not defined in global namespace.
Alternatively, you can use metaclass:
class Decorate(type):
def __new__(mcls, name, bases, attrs):
return decorator(super().__new__(name, bases, attrs))
class Base(metaclass=Decorate):
pass
When you write class Base(metaclass=Decorate):, Python uses Decorate to create Base and its subclasses.
All that Decorate does, is to decorate class using decorator before returning it.
If you use this, you will probably have a problem if you try to inherit from 2 (or more) classes, each with different metaclass.

I used the answer by #GingerPlusPlus, and created the following function, to apply a decorator to all subclasses of a class:
def apply_decorator_to_all_subclasses(globals_, base_class, decorator):
"""
Given a 'globals_' dictionary, a base class, and a decorator - this function applies the decorator to all the defined classes that derive from the base class
Note!: this function should be called only *after* the subclassess were declared
:param globals_: the given output of globals(), in the caller's context
:param base_class: the class whose descendants require the decorator
:param decorator: the decorator to apply
"""
for name, cls in globals_.items():
# Applying only on *class* items, that are descandants of base_class
if inspect.isclass(cls) and issubclass(cls, base_class) and cls != base_class:
globals_[name] = decorator(cls)

Python: make every instance have a different superclass

In Python, I currently have instances of a class like MyClass('name1'), MyClass('name2') and so on.
I want to make it so that each instance has its own superclass, i.e., I want MyClass('name1') to be an instance of Name1MyClass and MyClass('name2') to be an instance of Name2MyClass. Name1MyClass and Name2MyClass would be dynamically generated subclasses of MyClass. I can't figure out how to do this, because it seems that Python always makes whatever is returned from __new__ an instance of that class. It isn't clear to me how to do it in a metaclass either.
The reason I want to do this is that I want to define __doc__ docstrings on the instances. But it seems that help completely ignores __doc__ on instances; it only looks on classes. So to put a different docstring on each instance, I need to make each instance have its own custom class.

I could be wrong, but I don't think you want a metaclass here. __metaclass__es are used when the class is created, not when you call the class to construct a new instance of the class (or something else).
Here's an answer using __new__ without a metaclass. It feels a bit hacky, but it seems to work:
_sentinel = Ellipsis
class MyClass(object):
def __new__(cls, name):
if name is _sentinel:
return object.__new__(cls)
else:
instance = type(name + cls.__name__, (MyClass,), {})(_sentinel)
# Initialization goes here.
return instance
print type(MyClass('name1'))
print type(MyClass('name2'))
There's a catch here -- All the business logic of initializing then new instance must be done in __new__. Since __new__ is returning a different type than the class it is bound to, __init__ won't get called.
Another option is to create a class factory:
class MyClass(object):
pass
def class_factory(name):
new_cls = type(name + MyClass.__name__, (MyClass,), {})
return new_cls() # Or pass whatever you want in here...
print type(class_factory('name1'))
print type(class_factory('name2'))
Finally, you could even create a non-__new__ class method:
class MyClass(object):
#classmethod
def class_factory(cls, name):
new_cls = type(name + cls.__name__, (cls,), {})
return new_cls() # Or pass whatever you want in here...
print type(MyClass.class_factory('name1'))
print type(MyClass.class_factory('name2'))

How do I get a reference for the current class object?

In Python, how do I get a reference to the current class object within a class statement? Example:
def setup_class_members(cls, prefix):
setattr(cls, prefix+"_var1", "hello")
setattr(cls, prefix+"_var2", "goodbye")
class myclass(object):
setup_class_members(cls, "coffee") # How to get "cls"?
def mytest(self):
print(self.coffee_var1)
print(self.coffee_var2)
x = myclass()
x.mytest()
>>> hello
>>> goodbye
Alternatives that I've written off are:
Use locals(): This gives a dict in a class statement that can be written to. This seems to work for classes, however the documentation tells you not to do this. (I might be tempted to go with this alternative if someone can assure me that this will continue to work for some time.)
Add members to the class object after the class statement: My actual application is to derive a PyQt4 QWidget class with dynamically created pyqtProperty class attributes. QWidget is unusual in that it has a custom metaclass. Very roughly, the metaclass compiles a list of pyqtProperties and stores it as additional member. For this reason, properties that are added to the class after creation have no effect. An example to clear this up:
from PyQt4 import QtCore, QtGui
# works
class MyWidget1(QtGui.QWidget):
myproperty = QtCore.pyqtProperty(int)
# doesn't work because QWidget's metaclass doesn't get to "compile" myproperty
class MyWidget2(QtGui.QWidget):
pass
MyWidget2.myproperty = QtCore.pyqtProperty(int)
Please note that the above will work for most programming cases; my case just happens to be one of those unusual corner cases.

For Python 3, the class must be declared as
class myclass(object, metaclass = Meta):
prefix = "coffee"
...
A few other points:
The metaclass may be a callable, not just a class (Python 2&3)
If the base class of your class already has a non-standard metaclass, you have to make sure you call it's __init__() and __new__() methods instead of type's.
The class statement accepts keyword parameters that are passed on to the metaclass (Python 3 only)
A rewrite of mouad's solution in Python 3 using all of the above is...
def MetaFun(name, bases, attr, prefix=None):
if prefix:
attr[prefix+"_var1"] = "hello"
attr[prefix+"_var2"] = "goodbye"
return object.__class__(name, bases, attr)
class myclass(object, metaclass = MetaFun, prefix="coffee"):
def mytest(self):
print(self.coffee_var1)
print(self.coffee_var2)

AFAIK there is two way to do what you want:
Using metaclass, this will create your two variables in class creation time (which i think is what you want):
class Meta(type):
def __new__(mcs, name, bases, attr):
prefix = attr.get("prefix")
if prefix:
attr[prefix+"_var1"] = "hello"
attr[prefix+"_var2"] = "goodbye"
return type.__new__(mcs, name, bases, attr)
class myclass(object):
__metaclass__ = Meta
prefix = "coffee"
def mytest(self):
print(self.coffee_var1)
print(self.coffee_var2)
Create your two class variable in instantiation time:
class myclass(object):
prefix = "coffee"
def __init__(self):
setattr(self.__class__, self.prefix+"_var1", "hello")
setattr(self.__class__, self.prefix+"_var2", "goodbye")
def mytest(self):
print(self.coffee_var1)
print(self.coffee_var2)
N.B: I'm not sure what you want to achieve because if you want to create dynamic variables depending on the prefix variable why are you accessing like you do in your mytest method ?! i hope it was just an example.

Two more approaches you might use:
A class decorator.
def setup_class_members(prefix):
def decorator(cls):
setattr(cls, prefix+"_var1", "hello")
setattr(cls, prefix+"_var2", "goodbye")
return cls
return decorator
#setup_class_members("coffee")
class myclass(object):
# ... etc
Especially if you need to add attributes in various combinations, the decorator approach is nice because it does not have any effect on inheritance.
If you are dealing with a small set of of attributes that you wish to combine in various ways, you can use mixin classes. A mixin class is a regular class, it's just intended to "mix in" various attributes to some other class.
class coffee_mixin(object):
coffee_var1 = "hello"
coffee_var2 = "goodbye"
class tea_mixin(object):
tea_var1 = "good morning old bean"
tea_var2 = "pip pip cheerio"
class myclass(coffee_mixin, tea_mixin):
# ... etc

See zope.interface.declarations._implements for an example of doing this kind of magic. Just be warned that it's a serious maintainability and portability risk.

How to auto register a class when it's defined

I want to have an instance of class registered when the class is defined. Ideally the code below would do the trick.
registry = {}
def register( cls ):
registry[cls.__name__] = cls() #problem here
return cls
#register
class MyClass( Base ):
def __init__(self):
super( MyClass, self ).__init__()
Unfortunately, this code generates the error NameError: global name 'MyClass' is not defined.
What's going on is at the #problem here line I'm trying to instantiate a MyClass but the decorator hasn't returned yet so it doesn't exist.
Is the someway around this using metaclasses or something?

Yes, meta classes can do this. A meta class' __new__ method returns the class, so just register that class before returning it.
class MetaClass(type):
def __new__(cls, clsname, bases, attrs):
newclass = super(MetaClass, cls).__new__(cls, clsname, bases, attrs)
register(newclass) # here is your register function
return newclass
class MyClass(object):
__metaclass__ = MetaClass
The previous example works in Python 2.x. In Python 3.x, the definition of MyClass is slightly different (while MetaClass is not shown because it is unchanged - except that super(MetaClass, cls) can become super() if you want):
#Python 3.x
class MyClass(metaclass=MetaClass):
pass
As of Python 3.6 there is also a new __init_subclass__ method (see PEP 487) that can be used instead of a meta class (thanks to #matusko for his answer below):
class ParentClass:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
register(cls)
class MyClass(ParentClass):
pass
[edit: fixed missing cls argument to super().__new__()]
[edit: added Python 3.x example]
[edit: corrected order of args to super(), and improved description of 3.x differences]
[edit: add Python 3.6 __init_subclass__ example]

Since python 3.6 you don't need metaclasses to solve this
In python 3.6 simpler customization of class creation was introduced (PEP 487).
An __init_subclass__ hook that initializes all subclasses of a given class.
Proposal includes following example of subclass registration
class PluginBase:
subclasses = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.subclasses.append(cls)
In this example, PluginBase.subclasses will contain a plain list of
all subclasses in the entire inheritance tree. One should note that
this also works nicely as a mixin class.

The problem isn't actually caused by the line you've indicated, but by the super call in the __init__ method. The problem remains if you use a metaclass as suggested by dappawit; the reason the example from that answer works is simply that dappawit has simplified your example by omitting the Base class and therefore the super call. In the following example, neither ClassWithMeta nor DecoratedClass work:
registry = {}
def register(cls):
registry[cls.__name__] = cls()
return cls
class MetaClass(type):
def __new__(cls, clsname, bases, attrs):
newclass = super(cls, MetaClass).__new__(cls, clsname, bases, attrs)
register(newclass) # here is your register function
return newclass
class Base(object):
pass
class ClassWithMeta(Base):
__metaclass__ = MetaClass
def __init__(self):
super(ClassWithMeta, self).__init__()
#register
class DecoratedClass(Base):
def __init__(self):
super(DecoratedClass, self).__init__()
The problem is the same in both cases; the register function is called (either by the metaclass or directly as a decorator) after the class object is created, but before it has been bound to a name. This is where super gets gnarly (in Python 2.x), because it requires you to refer to the class in the super call, which you can only reasonably do by using the global name and trusting that it will have been bound to that name by the time the super call is invoked. In this case, that trust is misplaced.
I think a metaclass is the wrong solution here. Metaclasses are for making a family of classes that have some custom behaviour in common, exactly as classes are for making a family of instances that have some custom behavior in common. All you're doing is calling a function on a class. You wouldn't define a class to call a function on a string, neither should you define a metaclass to call a function on a class.
So, the problem is a fundamental incompatibility between: (1) using hooks in the class creation process to create instances of the class, and (2) using super.
One way to resolve this is to not use super. super solves a hard problem, but it introduces others (this is one of them). If you're using a complex multiple inheritance scheme, super's problems are better than the problems of not using super, and if you're inheriting from third-party classes that use super then you have to use super. If neither of those conditions are true, then just replacing your super calls with direct base class calls may actually be a reasonable solution.
Another way is to not hook register into class creation. Adding register(MyClass) after each of your class definitions is pretty equivalent to adding #register before them or __metaclass__ = Registered (or whatever you call the metaclass) into them. A line down the bottom is much less self-documenting than a nice declaration up the top of the class though, so this doesn't feel great, but again it may actually be a reasonable solution.
Finally, you can turn to hacks that are unpleasant, but will probably work. The problem is that a name is being looked up in a module's global scope just before it's been bound there. So you could cheat, as follows:
def register(cls):
name = cls.__name__
force_bound = False
if '__init__' in cls.__dict__:
cls.__init__.func_globals[name] = cls
force_bound = True
try:
registry[name] = cls()
finally:
if force_bound:
del cls.__init__.func_globals[name]
return cls
Here's how this works:
We first check to see whether __init__ is in cls.__dict__ (as opposed to whether it has an __init__ attribute, which will always be true). If it's inherited an __init__ method from another class we're probably fine (because the superclass will already be bound to its name in the usual way), and the magic we're about to do doesn't work on object.__init__ so we want to avoid trying that if the class is using a default __init__.
We lookup the __init__ method and grab it's func_globals dictionary, which is where global lookups (such as to find the class referred to in a super call) will go. This is normally the global dictionary of the module where the __init__ method was originally defined. Such a dictionary is about to have the cls.__name__ inserted into it as soon as register returns, so we just insert it ourselves early.
We finally create an instance and insert it into the registry. This is in a try/finally block to make sure we remove the binding we created whether or not creating an instance throws an exception; this is very unlikely to be necessary (since 99.999% of the time the name is about to be rebound anyway), but it's best to keep weird magic like this as insulated as possible to minimise the chance that someday some other weird magic interacts badly with it.
This version of register will work whether it's invoked as a decorator or by the metaclass (which I still think is not a good use of a metaclass). There are some obscure cases where it will fail though:
I can imagine a weird class that doesn't have an __init__ method but inherits one that calls self.someMethod, and someMethod is overridden in the class being defined and makes a super call. Probably unlikely.
The __init__ method might have been defined in another module originally and then used in the class by doing __init__ = externally_defined_function in the class block. The func_globals attribute of the other module though, which means our temporary binding would clobber any definition of this class' name in that module (oops). Again, unlikely.
Probably other weird cases I haven't thought of.
You could try to add more hacks to make it a little more robust in these situations, but the nature of Python is both that these kind of hacks are possible and that it's impossible to make them absolutely bullet proof.

The answers here didn't work for me in python3, because __metaclass__ didn't work.
Here's my code registering all subclasses of a class at their definition time:
registered_models = set()
class RegisteredModel(type):
def __new__(cls, clsname, superclasses, attributedict):
newclass = type.__new__(cls, clsname, superclasses, attributedict)
# condition to prevent base class registration
if superclasses:
registered_models.add(newclass)
return newclass
class CustomDBModel(metaclass=RegisteredModel):
pass
class BlogpostModel(CustomDBModel):
pass
class CommentModel(CustomDBModel):
pass
# prints out {<class '__main__.BlogpostModel'>, <class '__main__.CommentModel'>}
print(registered_models)

Calling the Base class directly should work (instead of using super()):
def __init__(self):
Base.__init__(self)

It can be also done with something like this (without a registry function)
_registry = {}
class MetaClass(type):
def __init__(cls, clsname, bases, methods):
super().__init__(clsname, bases, methods)
_registry[cls.__name__] = cls
class MyClass1(metaclass=MetaClass): pass
class MyClass2(metaclass=MetaClass): pass
print(_registry)
# {'MyClass1': <class '__main__.MyClass1'>, 'MyClass2': <class '__main__.MyClass2'>}
Additionally, if we need to use a base abstract class (e.g. Base() class), we can do it this way (notice the metacalss inherits from ABCMeta instead of type)
from abc import ABCMeta
_registry = {}
class MetaClass(ABCMeta):
def __init__(cls, clsname, bases, methods):
super().__init__(clsname, bases, methods)
_registry[cls.__name__] = cls
class Base(metaclass=MetaClass): pass
class MyClass1(Base): pass
class MyClass2(Base): pass
print(_registry)
# {'Base': <class '__main__.Base'>, 'MyClass1': <class '__main__.MyClass1'>, 'MyClass2': <class '__main__.MyClass2'>}

Using the docstring from one method to automatically overwrite that of another method

The problem: I have a class which contains a template method execute which calls another method _execute. Subclasses are supposed to overwrite _execute to implement some specific functionality. This functionality should be documented in the docstring of _execute.
Advanced users can create their own subclasses to extend the library. However, another user dealing with such a subclass should only use execute, so he won't see the correct docstring if he uses help(execute).
Therefore it would be nice to modify the base class in such a way that in a subclass the docstring of execute is automatically replaced with that of _execute. Any ideas how this might be done?
I was thinking of metaclasses to do this, to make this completely transparent to the user.

Well, if you don't mind copying the original method in the subclass, you can use the following technique.
import new
def copyfunc(func):
return new.function(func.func_code, func.func_globals, func.func_name,
func.func_defaults, func.func_closure)
class Metaclass(type):
def __new__(meta, name, bases, attrs):
for key in attrs.keys():
if key[0] == '_':
skey = key[1:]
for base in bases:
original = getattr(base, skey, None)
if original is not None:
copy = copyfunc(original)
copy.__doc__ = attrs[key].__doc__
attrs[skey] = copy
break
return type.__new__(meta, name, bases, attrs)
class Class(object):
__metaclass__ = Metaclass
def execute(self):
'''original doc-string'''
return self._execute()
class Subclass(Class):
def _execute(self):
'''sub-class doc-string'''
pass

Is there a reason you can't override the base class's execute function directly?
class Base(object):
def execute(self):
...
class Derived(Base):
def execute(self):
"""Docstring for derived class"""
Base.execute(self)
...stuff specific to Derived...
If you don't want to do the above:
Method objects don't support writing to the __doc__ attribute, so you have to change __doc__ in the actual function object. Since you don't want to override the one in the base class, you'd have to give each subclass its own copy of execute:
class Derived(Base):
def execute(self):
return Base.execute(self)
class _execute(self):
"""Docstring for subclass"""
...
execute.__doc__= _execute.__doc__
but this is similar to a roundabout way of redefining execute...

Look at the functools.wraps() decorator; it does all of this, but I don't know offhand if you can get it to run in the right context

Well the doc-string is stored in __doc__ so it wouldn't be too hard to re-assign it based on the doc-string of _execute after the fact.
Basically:
class MyClass(object):
def execute(self):
'''original doc-string'''
self._execute()
class SubClass(MyClass):
def _execute(self):
'''sub-class doc-string'''
pass
# re-assign doc-string of execute
def execute(self,*args,**kw):
return MyClass.execute(*args,**kw)
execute.__doc__=_execute.__doc__
Execute has to be re-declared to that the doc string gets attached to the version of execute for the SubClass and not for MyClass (which would otherwise interfere with other sub-classes).
That's not a very tidy way of doing it, but from the POV of the user of a library it should give the desired result. You could then wrap this up in a meta-class to make it easier for people who are sub-classing.

I agree that the simplest, most Pythonic way of approaching this is to simply redefine execute in your subclasses and have it call the execute method of the base class:
class Sub(Base):
def execute(self):
"""New docstring goes here"""
return Base.execute(self)
This is very little code to accomplish what you want; the only downside is that you must repeat this code in every subclass that extends Base. However, this is a small price to pay for the behavior you want.
If you want a sloppy and verbose way of making sure that the docstring for execute is dynamically generated, you can use the descriptor protocol, which would be significantly less code than the other proposals here. This is annoying because you can't just set a descriptor on an existing function, which means that execute must be written as a separate class with a __call__ method.
Here's the code to do this, but keep in mind that my above example is much simpler and more Pythonic:
class Executor(object):
__doc__ = property(lambda self: self.inst._execute.__doc__)
def __call__(self):
return self.inst._execute()
class Base(object):
execute = Executor()
class Sub(Base):
def __init__(self):
self.execute.inst = self
def _execute(self):
"""Actually does something!"""
return "Hello World!"
spam = Sub()
print spam.execute.__doc__ # prints "Actually does something!"
help(spam) # the execute method says "Actually does something!"