In python, how can I setup a parent class to track methods with a specific decorator for each child seperatly? A quick code snippet of what I am trying to do:
class Parent:
decorated_func_dict = {} #dictionary that stores name->func for decorated functions
def get_func_by_decorator_name(self, name):
#stuff
pass
class Child1(Parent):
#func_name("Bob")
def bob_func(self, *args):
pass
#func_name("Tom")
def func2(self, *args):
pass
class Child2(Parent):
#func_name("Bob")
def func_bob2(self, *args):
pass
foo = Child1()
bar = Child2()
foo.get_func_by_decorator_name("Bob")
#Returns foo.bob_func
bar.get_func_by_decorator_name("Bob")
#Returns bar.func_bob2
Using Python 3.9.
A decorator is not something that makes a function look pretty. It is a callable that ingests an object (not only functions), does some arbitrary operations, and returns a replacement object.
In this case, your decorator should be storing references to function objects in a dictionary somewhere. The problem is that you won't be able to reference the class in which the functions are defined until it is created, which happens well after the decorator is run. You can avoid this by storing the name of the class as well as the name of the function.
The final step here is to properly bind the function objects to methods on the right object. That is something that get_func_by_decorated_name can do for you.
In sum, you can write something like this:
decorated_func_dict = {}
def func_name(cls_name, func_name):
def decorator(func):
decorated_func_dict.setdefault(cls_name, {})[func_name] = func
return func
return decorator
class Parent:
def get_func_by_decorator_name(self, name):
return decorated_func_dict[type(self).__name__][name].__get__(self)
class Child1(Parent):
#func_name("Child1", "Bob")
def bob_func(self, *args):
pass
#func_name("Child1", "Tom")
def func2(self, *args):
pass
class Child2(Parent):
#func_name("Child2", "Bob")
def func_bob2(self, *args):
pass
And indeed you get:
>>> foo.get_func_by_decorator_name("Bob")
<bound method Child1.bob_func of <__main__.Child1 object at 0x000001D58181E070>>
>>> bar.get_func_by_decorator_name("Bob")
<bound method Child2.func_bob2 of <__main__.Child2 object at 0x000001D582041F10>>
Another way to do this is to give your functions a name attribute, which you can then aggregate into a mapping in __init_subclass__ in Parent. This allows you to make an interface a bit closer to what you originally intended:
def func_name(func_name):
def decorator(func):
func.special_name = func_name
return func
return decorator
class Parent:
def __init_subclass__(cls):
cls.decorated_func_dict = {}
for item in cls.__dict__.values():
if hasattr(item, 'special_name'):
cls.decorated_func_dict[item.special_name] = item
del item.special_name # optional
def get_func_by_decorator_name(self, name):
return self.decorated_func_dict[name].__get__(self)
class Child1(Parent):
#func_name("Bob")
def bob_func(self, *args):
pass
#func_name("Tom")
def func2(self, *args):
pass
class Child2(Parent):
#func_name("Bob")
def func_bob2(self, *args):
pass
The results are identical to the first example.
The easiest way would of course be to get access to the child's namespace before the class is created, e.g. with a metaclass.
Related
Need some help to implement/understand how decorators as a class work in Python. Most examples I've found are either decorating a class, but implementend as a function, or implemented as a class, but decorating a function. My goal is to create decorators implemented as classes and decorate classes.
To be more specific, I want to create a #Logger decorator and use it in some of my classes. What this decorator would do is simply inject a self.logger attribute in the class, so everytime I decorate a class with #Logger I'll be able to self.logger.debug() in its methods.
Some initial questions:
What does the decorator's __init__ receive as parameters? I it would receive only the decorated class and some eventual decorator parameters, and that's actually what happens for most of the cases, but please take a look at the output below for the DOMElementFeatureExtractor. Why does it received all those parameters?
What about the __call__ method? What will it receive?
How can I provide a parameter for the decorator (#Logger(x='y'))? Will it be passed to the __init__ method?
Should I really be returning an instance of the class in the __call__ method? (only way I could make it work)
What about chaining decorators? How would that work if the previous decorator already returned an instance of the class? What should I fix in the example below in order to be able to #Logger #Counter MyClass:?
Please take a look at this example code. I've created some dummy examples, but in the end you can see some code from my real project.
You can find the output at the end.
Any help to understand Python classes decorators implemented as a class would be much appreciated.
Thank you
from abc import ABC, abstractmethod
class ConsoleLogger:
def __init__(self):
pass
def info(self, message):
print(f'INFO {message}')
def warning(self, message):
print(f'WARNING {message}')
def error(self, message):
print(f'ERROR {message}')
def debug(self, message):
print(f'DEBUG {message}')
class Logger(object):
""" Logger decorator, adds a 'logger' attribute to the class """
def __init__(self, cls, *args, **kwargs):
print(cls, *args, **kwargs)
self.cls = cls
def __call__(self, *args, **kwargs):
print(self.cls.__name__)
logger = ConsoleLogger()
setattr(self.cls, 'logger', logger)
return self.cls(*args, **kwargs)
class Counter(object):
""" Counter decorator, counts how many times a class has been instantiated """
count = 0
def __init__(self, cls, *args, **kwargs):
self.cls = cls
def __call__(self, *args, **kwargs):
count += 1
print(f'Class {self.cls} has been initialized {count} times')
return self.cls(*args, **kwargs)
#Logger
class A:
""" Simple class, no inheritance, no arguments in the constructor """
def __init__(self):
self.logger.info('Class A __init__()')
class B:
""" Parent class for B1 """
def __init__(self):
pass
#Logger
class B1(B):
""" Child class, still no arguments in the constructor """
def __init__(self):
super().__init__()
self.logger.info('Class B1 __init__()')
class C(ABC):
""" Abstract class """
def __init__(self):
super().__init__()
#abstractmethod
def do_something(self):
pass
#Logger
class C1(C):
""" Concrete class, implements C """
def __init__(self):
self.logger.info('Class C1 __init__()')
def do_something(self):
self.logger.info('something')
#Logger
class D:
""" Class receives parameter on intantiation """
def __init__(self, color):
self.color = color
self.logger.info('Class D __init__()')
self.logger.debug(f'color = {color}')
class AbstractGenerator(ABC):
def __init__(self):
super().__init__()
self.items = None
self.next_item = None
#abstractmethod
def __iter__(self):
pass
def __next__(self):
pass
def __len__(self):
pass
def __getitem__(self, key):
pass
class AbstractDOMElementExtractor(AbstractGenerator):
def __init__(self, parameters, content):
super().__init__()
self.parameters = parameters
self.content = content
#Logger
class DOMElementExtractor(AbstractDOMElementExtractor):
def __init__(self, parameters, content):
super().__init__(parameters, content)
def __iter__(self):
self.logger.debug('__iter__')
def __next__(self):
self.logger.debug('__next__')
def __len__(self):
self.logger.debug('__len__')
def __getitem__(self, key):
self.logger.debug('__getitem__')
class DOMElementFeatureExtractor(DOMElementExtractor):
def __init__(self, parameters, content):
super().__init__(parameters, content)
class DocumentProcessor:
def __init__(self):
self.dom_element_extractor = DOMElementExtractor(parameters={}, content='')
def process(self):
self.dom_element_extractor.__iter__()
a = A()
b1 = B1()
c1 = C1()
c1.do_something()
d = D(color='Blue')
document_processor = DocumentProcessor()
document_processor.process()
Output:
<class '__main__.A'>
<class '__main__.B1'>
<class '__main__.C1'>
<class '__main__.D'>
<class '__main__.DOMElementExtractor'>
DOMElementFeatureExtractor (<__main__.Logger object at 0x7fae27c26400>,) {'__module__': '__main__', '__qualname__': 'DOMElementFeatureExtractor', '__init__': <function DOMElementFeatureExtractor.__init__ at 0x7fae27c25840>, '__classcell__': <cell at 0x7fae27cf09d8: empty>}
A
INFO Class A __init__()
B1
INFO Class B1 __init__()
C1
INFO Class C1 __init__()
INFO something
D
INFO Class D __init__()
DEBUG color = Blue
DOMElementExtractor
DEBUG __iter__
Won't be a full answer, but I think it's helpful to review the basics of a decorator. This is what decorating looks like:
#Logger
class A:
# A's code
By definition, it's equivalent to doing this:
class A
# A's code
A = Logger(A) # Logger has to be callable because...it's called
Sources often say that decorators "modify", but that's really just the intended use. Technically, all you need is A to have a definition (so a function, method, or class) and Logger to be callable. If Logger returned "Hello, World", that's what A becomes.
Okay, let's pretend we didn't decorate A for a bit and think about what it would take for Logger(A) to be "modifying." Well, A is a class, and you call a class to create instances: A(*args). Therefore, Logger(A)(*args) must also be instances of A. But Logger(A) isn't the class A, it's an instance of Logger. Luckily, you can make instances callable by defining the __call__ method in its class. Logger's __call__ method calls the class stored in its cls attribute and returns the instance.
As for parameters in a decorator, it also helps to think about what it's equivalent to. You're interested in doing this:
#Logger(x='y')
class A:
# A code
So it's equivalent to this:
class A:
# A code
A = Logger(x = 'y')(A)
Note that Logger itself is not taking A as an argument. It's taking 'y' as an argument and returning another callable that takes A as an argument. So if Logger is a class, Logger(x = 'y') would be a Logger instance. Instances of a class can also serve as decorators if the class has a __call__ method!
I'm trying to store specific actions that are defined within a class.
To reduce code duplication, I would like to make use of a mixin class that stores all the actions based on a decorator.
The idea is that it should be straightforward for other people to extend the classes with new actions. I especially want to avoid that these actions are explicitly listed in the source code (this should be handled by the decorator).
This is what I came up with. Unfortunately, in all .actions lists, all the actions from all the classes are listed.
However, I would like to have a solution that only the actions of the specific class are listed.
class ActionMixin:
actions = []
#staticmethod
def action(fun):
ActionMixin.actions.append(fun)
return fun
class Human(ActionMixin):
#ActionMixin.action
def talk(self):
pass
class Dog(ActionMixin):
#ActionMixin.action
def wuff(self):
pass
class Cat(ActionMixin):
#ActionMixin.action
def miau(self):
pass
if __name__ == "__main__":
party = [Human(), Dog()]
possible_actions = [action for memer in party for action in member.actions]
# I would like that possible_actions is now only Human.talk() and Dog.wuff()
# instead it is 2 times all actions
print(len(possible_actions)) # == 6
I would just write my own descriptor here. So:
class Registry:
def __init__(self):
self._registered = []
def __call__(self, func):
self._registered.append(func)
return func
def __get__(self, obj, objtype=None):
return self._registered
class Human:
actions = Registry()
#actions
def talk(self):
pass
class Dog:
actions = Registry()
#actions
def wuff(self):
pass
class Cat:
actions = Registry()
#actions
def miau(self):
pass
So, instead of inheriting from a mixin, just initialize the descriptor object. Then that object itself can be used as the decorator (the __call__ method!).
Note, the decorator would be whatever name you assigned it, and it would be the name of the attribute where the actions are stored.
In the REPL:
In [11]: party = [Human(), Dog()]
In [12]: [action for member in party for action in member.actions]
Out[12]: [<function __main__.Human.talk(self)>, <function __main__.Dog.wuff(self)>]
EDIT:
You would have to change the implementation if you want this to live in a base class. Basically, use a dict to keep track of the registries, unfortunately, we have to rely on the brittle __qualname__ to get the class in __call__:
class ActionsRegistry:
def __init__(self):
self._registry = {}
def __call__(self, func):
klass_name, func_name = func.__qualname__.rsplit('.', 1)
if klass_name not in self._registry:
self._registry[klass_name] = []
self._registry[klass_name].append(func)
return func
def __get__(self, obj, objtype=None):
if obj is None:
return self
return self._registry[objtype.__qualname__]
class Base:
actions = ActionsRegistry()
class Human(Base):
#Base.actions
def talk(self):
pass
class Dog(Base):
#Base.actions
def wuff(self):
pass
class Cat(Base):
#Base.actions
def miau(self):
pass
I want to do something like:
class A(Resource):
#dec(from_file=A.docpath)
def get(self):
pass
class B(A):
docpath = './docs/doc_for_get_b.json'
class C(A):
docpath = './docs/doc_for_get_c.json'
def dec(*args, **kwargs):
def inner(f):
docpath = kwargs.get('from_file')
f.__kwargs__ = open(path, 'r').read()
return f
return inner
The functions that will be called are B.get and C.get, never A.get.
How can I access the custom attribute docpath defined in class B or class C and pass it to the decorator of the get function in class A ?
Current solution: Put the decorator on each derived class ...
class A(Resource):
def _get(self):
pass
class B(A):
#dec(from_file='./docs/doc_for_get_b.json')
def get(self):
return self._get()
class C(A)
#dec(from_file='./docs/doc_for_get_c.json')
def get(self):
return self._get()
This works but it's pretty ugly compared to the one-line declaration of the classes in the previous code.
To access a class's attributes inside the decorator is easy:
def decorator(function):
def inner(self):
self_type = type(self)
# self_type is now the class of the instance of the method that this
# decorator is wrapping
print('The class attribute docpath is %r' % self_type.docpath)
# need to pass self through because at the point function is
# decorated it has not been bound to an instance, and so it is just a
# normal function which takes self as the first argument.
function(self)
return inner
class A:
docpath = "A's docpath"
#decorator
def a_method(self):
print('a_method')
class B(A):
docpath = "B's docpath"
a = A()
a.a_method()
b = B()
b.a_method()
In general I've found using multiple levels of decorators, i.e. decorator factory functions that create decorators such as you've used and such as:
def decorator_factory(**kwargs):
def decorator_function(function):
def wrapper(self):
print('Wrapping function %s with kwargs %s' % (function.__name__, kwargs))
function(self)
return wrapper
return decorator_function
class A:
#decorator_factory(a=2, b=3)
def do_something(self):
print('do_something')
a = A()
a.do_something()
a difficult thing to get right and not easy to comprehend when reading code, so I would err towards using class attributes and generic superclass methods in favour of lots of decorators.
So in your case, don't pass the file path in as an argument to your decorator factory, but set it as a class attribute on your derived classes, and then write a generic method in your superclass that reads the class attribute from the instance's class.
I have a big class which has a lot of functions and attributes.
the instances are created from data in a remote database.
the process of creating each instance is very long and heavy.
In performance sake ive created a bunch class from this heavy class.
so accessing the attributed is easy and works great .
the problem is how to use the methods from that class.
ex :
class clsA():
def __init__(self,obj):
self.attrA=obj.attrA
def someFunc(self):
print self
class bunchClsA(bunch):
def __getattr__(self, attr):
# this is the problem:
try:
#try and return a func
func = clsA.attr
return func
except:
# return simple attribute
return self.attr
Clearly this dosent work , Is there a way i could access the instance function staticly and override the "self" var ?
Found out a nice solution to the problem :
from bunch import Bunch
import types
#Original class:
class A():
y=6
def __init__(self,num):
self.x=num
def funcA(self):
print self.x
#class that wraps A using Bunch(thats what i needed .. u can use another):
class B(Bunch):
def __init__(self, data, cls):
self._cls = cls # notice, not an instance just the class it self
super(B, self).__init__(data)
def __getattr__(self, attr):
# Handles normal Bunch, dict attributes
if attr in self.keys():
return self[attr]
else:
res = getattr(self._cls, attr)
if isinstance(res, types.MethodType):
# returns the class func with self overriden
return types.MethodType(res.im_func, self, type(self))
else:
# returns class attributes like y
return res
data = {'x': 3}
ins_b = B(data, A)
print ins_b.funcA() # returns 3
print ins_b.y # returns 6
And this solves my issue, its a hack and if you have the privileges, redesign the code.
I have a class that is a super-class to many other classes. I would like to know (in the __init__() of my super-class) if the subclass has overridden a specific method.
I tried to accomplish this with a class method, but the results were wrong:
class Super:
def __init__(self):
if self.method == Super.method:
print 'same'
else:
print 'different'
#classmethod
def method(cls):
pass
class Sub1(Super):
def method(self):
print 'hi'
class Sub2(Super):
pass
Super() # should be same
Sub1() # should be different
Sub2() # should be same
>>> same
>>> different
>>> different
Is there any way for a super-class to know if a sub-class has overridden a method?
It seems simplest and sufficient to do this by comparing the common subset of the dictionaries of an instance and the base class itself, e.g.:
def detect_overridden(cls, obj):
common = cls.__dict__.keys() & obj.__class__.__dict__.keys()
diff = [m for m in common if cls.__dict__[m] != obj.__class__.__dict__[m]]
print(diff)
def f1(self):
pass
class Foo:
def __init__(self):
detect_overridden(Foo, self)
def method1(self):
print("Hello foo")
method2=f1
class Bar(Foo):
def method1(self):
print("Hello bar")
method2=f1 # This is pointless but not an override
# def method2(self):
# pass
b=Bar()
f=Foo()
Runs and gives:
['method1']
[]
If you want to check for an overridden instance method in Python 3, you can do this using the type of self:
class Base:
def __init__(self):
if type(self).method == Base.method:
print('same')
else:
print('different')
def method(self):
print('Hello from Base')
class Sub1(Base):
def method(self):
print('Hello from Sub1')
class Sub2(Base):
pass
Now Base() and Sub2() should both print "same" while Sub1() prints "different". The classmethod decorator causes the first parameter to be bound to the type of self, and since the type of a subclass is by definition different to its base class, the two class methods will compare as not equal. By making the method an instance method and using the type of self, you're comparing a plain function against another plain function, and assuming functions (or unbound methods in this case if you're using Python 2) compare equal to themselves (which they do in the C Python implementation), the desired behavior will be produced.
You can use your own decorator. But this is a trick and will only work on classes where you control the implementation.
def override(method):
method.is_overridden = True
return method
class Super:
def __init__(self):
if hasattr(self.method, 'is_overridden'):
print 'different'
else:
print 'same'
#classmethod
def method(cls):
pass
class Sub1(Super):
#override
def method(self):
print 'hi'
class Sub2(Super):
pass
Super() # should be same
Sub1() # should be different
Sub2() # should be same
>>> same
>>> different
>>> same
In reply to answer https://stackoverflow.com/a/9437273/1258307, since I don't have enough credits yet to comment on it, it will not work under python 3 unless you replace im_func with __func__ and will also not work in python 3.4(and most likely onward) since functions no longer have the __func__ attribute, only bound methods.
EDIT: Here's the solution to the original question(which worked on 2.7 and 3.4, and I assume all other version in between):
class Super:
def __init__(self):
if self.method.__code__ is Super.method.__code__:
print('same')
else:
print('different')
#classmethod
def method(cls):
pass
class Sub1(Super):
def method(self):
print('hi')
class Sub2(Super):
pass
Super() # should be same
Sub1() # should be different
Sub2() # should be same
And here's the output:
same
different
same
You can compare whatever is in the class's __dict__ with the function inside the method
you can retrieve from the object -
the "detect_overriden" functionbellow does that - the trick is to pass
the "parent class" for its name, just as one does in a call to "super" -
else it is not easy to retrieve attributes from the parentclass itself
instead of those of the subclass:
# -*- coding: utf-8 -*-
from types import FunctionType
def detect_overriden(cls, obj):
res = []
for key, value in cls.__dict__.items():
if isinstance(value, classmethod):
value = getattr(cls, key).im_func
if isinstance(value, (FunctionType, classmethod)):
meth = getattr(obj, key)
if not meth.im_func is value:
res.append(key)
return res
# Test and example
class A(object):
def __init__(self):
print detect_overriden(A, self)
def a(self): pass
#classmethod
def b(self): pass
def c(self): pass
class B(A):
def a(self): pass
##classmethod
def b(self): pass
edit changed code to work fine with classmethods as well:
if it detects a classmethod on the parent class, extracts the underlying function before proceeding.
--
Another way of doing this, without having to hard code the class name, would be to follow the instance's class ( self.__class__) method resolution order (given by the __mro__ attribute) and search for duplicates of the methods and attributes defined in each class along the inheritance chain.
I'm using the following method to determine if a given bound method is overridden or originates from the parent class
class A():
def bla(self):
print("Original")
class B(A):
def bla(self):
print("Overridden")
class C(A):
pass
def isOverriddenFunc(func):
obj = func.__self__
prntM = getattr(super(type(obj), obj), func.__name__)
return func.__func__ != prntM.__func__
b = B()
c = C()
b.bla()
c.bla()
print(isOverriddenFunc(b.bla))
print(isOverriddenFunc(c.bla))
Result:
Overridden
Original
True
False
Of course, for this to work, the method must be defined in the base class.
You can also check if something is overridden from its parents, without knowing any of the classes involved using super:
class A:
def fuzz(self):
pass
class B(A):
def fuzz(self):
super().fuzz()
class C(A):
pass
>>> b = B(); c = C()
>>> b.__class__.fuzz is super(b.__class__, b).fuzz.__func__
False
>>> c.__class__.fuzz is super(c.__class__, c).fuzz.__func__
True
See this question for some more nuggets of information.
A general function:
def overrides(instance, function_name):
return getattr(instance.__class__, function_name) is not getattr(super(instance.__class__, instance), function_name).__func__
>>> overrides(b, "fuzz")
True
>>> overrides(c, "fuzz")
False
You can check to see if the function has been overridden by seeing if the function handle points to the Super class function or not. The function handler in the subclass object points either to the Super class function or to an overridden function in the Subclass. For example:
class Test:
def myfunc1(self):
pass
def myfunc2(self):
pass
class TestSub(Test):
def myfunc1(self):
print('Hello World')
>>> test = TestSub()
>>> test.myfunc1.__func__ is Test.myfunc1
False
>>> test.myfunc2.__func__ is Test.myfunc2
True
If the function handle does not point to the function in the Super class, then it has been overridden.
Not sure if this is what you're looking for but it helped me when I was looking for a similar solution.
class A:
def fuzz(self):
pass
class B(A):
def fuzz(self):
super().fuzz()
assert 'super' in B.__dict__['fuzz'].__code__.co_names
The top-trending answer and several others use some form of Sub.method == Base.method. However, this comparison can return a false negative if Sub and Base do not share the same import syntax. For example, see discussion here explaining a scenario where issubclass(Sub, Base) -> False.
This subtlety is not apparent when running many of the minimal examples here, but can show up in a more complex code base. The more reliable approach is to compare the method defined in the Sub.__bases__ entry corresponding to Base because __bases__ is guaranteed to use the same import path as Sub
import inspect
def method_overridden(cls, base, method):
"""Determine if class overriddes the implementation of specific base class method
:param type cls: Subclass inheriting (and potentially overriding) the method
:param type base: Base class where the method is inherited from
:param str method: Name of the inherited method
:return bool: Whether ``cls.method != base.method`` regardless of import
syntax used to create the two classes
:raises NameError: If ``base`` is not in the MRO of ``cls``
:raises AttributeError: If ``base.method`` is undefined
"""
# Figure out which base class from the MRO to compare against
base_cls = None
for parent in inspect.getmro(cls):
if parent.__name__ == base.__name__:
base_cls = parent
break
if base_cls is None:
raise NameError(f'{base.__name__} is not in the MRO for {cls}')
# Compare the method implementations
return getattr(cls, method) != getattr(base_cls, method)