I am trying to write a decorator for a method that will call a second method. When I run the code I receive the error:
AttributeError: 'Backoff' object has no attribute 'formatter'
Simplified, the code is:
class Backoff:
def __init__(self, f):
self.f = f
def __call__(self, *args, **kwargs):
n = 1
while n < 11:
try:
return self.f(self, *args, **kwargs)
except FooError as e:
<handle error>
time.sleep((2 ** n) + (random.randint(0, 1000) / 1000))
n = n + 1
class SomeClass:
def __init__(self):
pass
#Backoff
def first_method(self, foo, bar):
return self.formatter(foo, bar)
def formatter(self, x, y):
return some_function_to_format(x, y)
How can I pass the second method to the first method in a way that the decorator can recognise it?
Any help would be amazing!
You are passing a method to Backoff, assigning it as an instance variable, and then calling it. This not bound to an instance until it is being called in Backoff.__call__, where it is then bound to an instance of Backoff which does not have a formatter property.
There might be a straightforward solution to this depending on what your instance method needs access to (i.e. if it only needs to reference a class or static method, it can just call the fully qualified name directly). However, if you need the instance method to reference an instance property, I would suggest not using a class decorator at all. Using a function decorator will not run into these issues, you can create a closure and return a function with the same call signature.
Related
I am new to decorators but ideally I wan to use them to simply define a bunch of class functions within class OptionClass, each representing some particular option with a name and description and if it's required. I don't want to modify the operation of the class function at all if that makes sense, I only want to use the decorator to define name, description, and if it's required.
Problem 1: I construct an OptionClass() and I want to call it's option_1. When I do this I receive a TypeError as the call decorator is not receiving an instance of OptionClass. Why is this? When I call option_1 passing the instance of OptionClass() it works. How do I call option_1 without needing to always pass the instance as self.
The error when received is:
Traceback (most recent call last):
File "D:/OneDrive_P/OneDrive/projects/python/examples/dec_ex.py", line 110, in <module>
print(a.option_1("test")) # TypeError: option1() missing 1 required positional argument: 'test_text'
File "D:/OneDrive_P/OneDrive/projects/python/examples/dec_ex.py", line 80, in __call__
return self.function_ptr(*args, **kwargs)
TypeError: option_1() missing 1 required positional argument: 'test_text'
Problem 2: How would I run or call methods on the decorator to set_name, set_description, set_required?
Problem 3: Although this is a sample I intend to code an option class using async functions and decorate them. Do I need to make the decorator call be async def __call__() or is it fine since it's just returning the function?
class option_decorator(object):
def __init__(self, function_pt):
self.function_ptr = function_pt
self.__required = True
self.__name = ""
self.__description = ""
def set_name(self, text):
self.__name = text
def set_description(self, text):
self.__description = text
def set_required(self,flag:bool):
self.__required = flag
def __bool__(self):
"""returns if required"""
return self.__required
def __call__(self, *args, **kwargs):
return self.function_ptr(*args, **kwargs)
def __str__(self):
"""prints a description and name of the option """
return "{} - {}".format(self.__name, self.__description)
class OptionClass(object):
"""defines a bunch of options"""
#option_decorator
def option_1(self,test_text):
return("option {}".format(test_text))
#option_decorator
def option_2(self):
print("option 2")
def get_all_required(self):
"""would return a list of option functions within the class that have their decorator required flag set to true"""
pass
def get_all_available(self):
"""would return all options regardless of required flag set"""
pass
def print_all_functions(self):
"""would call str(option_1) and print {} - {} for example"""
pass
a = OptionClass()
print(a.option_1("test")) # TypeError: option1() missing 1 required positional argument: 'test_text'
print(a.option_1(a,"test")) #Prints: option test
Problem 1
You implemented the method wrapper as a custom callable instead of as a normal function object. This means that you must implement the __get__() descriptor that transforms a function into a method yourself. (If you had used a function this would already be present.)
from types import MethodType
class Dec:
def __init__(self, f):
self.f = f
def __call__(self, *a, **kw):
return self.f(*a, **kw)
def __get__(self, obj, objtype=None):
return self if obj is None else MethodType(self, obj)
class Foo:
#Dec
def opt1(self, text):
return 'foo' + text
>>> Foo().opt1('two')
'footwo'
See the Descriptor HowTo Guide
Problem 2
The callable option_decorator instance replaces the function in the OptionClass dict. That means that mutating the callable instance affects all instances of OptionClass that use that callable object. Make sure that's what you want to do, because if you want to customize the methods per-instance, you'll have to build this differently.
You could access it in class definition like
class OptionClass(object):
"""defines a bunch of options"""
#option_decorator
def option_1(self,test_text):
return("option {}".format(test_text))
option_1.set_name('foo')
Problem 3
The __call__ method in your example isn't returning a function. It's returning the result of the function_ptr invocation. But that will be a coroutine object if you define your options using async def, which you would have to do anyway if you're using the async/await syntax in the function body. This is similar to the way that yield transforms a function into a function that returns a generator object.
I have this class:
class SomeClass(object):
def __init__(self):
self.cache = {}
def check_cache(method):
def wrapper(self):
if method.__name__ in self.cache:
print('Got it from the cache!')
return self.cache[method.__name__]
print('Got it from the api!')
self.cache[method.__name__] = method(self)
return self.cache[method.__name__]
return wrapper
#check_cache
def expensive_operation(self):
return get_data_from_api()
def get_data_from_api():
"This would call the api."
return 'lots of data'
The idea is that I can use the #check_cache decorator to keep the expensive_operation method from calling an api additional times if the result is already cached.
This works fine, it seems.
>>> sc.expensive_operation()
Got it from the api!
'lots of data'
>>> sc.expensive_operation()
Got it from the cache!
'lots of data'
But I would love to be able to test it with another decorator:
import unittest
class SomeClassTester(SomeClass):
def counted(f):
def wrapped(self, *args, **kwargs):
wrapped.calls += 1
return f(self, *args, **kwargs)
wrapped.calls = 0
return wrapped
#counted
def expensive_operation(self):
return super().expensive_operation()
class TestSomeClass(unittest.TestCase):
def test_api_is_only_called_once(self):
sc = SomeClassTester()
sc.expensive_operation()
self.assertEqual(sc.expensive_operation.calls, 1) # is 1
sc.expensive_operation()
self.assertEqual(sc.expensive_operation.calls, 1) # but this goes to 2
unittest.main()
The problem is that the counted decorator counts the number of times the wrapper function is called, not this inner function.
How do I count that from SomeClassTester?
There's no easy way to do this. Your current test applies the decorators in the wrong order. You want check_cache(counted(expensive_operation)), but you're getting the counted decorator on the outside instead: counted(check_cache(expensive_operation)).
There's no easy way to fix this within the counted decorator, because by the time it gets called, the original function is already wrapped up by the check_cache decorator, and there's no easy way to change the wrapper (it holds its reference to the original function in a closure cell, which is read-only from the outside).
One possible way to make it work is to rebuild the whole method with the decorators in the desired order. You can get a reference to the original method from the closure cell:
class SomeClassTester(SomeClass):
def counted(f):
def wrapped(self, *args, **kwargs):
wrapped.calls += 1
return f(self, *args, **kwargs)
wrapped.calls = 0
return wrapped
expensive_operation = SomeClass.check_cache(
counted(SomeClass.expensive_operation.__closure__[0].cell_value)
)
This is of course far from ideal, since you need to know exactly what decorators are being applied on the method in SomeClass in order to apply them again properly. You also need to know the internals of those decorators so that you can get the right closure cell (the [0] index may not be correct if the other decorator gets changed to differently).
Another (perhaps better) approach might be to change SomeClass in such a way that you can inject your counting code in between the changed method and the expensive bit you want to count. For example, you could have the real expensive part be in _expensive_method_implementation, while the decorated expensive_method is just a simple wrapper that calls it. The test class can override the _implementation method with its own decorated version (which might even skip the actually expensive part and just return dummy data). It doesn't need to override the regular method or mess with its decorators.
It is impossible to do this, without modifying the base class to provide hooks or changing the whole decorated function in derived class based on internal knowledge of base class. Though there is a third way based on internal working of cache decorator, basically change your cache dict so that it counts
class CounterDict(dict):
def __init__(self, *args):
super().__init__(*args)
self.count = {}
def __setitem__(self, key, value):
try:
self.count[key] += 1
except KeyError:
self.count[key] = 1
return super().__setitem__(key, value)
class SomeClassTester(SomeClass):
def __init__(self):
self.cache = CounterDict()
class TestSomeClass(unittest.TestCase):
def test_api_is_only_called_once(self):
sc = SomeClassTester()
sc.expensive_operation()
self.assertEqual(sc.cache.count['expensive_operation'], 1) # is 1
sc.expensive_operation()
self.assertEqual(sc.cache.count['expensive_operation'], 1) # is 1
I have a simple method which accepts a function to call this back later:
def SimpleFunc(parm1):
print(parm1)
class CallMe:
def __init__(self, func):
self.func = func
def Call(self, parm):
self.func(parm)
caller = CallMe(SimpleFunc)
caller.Call("Hallo")
That works fine!
But I want to use a class method and want to call the method on a defined object as callback:
class WithClassMethod:
def __init__( self, val ):
self.val = val
def Func(self, parm):
print( "WithClass: ", self.val, parm )
obj = WithClassMethod(1)
caller = CallMe( ??? )
caller.Call("Next")
How can I bind an object/method pair to a callable object?
Attention: The code from CallMe is not under my control. It comes from a webserver which needs a handler function.
You could simply pass the method object to the class:
called = CallMe(obj.Func)
To expand a bit, instance methods are really just the original class function:
>>> obj.Func.__func__
<function __main__.WithClassMethod.Func>
which, during access on an instance (obj.Func) are transformed via a descriptor (__get__) that attaches self (the instance) to them:
>>> obj.Func.__self__
<__main__.WithClassMethod at 0x7fbe740ce588>
so you can pretty much do anything you want with methods as with functions.
I'm trying to add a decorator that adds callable attributes to functions that return slightly different objects than the return value of the function, but will execute the function at some point.
The problem I'm running into is that when the function object is passed into the decorator, it is unbound and doesn't contain the implicit self argument. When I call the created attribute function (ie. string()), I don't have access to self and can't pass it into the original function.
def deco(func):
"""
Add an attribute to the function takes the same arguments as the
function but modifies the output.
"""
def string(*args, **kwargs):
return str(func(*args, **kwargs))
func.string = string
return func
class Test(object):
def __init__(self, value):
self._value = 1
#deco
def plus(self, n):
return self._value + n
When I go to execute the attribute created by the decorator, this is the error I get, because args doesn't contain the self reference.
>>> t = Test(100)
>>> t.plus(1) # Gets passed self implicitly
101
>>> t.plus.string(1) # Does not get passed self implicitly
...
TypeError: plus() takes exactly 2 arguments (1 given)
Is there a way to create a decorator like this that can get a reference to self? Or is there a way to bind the added attribute function (string()) so that it also gets called with the implicit self argument?
You can use descriptors here:
class deco(object):
def __init__(self, func):
self.func = func
self.parent_obj = None
def __get__(self, obj, type=None):
self.parent_obj = obj
return self
def __call__(self, *args, **kwargs):
return self.func(self.parent_obj, *args, **kwargs)
def string(self, *args, **kwargs):
return str(self(*args, **kwargs))
class Test(object):
def __init__(self, value):
self._value = value
#deco
def plus(self, n):
return self._value + n
so that:
>>> test = Test(3)
>>> test.plus(1)
4
>>> test.plus.string(1)
'4'
This warrants an explanation. deco is a decorator, but it is also a descriptor. A descriptor is an object that defines alternative behavior that is to be invoked when the object is looked up as an attribute of its parent. Interestingly, bounds methods are themselves implemented using the descriptor protocol
That's a mouthful. Let's look at what happens when we run the example code. First, when we define the plus method, we apply the deco decorator. Now normally we see functions as decorators, and the return value of the function is the decorated result. Here we are using a class as a decorator. As a result, Test.plus isn't a function, but rather an instance of the deco type. This instance contains a reference to the plus function that we wish to wrap.
The deco class has a __call__ method that allows instances of it to act like functions. This implementation simply passes the arguments given to the plus function it has a reference to. Note that the first argument will be the reference to the Test instance.
The tricky part comes in implementing test.plus.string(1). To do this, we need a reference to the test instance of which the plus instance is an attribute. To accomplish this, we use the descriptor protocol. That is, we define a __get__ method which will be invoked whenever the deco instance is accessed as an attribute of some parent class instance. When this happens, it stores the parent object inside itself. Then we can simply implement plus.string as a method on the deco class, and use the reference to the parent object stored within the deco instance to get at the test instance to which plus belongs.
This is a lot of magic, so here's a disclaimer: Though this looks cool, it's probably not a great idea to implement something like this.
You need to decorate your function at instantiation time (before creating the instance method). You can do this by overriding the __new__ method:
class Test(object):
def __new__(cls, *args_, **kwargs_):
def deco(func):
def string(*args, **kwargs):
return "my_str is :" + str(func(*args, **kwargs))
# *1
func.__func__.string = string
return func
obj = object.__new__(cls, *args_, **kwargs_)
setattr(obj, 'plus', deco(getattr(obj, 'plus')))
return obj
def __init__(self, value):
self._value = 1
def plus(self, n):
return self._value + n
Demo:
>>> t = Test(100)
>>> t.plus(1)
>>> t.plus.string(5)
>>> 'my_str is :6'
1. Since python doesn't let you access the real instance attribute at setting time you can use __func__ method in order to access the real function object of the instance method.
I have a class where I have multiple methods. I want to use one of the methods as a decorator for other methods. For this I am using following syntax:
#self.action
def execute(self,req):
where action is other method in my class. But it doesn't work and throws exception as
name 'self' is not defined
You cannot use a method of the class while defining it; there is no self within the class nor is the class 'baked' yet to even access any class.
You can treat methods as functions to use as a decorator:
class SomeClass():
def action(func):
# decorate
return wrapper
#action
def execute(self, req):
# something
If action is defined on a base class, then you'd have to refer to the name via the base class:
class Base():
#staticmethod
def action(func):
# decorate
return wrapper
class Derived(Base):
#Base.action
def execute(self, req):
# something
For Python 2, you'd have to make action a static method here, as otherwise you get an unbound method that'll complain you cannot call it without an instance as the first argument. In Python 3, you can leave off the #staticmethod decorator there, at least for the purposes of the decorator.
But note that action cannot then be used as a method directly; perhaps it should not be part of the class at all at that point. It is not part of the end-user API here, presumably the decorator is not used by consumers of the instances of these classes.
Just beware that both the decorator and the decorated function are unbound methods, so you can only access the self (or cls for classmethods) in the inner scope of the decorator, and must manually bind the decorated method to the instance bound in the inner decorator.
class A:
x = 5
y = 6
def decorate(unbound):
def _decorator(self):
bound = unbound.__get__(self)
return bound() * self.x
return _decorator
#decorate
def func(self):
return self.y
A().func() # 30!!
Still trying to wrap my head around how decorators could be inherited and overridden.
Beware that for the decorator to work it can't be bound to an instance. That is: there is no way to make this work
a = A()
#a.decorate
def func(*args):
return 1
Despite this pattern is much more common than the asked here.
At this point the question raises: is it a method at all or just code that you happen to hide in a class?
The only way to prevent the decorator being wrongfully bound is to declare it as a staticmethod, but then it must be in a previous super class because to be used it must be bound to the static class reference which would not be yet defined, just as the self.
class A:
x = 1
#staticmethod
def decorate(unbound):
def _decorator(self):
bound = unbound.__get__(self)
return bound() * self.x
return _decorator
class B(A):
#A.decorate
def func(self):
return 1
class C():
x = 2
#B.decorate
def func(self):
return 1
a = A()
class D():
x = 3
#a.decorate
def func(self):
return 1
B().func() # 1
C().func() # 2
D().func() # 3
But as you can see, there is no way for the decorator to use the state of its own class. class A from this last example just happens to be a mixin with a default x variable and an "unrelated" static decorator.
So, again, is it a method?
To overcome all of this, you can bind the staticmethod in your same class to an arbitrary type. Namely, the builtin type will do.
class A:
x = 1
#staticmethod
def decorate(unbound):
def _decorator(self):
bound = unbound.__get__(self)
return bound() * self.x
return _decorator
#decorate.__get__(type)
def func(self):
return 1
class B:
x = 2
#A.decorate
def func(self):
return 1
class C:
x = 3
#(A().decorate) # Only for Python 3.9+, see PEP-614
def func(self):
return 1
A().func() # 1
B().func() # 2
C().func() # 3
But this features too much magic for my taste. And still not a method for my gut.
In python "self" is passed to instance methods as an argument (the first), "self" is just a convention is possible to call it "foobarbaz" (of course it would be silly)… the point is that, from the outside "self" is not defined (because its scope is the method)… you can't decorate class methods with other class methods, instead you have to write a separate class!