I'm trying to add a decorator that adds callable attributes to functions that return slightly different objects than the return value of the function, but will execute the function at some point.
The problem I'm running into is that when the function object is passed into the decorator, it is unbound and doesn't contain the implicit self argument. When I call the created attribute function (ie. string()), I don't have access to self and can't pass it into the original function.
def deco(func):
"""
Add an attribute to the function takes the same arguments as the
function but modifies the output.
"""
def string(*args, **kwargs):
return str(func(*args, **kwargs))
func.string = string
return func
class Test(object):
def __init__(self, value):
self._value = 1
#deco
def plus(self, n):
return self._value + n
When I go to execute the attribute created by the decorator, this is the error I get, because args doesn't contain the self reference.
>>> t = Test(100)
>>> t.plus(1) # Gets passed self implicitly
101
>>> t.plus.string(1) # Does not get passed self implicitly
...
TypeError: plus() takes exactly 2 arguments (1 given)
Is there a way to create a decorator like this that can get a reference to self? Or is there a way to bind the added attribute function (string()) so that it also gets called with the implicit self argument?
You can use descriptors here:
class deco(object):
def __init__(self, func):
self.func = func
self.parent_obj = None
def __get__(self, obj, type=None):
self.parent_obj = obj
return self
def __call__(self, *args, **kwargs):
return self.func(self.parent_obj, *args, **kwargs)
def string(self, *args, **kwargs):
return str(self(*args, **kwargs))
class Test(object):
def __init__(self, value):
self._value = value
#deco
def plus(self, n):
return self._value + n
so that:
>>> test = Test(3)
>>> test.plus(1)
4
>>> test.plus.string(1)
'4'
This warrants an explanation. deco is a decorator, but it is also a descriptor. A descriptor is an object that defines alternative behavior that is to be invoked when the object is looked up as an attribute of its parent. Interestingly, bounds methods are themselves implemented using the descriptor protocol
That's a mouthful. Let's look at what happens when we run the example code. First, when we define the plus method, we apply the deco decorator. Now normally we see functions as decorators, and the return value of the function is the decorated result. Here we are using a class as a decorator. As a result, Test.plus isn't a function, but rather an instance of the deco type. This instance contains a reference to the plus function that we wish to wrap.
The deco class has a __call__ method that allows instances of it to act like functions. This implementation simply passes the arguments given to the plus function it has a reference to. Note that the first argument will be the reference to the Test instance.
The tricky part comes in implementing test.plus.string(1). To do this, we need a reference to the test instance of which the plus instance is an attribute. To accomplish this, we use the descriptor protocol. That is, we define a __get__ method which will be invoked whenever the deco instance is accessed as an attribute of some parent class instance. When this happens, it stores the parent object inside itself. Then we can simply implement plus.string as a method on the deco class, and use the reference to the parent object stored within the deco instance to get at the test instance to which plus belongs.
This is a lot of magic, so here's a disclaimer: Though this looks cool, it's probably not a great idea to implement something like this.
You need to decorate your function at instantiation time (before creating the instance method). You can do this by overriding the __new__ method:
class Test(object):
def __new__(cls, *args_, **kwargs_):
def deco(func):
def string(*args, **kwargs):
return "my_str is :" + str(func(*args, **kwargs))
# *1
func.__func__.string = string
return func
obj = object.__new__(cls, *args_, **kwargs_)
setattr(obj, 'plus', deco(getattr(obj, 'plus')))
return obj
def __init__(self, value):
self._value = 1
def plus(self, n):
return self._value + n
Demo:
>>> t = Test(100)
>>> t.plus(1)
>>> t.plus.string(5)
>>> 'my_str is :6'
1. Since python doesn't let you access the real instance attribute at setting time you can use __func__ method in order to access the real function object of the instance method.
Related
Is there a way to access a class (where function is defined as a method) before there is an instance of that class?
class MyClass:
def method(self):
print("Calling me")
m1 = MyClass.method
instance = MyClass()
m2 = instance.method
print(m2.__self__.__class__) # <class 'MyClass'>
# how to access `MyClass` from `m1`?
For example I have m1 variable somewhere in my code and want to have a reference to MyClass the same way I can access it from bound method m2.__self__.__class__.
print(m1.__qualname__) # 'MyClass.method'
The only option I was able to find is __qualname__ which is a string containing name of the class.
The attribute __self__ itself is annotated by Python when the function is bound to an instance and become a method. (The code to that is run somewhere when running the __get__ code in the function, but passing an instance different than None).
So, as people pointed out, you have the option of getting the classname as a string by going through __qualname__. Otherwise, if the functions/methods for which you will need this feature are known beforehand, it is possible to create a decorator that will annotate their class when they are retrieved as a class attribute (in contrast to the native annotation which only takes place when retrieving then as an instance attribute):
class unboundmethod:
def __init__(self, func, cls):
self.__func__ = func
self.class_ = cls
self.__self__ = None
def __call__(self, instance, *args, **kw):
if not isinstance(instance, self.class_):
# This check is actually optional fancy stuff, since we are here! :-)
raise TypeError(f"First parameter fo {self.__func__.__name__} must be an instance of {self.class_}")
return self.__func__(instance, *args, **kw)
def __repr__(self):
return f"Unbound method {self.__func__!r} related to {self.class_}"
class clsbind:
def __init__(self, func):
self.func = func
def __get__(self, instance, owner):
if instance is None:
# the function is being retrieved from the class:
return unboundmethod(self.func, owner)
# return control to usual method creation codepath:
return self.func.__get__(instance, owner)
class MyClass:
#clsbind
def method(self):
print("Calling me")
And on the REPL you can have this:
In [136]: m1 = MyClass.method
In [137]: m1.class_
Out[137]: __main__.MyClass
In [138]: m1(MyClass())
Calling me
You can get the class instance using the __qualname__
my_class = eval(m1.__qualname__.split('.')[-2])
print(my_class)
Not the most generic and safest approach, but should work for this simple scenario.
I understand what I am asking here is probably not the best code design, but the reason for me asking is strictly academic. I am trying to understand how to make this concept work.
Typically, I will return self from a class method so that the following methods can be chained together. My understanding is by returning self, I am simply returning an instance of the class, for the following methods to work on.
But in this case, I am trying to figure out how to return both self and another value from the method. The idea is if I do not want to chain, or I do not call any class attributes, I want to retrieve the data from the method being called.
Consider this example:
class Test(object):
def __init__(self):
self.hold = None
def methoda(self):
self.hold = 'lol'
return self, 'lol'
def newmethod(self):
self.hold = self.hold * 2
return self, 2
t = Test()
t.methoda().newmethod()
print(t.hold)
In this case, I will get an AttributeError: 'tuple' object has no attribute 'newmethod' which is to be expected because the methoda method is returning a tuple which does not have any methods or attributes called newmethod.
My question is not about unpacking multiple returns, but more about how can I continue to chain methods when the preceding methods are returning multiple values. I also understand that I can control the methods return with an argument to it, but that is not what I am trying to do.
As mentioned previously, I do realize this is probably a bad question, and I am happy to delete the post if the question doesnt make any sense.
Following the suggestion by #JohnColeman, you can return a special tuple with attribute lookup delegated to your object if it is not a normal tuple attribute. That way it acts like a normal tuple except when you are chaining methods.
You can implement this as follows:
class ChainResult(tuple):
def __new__(cls, *args):
return super(ChainResult, cls).__new__(cls, args)
def __getattribute__(self, name):
try:
return getattr(super(), name)
except AttributeError:
return getattr(super().__getitem__(0), name)
class Test(object):
def __init__(self):
self.hold = None
def methoda(self):
self.hold = 'lol'
return ChainResult(self, 'lol')
def newmethod(self):
self.hold = self.hold * 2
return ChainResult(self, 2)
Testing:
>>> t = Test()
>>> t.methoda().newmethod()
>>> print(t.hold)
lollol
The returned result does indeed act as a tuple:
>>> t, res = t.methoda().newmethod()
>>> print(res)
2
>>> print(isinstance(t.methoda().newmethod(), tuple))
True
You could imagine all sorts of semantics with this, such as forwarding the returned values to the next method in the chain using closure:
class ChainResult(tuple):
def __new__(cls, *args):
return super(ChainResult, cls).__new__(cls, args)
def __getattribute__(self, name):
try:
return getattr(super(), name)
except AttributeError:
attr = getattr(super().__getitem__(0), name)
if callable(attr):
chain_results = super().__getitem__(slice(1, None))
return lambda *args, **kw: attr(*(chain_results+args), **kw)
else:
return attr
For example,
class Test:
...
def methodb(self, *args):
print(*args)
would produce
>>> t = Test()
>>> t.methoda().methodb('catz')
lol catz
It would be nice if you could make ChainResults invisible. You can almost do it by initializing the tuple base class with the normal results and saving your object in a separate attribute used only for chaining. Then use a class decorator that wraps every method with ChainResults(self, self.method(*args, **kw)). It will work okay for methods that return a tuple but a single value return will act like a length 1 tuple, so you will need something like obj.method()[0] or result, = obj.method() to work with it. I played a bit with delegating to tuple for a multiple return or to the value itself for a single return; maybe it could be made to work but it introduces so many ambiguities that I doubt it could work well.
I am new to decorators but ideally I wan to use them to simply define a bunch of class functions within class OptionClass, each representing some particular option with a name and description and if it's required. I don't want to modify the operation of the class function at all if that makes sense, I only want to use the decorator to define name, description, and if it's required.
Problem 1: I construct an OptionClass() and I want to call it's option_1. When I do this I receive a TypeError as the call decorator is not receiving an instance of OptionClass. Why is this? When I call option_1 passing the instance of OptionClass() it works. How do I call option_1 without needing to always pass the instance as self.
The error when received is:
Traceback (most recent call last):
File "D:/OneDrive_P/OneDrive/projects/python/examples/dec_ex.py", line 110, in <module>
print(a.option_1("test")) # TypeError: option1() missing 1 required positional argument: 'test_text'
File "D:/OneDrive_P/OneDrive/projects/python/examples/dec_ex.py", line 80, in __call__
return self.function_ptr(*args, **kwargs)
TypeError: option_1() missing 1 required positional argument: 'test_text'
Problem 2: How would I run or call methods on the decorator to set_name, set_description, set_required?
Problem 3: Although this is a sample I intend to code an option class using async functions and decorate them. Do I need to make the decorator call be async def __call__() or is it fine since it's just returning the function?
class option_decorator(object):
def __init__(self, function_pt):
self.function_ptr = function_pt
self.__required = True
self.__name = ""
self.__description = ""
def set_name(self, text):
self.__name = text
def set_description(self, text):
self.__description = text
def set_required(self,flag:bool):
self.__required = flag
def __bool__(self):
"""returns if required"""
return self.__required
def __call__(self, *args, **kwargs):
return self.function_ptr(*args, **kwargs)
def __str__(self):
"""prints a description and name of the option """
return "{} - {}".format(self.__name, self.__description)
class OptionClass(object):
"""defines a bunch of options"""
#option_decorator
def option_1(self,test_text):
return("option {}".format(test_text))
#option_decorator
def option_2(self):
print("option 2")
def get_all_required(self):
"""would return a list of option functions within the class that have their decorator required flag set to true"""
pass
def get_all_available(self):
"""would return all options regardless of required flag set"""
pass
def print_all_functions(self):
"""would call str(option_1) and print {} - {} for example"""
pass
a = OptionClass()
print(a.option_1("test")) # TypeError: option1() missing 1 required positional argument: 'test_text'
print(a.option_1(a,"test")) #Prints: option test
Problem 1
You implemented the method wrapper as a custom callable instead of as a normal function object. This means that you must implement the __get__() descriptor that transforms a function into a method yourself. (If you had used a function this would already be present.)
from types import MethodType
class Dec:
def __init__(self, f):
self.f = f
def __call__(self, *a, **kw):
return self.f(*a, **kw)
def __get__(self, obj, objtype=None):
return self if obj is None else MethodType(self, obj)
class Foo:
#Dec
def opt1(self, text):
return 'foo' + text
>>> Foo().opt1('two')
'footwo'
See the Descriptor HowTo Guide
Problem 2
The callable option_decorator instance replaces the function in the OptionClass dict. That means that mutating the callable instance affects all instances of OptionClass that use that callable object. Make sure that's what you want to do, because if you want to customize the methods per-instance, you'll have to build this differently.
You could access it in class definition like
class OptionClass(object):
"""defines a bunch of options"""
#option_decorator
def option_1(self,test_text):
return("option {}".format(test_text))
option_1.set_name('foo')
Problem 3
The __call__ method in your example isn't returning a function. It's returning the result of the function_ptr invocation. But that will be a coroutine object if you define your options using async def, which you would have to do anyway if you're using the async/await syntax in the function body. This is similar to the way that yield transforms a function into a function that returns a generator object.
I am trying to write a decorator for a method that will call a second method. When I run the code I receive the error:
AttributeError: 'Backoff' object has no attribute 'formatter'
Simplified, the code is:
class Backoff:
def __init__(self, f):
self.f = f
def __call__(self, *args, **kwargs):
n = 1
while n < 11:
try:
return self.f(self, *args, **kwargs)
except FooError as e:
<handle error>
time.sleep((2 ** n) + (random.randint(0, 1000) / 1000))
n = n + 1
class SomeClass:
def __init__(self):
pass
#Backoff
def first_method(self, foo, bar):
return self.formatter(foo, bar)
def formatter(self, x, y):
return some_function_to_format(x, y)
How can I pass the second method to the first method in a way that the decorator can recognise it?
Any help would be amazing!
You are passing a method to Backoff, assigning it as an instance variable, and then calling it. This not bound to an instance until it is being called in Backoff.__call__, where it is then bound to an instance of Backoff which does not have a formatter property.
There might be a straightforward solution to this depending on what your instance method needs access to (i.e. if it only needs to reference a class or static method, it can just call the fully qualified name directly). However, if you need the instance method to reference an instance property, I would suggest not using a class decorator at all. Using a function decorator will not run into these issues, you can create a closure and return a function with the same call signature.
I'm attempting to implement a decorator on certain methods in a class so that if the value has NOT been calculated yet, the method will calculate the value, otherwise it will just return the precomputed value, which is stored in an instance defaultdict. I can't seem to figure out how to access the instance defaultdict from inside of a decorator declared outside of the class. Any ideas on how to implement this?
Here are the imports (for a working example):
from collections import defaultdict
from math import sqrt
Here is my decorator:
class CalcOrPass:
def __init__(self, func):
self.f = func
#if the value is already in the instance dict from SimpleData,
#don't recalculate the values, instead return the value from the dict
def __call__(self, *args, **kwargs):
# can't figure out how to access/pass dict_from_SimpleData to here :(
res = dict_from_SimpleData[self.f.__name__]
if not res:
res = self.f(*args, **kwargs)
dict_from_SimpleData[self.f__name__] = res
return res
And here's the SimpleData class with decorated methods:
class SimpleData:
def __init__(self, data):
self.data = data
self.stats = defaultdict() #here's the dict I'm trying to access
#CalcOrPass
def mean(self):
return sum(self.data)/float(len(self.data))
#CalcOrPass
def se(self):
return [i - self.mean() for i in self.data]
#CalcOrPass
def variance(self):
return sum(i**2 for i in self.se()) / float(len(self.data) - 1)
#CalcOrPass
def stdev(self):
return sqrt(self.variance())
So far, I've tried declaring the decorator inside of SimpleData, trying to pass multiple arguments with the decorator(apparently you can't do this), and spinning around in my swivel chair while trying to toss paper airplanes into my scorpion tank. Any help would be appreciated!
The way you define your decorator the target object information is lost. Use a function wrapper instead:
def CalcOrPass(func):
#wraps(func)
def result(self, *args, **kwargs):
res = self.stats[func.__name__]
if not res:
res = func(self, *args, **kwargs)
self.stats[func.__name__] = res
return res
return result
wraps is from functools and not strictly necessary here, but very convenient.
Side note: defaultdict takes a factory function argument:
defaultdict(lambda: None)
But since you're testing for the existence of the key anyway, you should prefer a simple dict.
You can't do what you want when your function is defined, because it is unbound. Here's a way to achieve it in a generic fashion at runtime:
class CalcOrPass(object):
def __init__(self, func):
self.f = func
def __get__(self, obj, type=None): # Cheat.
return self.__class__(self.f.__get__(obj, type))
#if the value is already in the instance dict from SimpleData,
#don't recalculate the values, instead return the value from the dict
def __call__(self, *args, **kwargs):
# I'll concede that this doesn't look very pretty.
# TODO handle KeyError here
res = self.f.__self__.stats[self.f.__name__]
if not res:
res = self.f(*args, **kwargs)
self.f.__self__.stats[self.f__name__] = res
return res
A short explanation:
Our decorator defines __get__ (and is hence said to be a descriptor). Whereas the default behaviour for an attribute access is to get it from the object's dictionary, if the descriptor method is defined, Python will call that instead.
The case with objects is that object.__getattribute__ transforms an access like b.x into type(b).__dict__['x'].__get__(b, type(b))
This way we can access the bound class and its type from the descriptor's parameters.
Then we create a new CalcOrPass object which now decorates (wraps) a bound method instead of the old unbound function.
Note the new style class definition. I'm not sure if this will work with old-style classes, as I haven't tried it; just don't use those. :) This will work for both functions and methods, however.
What happens to the "old" decorated functions is left as an exercise.