I have a memoization decorator that looks like this:
def memoize(obj):
from functools import wraps
cache = {}
#wraps(obj)
def memoizer(*args, **kwargs):
if args not in cache:
cache[args] = obj(*args, **kwargs)
return cache[args]
return memoizer
However, I'm not sure this function works properly because it seems like to me it recreates cache as an empty dictionary every time the decorated function is called. When I test it with a simple fibonacci function, it does appear to memoize correctly. So is cache not recreated every time?
The python wiki has a version that includes this line:
cache = obj.cache = {}
So I am not sure what this does. I guess that python functions are objects so it is creating a new attribute associated with the function and is publically available/usable every time the function is called.
In either version, if I make repeated calls to the function, as in a recursive definition or just through repeated calls, how is the cache treated? Is it associated with the function, does it become a "global" variable, or something else?
1) A new cache is created for each call to memoize(). But memoize() is called only once for each decorated function. Thus each decorated function has its own cache.
2) Concerning cache = obj.cache = {}:
Functions are also objects in Python. And as you already assumed, obj.cache = {} will create a new attribute on the wrapped function. The local cache object and the cache attribute on the wrapped function will point at the same dictionary.
This does not man that cache is a global variable - it's local in the memoize function and it's a attribute on the function.
If you have a decorated function f you could access f.cache on a global scope, if that's what you mean with a global variable.
If you call the decorated function repeatedly, it will always access the same cache specific to the decorated function.
To work around inner functions not beeing "memoized".
def outer():
# do not add the decorator here!
def inner():
return 5
if(not hasattr(outer, "inner")):
# the "#" decorator is only syntactical sugar,
# we can simply call the decorator function to wrap another function
outer.inner = memoize( inner )
outer.inner()
return outer.inner()
outer()
outer()
Related
The following code is from Matplotlib's __init__ file:
def _logged_cached(fmt, func=None):
"""
Decorator that logs a function's return value, and memoizes that value.
After ::
#_logged_cached(fmt)
def func(): ...
the first call to *func* will log its return value at the DEBUG level using
%-format string *fmt*, and memoize it; later calls to *func* will directly
return that value.
"""
if func is None: # Return the actual decorator.
return functools.partial(_logged_cached, fmt)
called = False
ret = None
#functools.wraps(func)
def wrapper(**kwargs):
nonlocal called, ret
if not called:
ret = func(**kwargs)
called = True
_log.debug(fmt, ret)
return ret
return wrapper
Question: How does the caching in the above function work? Won't the local variables of _logged_cached be reinitialised on each call?
As I understand it, the local variables are deleted after the function returns (right?). If so, then the caching won't work. Yet, the documentation says the decorated function will return the same object on each call -- and it does. How, though?
The reason the caching works is because of a concept called closures. Your understanding is correct that local variables are deleted after a function exits but the variables used here for caching aren't local to the inner function.
If you use the _logged_cached decorator on a function, the function will essentially be replaced by the nested function wrapper. This wrapper function when returned becomes a closure (a function object) which will remember the non-local variables in the enclosing scope. So it will essentially have a mapping that remembers the variables called and ret. Each time you call the function object (the decorated function) the inner scope will have called and ret available to use.
Normally they would only be usable as read-only in the inner function but by using the keyword nonlocal, the inner function is actually able to change/update the values of the variables called and ret in the closure. Thus that mapping we mentioned before saved in the function object gets its values updated and each subsequential call to the decorated function will make use of those same values.
I can't figure out how to do this, and frankly, I don't know if it's possible.
I want to write a decorator that changes the way a function is called. It's easiest to see with example code:
def my_print(*args, **kwargs):
print(args[0].upper())
#reroute_decorator('print', my_print)
def my_func():
print('normally this print function is just a print function...')
print('but since my_func is decorated with a special reroute_decorator...')
print('it is replaced with a different function, and its args sent there.')
my_func()
# NORMALLY THIS PRINT FUNCTION IS JUST A PRINT FUNCTION...
# BUT SINCE MY_FUNC IS DECORATED WITH A SPECIAL REROUTE_DECORATOR...
# IT IS REPLACED WITH A DIFFERENT FUNCTION, AND ITS ARGS SENT THERE.
Is a decorator with this kind of functionality even possible in python?
Now, I don't really need this if it's too complex, I just can't figure out how to do it in a simple way.
Is this kind of a problem trivial? Or is it really complex?
You can create a new function with an updated globals dictionary so that to that function it appears that the global was bound to the desired value.
Note that this is weaker than actual dynamic scope as any functions called by the function will see the original bindings and not the modified one.
See namespaced_function referenced in How does Python's types.FunctionType create dynamic Functions?
To elaborate on #Dan D.'s answer, you would create a new function object to replace the original, something like this:
from types import FunctionType
def reroute_decorator(**kwargs):
def actual_decorator(func):
globals = func.__globals__.copy()
globals.update(kwargs)
new_func = FunctionType(
func.__code__, globals, name=func.__name__,
argdefs=func.__defaults__, closure=func.__closure__)
new_func.__dict__.update(func.__dict__)
return new_func
return actual_decorator
The only catch here is that the updated function object is the only one that will see whatever kwargs you passed in, since they will be spoofed into globals. Additionally, any modifications you make to the module after calling the decorator function will not be visible to the decorated function, but that should not be an issue. You can go a layer deeper and create a proxy dictionary that would allow you to interact normally with the original, except for keys you explicitly defined, like print, but that's a bit out of scope here.
I've updated your print implementation to be a bit more general, and made the input to the decorator function more pythonic (less MATLABy):
def my_print(*args, **kwargs):
print(*(str(x).upper() for x in args), **kwargs)
#reroute_decorator(print=my_print)
def my_func():
print('normally this print function is just a print function...')
print('but since my_func is decorated with a special reroute_decorator...')
print('it is replaced with a different function, and its args sent there.')
Which results in:
>>> my_func()
NORMALLY THIS PRINT FUNCTION IS JUST A PRINT FUNCTION...
BUT SINCE MY_FUNC IS DECORATED WITH A SPECIAL REROUTE_DECORATOR...
IT IS REPLACED WITH A DIFFERENT FUNCTION, AND ITS ARGS SENT THERE.
Can you explain me how the following decorator works:
def set_ev_cls(ev_cls, dispatchers=None):
def _set_ev_cls_dec(handler):
if 'callers' not in dir(handler):
handler.callers = {}
for e in _listify(ev_cls):
handler.callers[e] = _Caller(_listify(dispatchers), e.__module__)
return handler
return _set_ev_cls_dec
#set_ev_cls(ofp_event.EventOFPSwitchFeatures, CONFIG_DISPATCHER)
def _switch_features_handler(self, ev):
datapath = ev.msg.datapath
....
Please, don't go into details on what's going on inside the function. I'm interested in how the decorator with parameters wrap methods here. By the way, it's a code snippet from Ryu (event registration mechanism).
Thank you in advance
First, a decorator is just a function that gets called with a function. In particular, the following are (almost) the same thing:
#spam
def eggs(arg): pass
def eggs(arg): pass
eggs = spam(eggs)
So, what happens when the decorator takes parameters? Same thing:
#spam(arg2)
def eggs(arg): pass
def eggs(arg): pass
eggs = spam(arg2)(eggs)
Now, notice that the function _set_ev_cls_dec, which is ultimately returned and used in place of _switch_features_handler, is a local function, defined inside the decorator. That means it can be a closure over variables from the outer function—including the parameters of the outer function. So, it can use the handler argument at call time, plus the ev_cls and dispatchers arguments that it got at decoration time.
So:
set_ev_cls_dev creates a local function and returns a closure around its ev_cls and dispatchers arguments, and returns that function.
That closure gets called with _switch_features_handler as its parameter, and it modifies and returns that parameter by adding a callers attribute, which is a dict of _Caller objects built from that closed-over dispatchers parameter and keyed off that closed-over ev_cls parameter.
Explain how it works without detailing what's going on inside? That kind of sounds like "explain without explaining," but here's a rough walkthrough:
Think of set_ev_cls as a factory for decorators. It's there to catch the arguments at the time the decorator is invoked:
#set_ev_cls(ofp_event.EventOFPSwitchFeatures, CONFIG_DISPATCHER)
And return a function, _set_ev_cls_dec that has its variables bound to:
ev_cls = ofp_event.EventOFPSwitchFeatures
dispatchers = CONFIG_DISPATCHER
Or put another way, you now have a 'customized' or 'parametrized' dispatcher that's logically equivalent to:
def custom_decorator(handler):
if 'callers' not in dir(handler):
handler.callers = {}
for e in _listify(ofp_event.EventOFPSwitchFeatures):
handler.callers[e] = _Caller(_listify(CONFIG_DISPATCHER), e.__module__)
return handler
(If you captured the values of ofp_event.EventOFPSwitchFeatures and CONFIG_DISPATCHER at the moment the #set_ev_cls(...) was called).
The custom_decorator of step 1 is applied to _switch_features_handleras a more traditional unparameterized decorator.
I thought I would be clever and write a wrapper that called the session variables (many are present) and add that to the (django) views requiring session variables. However I seem to not be understanding the scope of the variables, or am writing this incorrectly.
The wrapper I have is:
def s_vars(func_to_decorate):
#wraps(func_to_decorate)
def wrapper(request, *args, **kwargs):
#add all the session variables to kwargs and accessible for the decorated function.
user_obj = request.user
datepicker = request.session['datepicker']
date_format = request.session['date_format']
.........
country = request.session['country']
metric = request.session['metric']
qrydtm = request.session.get("qrydtm",date.today())
result = func_to_decorate(request, *args, **kwargs)
#any post view checks to be done go here
#return to the function to be decorated.
return result
return wrapper
Then for the view I have something like:
#s_vars
def main(request, template_name='placeholder.html'):
return render_to_response(template_name, RequestContext(request,{
'user':user_obj
}))
But this leads to the error that user_obj is not accessible inside the method "main". My understanding was that this is an inner function and therefore the variables in the list under the "wrapper" method would be accessible to this inner function "main". What am I missing here?
The syntax
#spam
def ham():
pass
is precisely equivalent to the syntax
def ham():
pass
ham = spam(ham)
Does that clarify why what you are doing doesn't work?
If you want to pass stuffto a function from a decorator, the usual idiom is to send extra arguments to the function. This can be a little icky, because it means that the argspec that looks right is actually not.
Nested functions only take scoped variables from the scope where they are defined, and the binding takes place at compile time.
You cannot add scoped variables later on, and certainly not with a simple wrapper.
At the point where the inner function is called, it is just called by the outer scope function, not defined in it.
That distinction (also made by interpreter vs runtime) is definitely important in scoping. Take a look at the dis (disassembly) of the s_vars wrapper (or a reduced simple example of the same behaviour). The code is not reinterpreted for different values (it is just a value here) of func_to_decorate.
If you want to make a list of variables available to the inner function, perhaps an object passed in would make more sense. The wrapper could ensure that the external API is without it.
Python doesn't support complicated anonymous functions. What's a good alternative? For example:
class Calculation:
def __init__(self, func):
self.func = func
def __call__(self, data):
try:
# check if the value has already been calculated
# if it has, it would be cached under key = self.func
return data[self.func]
except KeyError:
pass # first-time call; calculate and cache the values
data[self.func] = self.func(data)
return data[self.func]
# with a simple function, which can be represented using lambda, this works great
f1 = Calculation(lambda data : data['a'] * data['b'])
# with a complicated function, I can do this:
def f2_aux:
# some complicated calculation, which isn't suitable for a lambda one-liner
f2 = Calculation(f2_aux)
Is this a reasonable design to begin with?
If so, is there a way to avoid the ugliness of f*_aux for each f* that I define in the module?
UPDATE:
Example of use:
d = {'a' : 3, 'b' : 6}
# computes 3 * 6
# stores 18 in d under a key <function <lambda> at ...>
# returns 18
f1(d)
# retrieves 18 from d[<function <lambda> at ...>]
# returns 18, without having to recalculate it
f1(d)
UPDATE:
Just for my understanding, I added a version that uses the inner function.
def memoize(func):
def new_func(data):
try:
# check if the value has already been calculated
# if it has, it would be cached under key = self.func
return data[func]
except KeyError:
pass # first-time call; calculate and cache the values
data[func] = func(data)
return data[func]
return new_func
#memoize
def f1(data):
return data['a'] * data['b']
You don't need anonymous functions. Also, memoization has been done better than this, there's probably no reason for you to roll your own.
But to answer the question: You can use your class as a decorator.
#Calculation
def f2():
...
This simply defined the function, wraps it in Calculation and stored the result of that as f2.
The decorator syntax is defined to be equivalent to:
_decorator = Calculation # a fresh identifier
# not needed here, but in other cases (think properties) it's useful
def f2():
...
f2 = _decorator(f2)
The alternative to an anonymous function is a non-anonymous function. An anonymous function is only anonymous in the context where it was defined. But it is not truly anonymous, because then you could not use it.
In Python you make anonymous functions with the lambda statement. You can for example do this:
output = mysort(input, lambda x: x.lastname)
The lambda will create a function, but that function has no name in the local space, and it's own name for itself is just '<lambda>'. But if we look at mysort, it would have to be defined something like this:
def mysort(input, getterfunc):
blahblahblah
As we see here, in this context the function isn't anonymous at all. It has a name, getterfunc. From the viewpoint of this function it does not matter if the function passed in are anonymous or not. This works just as well, and is exactly equivalent in all significant ways:
def get_lastname(x):
return x.lastname
output = mysort(input, get_lastname)
Sure, it uses more code, but it is not slower or anything like that. In Python, therefore anonymous functions are nothing but syntactic sugar for ordinary functions.
A truly anonymous function would be
lambda x: x.lastname
But as we don't assign the resulting function to anything, we do not get a name for the function, and then we can't use it. All truly anonymous functions are unusable.
For that reason, if you need a function that can't be a lambda, make it an ordinary function. It can never be anonymous in any meaningful way, so why bother making it anonymous at all? Lambdas are useful when you want a small one-line function and you don't want to waste space by defining a full function. That they are anonymous are irrelevant.
A closure can be a succinct alternative to writing a class such as the one in your example. The technique involves putting a def inside another def. The inner function can have access to the variable in the enclosing function. In Python 3, the nonlocal keyword gives you write access to that variable. In Python 2, you need to use a mutable value for the nonlocal variable in order to be able to update it from the inner function.
About the question regarding anonymous functions, the language intentionally pushes you back to use def for anything more complicated than a lambda can handle.