Python doesn't support complicated anonymous functions. What's a good alternative? For example:
class Calculation:
def __init__(self, func):
self.func = func
def __call__(self, data):
try:
# check if the value has already been calculated
# if it has, it would be cached under key = self.func
return data[self.func]
except KeyError:
pass # first-time call; calculate and cache the values
data[self.func] = self.func(data)
return data[self.func]
# with a simple function, which can be represented using lambda, this works great
f1 = Calculation(lambda data : data['a'] * data['b'])
# with a complicated function, I can do this:
def f2_aux:
# some complicated calculation, which isn't suitable for a lambda one-liner
f2 = Calculation(f2_aux)
Is this a reasonable design to begin with?
If so, is there a way to avoid the ugliness of f*_aux for each f* that I define in the module?
UPDATE:
Example of use:
d = {'a' : 3, 'b' : 6}
# computes 3 * 6
# stores 18 in d under a key <function <lambda> at ...>
# returns 18
f1(d)
# retrieves 18 from d[<function <lambda> at ...>]
# returns 18, without having to recalculate it
f1(d)
UPDATE:
Just for my understanding, I added a version that uses the inner function.
def memoize(func):
def new_func(data):
try:
# check if the value has already been calculated
# if it has, it would be cached under key = self.func
return data[func]
except KeyError:
pass # first-time call; calculate and cache the values
data[func] = func(data)
return data[func]
return new_func
#memoize
def f1(data):
return data['a'] * data['b']
You don't need anonymous functions. Also, memoization has been done better than this, there's probably no reason for you to roll your own.
But to answer the question: You can use your class as a decorator.
#Calculation
def f2():
...
This simply defined the function, wraps it in Calculation and stored the result of that as f2.
The decorator syntax is defined to be equivalent to:
_decorator = Calculation # a fresh identifier
# not needed here, but in other cases (think properties) it's useful
def f2():
...
f2 = _decorator(f2)
The alternative to an anonymous function is a non-anonymous function. An anonymous function is only anonymous in the context where it was defined. But it is not truly anonymous, because then you could not use it.
In Python you make anonymous functions with the lambda statement. You can for example do this:
output = mysort(input, lambda x: x.lastname)
The lambda will create a function, but that function has no name in the local space, and it's own name for itself is just '<lambda>'. But if we look at mysort, it would have to be defined something like this:
def mysort(input, getterfunc):
blahblahblah
As we see here, in this context the function isn't anonymous at all. It has a name, getterfunc. From the viewpoint of this function it does not matter if the function passed in are anonymous or not. This works just as well, and is exactly equivalent in all significant ways:
def get_lastname(x):
return x.lastname
output = mysort(input, get_lastname)
Sure, it uses more code, but it is not slower or anything like that. In Python, therefore anonymous functions are nothing but syntactic sugar for ordinary functions.
A truly anonymous function would be
lambda x: x.lastname
But as we don't assign the resulting function to anything, we do not get a name for the function, and then we can't use it. All truly anonymous functions are unusable.
For that reason, if you need a function that can't be a lambda, make it an ordinary function. It can never be anonymous in any meaningful way, so why bother making it anonymous at all? Lambdas are useful when you want a small one-line function and you don't want to waste space by defining a full function. That they are anonymous are irrelevant.
A closure can be a succinct alternative to writing a class such as the one in your example. The technique involves putting a def inside another def. The inner function can have access to the variable in the enclosing function. In Python 3, the nonlocal keyword gives you write access to that variable. In Python 2, you need to use a mutable value for the nonlocal variable in order to be able to update it from the inner function.
About the question regarding anonymous functions, the language intentionally pushes you back to use def for anything more complicated than a lambda can handle.
Related
While there is a lot of discussion around construction and use of nested functions, there is very little info on when to use them. What I'm trying to find out is: when is it idiomatic to use a nested function? When should you use lambda x: <function> over a nested function?
Python is a language that lets you do a lot of things you shouldn't (ie: using globals). Is this a feature that, while you can use it, you should not?
A scenario I envisioned for using one is in unit testing. Let's say you a method defined like this:
def FunctionOne(varone: int, vartwo: object) -> None:
assert varone == vartwo()
vartwo is a function that is passed into the function. Now we want to unittest this. So we write a (partial) unit test that looks like this:
def unittest() -> None:
def test_function() -> int:
return 1
FuncitonOne(1, test_function)
The specifics here are:
The function is only needed in the scope of the unit test function.
The function is never called elsewhere.
Therefore, to summarize: when should you (if ever) use a nested function with Python?
First of all, notice that a nested function offers features that a lambda or a global function do not have.
It can be multi-statements, by opposition to a lambda
It can have a closure, by opposition to a global function
Thus, a purely feature-oriented answer would be to use a nested function whenever you need to define a multi-statements function with a closure.
Here is an example of such a function.
def create_counter(step):
start = 0
def count():
print(start)
start += step
return count
Note that another recurring example of that use-case are decorators.
def decorator(f):
def wrapper(*args):
...
f(*args)
...
return wrapper
#decorator
def my_function(*args):
...
Another reason you might want to use a nested function is to make it private in some sense, since the function will then be local and cannot be accessed globally. Although, I see very few cases where this is useful since by choosing proper function names, it should not bother you to have helper functions in your namespace.
I can't figure out how to do this, and frankly, I don't know if it's possible.
I want to write a decorator that changes the way a function is called. It's easiest to see with example code:
def my_print(*args, **kwargs):
print(args[0].upper())
#reroute_decorator('print', my_print)
def my_func():
print('normally this print function is just a print function...')
print('but since my_func is decorated with a special reroute_decorator...')
print('it is replaced with a different function, and its args sent there.')
my_func()
# NORMALLY THIS PRINT FUNCTION IS JUST A PRINT FUNCTION...
# BUT SINCE MY_FUNC IS DECORATED WITH A SPECIAL REROUTE_DECORATOR...
# IT IS REPLACED WITH A DIFFERENT FUNCTION, AND ITS ARGS SENT THERE.
Is a decorator with this kind of functionality even possible in python?
Now, I don't really need this if it's too complex, I just can't figure out how to do it in a simple way.
Is this kind of a problem trivial? Or is it really complex?
You can create a new function with an updated globals dictionary so that to that function it appears that the global was bound to the desired value.
Note that this is weaker than actual dynamic scope as any functions called by the function will see the original bindings and not the modified one.
See namespaced_function referenced in How does Python's types.FunctionType create dynamic Functions?
To elaborate on #Dan D.'s answer, you would create a new function object to replace the original, something like this:
from types import FunctionType
def reroute_decorator(**kwargs):
def actual_decorator(func):
globals = func.__globals__.copy()
globals.update(kwargs)
new_func = FunctionType(
func.__code__, globals, name=func.__name__,
argdefs=func.__defaults__, closure=func.__closure__)
new_func.__dict__.update(func.__dict__)
return new_func
return actual_decorator
The only catch here is that the updated function object is the only one that will see whatever kwargs you passed in, since they will be spoofed into globals. Additionally, any modifications you make to the module after calling the decorator function will not be visible to the decorated function, but that should not be an issue. You can go a layer deeper and create a proxy dictionary that would allow you to interact normally with the original, except for keys you explicitly defined, like print, but that's a bit out of scope here.
I've updated your print implementation to be a bit more general, and made the input to the decorator function more pythonic (less MATLABy):
def my_print(*args, **kwargs):
print(*(str(x).upper() for x in args), **kwargs)
#reroute_decorator(print=my_print)
def my_func():
print('normally this print function is just a print function...')
print('but since my_func is decorated with a special reroute_decorator...')
print('it is replaced with a different function, and its args sent there.')
Which results in:
>>> my_func()
NORMALLY THIS PRINT FUNCTION IS JUST A PRINT FUNCTION...
BUT SINCE MY_FUNC IS DECORATED WITH A SPECIAL REROUTE_DECORATOR...
IT IS REPLACED WITH A DIFFERENT FUNCTION, AND ITS ARGS SENT THERE.
I'm trying to wrap my head around the scope of Lambda functions. I noticed that I can create a lambda function in one module A and pass it to a function in another module B, but able to call functions from module A .
Is this bad practice to pass lambda functions around like this, or is there a more preferred (Best Practice) method for handling this?
target.py
class TestLambda():
def __init__(self,name):
self.name = name
def call(self,func):
func(self.name)
source.py
from target import TestLambda
def sayHello(name):
print("Hello {}".format(name))
func = lambda n: sayHello(n)
l = TestLambda("John")
l.call(func)
output
➜ lambda-test python3 source.py
Hello John
The key here is that every function object keeps a reference to the scope in which the function was defined.
>>> func.__globals__['sayHello']
<function sayHello at 0x1085f2680>
This is what lets func still call sayHello even when called from a function with a different global scope. Name lookups are static; they depend only on the lexical (static) context in which the names appear, not the runtime (dynamic) context. There are languages (Lisp, shell, etc) that use dynamic scoping; lexical or static scoping is widely regarded as easier to reason about, though.
I don't think there are any issues with creating and passing lambda functions around, as opposed to using
def func(name):
but I don't see the point in defining a function as a lambda expression if you are going to use it in a different module.
The result is the same, the only difference is that you're not consistent with your function definitions.
The Python docs specifically discourage this:
Always use a def statement instead of an assignment statement that
binds a lambda expression directly to an identifier.
Yes:
def f(x): return 2*x
No:
f = lambda x: 2*x
The first form means that the name of the resulting
function object is specifically 'f' instead of the generic ''.
This is more useful for tracebacks and string representations in
general. The use of the assignment statement eliminates the sole
benefit a lambda expression can offer over an explicit def statement
(i.e. that it can be embedded inside a larger expression)
Can you explain me how the following decorator works:
def set_ev_cls(ev_cls, dispatchers=None):
def _set_ev_cls_dec(handler):
if 'callers' not in dir(handler):
handler.callers = {}
for e in _listify(ev_cls):
handler.callers[e] = _Caller(_listify(dispatchers), e.__module__)
return handler
return _set_ev_cls_dec
#set_ev_cls(ofp_event.EventOFPSwitchFeatures, CONFIG_DISPATCHER)
def _switch_features_handler(self, ev):
datapath = ev.msg.datapath
....
Please, don't go into details on what's going on inside the function. I'm interested in how the decorator with parameters wrap methods here. By the way, it's a code snippet from Ryu (event registration mechanism).
Thank you in advance
First, a decorator is just a function that gets called with a function. In particular, the following are (almost) the same thing:
#spam
def eggs(arg): pass
def eggs(arg): pass
eggs = spam(eggs)
So, what happens when the decorator takes parameters? Same thing:
#spam(arg2)
def eggs(arg): pass
def eggs(arg): pass
eggs = spam(arg2)(eggs)
Now, notice that the function _set_ev_cls_dec, which is ultimately returned and used in place of _switch_features_handler, is a local function, defined inside the decorator. That means it can be a closure over variables from the outer function—including the parameters of the outer function. So, it can use the handler argument at call time, plus the ev_cls and dispatchers arguments that it got at decoration time.
So:
set_ev_cls_dev creates a local function and returns a closure around its ev_cls and dispatchers arguments, and returns that function.
That closure gets called with _switch_features_handler as its parameter, and it modifies and returns that parameter by adding a callers attribute, which is a dict of _Caller objects built from that closed-over dispatchers parameter and keyed off that closed-over ev_cls parameter.
Explain how it works without detailing what's going on inside? That kind of sounds like "explain without explaining," but here's a rough walkthrough:
Think of set_ev_cls as a factory for decorators. It's there to catch the arguments at the time the decorator is invoked:
#set_ev_cls(ofp_event.EventOFPSwitchFeatures, CONFIG_DISPATCHER)
And return a function, _set_ev_cls_dec that has its variables bound to:
ev_cls = ofp_event.EventOFPSwitchFeatures
dispatchers = CONFIG_DISPATCHER
Or put another way, you now have a 'customized' or 'parametrized' dispatcher that's logically equivalent to:
def custom_decorator(handler):
if 'callers' not in dir(handler):
handler.callers = {}
for e in _listify(ofp_event.EventOFPSwitchFeatures):
handler.callers[e] = _Caller(_listify(CONFIG_DISPATCHER), e.__module__)
return handler
(If you captured the values of ofp_event.EventOFPSwitchFeatures and CONFIG_DISPATCHER at the moment the #set_ev_cls(...) was called).
The custom_decorator of step 1 is applied to _switch_features_handleras a more traditional unparameterized decorator.
I have only started learning Python recently. Let me explain what I am trying to accomplish. I have this .py script that basically has several functions (hard-coded into the script) that all need to be added to a single list, so that I can get the function I require by simply using the index operator as follows:
needed_function = function_list[needed_function_index]
My first attempt at implementing this resulted in the following code structure:
(imports)
function_list = []
(other global variables)
def function_0 = (...)
function_list.append(function_0)
def function_1 = (...)
function_list.append(function_1)
def function_2 = (...)
function_list.append(function_2)
(rest of code)
But I don't like that solution since it isn't very elegant. My goal is to be able to simply add the function definition to the script (without the append call) and the script will automatically add it to the list of functions.
I've thought of defining all the functions within another function, but I don't think I'd get anywhere with those. I thought of maybe "tagging" each function with a decorator but I realized that decorators (if I understand them correctly) are called every time a function is called, and not just once.
After some time I came up with this solution:
(imports)
(global variables)
def function_0 = (...)
def function_1 = (...)
def function_2 = (...)
function_list= [globals()[x] for x in globals() if re.match('^function_[0-9]+$', x)]
(rest of code)
I like it a bit more as a solution, but my only qualm with it is that I would prefer, for cleanliness purposes, to completely define function_list at the top of the script. However, I cannot do that since an invocation of globals() at the top of the script would not contain the functions since they have not been defined yet.
Perhaps I should simply settle for a less elegant solution, or maybe I am not writing my script in an idiomatic way. Whatever the case, any input and suggestions are appreciated.
You are mistaken about decorators. They are invoked once when the function is defined, and the function they return is then the value assigned to the function name, and it is that function that is invoked each time. You can do what you want in a decorator without incurring runtime overhead.
my_functions = []
def put_in_list(fn):
my_functions.append(fn)
return fn
#put_in_list
def function1():
pass
#put_in_list
def function2():
pass
PS: You probably don't need to worry about runtime overhead anyway.
PPS: You are also trying to optimize odd things, you might be better off simply maintaining a list in your file. How often are you adding functions, and with how little thought? A list is not difficult to update in the source file.
Example of using a decorator that does not add any overhead to the function call:
my_list = []
def add_to_my_list(func):
print 'decorator called'
my_list.append(func)
return func
#add_to_my_list
def foo():
print 'foo called'
#add_to_my_list
def bar():
print 'foo called'
print '-- done defining functions --'
my_list[0]()
my_list[1]()
One way to solve this problem would be to put all those functions into a single container, then extract the functions from the container to build your list.
The most Pythonic container would be a class. I'm not saying to make them member functions of the class; just define them in the class.
class MyFunctions(object):
def func0():
pass
def func1():
pass
lst_funcs = [x for x in MyFunctions.__dict__ if not x.startswith('_')]
But I like the decorator approach even better; that's probably the most Pythonic solution.