This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 6 months ago.
I need a callback function that is almost exactly the same for a series of gui events. The function will behave slightly differently depending on which event has called it. Seems like a simple case to me, but I cannot figure out this weird behavior of lambda functions.
So I have the following simplified code below:
def callback(msg):
print msg
#creating a list of function handles with an iterator
funcList=[]
for m in ('do', 're', 'mi'):
funcList.append(lambda: callback(m))
for f in funcList:
f()
#create one at a time
funcList=[]
funcList.append(lambda: callback('do'))
funcList.append(lambda: callback('re'))
funcList.append(lambda: callback('mi'))
for f in funcList:
f()
The output of this code is:
mi
mi
mi
do
re
mi
I expected:
do
re
mi
do
re
mi
Why has using an iterator messed things up?
I've tried using a deepcopy:
import copy
funcList=[]
for m in ('do', 're', 'mi'):
funcList.append(lambda: callback(copy.deepcopy(m)))
for f in funcList:
f()
But this has the same problem.
When a lambda is created, it doesn't make a copy of the variables in the enclosing scope that it uses. It maintains a reference to the environment so that it can look up the value of the variable later. There is just one m. It gets assigned to every time through the loop. After the loop, the variable m has value 'mi'. So when you actually run the function you created later, it will look up the value of m in the environment that created it, which will by then have value 'mi'.
One common and idiomatic solution to this problem is to capture the value of m at the time that the lambda is created by using it as the default argument of an optional parameter. You usually use a parameter of the same name so you don't have to change the body of the code:
for m in ('do', 're', 'mi'):
funcList.append(lambda m=m: callback(m))
The problem here is the m variable (a reference) being taken from the surrounding scope.
Only parameters are held in the lambda scope.
To solve this you have to create another scope for lambda:
def callback(msg):
print msg
def callback_factory(m):
return lambda: callback(m)
funcList=[]
for m in ('do', 're', 'mi'):
funcList.append(callback_factory(m))
for f in funcList:
f()
In the example above, lambda also uses the surounding scope to find m, but this
time it's callback_factory scope which is created once per every callback_factory
call.
Or with functools.partial:
from functools import partial
def callback(msg):
print msg
funcList=[partial(callback, m) for m in ('do', 're', 'mi')]
for f in funcList:
f()
Python does uses references of course, but it does not matter in this context.
When you define a lambda (or a function, since this is the exact same behavior), it does not evaluate the lambda expression before runtime:
# defining that function is perfectly fine
def broken():
print undefined_var
broken() # but calling it will raise a NameError
Even more surprising than your lambda example:
i = 'bar'
def foo():
print i
foo() # bar
i = 'banana'
foo() # you would expect 'bar' here? well it prints 'banana'
In short, think dynamic: nothing is evaluated before interpretation, that's why your code uses the latest value of m.
When it looks for m in the lambda execution, m is taken from the topmost scope, which means that, as others pointed out; you can circumvent that problem by adding another scope:
def factory(x):
return lambda: callback(x)
for m in ('do', 're', 'mi'):
funcList.append(factory(m))
Here, when the lambda is called, it looks in the lambda' definition scope for a x. This x is a local variable defined in factory's body. Because of this, the value used on lambda execution will be the value that was passed as a parameter during the call to factory. And doremi!
As a note, I could have defined factory as factory(m) [replace x by m], the behavior is the same. I used a different name for clarity :)
You might find that Andrej Bauer got similar lambda problems. What's interesting on that blog is the comments, where you'll learn more about python closure :)
Yes, that's a problem of scope, it binds to the outer m, whether you are using a lambda or a local function. Instead, use a functor:
class Func1(object):
def __init__(self, callback, message):
self.callback = callback
self.message = message
def __call__(self):
return self.callback(self.message)
funcList.append(Func1(callback, m))
the soluiton to lambda is more lambda
In [0]: funcs = [(lambda j: (lambda: j))(i) for i in ('do', 're', 'mi')]
In [1]: funcs
Out[1]:
[<function __main__.<lambda>>,
<function __main__.<lambda>>,
<function __main__.<lambda>>]
In [2]: [f() for f in funcs]
Out[2]: ['do', 're', 'mi']
the outer lambda is used to bind the current value of i to j
at the
each time the outer lambda is called it makes an instance of the inner lambda with j bound to the current value of i as i's value
First, what you are seeing is not a problem, and not related to call-by-reference or by-value.
The lambda syntax you defined has no parameters, and as such, the scope you are seeing with parameter m is external to the lambda function. This is why you are seeing these results.
Lambda syntax, in your example is not necessary, and you would rather be using a simple function call:
for m in ('do', 're', 'mi'):
callback(m)
Again, you should be very precise about what lambda parameters you are using and where exactly their scope begins and ends.
As a side note, regarding parameter passing. Parameters in python are always references to objects. To quote Alex Martelli:
The terminology problem may be due to
the fact that, in python, the value of
a name is a reference to an object.
So, you always pass the value (no
implicit copying), and that value is
always a reference. [...] Now if you
want to coin a name for that, such as
"by object reference", "by uncopied
value", or whatever, be my guest.
Trying to reuse terminology that is
more generally applied to languages
where "variables are boxes" to a
language where "variables are post-it
tags" is, IMHO, more likely to confuse
than to help.
The variable m is being captured, so your lambda expression always sees its "current" value.
If you need to effectively capture the value at a moment in time, write a function takes the value you want as a parameter, and returns a lambda expression. At that point, the lambda will capture the parameter's value, which won't change when you call the function multiple times:
def callback(msg):
print msg
def createCallback(msg):
return lambda: callback(msg)
#creating a list of function handles with an iterator
funcList=[]
for m in ('do', 're', 'mi'):
funcList.append(createCallback(m))
for f in funcList:
f()
Output:
do
re
mi
there are actually no variables in the classic sense in Python, just names that have been bound by references to the applicable object. Even functions are some sort of object in Python, and lambdas do not make an exception to the rule :)
As a side note, map, although despised by some well known Python figure, forces a construction which prevents this pitfall.
fs = map (lambda i: lambda: callback (i), ['do', 're', 'mi'])
NB : the first lambda i acts like the factory in other answers.
Related
I am trying out lambda in python and came across this question:
def foo(y):
return lambda x: x(x(y))
def bar(x):
return lambda y: x(y)
print((bar)(bar)(foo)(2)(lambda x:x+1))
can someone explain/breakdown how this code works? I am having problems trying to figure out what is x and y.
Lambda functions are just functions. They're almost syntatic sugar, as you can think of this structure:
anony_mouse = lambda x: x # don't actually assign lambdas
as equivalent to this structure:
def anony_mouse(x):
return x
(Almost, as there is no other way of getting a function without assigning it to some variable, and the syntax prevents you doing some things with them, such as using multiple lines.)
Thus let's write out the top example using standard function notation:
def foo(y):
# note that y exists here
def baz(x):
return x(x(y))
return baz
So we have a factory function, which generates a function which... expects to be called with a function (x), and returns x(x(arg_to_factory_function)). Consider:
>>> def add_six(x):
return x + 6
>>> bazzer = foo(3)
>>> bazzer(add_six) # add_six(add_six(3)) = 6+(6+3)
I could go on, but does that make it clearer?
Incidentally that code is horrible, and almost makes me agree with Guido that lambdas are bad.
The 1st ‘(bar)’ is equal to just ‘bar’ so it is an ordinary function call, the 2nd — argument to that call, i.e. bar(bar) — substitute ‘x’ to ‘bar’ there any you will get what is result of bar(bar); the’(foo)’ argument passing to the result of bar(bar) it will be a lambda-function with some arg. — substitute it to ‘foo’ and get result and so on until you reach the end of expression
I slightly modify your original function to make clearer what's going on (so it should be clearer which parameter is callable!)
# given a function it evaluates it at value p
def eval(func): # your foo
return lambda p: func(p)
# given a value p perform a double composition of the function at this value (2-step recursion)
def iter_2(p): # your bar
return lambda func: func(func(p))
increment = lambda x: x + 1 # variable binding only for readability
This example is quite hard to understand because one of the function, eval just do nothing special, and it composition is equivalent to the identity! ... so it could be quite confusing.
(foo)(2)(lambda x:x+1)):
x = 2
iter_2(x)(increment) # increment by 2 because iter_2 calls increment twice
# 4
idempotency: (or composition with itself return the identity function)
increment(3) == eval(increment)(3)
# True
# idempotency - second composition is equivalent to the identity
eval(increment)(3) == eval(eval)(increment)(3)
# True
eval(increment)(3) == eval(eval)(eval)(increment)(3)
# True
# ... and so on
final: consequence of idempotency -> bar do nothing, just confusion
eval(eval)(iter_2)(x)(increment) == iter_2(x)(increment)
# True
Remark:
in (bar)(bar)(foo)(2)(lambda x:x+1) you can omit the brackets around the 1st term, just bar(bar)(foo)(2)(lambda x:x+1)
Digression: [since you example is quite scaring]
Lambda functions are also known as anonymous function. Why this? Simply because that they don't need to be declared. They are designed to be single purpose, so you should "never" assign to a variable. The arise for example in the context of functional programming where the basic ingredients are... functions! They are used to modify the behavior of other functions (for example by decoration!). Your example it is just a standalone syntactical one... essentially a nonsense example which hides the truth "power" of the lambda functions. There is also a branch mathematics which based on them called lambda calculus.
Here a totally different example of application of the lambda functions, useful for decoration (but this is another story):
def action(func1):
return lambda func2: lambda p: func2(p, func1())
def save(path, content):
print(f'content saved to "{path}"')
def content():
return 'content' # i.e. from a file, url, ...
# call
action(content)(save)('./path')
# with each key-parameter would be
action(func1=content)(func2=save)(p='./path')
Output
content saved to "./path"
I got the following code:
g = lambda x: x+7
foo = lambda f: (lambda x: f(x+1)*2)
print( g(3), (foo(g))(3), (foo(foo(g))((3) )
Could I get an explanation on how (foo(foo(g))((3) works?
The first thing to remember is that lambdas are regular functions that:
Don't automatically have names
Can be used as expressions
Must consist of a single expression
Implicitly return the results of that expression
So you can always rewrite them as normal def functions with names if you're confused. For example, foo can become:
def foo(f):
def foo_inner(x):
return f(x + 1) * 2
return foo_inner
So calling foo with any function (f) returns a new function which takes a numeric type, adds one to it, calls f with the value, and doubles the result.
All the rest of it is just tracing the multiple layers of wrapping here, which I'll leave to you; this isn't an interesting problem in general. In real code that uses factory functions like this, the intent and behavior is generally much more clear (because it's being done for a purpose, rather than as a brainteaser).
I'm curious about the difference between lambda function and a regular function (defined with def) - in the python level. (I know what is the difference for programmers and when to use each one.)
>>> def a():
return 1
>>> b = lambda: 1
>>> a
<function a at 0x0000000004036F98>
>>> b
<function <lambda> at 0x0000000004031588>
As we can see - python knows that b is a lambda function and a is a regular function. why is that? what is the difference between them to python?
They are the same type so they are treated the same way:
>>> type(a)
<type 'function'>
>>> type(b)
<type 'function'>
Python also knows that b was defined as a lambda function and it sets that as function name:
>>> a.func_name
'a'
>>> b.func_name
'<lambda>'
In other words, it influences the name that the function will get but as far as Python is concerned, both are functions which means they can be mostly used in the same way. See mgilson's comment below for an important difference between functions and lambda functions regarding pickling.
The only difference is that (a) the body of a lambda can consist of only a single expression, the result of which is returned from the function created and (b) a lambda expression is an expression which evaluates to a function object, while a def statement has no value, and creates a function object and binds it to a name.
In all other material respects they result in identical objects - the same scope and capture rules apply. (Immaterial differences are that lambda-created functions have a default func_name of "<lambda>". This may affect operation in esoteric cases - e.g. attempts to pickle functions.).
Both lambda and def create the same kind of function – they have the same kind of metadata and capabilities. Their technical difference is syntactical:
A lambda is an expression producing a function.
A def is a statement producing a function.
This is everything that dictates how they can be used. Other apparent differences simply come from the information lambda/def can capture.
>>> def def_func(): pass
>>> lambda_func = lambda: None
>>> type(def_func) == type(lambda_func)
True
Usage: Expression vs. Statement
A lambda is more flexible as expressions can be part of more language constructs.
# v--------------v arguments must be expressions
sort(values, key=lambda x: abs(x))
In contrast, a def is more powerful as it can consist of more language constructs.
def encode(num, base):
while num: # statements must be inside statements
num, bit = divmod(num, base)
yield bit
These differences derive directly from one being an expression and the other being a statement. Python has no special rules to decide where a lambda/def may be used.
Where the wild <lambda>s grow
The primary reason to assume lambda and def correspond to different kinds of function is metadata: lambda is often referred to as an "anonymous function" and miraculously it always produces a function <lambda>. Other quirks include "lambda functions can't be pickled", and recently typing also does "not work" for lambda.
That is because compared to def syntax, the lambda syntax has no way of specifying name, type annotations and similar. As such, Python simply fills in sane defaults for either: the name becomes <lambda> and annotations are left empty.
>>> identity = lambda a: a
>>> identity.__qualname__
'<lambda>'
>>> identity.__annotations__
{}
Since <lambda> is not a valid identifier, everything using this metadata to find the function – most prominently pickle – fails.
However, that does not make the function an "anonymous function" type. The metadata can be patched up to insert what def would provide:
>>> identity.__qualname__ = identity.__name__ = 'identity'
>>> identity
<function __main__.identity(a)>
Of course at that one point one can just use def…
First consider the diff b/w the two.
Lambda functions: are operator can have any number of arguments, but it can have only one expression. It cannot contain any statements and it returns a function object which can be assigned to any variable. They can be used in the block they were created.
def functions: Functions help break our program into smaller and modular chunks. As our program grows larger and larger, functions make it more organised and manageable. They can be called and used anywhere we want.
Here you can get more clear difference by following example.
Defining a function
def add(a,b):
return a+b
print(add(4,5))
Defining a lambda
add = lambda x, y : x + y
print(add(4,5))
Lambda is an inline function where we can do any functionality without a function name.
It is helpful when we use it as an argument to a higher-order function.
Eg: A function that takes in other functions as arguments.
Example of Function definition:
>>> def func(a, b):
return a * b
>>> func(2,3)
6
>>> type(func)
<class 'function'>
>>> func
<function func at 0x034B6E88>
Example of Lambda expression:
>>> multiply = lambda a, b: a * b
>>> multiply(2, 3)
6
>>> type(multiply)
<class 'function'>
>>> multiply
<function <lambda> at 0x034B6ED0>
Both returns same output value. Only object returned are different. "func" name for Function and for Lambda.
lambda creates an anonymous function. This idea has been taken from functional programming languages. In this way you can create and pass the function to other functions like map and filter. (look here)
You can pass normal functions to these functions too, but since mostly they are simple and they are not used anywhere else, it's inconvenient to go through the whole process of definfing a new function.
As an example take a look at this:
>>> a = [1, 2, 3, 4]
>>> print map( lambda x : x*2 + 1, a )
[3, 5, 7, 9, 11]
What does the following code do?
a = lambda _:True
From what I read and tested in the interactive prompt, it seems to be a function that returns always True.
Am I understanding this correctly? I hope to understand why an underscore (_) was used as well.
The _ is variable name. Try it.
(This variable name is usually a name for an ignored variable. A placeholder so to speak.)
Python:
>>> l = lambda _: True
>>> l()
<lambda>() missing 1 required positional argument: '_'
>>> l("foo")
True
So this lambda does require one argument. If you want a lambda with no argument that always returns True, do this:
>>> m = lambda: True
>>> m()
True
Underscore is a Python convention to name an unused variable (e.g. static analysis tools does not report it as unused variable). In your case lambda argument is unused, but created object is single-argument function which always returns True. So your lambda is somewhat analogous to Constant Function in math.
it seems to be a function that returns True regardless.
Yes, it is a function (or lambda) that returns True. The underscore, which is usually a placeholder for an ignored variable, is unnecessary in this case.
An example use case for such a function (that does almost nothing):
dd = collections.defaultdict(lambda: True)
When used as the argument to a defaultdict, you can have True as a general default value.
Below is the line of code in question:
a = lambda _:True
It creates a function having one input parameter: _. Underscore is a rather strange choice of variable name, but it is just a variable name. You can use _ anywhere, even when not using lambda functions. For example, instead of....
my_var = 5
print(my_var)
You could write:
_ = 5
print(_)
However, there was a reason that _ was used as the name of parameter name instead of something like x or input. We'll get to that in a moment.
First, we need to know that the lambda-keyword constructs a function, similar to def, but with different syntax. The definition of the lambda function, a = lambda _:True, is similar to writing:
def a(_):
return True
It creates a function named a with an input parameter _, and it returns True. One could have just as easily written a = lambda x:True, with an x instead of an underscore. However, the convention is to use _ as a variable name when we do not intend to use that variable. Consider the following:
for _ in range(1, 11):
print('pear')
Notice that the loop index is never used inside of the loop-body. We simply want the loop to execute a specified number of times. As winklerrr has written, "the variable name _ is [...] like a "throw-away-variable", just a placeholder which is of no use. "
Likewise, with ``a = lambda x:True the input parameter is not used inside the body of the function. It does not really matter what the input argument is, as long as there is one. The author of that lambda-function wrote _ instead of something like x, to indicate that the variable would not be used.
Note that the lambda does have an argument; So, writing
a(), will raise an error.
If you want a lambda with no argument write something like this:
bar = lambda: True
Now calling bar(), with no args, will work just fine.
A lambda which takes no arguments need not always return the same value:
import random
process_fruit = lambda : random.random()
The lambda function above is more complex that just a something which always returns the same constant.
One reason that programmers sometimes us the lambda keyword instead of def is for functions which are especially short and simple. Note that a lambda definition can usually fit all on one line, whereas, it is difficult to do the same with a def statement. Another reason to use lambda instead of def sf when the function will not be used again. If we don't want to call the function again later, then there is no need to give the function a name. For example consider the following code:
def apply_to_each(transform, in_container):
out_container = list()
for idx, item in enumerate(container, 0):
out_container[idx] = transform(item)
return out_container
Now we make the following call:
squares = apply_to_each(lambda x: x**2 range(0, 101))
Notice that lambda x: x**2 is not given a label. This is because we probably won't call it again later, it was just something short and simple we needed temporarily.
The fact that lambda functions need not be given a name is the source of another name to describe them: "anonymous functions."
Also note that lambda-statements are like a function-call in that they return a reference to the function they create. The following is illegal:
apply_to_each(def foo(x): x**2 , range(0, 101))
Whereas, apply_to_each(lambda x: x**2 range(0, 101)) is just fine.
So, we use lambda instead of def and _ instead of a long variable name when we want something short, sweet and probably won't want use again later.
Lambda means a function.
The above statement is same as writing
def f(_):
return True
For lambda a variable needs to be present. So you pass it a variable called _(Similarly you could pass x, y..)
Underscore _ is a valid identifier and is used here as a variable name. It will always return True for the argument passed to the function.
>>>a('123')
True
Is there a way to make function B to be able to access a non global variable that was declared in only in function A, without return statements from function A.
As asked, the question:
Define two functions:
p: prints the value of a variable
q: increments the variable
such that
Initial value of the variable is 0. You can't define the variable in the global
enviroment.
Variable is not located in the global environment and the only way to change it is by invoking q().
The global enviroment should know only p() and q().
Tip: 1) In python, a function can return more than 1 value. 2) A function can be
assigned to a variable.
# Example:
>>> p()
0
>>> q()
>>> q()
>>> p()
2
The question says the global enviroment should know only p and q.
So, taking that literally, it could be done inline using a single function scope:
>>> p, q = (lambda x=[0]: (lambda: print(x[0]), lambda: x.__setitem__(0, x[0] + 1)))()
>>> p()
0
>>> q()
>>> q()
>>> p()
2
Using the tips provided as clues, it could be done something like this:
def make_p_and_q():
context = {'local_var': 0}
def p():
print('{}'.format(context['local_var']))
def q():
context['local_var'] += 1
return p, q
p, q = make_p_and_q()
p() # --> 0
q()
q()
p() # --> 2
The collection of things that functions can access is generally called its scope. One interpretation of your question is whether B can access a "local variable" of A; that is, one that is defined normally as
def A():
x = 1
The answer here is "not easily": Python lets you do a lot, but local variables are one of the things that are not meant to be accessed inside a function.
I suspect what your teacher is getting at is that A can modify things outside of its scope, in order to send information out without sending it through the return value. (Whether this is good coding practise is another matter.) For example, functions are themselves Python objects, and you can assign arbitrary properties to Python objects, so you can actually store values on the function object and read them from outside it.
def a():
a.key = "value"
a()
print a.key
Introspection and hacking with function objects
In fact, you can sort of get at the constant values defined in A by looking at the compiled Python object generated when you define a function. For example, in the example above, "value" is a constant, and constants are stored on the code object:
In [9]: a.func_code.co_consts
Out[9]: (None, 'value')
This is probably not what you meant.
Firstly, it's bad practise to do so. Such variables make debugging difficult and are easy to lose track of, especially in complex code.
Having said that, you can accomplish what you want by declaring a variable as global:
def funcA():
global foo
foo = 3
def funcB():
print foo # output is 3
That's one weird homework assignment; especially the tips make me suspect that you've misunderstood or left out something.
Anyway, here's a simpler solution than the accepted answer: Since calls to q increment the value of the variable, it must be a persistent ("static") variable of some sort. Store it somewhere other than the global namespace, and tell p about it. The obvious place to store it is as an attribute of q:
def q():
q.x += 1
q.x = 0 # Initialize
def p():
print(q.x)