This question already has answers here:
What do lambda function closures capture?
(7 answers)
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 6 months ago.
I've been working on a basic testing framework for an automated build. The piece of code below represents a simple test of communication between two machines using different programs. Before I actually do any tests, I want to completely define them - so this test below is not actually run until after all the tests have been declared. This piece of code is simply a declaration of a test.
remoteTests = []
for client in clients:
t = Test(
name = 'Test ' + str(host) + ' => ' + str(client),
cmds = [
host.start(CMD1),
client.start(CMD2),
host.wait(5),
host.stop(CMD1),
client.stop(CMD2),
],
passIf = lambda : client.returncode(CMD2) == 0
)
remoteTests.append(t)
Anyhow, after the test is run, it runs the function defined by 'passIf'. Since I want to run this test for multiple clients, I'm iterating them and defining a test for each - no big deal. However, after running the test on the first client, the 'passIf' evaluates on the last one in the clients list, not the 'client' at the time of the lambda declaration.
My question, then: when does python bind variable references in lambdas? I figured if using a variable from outside the lambda was not legal, the interpretor would have no idea what I was talking about. Instead, it silently bound to the instance of the last 'client'.
Also, is there a way to force the resolution the way I intended it?
The client variable is defined in the outer scope, so by the time the lambda is run it will always be set to the last client in the list.
To get the intended result, you can give the lambda an argument with a default value:
passIf = lambda client=client: client.returncode(CMD2) == 0
Since the default value is evaluated at the time the lambda is defined, its value will remain correct.
Another way is to create the lambda inside a function:
def createLambda(client):
return lambda: client.returncode(CMD2) == 0
#...
passIf = createLambda(client)
Here the lambda refers to the client variable in the createLambda function, which has the correct value.
What happens is that your passIf argument, the lambda, refers to the variable client from the enclosing scope. It doesn't refer to the object the variable client refers to when it is created, but the variable itself. If you call these passIf after the loop has ended, that means they all refer to the last value in the loop. (In closure terminology, Python's closures are late-binding, not early-binding.)
Fortunately it's fairly easy to make a late-binding closure into an early-binding closure. You can do it by simply giving the lambda an argument with as default the value you want to bind:
passIf = lambda client=client: client.returncode(CMD2) == 0
That does mean the function gets that extra argument, and might mess things up if it gets called with an argument by accident -- or when you want the function to take arbitrary arguments. So another technique is to do it like this:
# Before your loop:
def make_passIf(client):
return lambda: client.returncode(CMD2) == 0
# In the loop
t = Test(
...
passIf = make_passIf(client)
)
Related
I am new to Python so I am unsure about the difference in variable assignment before or after the function definition.
Specifically, the first example was adopted from Lutz's book.
def tester(start):
print("inside tester")
def nested(label):
print("inside nested")
print(label,nested.state)
nested.state += 1
print("done with nested")
nested.state = start
print("done with tester")
return nested
F = tester(0)
F('spam')
F.state
F.state
The objective of the code is to store the state information without using nonlocal.
I am unsure what nested.state means here. I am unsure because nested.state is used inside nested() function (i.e. nested.state +=1) and outside nested() function (i.e. nested.state = start).
I modified the code above to see whether Python accepts assigning variable after function declaration for nested() and to see whether there is any concept I am missing relating to function.variable call (i.e. nested.state call).
def tester(start):
def nested(label):
print(label, state)
state += 1 #replaced 'nested.state' with 'state' here
state = start #replaced 'nested.state' with 'state' here
return nested
F=tester(0)
F('spam')
F('ham')
Unfortunately, above code generates error local variable 'state' referenced before assignment. This tells me that I am missing some concept about function.variable (i.e. nested.state).
Can someone please help me understand three things:
I. why it is that the code with nested.state doesn't generate any error but state does?
II. what does nested.state mean? If nested.state is a mechanism to access function's variables, why is it that the call inside nested() function also uses nested.state and not state?
III. If nested.state is a mechanism to access variable inside function, then why is it that PyCharm fails to show state under dropdown when I type nested.?
I'd appreciate any help. I research SO, and couldn't find any explanation on such problems.
The reason the first code example worked is because it was assigning and referencing an attribute of the nested function object. The key concept to understand here, is that Python allows you to assign new, arbitrary attributes to objects - including functions:
>>> def func(a, b):
return a + b
>>> func(1, 2)
3
>>> func.attr = 5
>>> func.attr
5
The first code example takes advantage of this fact by using the nested function object to store the necessary state. This is the same concept as using any other object to store the state. It's particularly convenient, however, to use a function since it's readily available.
In the second example, a normal variable is used. Because of this, normal scoping rules apply which means simply that the state variable defined in tester is not the state variable being referenced in nested. Thus, an error is raised.
Actually, I think you're asking a question about scope in Python, ignoring your code, check this:
def scope_level_1():
variable = 'Scope_level_1'
def scope_level_2():
variable = 'Scope_level_2'
def core():
nonlocal variable
variable += '_TOUCHED_AND_MODIFIED_BY_CORE'
print(variable)
return core()
return scope_level_2()
scope_level_1()
# 'Scope_level_2_TOUCHED_AND_MODIFIED_BY_CORE'
Don't worry about the keyword nonlocal, treat it just as a declaring to make code more readable.
First, remember a += b is the same as a = a + b. So a must exist before getting to the +=.
Simply put, in the first example the function nested has an attribute called state (accessed by nested.state). It is an attribute, which means that once you tell nested that it has an attribute called state (you are doing this in line 9 when nested.state = start) it keep that attribute. So, in the first example nested.state exists when you get to the +=.
In the second example, you are declaring a variable called state in tester, and another variable called state in nested. The one in nested could be called potato for all that matters, because it is not the same variable. Therefore when you arrive to the +=, the variable state does not exist!
I understand the normal lambda expression, such as
g = lambda x: x**2
However, for some complex ones, I am a little confused about them. For example:
for split in ['train', 'test']:
sets = (lambda split=split: newspaper(split, newspaper_devkit_path))
def get_imdb():
return sets()
Where newspaper is a function. I was wondering what actually the sets is and why the get_imdb function can return the value sets()
Thanks for your help!
Added:
The codes are actually from here factory.py
sets is being assigned a lambda that is not really supposed to accept inputs, which you see from the way it is invoked. Lambdas in general behave like normal functions, and can therefore be assigned to variables like g or sets. The definition of sets is surrounded by an extra set of parentheses for no apparent reason. You can ignore those outer parens.
Lambdas can have all the same types of positional, keyword and default arguments a normal function can. The lambda sets has a default parameter named split. This is a common idiom to ensure that sets in each iteration of the loop gets the value of split corresponding to that iteration rather than just the one from the last iteration in all cases.
Without a default parameter, split would be evaluated within the lambda based on the namespace at the time it was called. Once the loop completes, split in the outer function's namespace will just be the last value it had for the loop.
Default parameters are evaluated immediately when a function object is created. This means that the value of the default parameter split will be wherever it is in the iteration of the loop that creates it.
Your example is a bit misleading because it discards all the actual values of sets besides the last one, making the default parameter to the lambda meaningless. Here is an example illustrating what happens if you keep all the lambdas. First with the default parameter:
sets = []
for split in ['train', 'test']:
sets.append(lambda split=split: split)
print([fn() for fn in sets])
I have truncated the lambdas to just return their input parameter for purposes of illustration. This example will print ['train', 'test'], as expected.
If you do the same thing without the default parameter, the output will be ['test', 'test'] instead:
sets = []
for split in ['train', 'test']:
sets.append(lambda: split)
print([fn() for fn in sets])
This is because 'test' is the value of split when all the lambdas get evaluated.
A lambda function:
func = lambda x: x**2
can be rewritten almost equivalently:
def func(x):
return x**2
Using either way, you can call the function in this manner:
func(4)
In your example,
sets = lambda split=split: newspaper(split, newspaper_devkit_path)
can be rewritten:
def sets(split=split):
return newspaper(split, newspaper_devkit_path)
and so can be called:
sets()
When you write the following:
def get_imdb():
return sets()
you are defining a "closure". A reference to the function sets is saved within get_imdb so that it can be called later wherever get_imdb is called.
Maybe you are confused about the split=split part. This has the same meaning as it would have in a regular function: the split on the left is a parameter of the lambda function and the split on the right is the default value the left split takes when no value is provided. In this case, the default value would be the variable split defined in the for loop.
So, answering your first question (what is sets?):
sets is a variable to which an anonymous function (or lambda function) is assigned. This allows the lambda function to be referenced and used via the variable sets.
To your second question (why can sets() be returned?), I respond:
Since sets is a variable that acts as a function, adding parenthesis after it calls the lambda function. Because no parameters are given, the parameter split takes the value 'test', which is the last value the for loop variable split takes. It is worth noting here that, since sets is not defined inside the function get_imdb, the interpreter looks for a definition of sets outside the scope of get_imdb (and finds the one that refers to the lambda function).
This question already has answers here:
masking the built-in variable with its magic behavior?
(2 answers)
Closed 6 years ago.
In the python tutorial:
In interactive mode, the last printed expression is assigned to the variable _. This means that when you are using Python as a desk calculator, it is somewhat easier to continue calculations, for example:
>>> tax = 12.5 / 100
>>> price = 100.50
>>> price * tax
12.5625
>>> price + _
113.0625
>>> round(_, 2)
113.06
This variable should be treated as read-only by the user. Don’t explicitly assign a value to it — you would create an independent local variable with the same name masking the built-in variable with its magic behavior.
What is masking here?
In general what is meant by masking or mask variable in python?
THanks
A variable is masked if some other variable with the same name exists, which is preferred. An example is having a global variable x. If you're in a function which also has a local variable x, then the global won't be accessible, without special measures the local variable is preferred whenever you use identifier x inside the function.
It's as if you have two kids in your classroom both called Mary, one of which is closer, and will hear your voice earlier, so always gives the quickest answer. How to give the other girl a chance?
Overwriting/reassigning a previously named identifier in a new scope
let's look at this code for example
def f():
return 3
def g():
f = 5
f()
try:
print(g())
except TypeError:
print(f())
within the scope of g, the identifier f has been reassigned from a function that returns an integer, to just an integer. so when you try calling f inside of g you should get a type error because you can't call an int. however outside of g, f is still a function that returns 3
the interpreter has its own scope so when you assign _ to another value, it forgets it's previous functionality in replace of the new value, until you restart the interpreter, or in this special case delete it with the del keyword
The _ is built-in to the interpreter, it has special meaning. It only acts like a read-only variable. When you read from it, the interpreter replaces it with the last calculated value. If you do something like
_ = 1 + 1
You are actually creating a new variable named _ in the current namespace (try running locals()function before and after creating the variable). Since there is now a variable with the same name as the interpreter built-in _ that variable is accessed first when you do subsequent reads, thus hiding the meaning of the built-in, or masking it.
You can also do what you're doing with Python built-ins as well, such as defining a function parameter namedint. In the the local function namespace, if you tried to call theint() function on a value, it would attempt to use your parameterint as a callable because there is a local variable with that name masking the built-in function int. Some IDEs will warn you when you try to do something like this because in practice it is generally a bad idea to mask the name of a built-in like that because you're eliminating the behavior of that what that symbol is intended for.
Where making comes in its if you do something like
val = 20
def do_stuff(val):
# parameter masks the value defined in outer scope
print('Value of a inside function: {}'.f format(val))
do_stuff(10) # prints 10
This assigns a variable val a value of 20, but when the function do_stuff is called it had it's own local variable namedval. Accessingval from within the function will give you the value passed in, 10 in this case, not 20 from the outer variable because thea within the function masks the one in the outer scope.
This question already has answers here:
tkinter creating buttons in for loop passing command arguments
(3 answers)
Closed 10 months ago.
from tkinter import *
F=Tk()
i=1
while i<10:
newButton = Button(F,text="Show Number",command=lambda:showNumber(i))
newButton.pack(side=TOP)
i+=1
def showNumber(nb):
print(nb)
F.mainloop()
All buttons return 10. Why ?
I want button 1 return 1, button 2 return 2...
Thank you very much for helping me
Your anonymous lambda functions are can be though of as closures (as #abernert points out, they're not actually closures in Python's case) - they "close over" the variable i, to reference it later. However, they don't look up the value at the time of definition, but rather at the time of calling, which is some time after the entire while loop is over (at which point, i is equal to 10).
To fix this, you need to re-bind the value of i to a something else for the lambda to use. You can do this in many ways - here's one:
...
i = 1
while i < 10:
# Give a parameter to the lambda, defaulting to i (function default
# arguments are bound at time of declaration)
newButton = Button(F, text="Show Number",
command=lambda num=i: showNumber(num))
...
This is explained in the Python FAQ: Why do lambdas defined in a loop with different values all return the same result?.
Quoting the FAQ answer:
This happens because x is not local to the lambdas, but is defined in the outer scope, and it is accessed when the lambda is called — not when it is defined…
In order to avoid this, you need to save the values in variables local to the lambdas, so that they don’t rely on the value of the global…
In other words, your new functions aren't storing the value of i, they're storing the variable i. And they're all storing the same variable i, which has the value 10 at the end of your loop. In fact, if you add an i = 'spam' right before F.mainloop(), you'll see that all the buttons now print out the string spam instead of a number.
This is very useful when you're trying to create closures—functions that can affect their defining environment.* But when you're not trying to do so, that can get in the way.
The simplest way around this is to use a parameter with a default value. Default values don't hold variables; just values, which are evaluated at the time the function is defined. So:
newButton = Button(F,text="Show Number", command=lambda num=i: showNumber(num))
* Note that in this case, there aren't actually any closures involved, because i is a global, rather than a local in the enclosing scope. But really, this is just because Python has special handling for globals and doesn't need a closure here; conceptually, if you think of there being one, you won't get into any trouble unless you start looking at the __closure__ or __code__ attributes.
This question already has answers here:
What do lambda function closures capture?
(7 answers)
Closed 6 months ago.
I understand what are lambda functions in Python, but I can't find what is the meaning of "lambda binding" by searching the Python docs.
A link to read about it would be great.
A trivial explained example would be even better.
Thank you.
First, a general definition:
When a program or function statement
is executed, the current values of
formal parameters are saved (on the
stack) and within the scope of the
statement, they are bound to the
values of the actual arguments made in
the call. When the statement is
exited, the original values of those
formal arguments are restored. This
protocol is fully recursive. If within
the body of a statement, something is
done that causes the formal parameters
to be bound again, to new values, the
lambda-binding scheme guarantees that
this will all happen in an orderly
manner.
Now, there is an excellent python example in a discussion here:
"...there is only one binding for x: doing x = 7 just changes the value in the pre-existing binding. That's why
def foo(x):
a = lambda: x
x = 7
b = lambda: x
return a,b
returns two functions that both return 7; if there was a new binding after the x = 7, the functions would return different values [assuming you don't call foo(7), of course. Also assuming nested_scopes]...."
I've never heard that term, but one explanation could be the "default parameter" hack used to assign a value directly to a lambda's parameter. Using Swati's example:
def foo(x):
a = lambda x=x: x
x = 7
b = lambda: x
return a,b
aa, bb = foo(4)
aa() # Prints 4
bb() # Prints 7
Where have you seen the phrase used?
"Binding" in Python generally refers to the process by which a variable name ends up pointing to a specific object, whether by assignment or parameter passing or some other means, e.g.:
a = dict(foo="bar", zip="zap", zig="zag") # binds a to a newly-created dict object
b = a # binds b to that same dictionary
def crunch(param):
print param
crunch(a) # binds the parameter "param" in the function crunch to that same dict again
So I would guess that "lambda binding" refers to the process of binding a lambda function to a variable name, or maybe binding its named parameters to specific objects? There's a pretty good explanation of binding in the Language Reference, at http://docs.python.org/ref/naming.html