This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
“Least Astonishment” in Python: The Mutable Default Argument
I'm kind of confused about how optional parameters work in Python functions/methods.
I have the following code block:
>>> def F(a, b=[]):
... b.append(a)
... return b
...
>>> F(0)
[0]
>>> F(1)
[0, 1]
>>>
Why F(1) returns [0, 1] and not [1]?
I mean, what is happening inside?
Good doc from PyCon a couple years back - Default parameter values explained. But basically, since lists are mutable objects, and keyword arguments are evaluated at function definition time, every time you call the function, you get the same default value.
The right way to do this would be:
def F(a, b=None):
if b is None:
b = []
b.append(a)
return b
Default parameters are, quite intuitively, somewhat like member variables on the function object.
Default parameter values are evaluated when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that same “pre-computed” value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified.
http://docs.python.org/reference/compound_stmts.html#function
Lists are a mutable objects; you can change their contents. The correct way to get a default list (or dictionary, or set) is to create it at run time instead, inside the function:
def good_append(new_item, a_list=None):
if a_list is None:
a_list = []
a_list.append(new_item)
return a_list
Related
This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 1 year ago.
I'm a bit confused about function variable scope, sometimes the function keeps track of its local variables so that if it's called again it recalls the previous values of the variables - and other times it doesn't keep track of the values.
In the following code, the function keeps track of the mylist and appends to it at every call.
def test(mylist=[]):
mylist.append(1)
print (mylist)
test()
test()
test()
The output:
[1]
[1, 1]
[1, 1, 1]
While in the following code, x is set to zero each time the function is called.
def test(x=0):
x+=1
print (x)
test()
test()
test()
The output:
1
1
1
What is the explanation of this behavior?
Also, is there a way to take a look at the current values of variables inside the functions from outside?
[Update]
After comments, I now understand why mylist is updated, but what about x in the second case? why is it not updated? is it because x is immutable while mylist is mutable?
It's not mylist that's kept between invocations of the function, but rather the default value [], which is a single anonymous array that's assigned by reference to mylist each time.
Appending to mylist of course appends to the array that it refers to.
In the first case, x contains a reference to an array, and when you "add to x" you modify that array.
In the second case, x simply contains a number 0, and when you "add to x", you change what x contains.
The key point is that the expression for the default parameter value is evaluated only once, and re-used between calls to the function.
Welcome to Python :-(
What is the difference between using *args and getting a list as input for having a dynamic number of inputs for a function.
Do these two ways have any advantages in comparison with the other?
It's just a difference in what type of arguments the function expects. They are fairly interchangeable; which you choose is mostly a matter of preference (though see the exceptions at the end of the answer).
Given the following definitions:
# Separate arguments
def foo(*args):
pass
# Single sequence of arguments
def bar(list_of_args):
pass
x = [1, 2, 3]
These two calls are equivalent:
foo(1, 2, 3)
foo(*x) # Unpack a list before calling
And these two calls are equivalent:
bar(x)
bar([1, 2, 3]) # Explicitly pack individual arguments
You mentioned **args in your question. The name of the parameter isn't really important; the special behavior is indicated by a parameter name beginning with one or two *s. One * indicates arbitrary positional arguments, as shown above. Two *s indicates arbitrary keyword arguments. Conventionally, the two most common names are *args and **kwargs.
A brief example similar to the above:
# Arbitrary keyword arguments
def foo(**kwargs):
pass
# Single dict d
def bar(d):
pass
d = {'a': 1, 'b': 2}
# Equivalent
foo(*d)
foo(a=1, b=2)
# Equivalent
bar(d)
bar(dict(a=1, b=2))
Exception: if the function actually expects to mutate a value, you can't replace that effectively with a */** parameter. Example:
def foo(**kwargs):
kwargs['foo'] = 3
This adds a mapping of 'foo' to 3 to the dictionary, but it is a temporary dictionary created just for the call to foo. Nothing outside the function has a reference to that dictionary, so the update doesn't have any effect after foo returns.
Similar logic applies to *args, but since args is a tuple, and tuples are immutable, there's little you could do with the argument anyway. That said, if the function really expects a list, and not just an arbitrary sequence of arguments, you have to define your function to handle that.
def foo(some_list): # can't use *args here
some_list.append(3)
You can't effectively define that as
def foo(*args):
args.append(3)
because foo receives a temporary tuple, not a list object that
This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 6 months ago.
I am trying to use closures to eliminate a variable from a function signature (the application is to make writing all the functions needed for connecting Qt signals for an interface to control a largish number of parameters to the dictionary that stores the values ).
I do not understand why the case of using the lambda not wrapped in another function returns the last name for all cases.
names = ['a', 'b', 'c']
def test_fun(name, x):
print(name, x)
def gen_clousure(name):
return lambda x: test_fun(name, x)
funcs1 = [gen_clousure(n) for n in names]
funcs2 = [lambda x: test_fun(n, x) for n in names]
# this is what I want
In [88]: for f in funcs1:
....: f(1)
a 1
b 1
c 1
# I do not understand why I get this
In [89]: for f in funcs2:
....: f(1)
c 1
c 1
c 1
The reason is that closures (lambdas or otherwise) close over names, not values. When you define lambda x: test_fun(n, x), the n is not evaluated, because it is inside the function. It is evaluated when the function is called, at which time the value that is there is the last value from the loop.
You say at the beginning that you want to "use closures to eliminate a variable from a function signature", but it doesn't really work that way. (See below, though, for a way that may satisfy you, depending on what you mean by "eliminate".) Variables inside the function body will not be evaluated when the function is defined. In order to get the function to take a "snapshot" of the variable as it exists at function-definition time, you must pass the variable as an argument. The usual way to do this is to give the function an argument whose default value is the variable from the outer scope. Look at the difference between these two examples:
>>> stuff = [lambda x: n+x for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
4
4
4
>>> stuff = [lambda x, n=n: n+x for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
2
3
4
In the second example, passing n as an argument to the function "locks in" the current value of n to that function. You have to do something like this if you want to lock in the value in this way. (If it didn't work this way, things like global variables wouldn't work at all; it's essential that free variables be looked up at the time of use.)
Note that nothing about this behavior is specific to lambdas. The same scoping rules are in effect if you use def to define a function that references variables from the enclosing scope.
If you really want to, you can avoid adding the extra argument to your returned function, but to do so you must wrap that function in yet another function, like so:
>>> def makeFunc(n):
... return lambda x: x+n
>>> stuff = [makeFunc(n) for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
2
3
4
Here, the inner lambda still looks up the value of n when it is called. But the n it refers to is no longer a global variable but a local variable inside the enclosing function makeFunc. A new value of this local variable is created every time makeFunc is called, and the returned lambda creates a closure that "saves" the local variable value that was in effect for that invocation of makeFunc. Thus each function created in the loop has its own "private" variable called x. (For this simple case, this can also be done using a lambda for the outer function --- stuff = [(lambda n: lambda x: x+n)(n) for n in [1, 2, 3]] --- but this is less readable.)
Notice that you still have to pass your n as an argument, it's just that, by doing it this way, you don't pass it as an argument to the same function that winds up going into the stuff list; instead you pass it as an argument to a helper function that creates the function you want to put into stuff. The advantage of using this two-function approach is that the returned function is "clean" and doesn't have the extra argument; this could be useful if you were wrapping functions that accepted a lot of arguments, in which case it could become confusing to remember where the n argument was in the list. The disadvantage is that, doing it this way, the process of making the functions is more complicated, since you need another enclosing function.
The upshot is that there is a tradeoff: you can make the function-creation process simpler (i.e., no need for two nested functions), but then you must make the resulting function a bit more complicated (i.e., it has this extra n=n argument). Or you can make the function simpler (i.e., it has no n=n argument), but then you must make the function-creation process more complicated (i.e., you need two nested functions to implement the mechanism).
This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 6 months ago.
#!/usr/bin/env python3.2
def f1(a, l=[]):
l.append(a)
return(l)
print(f1(1))
print(f1(1))
print(f1(1))
def f2(a, b=1):
b = b + 1
return(a+b)
print(f2(1))
print(f2(1))
print(f2(1))
In f1 the argument l has a default value assignment, and it is only evaluated once, so the three print output 1, 2, and 3. Why f2 doesn't do the similar?
Conclusion:
To make what I learned easier to navigate for future readers of this thread, I summarize as the following:
I found this nice tutorial on the topic.
I made some simple example programs to compare the difference between mutation, rebinding, copying value, and assignment operator.
This is covered in detail in a relatively popular SO question, but I'll try to explain the issue in your particular context.
When your declare your function, the default parameters get evaluated at that moment. It does not refresh every time you call the function.
The reason why your functions behave differently is because you are treating them differently. In f1 you are mutating the object, while in f2 you are creating a new integer object and assigning it into b. You are not modifying b here, you are reassigning it. It is a different object now. In f1, you keep the same object around.
Consider an alternative function:
def f3(a, l= []):
l = l + [a]
return l
This behaves like f2 and doesn't keep appending to the default list. This is because it is creating a new l without ever modifying the object in the default parameter.
Common style in python is to assign the default parameter of None, then assign a new list. This gets around this whole ambiguity.
def f1(a, l = None):
if l is None:
l = []
l.append(a)
return l
Because in f2 the name b is rebound, whereas in f1 the object l is mutated.
This is a slightly tricky case. It makes sense when you have a good understanding of how Python treats names and objects. You should strive to develop this understanding as soon as possible if you're learning Python, because it is central to absolutely everything you do in Python.
Names in Python are things like a, f1, b. They exist only within certain scopes (i.e. you can't use b outside the function that uses it). At runtime a name refers to a value, but can at any time be rebound to a new value with assignment statements like:
a = 5
b = a
a = 7
Values are created at some point in your program, and can be referred to by names, but also by slots in lists or other data structures. In the above the name a is bound to the value 5, and later rebound to the value 7. This has no effect on the value 5, which is always the value 5 no matter how many names are currently bound to it.
The assignment to b on the other hand, makes binds the name b to the value referred to by a at that point in time. Rebinding the name a afterwards has no effect on the value 5, and so has no effect on the name b which is also bound to the value 5.
Assignment always works this way in Python. It never has any effect on values. (Except that some objects contain "names"; rebinding those names obviously effects the object containing the name, but it doesn't affect the values the name referred to before or after the change)
Whenever you see a name on the left side of an assignment statement, you're (re)binding the name. Whenever you see a name in any other context, you're retrieving the (current) value referred to by that name.
With that out of the way, we can see what's going on in your example.
When Python executes a function definition, it evaluates the expressions used for default arguments and remembers them somewhere sneaky off to the side. After this:
def f1(a, l=[]):
l.append(a)
return(l)
l is not anything, because l is only a name within the scope of the function f1, and we're not inside that function. However, the value [] is stored away somewhere.
When Python execution transfers into a call to f1, it binds all the argument names (a and l) to appropriate values - either the values passed in by the caller, or the default values created when the function was defined. So when Python beings executing the call f3(5), the name a will be bound to the value 5 and the name l will be bound to our default list.
When Python executes l.append(a), there's no assignment in sight, so we're referring to the current values of l and a. So if this is to have any effect on l at all, it can only do so by modifying the value that l refers to, and indeed it does. The append method of a list modifies the list by adding an item to the end. So after this our list value, which is still the same value stored to be the default argument of f1, has now had 5 (the current value of a) appended to it, and looks like [5].
Then we return l. But we've modified the default list, so it will affect any future calls. But also, we've returned the default list, so any other modifications to the value we returned will affect any future calls!
Now, consider f2:
def f2(a, b=1):
b = b + 1
return(a+b)
Here, as before, the value 1 is squirreled away somewhere to serve as the default value for b, and when we begin executing f2(5) call the name a will become bound to the argument 5, and the name b will become bound to the default value 1.
But then we execute the assignment statement. b appears on the left side of the assignment statement, so we're rebinding the name b. First Python works out b + 1, which is 6, then binds b to that value. Now b is bound to the value 6. But the default value for the function hasn't been affected: 1 is still 1!
Hopefully that's cleared things up. You really need to be able to think in terms of names which refer to values and can be rebound to point to different values, in order to understand Python.
It's probably also worth pointing out a tricky case. The rule I gave above (about assignment always binding names with no effect on the value, so if anything else affects a name it must do it by altering the value) are true of standard assignment, but not always of the "augmented" assignment operators like +=, -= and *=.
What these do unfortunately depends on what you use them on. In:
x += y
this normally behaves like:
x = x + y
i.e. it calculates a new value with and rebinds x to that value, with no effect on the old value. But if x is a list, then it actually modifies the value that x refers to! So be careful of that case.
In f1 you are storing the value in an array or better yet in Python a list where as in f2 your operating on the values passed. Thats my interpretation on it. I may be wrong
Other answers explain why this is happening, but I think there should be some discussion of what to do if you want to get new objects. Many classes have the method .copy() that allows you create copies. For instance, if we rewrite f1 as
def f1(a, l=[]):
new_l = l.copy()
new_l.append(a)
return(new_l)
then it will continue to return [1]no matter how many times we call it. There is also the library https://docs.python.org/3/library/copy.html for managing copies.
Also, if you're looping through the elements of a container and mutating them one by one, using comprehensions not only is more Pythonic, but can avoid the issue of mutating the original object. For instance, suppose we have the following code:
data = [1,2,3]
scaled_data = data
for i, value in enumerate(scaled_data):
scaled_data[i] = value/sum(data)
This will set scaled_data to [0.16666666666666666, 0.38709677419354843, 0.8441754916792739]; each time you set a value of scaled_data to the scaled version, you also change the value in data. If you instead have
data = [1,2,3]
scaled_data = [x/sum(data) for x in data]
this will set scaled_data to [0.16666666666666666, 0.3333333333333333, 0.5] because you're not mutating the original object but creating a new one.
This question already has answers here:
Python: Is the split function evaluated multiple times in a list comprehension?
(3 answers)
Closed 7 years ago.
Which one's a better way of doing list comprehension in python (in terms of computation time & cpu cycles).
In example (1) is the value f(r) evaluated in each iteration or is it evaluated once and cached ?
y = [x*f(r) for x in xlist]
c = f(r)
y = [x*c for x in xlist]
where
def f(r):
... some arbitrary function ...
It evaluates for every iteration. Look at this:
>>> def f():
... print("func")
...
>>> [f() for i in range(4)]
func
func
func
func
[None, None, None, None]
As you say, if f() has no side effects, storing the return value on a variable and using that variable instead is a lot faster solution.
I would probably choose the latter because the Python compiler doesn't know if the function has side-effects so it is called for each element.
Here is an easy way to find out:
>>> def f():
... print "called"
... return 1
...
>>> [1+f() for x in xrange(5)]
called
called
called
called
called
[2, 2, 2, 2, 2]
so yes, if the function is going to be the same each time then it is better to call it once outside the list comprehension.
The function f will be called for every element.
It is very complex for the compiler/interpreter to determine that the function need not to be called many times. It is then very probable that the function is called many times. Thus, using the second solution is always the best solution.
Functions have a non-trivial execution time compared to a name lookup, and caching the value is considered acceptable if the function is called many times and the same value is expected each time.
Python is probably free to do it once or many times, I'm not sure I would rely on any observed behavior. It might change in the next version.
If it's important to you to make sure the function is only called once, call it yourself and save the results.