This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 5 years ago.
I am a python beginner, reading 'python tutorial', it says if we have a function:
def f(a, L=[]):
L.append(a)
return L
print f(1)
print f(2)
print f(3)
This will print
[1]
[1, 2]
[1, 2, 3]
Because the default value is evaluated only once and list is a mutable object. I can understand it.
And it says continue, if we don't want the default to be shared between subsquent calls, we can:
def f(a, L=None):
if L is None: #line 2
L = []
L.append(a)
return L
print f(1)
print f(2)
print f(3)
and this will output:
[1]
[2]
[3]
But why? How to explain this. We know default value is evaluated only once, and when we call f(2), L is not None and that if(in line 2) can not be true, so L.append(a) == [1, 2]. Could I guess the default value is evaluated again for some reason , but what is 'some reason', just because the python interpreter see if L is None: L = []
Python passes parameters to functions by value; So for objects, the value passed is a reference to the object, not a new copy of the object.
That, along with the following part of the official docs is what helped me understand it better (emphasis mine):
Default parameter values are evaluated [...] when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that the same “pre-computed” value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. [...] A way around this is to use None as the default, and explicitly test for it in the body of the function [...]
Putting it all together:
If you define the default for a parameter to be a mutable object (such as []) then the "pre-computed" value is the reference to that object, so each call to the function will always reference the same object, which can then be mutated across multiple invocations of the function.
However, since None is an immutable built-in type, the "pre-computed" value for a default of None is simply that. So the parameter will be None each time you call the function.
Hopefully that helps! I do think that the tutorial could have had better wording, because I was also confused by that at first.
"The default value is only evaluated once" does not mean that a parameter with a default retains its value between invocations of the function. It means that the expression which you specify (the None part of def f(a, L=None)) is evaluated once, and the object it results in is stored in a hidden location and re-used if no value for that parameter is given at call. Parameters are still reset to the value (default or not) at every invocation.
In your second example you have a variable L. At first L refers to None. You repoint it to a new empty list on each invocation, then mutate that new list. Remember L = [] is the same as L = list()
In your first example, however, L is set to the new list once at function declaration. L isn't reset to [] on each invocation of the function. So you are always mutating the same list.
What happens is as follows:
When a python function is called, it is evaluated in the environment in which it was defined and not the environment in which it was called although the second part is secondary ( no pun intended) for the purpose of answering your question.
The default arguments are evaluated only once at the time of function definition. This creates a closure. Think of closure as function code + environment in which the function was been defined.
So in this case when the function was defined, L was assigned to [] and now every subsequent call to the function will use this value of L.
The tutorial also mentions:
The default values are evaluated at the point of function definition in the defining scope (and the defining scope is part of the closure along with function code)
http://docs.python.org/2/tutorial/controlflow.html#default-argument-values
I think this is happening because a list is a mutable object while the value None is immutable.
For the first function, variable L is outside the function environment (in the function definition), and it refers to an empty list. Then you make changes to this list in the function environment, but since a list is mutable, the variable L that is outside the function environment refers to this now mutated list, and the change propagates each time you call the function.
For the second function, variable L is also outside the function environment (in the function definition), but this time it refers to None, which is immutable. Now, every change you make in the function environment will not affect what L refers to outside the function environment. The variable L inside the function environment refers to something different as you change it. First, it refers to an empty list, and then a list that gets a value appended to it. You then return this list. The next time you call the function, you call it with the variable L that is outside the function environment, which has not changed and still refers to None.
Hope this makes sense.
Related
I have the following code:
def test(a):
a.append(3)
test_list = [1,2]
test(test_list)
In the above code, when I pass test_list as an argument to test function, does python send the whole list object/data (which is a big byte size) or just the reference to the list (which would be much smaller since its just a pointer)?
From what I understood by looking at Python's pass by object reference, it only sends the reference, but do not know how to verify that it indeed is the case
It's passing an alias to the function; for the life of the function, or until the function intentionally rebinds a to point to something else (with a = something), the function's a is an alias to the same list bound to the caller's test_list.
The straightforward ways to confirm this are:
Print test_list after the call; if the list were actually copied, the append in the function would only affect the copy, the caller wouldn't see it. test_list will in fact have had a new element appended, so it was the same list in the function.
Print id(test_list) outside the function, and id(a) inside the function; ids are required to be unique at any given point in time, so the only way two different lists could have the same ID is if one was destroyed before the other was created; since test_list continues to exist before and after the function call, if a has the same id, it's definitionally the same list.
All function arguments are passed by reference in Python. So the a variable in the function test will refer to the same object as test_list does in the calling code. After the function returns, test_list will contain [1,2,3].
I've been looking up the information about why people should not use empty list as a function argument and pass none type as an argument and came up with a few questions.
Here is my code:
def add_employee(emp, emp_list=None):
if emp_list is None:
emp_list = []
emp_list.append(emp)
print(emp_list)
And here is code without second argument's type specified:
def add_employee(emp, emp_list):
emp_list.append(emp)
return emp_list
When I do not define emp_list as an empty list or none I can not utilize function's deafualt argument behavior: I can't call it like add_employee('Mark'), I had to add second variable to pass. Why is it good to have that backup default behaviour? Why couldn't I just leave it as emp_list.
Here is a great explanation of why you should avoid using mutable arguments in your defaults. Or at least use them sparingly: link
The general gist of it that the list will be created for the first time (in a location in memory) when you define the function. So as python reads the code and interprets it, you will end up creating that list once on the first 'read' of the function.
What this means is that you are not creating a fresh list each time you call that function, only when you define it.
To illustrate I will use the example from the link I shared above:
def some_func(default_arg=[]):
default_arg.append("some_string")
return default_arg
>>> some_func()
['some_string']
>>> some_func()
['some_string', 'some_string']
>>> some_func([])
['some_string']
>>> some_func()
['some_string', 'some_string', 'some_string']
If I understood your question correctly, you are asking why you're better off defining the emp_list explicitly rather than having it outside the function. In my mind it boils down to encapsulation. You're essentially trying to make sure that your function doesn't change the behavior of anything outside of its scope so you're forced to pass it things directly and be explicit. In practice if you have a variable outside of the scope named emp_list, it is absolutely fine to just append to it as long as you understand what the expected behavior is.
If you pass a list in the first bit of code as the emp_list, then the variable emp_list will contain your list a. The if statement will check if list a is None and since that check fails, it will skip the next line of assigning it a fresh empty list.
This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 6 years ago.
I am confused by the following example:
def append(elem, to=None):
if to is None:
to = []
to.append(elem)
return to
append(5)
# Out: [5]
append(3) # A new list is created
# Out: [3]
The point here is supposed to be that you are resetting "to" to an empty list at the beginning of every function call to avoid the following:
append(5)
# Out: [5]
append(3) # A new list is created
# Out: [5, 3]
But how can you check that "to" is None to set it to []? It seems to me that you are either pulling in the "to" defined in the definition, or you are pulling in the "to" modified by the last call. How does this work?
When you define a function with a default argument, the function uses that default values for that argument if it is not supplied. So in the case of append(5), to is not specified, so the function assumes the value of to to be None - it's effectively the same as calling append(5, None).
So now, the function checks if to is None, which it is, so to gets reassigned to an empty list. 5 gets appended to the list, and the list is returned.
When you make the second call append(3), it is again as if you called append(3, None). Again, the if to is None evaluates to True, and to is reassigned to an empty list. Then, 3 is appended to that empty list, and the list is returned.
Since the default argument (in this case None) is immutable, the operations on to do not persist the end of the function call. The function has its own chunk of memory which gets cleared when the function returns.
However, if the default argument was mutable (like for instance []), that value is created when the function is defined (i.e. when python sees def func(arg1, arg2=[]), it creates an empty list in memory and uses that list every time this function is called). Thus, any changes made to that list will persist the end of the function, since that mutable default argument was created when the function was defined (before the function was ever called).
I'll be quoting Common Gotchas — The Hitchhiker's Guide to Python here:
For example like:
def append_to(element, to=[]):
to.append(element)
return to
A new list is created once when the function is defined, and the same
list is used in each successive call.
Python’s default arguments are evaluated once when the function is
defined, not each time the function is called (like it is in say,
Ruby). This means that if you use a mutable default argument and
mutate it, you will and have mutated that object for all future calls
to the function as well.
Also if you don't like to use none ant consider it an antipattern, you can use more fancier kind of this:
def append_to(element, to=list):
if callable(to):
to = to()
to.append(element)
return to
I don't know if it's better though
This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 9 years ago.
In the following function L stores the values during every call.
For example, if I call f(1), L is now [1]. When I call it again the previous L is appended with the new value. So now L is [1,1].
def f(a, L=[]):
L.append(a)
return L
But in this function:
i = 5
def f(arg=i):
print arg
i = 6
No matter how many times I call this function, the argument is still 5 - it does not remain updated between calls.
What is the reason why this does not get updated but the list does?
This is a good question. The reason why this happens is because the default arguments to functions are stored as single objects in memory, not recreated every time you call the function.
So when you have a list [] as a default argument, there will only be one list forever for the duration of that program. So when you add to the list you are adding to that one copy of the list. This is still true for numbers like 5. However, numbers are immutable in Python, and so when you alter a default argument that starts at a number, you're really making it point to a new number and not editing that 5 object, whereas many operations on lists mutate the list in place rather than returning a new list.
http://docs.python.org/3/tutorial/controlflow.html#default-argument-values
Important warning: The default value is evaluated only once. This
makes a difference when the default is a mutable object such as a
list, dictionary, or instances of most classes.
The recommended solution, if you need the behaviour of an empty list default argument without having the same empty list default argument every call to the function, is to do this:
def f(a, L = None):
if L is None:
L = []
L.append(a)
return L
The creation of an empty list is now evaluated on individual calls to the function - not once.
a.append() is different from i = 6. The first changes the array but the array stays the same object (the same but not equal). The latter on the other hand assigns a totally new value to a variable. It does not change any object (ints aren't mutable anyway).
I'm at the point in learning Python where I'm dealing with the Mutable Default Argument problem.
# BAD: if `a_list` is not passed in, the default will wrongly retain its contents between successive function calls
def bad_append(new_item, a_list=[]):
a_list.append(new_item)
return a_list
# GOOD: if `a_list` is not passed in, the default will always correctly be []
def good_append(new_item, a_list=None):
if a_list is None:
a_list = []
a_list.append(new_item)
return a_list
I understand that a_list is initialized only when the def statement is first encountered, and that's why subsequent calls of bad_append use the same list object.
What I don't understand is why good_append works any different. It looks like a_list would still be initialized only once; therefore, the if statement would only be true on the first invocation of the function, meaning a_list would only get reset to [] on the first invocation, meaning it would still accumulate all past new_item values and still be buggy.
Why isn't it? What concept am I missing? How does a_list get wiped clean every time good_append runs?
It looks like a_list would still be initialized only once
"initialization" is not something that happens to variables in Python, because variables in Python are just names. "initialization" only happens to objects, and it's done via the class' __init__ method.
When you write a = 0, that is an assignment. That is saying "a shall refer to the object that is described by the expression 0". It is not initialization; a can name anything else of any type at any later time, and that happens as a result of assigning something else to a. Assignment is just assignment. The first one is not special.
When you write def good_append(new_item, a_list=None), that is not "initializing" a_list. It is setting up an internal reference to an object, the result of evaluating None, so that when good_append is called without a second parameter, that object is automatically assigned to a_list.
meaning a_list would only get reset to [] on the first invocation
No, a_list gets set to [] any time that a_list is None to begin with. That is, when either None is passed explicitly, or the argument is omitted.
The problem with [] occurs because the expression [] is only evaluated once in this context. When the function is compiled, [] is evaluated, a specific list object is created - that happens to be empty to start - and that object is used as the default.
How does a_list get wiped clean every time good_append runs?
It doesn't. It doesn't need to be.
You know how the problem is described as being with "mutable default arguments"?
None is not mutable.
The problem occurs when you modify the object that the parameter has as a default.
a_list = [] does not modify whatever object a_list previously referred to. It cannot; arbitrary objects cannot magically transform in-place into empty lists. a_list = [] means "a_list shall stop referring to what it previously referred to, and start referring to []". The previously-referred-to object is unchanged.
When the function is compiled, and one of the arguments has a default value, that value - an object - gets baked into the function (which is also, itself, an object!). When you write code that mutates an object, the object mutates. If the object being referred to happens to be the object baked into the function, it still mutates.
But you cannot mutate None. It is immutable.
You can mutate []. It is a list, and lists are mutable. Appending an item to a list mutates the list.
The default value of a_list (or any other default value, for that matter) is stored in the function's interiors once it has been initialized and thus can be modified in any way:
>>> def f(x=[]): return x
...
>>> f.func_defaults
([],)
>>> f.func_defaults[0] is f()
True
resp. for Python 3:
>>> def f(x=[]): return x
...
>>> f.__defaults__
([],)
>>> f.__defaults__[0] is f()
True
So the value in func_defaults is the same which is as well known inside function (and returned in my example in order to access it from outside.
In other words, what happens when calling f() is an implicit x = f.func_defaults[0]. If that object is modified subsequently, you'll keep that modification.
In contrast, an assignment inside the function gets always a new []. Any modification will last until the last reference to that [] has gone; on the next function call, a new [] is created.
In order words again, it is not true that [] gets the same object on every execution, but it is (in the case of default argument) only executed once and then preserved.
The problem only exists if the default value is mutable, which None is not. What gets stored along with the function object is the default value. When the function is called, the function's context is initialized with the default value.
a_list = []
just assigns a new object to the name a_list in the context of the current function call. It does not modify None in any way.
No, in good_insert a_list is not initalised only once.
Each time the function is called without specifying the a_list argument, the default is used and a new instance of list is used and returned, the new list does not replace the default value.
The python tutorial says that
the default value is evaluated only once.
The evaluated (only once) default value is stored internally (name it x for simplicity).
case []:
When you define the function with a_list defaulted to [], if you don't provide a_list, it is assigned the internal variable x when . Therefore, when you append to a_list, you are actually appending to x (because a_list and x refer to the same variable now). When you call the function again without a_list, the updated x is re-assigned to a_list.
case None:
The value None is evaluated once and stored in x. If you don't provide, a_list, the variable x is assigned to a_list. But you don't append to x of course. You reassign an empty array to a_list. At this point x and a_list are different variables. The same way when you call the function again without a_list, it first gets the value None from x but then a_list gets assigned to an empty array again.
Note that, for the a_list = [] case, if you provide an explicit value for a_list when you call the function, the new argument does not override x because that's evaluated only once.