Related
This question already has answers here:
What does "list comprehension" and similar mean? How does it work and how can I use it?
(5 answers)
Closed 7 months ago.
I saw some code like:
foo = [x for x in bar if x.occupants > 1]
What does this mean, and how does it work?
The current answers are good, but do not talk about how they are just syntactic sugar to some pattern that we are so used to.
Let's start with an example, say we have 10 numbers, and we want a subset of those that are greater than, say, 5.
>>> numbers = [12, 34, 1, 4, 4, 67, 37, 9, 0, 81]
For the above task, the below approaches below are totally identical to one another, and go from most verbose to concise, readable and pythonic:
Approach 1
result = []
for index in range(len(numbers)):
if numbers[index] > 5:
result.append(numbers[index])
print result #Prints [12, 34, 67, 37, 9, 81]
Approach 2 (Slightly cleaner, for-in loops)
result = []
for number in numbers:
if number > 5:
result.append(number)
print result #Prints [12, 34, 67, 37, 9, 81]
Approach 3 (Enter List Comprehension)
result = [number for number in numbers if number > 5]
or more generally:
[function(number) for number in numbers if condition(number)]
where:
function(x) takes an x and transforms it into something useful (like for instance: x*x)
if condition(x) returns any False-y value (False, None, empty string, empty list, etc ..) then the current iteration will be skipped (think continue). If the function return a non-False-y value then the current value makes it to the final resultant array (and goes through the transformation step above).
To understand the syntax in a slightly different manner, look at the Bonus section below.
For further information, follow the tutorial all other answers have linked: List Comprehension
Bonus
(Slightly un-pythonic, but putting it here for sake of completeness)
The example above can be written as:
result = filter(lambda x: x > 5, numbers)
The general expression above can be written as:
result = map(function, filter(condition, numbers)) #result is a list in Py2
It's a list comprehension
foo will be a filtered list of bar containing the objects with the attribute occupants > 1
bar can be a list, set, dict or any other iterable
Here is an example to clarify
>>> class Bar(object):
... def __init__(self, occupants):
... self.occupants = occupants
...
>>> bar=[Bar(0), Bar(1), Bar(2), Bar(3)]
>>> foo = [x for x in bar if x.occupants > 1]
>>> foo
[<__main__.Bar object at 0xb748516c>, <__main__.Bar object at 0xb748518c>]
So foo has 2 Bar objects, but how do we check which ones they are? Lets add a __repr__ method to Bar so it is more informative
>>> Bar.__repr__=lambda self:"Bar(occupants={0})".format(self.occupants)
>>> foo
[Bar(occupants=2), Bar(occupants=3)]
Since the programming part of question is fully answered by others it is nice to know its relation to mathematics (set theory). Actually it is the Python implementation of Set builder notation:
Defining a set by axiom of specification:
B = { x є A : S(x) }
English translation: B is a set where its members are chosen from A,
so B is a subset of A (B ⊂ A), where characteristic(s) specified by
function S holds: S(x) == True
Defining B using list comprehension:
B = [x for x in A if S(x)]
So to build B with list comprehension, member(s) of B (denoted by x) are chosen from set A where S(x) == True (inclusion condition).
Note: Function S which returns a boolean is called predicate.
This return a list which contains all the elements in bar which have occupants > 1.
The way this should work as far as I can tell is it checks to see if the list "bar" is empty (0) or consists of a singleton (1) via x.occupants where x is a defined item within the list bar and may have the characteristic of occupants. So foo gets called, moves through the list and then returns all items that pass the check condition which is x.occupant.
In a language like Java, you'd build a class called "x" where 'x' objects are then assigned to an array or similar. X would have a Field called "occupants" and each index would be checked with the x.occupants method which would return the number that is assigned to occupant. If that method returned greater than 1 (We assume an int here as a partial occupant would be odd.) the foo method (being called on the array or similar in question.) would then return an array or similar as defined in the foo method for this container array or what have you. The elements of the returned array would be the 'x' objects in the first array thingie that fit the criteria of "Greater than 1".
Python has built-in methods via list comprehension to deal with this in a much more succinct and vastly simplified way. Rather than implementing two full classes and several methods, I write that one line of code.
I'm trying to understand Python's approach to variable scope. In this example, why is f() able to alter the value of x, as perceived within main(), but not the value of n?
def f(n, x):
n = 2
x.append(4)
print('In f():', n, x)
def main():
n = 1
x = [0,1,2,3]
print('Before:', n, x)
f(n, x)
print('After: ', n, x)
main()
Output:
Before: 1 [0, 1, 2, 3]
In f(): 2 [0, 1, 2, 3, 4]
After: 1 [0, 1, 2, 3, 4]
See also: How do I pass a variable by reference?
Some answers contain the word "copy" in the context of a function call. I find it confusing.
Python doesn't copy objects you pass during a function call ever.
Function parameters are names. When you call a function, Python binds these parameters to whatever objects you pass (via names in a caller scope).
Objects can be mutable (like lists) or immutable (like integers and strings in Python). A mutable object you can change. You can't change a name, you just can bind it to another object.
Your example is not about scopes or namespaces, it is about naming and binding and mutability of an object in Python.
def f(n, x): # these `n`, `x` have nothing to do with `n` and `x` from main()
n = 2 # put `n` label on `2` balloon
x.append(4) # call `append` method of whatever object `x` is referring to.
print('In f():', n, x)
x = [] # put `x` label on `[]` ballon
# x = [] has no effect on the original list that is passed into the function
Here are nice pictures on the difference between variables in other languages and names in Python.
You've got a number of answers already, and I broadly agree with J.F. Sebastian, but you might find this useful as a shortcut:
Any time you see varname =, you're creating a new name binding within the function's scope. Whatever value varname was bound to before is lost within this scope.
Any time you see varname.foo() you're calling a method on varname. The method may alter varname (e.g. list.append). varname (or, rather, the object that varname names) may exist in more than one scope, and since it's the same object, any changes will be visible in all scopes.
[note that the global keyword creates an exception to the first case]
f doesn't actually alter the value of x (which is always the same reference to an instance of a list). Rather, it alters the contents of this list.
In both cases, a copy of a reference is passed to the function. Inside the function,
n gets assigned a new value. Only the reference inside the function is modified, not the one outside it.
x does not get assigned a new value: neither the reference inside nor outside the function are modified. Instead, x’s value is modified.
Since both the x inside the function and outside it refer to the same value, both see the modification. By contrast, the n inside the function and outside it refer to different values after n was reassigned inside the function.
I will rename variables to reduce confusion. n -> nf or nmain. x -> xf or xmain:
def f(nf, xf):
nf = 2
xf.append(4)
print 'In f():', nf, xf
def main():
nmain = 1
xmain = [0,1,2,3]
print 'Before:', nmain, xmain
f(nmain, xmain)
print 'After: ', nmain, xmain
main()
When you call the function f, the Python runtime makes a copy of xmain and assigns it to xf, and similarly assigns a copy of nmain to nf.
In the case of n, the value that is copied is 1.
In the case of x the value that is copied is not the literal list [0, 1, 2, 3]. It is a reference to that list. xf and xmain are pointing at the same list, so when you modify xf you are also modifying xmain.
If, however, you were to write something like:
xf = ["foo", "bar"]
xf.append(4)
you would find that xmain has not changed. This is because, in the line xf = ["foo", "bar"] you have change xf to point to a new list. Any changes you make to this new list will have no effects on the list that xmain still points to.
Hope that helps. :-)
If the functions are re-written with completely different variables and we call id on them, it then illustrates the point well. I didn't get this at first and read jfs' post with the great explanation, so I tried to understand/convince myself:
def f(y, z):
y = 2
z.append(4)
print ('In f(): ', id(y), id(z))
def main():
n = 1
x = [0,1,2,3]
print ('Before in main:', n, x,id(n),id(x))
f(n, x)
print ('After in main:', n, x,id(n),id(x))
main()
Before in main: 1 [0, 1, 2, 3] 94635800628352 139808499830024
In f(): 94635800628384 139808499830024
After in main: 1 [0, 1, 2, 3, 4] 94635800628352 139808499830024
z and x have the same id. Just different tags for the same underlying structure as the article says.
My general understanding is that any object variable (such as a list or a dict, among others) can be modified through its functions. What I believe you are not able to do is reassign the parameter - i.e., assign it by reference within a callable function.
That is consistent with many other languages.
Run the following short script to see how it works:
def func1(x, l1):
x = 5
l1.append("nonsense")
y = 10
list1 = ["meaning"]
func1(y, list1)
print(y)
print(list1)
It´s because a list is a mutable object. You´re not setting x to the value of [0,1,2,3], you´re defining a label to the object [0,1,2,3].
You should declare your function f() like this:
def f(n, x=None):
if x is None:
x = []
...
n is an int (immutable), and a copy is passed to the function, so in the function you are changing the copy.
X is a list (mutable), and a copy of the pointer is passed o the function so x.append(4) changes the contents of the list. However, you you said x = [0,1,2,3,4] in your function, you would not change the contents of x in main().
Python is copy by value of reference. An object occupies a field in memory, and a reference is associated with that object, but itself occupies a field in memory. And name/value is associated with a reference. In python function, it always copy the value of the reference, so in your code, n is copied to be a new name, when you assign that, it has a new space in caller stack. But for the list, the name also got copied, but it refer to the same memory(since you never assign the list a new value). That is a magic in python!
When you are passing the command n = 2 inside the function, it finds a memory space and label it as 2. But if you call the method append, you are basically refrencing to location x (whatever the value is) and do some operation on that.
Python is a pure pass-by-value language if you think about it the right way. A python variable stores the location of an object in memory. The Python variable does not store the object itself. When you pass a variable to a function, you are passing a copy of the address of the object being pointed to by the variable.
Contrast these two functions
def foo(x):
x[0] = 5
def goo(x):
x = []
Now, when you type into the shell
>>> cow = [3,4,5]
>>> foo(cow)
>>> cow
[5,4,5]
Compare this to goo.
>>> cow = [3,4,5]
>>> goo(cow)
>>> goo
[3,4,5]
In the first case, we pass a copy the address of cow to foo and foo modified the state of the object residing there. The object gets modified.
In the second case you pass a copy of the address of cow to goo. Then goo proceeds to change that copy. Effect: none.
I call this the pink house principle. If you make a copy of your address and tell a
painter to paint the house at that address pink, you will wind up with a pink house.
If you give the painter a copy of your address and tell him to change it to a new address,
the address of your house does not change.
The explanation eliminates a lot of confusion. Python passes the addresses variables store by value.
As jouell said. It's a matter of what points to what and i'd add that it's also a matter of the difference between what = does and what the .append method does.
When you define n and x in main, you tell them to point at 2 objects, namely 1 and [1,2,3]. That is what = does : it tells what your variable should point to.
When you call the function f(n,x), you tell two new local variables nf and xf to point at the same two objects as n and x.
When you use "something"="anything_new", you change what "something" points to. When you use .append, you change the object itself.
Somehow, even though you gave them the same names, n in the main() and the n in f() are not the same entity, they only originally point to the same object (same goes for x actually). A change to what one of them points to won't affect the other. However, if you instead make a change to the object itself, that will affect both variables as they both point to this same, now modified, object.
Lets illustrate the difference between the method .append and the = without defining a new function :
compare
m = [1,2,3]
n = m # this tells n to point at the same object as m does at the moment
m = [1,2,3,4] # writing m = m + [4] would also do the same
print('n = ', n,'m = ',m)
to
m = [1,2,3]
n = m
m.append(4)
print('n = ', n,'m = ',m)
In the first code, it will print n = [1, 2, 3] m = [1, 2, 3, 4], since in the 3rd line, you didnt change the object [1,2,3], but rather you told m to point to a new, different, object (using '='), while n still pointed at the original object.
In the second code, it will print n = [1, 2, 3, 4] m = [1, 2, 3, 4]. This is because here both m and n still point to the same object throughout the code, but you modified the object itself (that m is pointing to) using the .append method... Note that the result of the second code will be the same regardless of wether you write m.append(4) or n.append(4) on the 3rd line.
Once you understand that, the only confusion that remains is really to understand that, as I said, the n and x inside your f() function and the ones in your main() are NOT the same, they only initially point to the same object when you call f().
Please allow me to edit again. These concepts are my experience from learning python by try error and internet, mostly stackoverflow. There are mistakes and there are helps.
Python variables use references, I think reference as relation links from name, memory adress and value.
When we do B = A, we actually create a nickname of A, and now the A has 2 names, A and B. When we call B, we actually are calling the A. we create a ink to the value of other variable, instead of create a new same value, this is what we call reference. And this thought would lead to 2 porblems.
when we do
A = [1]
B = A # Now B is an alias of A
A.append(2) # Now the value of A had been changes
print(B)
>>> [1, 2]
# B is still an alias of A
# Which means when we call B, the real name we are calling is A
# When we do something to B, the real name of our object is A
B.append(3)
print(A)
>>> [1, 2, 3]
This is what happens when we pass arguments to functions
def test(B):
print('My name is B')
print(f'My value is {B}')
print(' I am just a nickname, My real name is A')
B.append(2)
A = [1]
test(A)
print(A)
>>> [1, 2]
We pass A as an argument of a function, but the name of this argument in that function is B.
Same one with different names.
So when we do B.append, we are doing A.append
When we pass an argument to a function, we are not passing a variable , we are passing an alias.
And here comes the 2 problems.
the equal sign always creates a new name
A = [1]
B = A
B.append(2)
A = A[0] # Now the A is a brand new name, and has nothing todo with the old A from now on.
B.append(3)
print(A)
>>> 1
# the relation of A and B is removed when we assign the name A to something else
# Now B is a independent variable of hisown.
the Equal sign is a statesment of clear brand new name,
this was the concused part of mine
A = [1, 2, 3]
# No equal sign, we are working on the origial object,
A.append(4)
>>> [1, 2, 3, 4]
# This would create a new A
A = A + [4]
>>> [1, 2, 3, 4]
and the function
def test(B):
B = [1, 2, 3] # B is a new name now, not an alias of A anymore
B.append(4) # so this operation won't effect A
A = [1, 2, 3]
test(A)
print(A)
>>> [1, 2, 3]
# ---------------------------
def test(B):
B.append(4) # B is a nickname of A, we are doing A
A = [1, 2, 3]
test(A)
print(A)
>>> [1, 2, 3, 4]
the first problem is
the left side of and equation is always a brand new name, new variable,
unless the right side is a name, like B = A, this create an alias only
The second problem, there are something would never be changed, we cannot modify the original, can only create a new one.
This is what we call immutable.
When we do A= 123 , we create a dict which contains name, value, and adress.
When we do B = A, we copy the adress and value from A to B, all operation to B effect the same adress of the value of A.
When it comes to string, numbers, and tuple. the pair of value and adress could never be change. When we put a str to some adress, it was locked right away, the result of all modifications would be put into other adress.
A = 'string' would create a protected value and adess to storage the string 'string' . Currently, there is no built-in functions or method cound modify a string with the syntax like list.append, because this code modify the original value of a adress.
the value and adress of a string, a number, or a tuple is protected, locked, immutable.
All we can work on a string is by the syntax of A = B.method , we have to create a new name to storage the new string value.
please extend this discussion if you still get confused.
this discussion help me to figure out mutable / immutable / refetence / argument / variable / name once for all, hopely this could do some help to someone too.
##############################
had modified my answer tons of times and realized i don't have to say anything, python had explained itself already.
a = 'string'
a.replace('t', '_')
print(a)
>>> 'string'
a = a.replace('t', '_')
print(a)
>>> 's_ring'
b = 100
b + 1
print(b)
>>> 100
b = b + 1
print(b)
>>> 101
def test_id(arg):
c = id(arg)
arg = 123
d = id(arg)
return
a = 'test ids'
b = id(a)
test_id(a)
e = id(a)
# b = c = e != d
# this function do change original value
del change_like_mutable(arg):
arg.append(1)
arg.insert(0, 9)
arg.remove(2)
return
test_1 = [1, 2, 3]
change_like_mutable(test_1)
# this function doesn't
def wont_change_like_str(arg):
arg = [1, 2, 3]
return
test_2 = [1, 1, 1]
wont_change_like_str(test_2)
print("Doesn't change like a imutable", test_2)
This devil is not the reference / value / mutable or not / instance, name space or variable / list or str, IT IS THE SYNTAX, EQUAL SIGN.
This question already has answers here:
What does "list comprehension" and similar mean? How does it work and how can I use it?
(5 answers)
Closed 8 months ago.
I saw some code like:
foo = [x for x in bar if x.occupants > 1]
What does this mean, and how does it work?
The current answers are good, but do not talk about how they are just syntactic sugar to some pattern that we are so used to.
Let's start with an example, say we have 10 numbers, and we want a subset of those that are greater than, say, 5.
>>> numbers = [12, 34, 1, 4, 4, 67, 37, 9, 0, 81]
For the above task, the below approaches below are totally identical to one another, and go from most verbose to concise, readable and pythonic:
Approach 1
result = []
for index in range(len(numbers)):
if numbers[index] > 5:
result.append(numbers[index])
print result #Prints [12, 34, 67, 37, 9, 81]
Approach 2 (Slightly cleaner, for-in loops)
result = []
for number in numbers:
if number > 5:
result.append(number)
print result #Prints [12, 34, 67, 37, 9, 81]
Approach 3 (Enter List Comprehension)
result = [number for number in numbers if number > 5]
or more generally:
[function(number) for number in numbers if condition(number)]
where:
function(x) takes an x and transforms it into something useful (like for instance: x*x)
if condition(x) returns any False-y value (False, None, empty string, empty list, etc ..) then the current iteration will be skipped (think continue). If the function return a non-False-y value then the current value makes it to the final resultant array (and goes through the transformation step above).
To understand the syntax in a slightly different manner, look at the Bonus section below.
For further information, follow the tutorial all other answers have linked: List Comprehension
Bonus
(Slightly un-pythonic, but putting it here for sake of completeness)
The example above can be written as:
result = filter(lambda x: x > 5, numbers)
The general expression above can be written as:
result = map(function, filter(condition, numbers)) #result is a list in Py2
It's a list comprehension
foo will be a filtered list of bar containing the objects with the attribute occupants > 1
bar can be a list, set, dict or any other iterable
Here is an example to clarify
>>> class Bar(object):
... def __init__(self, occupants):
... self.occupants = occupants
...
>>> bar=[Bar(0), Bar(1), Bar(2), Bar(3)]
>>> foo = [x for x in bar if x.occupants > 1]
>>> foo
[<__main__.Bar object at 0xb748516c>, <__main__.Bar object at 0xb748518c>]
So foo has 2 Bar objects, but how do we check which ones they are? Lets add a __repr__ method to Bar so it is more informative
>>> Bar.__repr__=lambda self:"Bar(occupants={0})".format(self.occupants)
>>> foo
[Bar(occupants=2), Bar(occupants=3)]
Since the programming part of question is fully answered by others it is nice to know its relation to mathematics (set theory). Actually it is the Python implementation of Set builder notation:
Defining a set by axiom of specification:
B = { x є A : S(x) }
English translation: B is a set where its members are chosen from A,
so B is a subset of A (B ⊂ A), where characteristic(s) specified by
function S holds: S(x) == True
Defining B using list comprehension:
B = [x for x in A if S(x)]
So to build B with list comprehension, member(s) of B (denoted by x) are chosen from set A where S(x) == True (inclusion condition).
Note: Function S which returns a boolean is called predicate.
This return a list which contains all the elements in bar which have occupants > 1.
The way this should work as far as I can tell is it checks to see if the list "bar" is empty (0) or consists of a singleton (1) via x.occupants where x is a defined item within the list bar and may have the characteristic of occupants. So foo gets called, moves through the list and then returns all items that pass the check condition which is x.occupant.
In a language like Java, you'd build a class called "x" where 'x' objects are then assigned to an array or similar. X would have a Field called "occupants" and each index would be checked with the x.occupants method which would return the number that is assigned to occupant. If that method returned greater than 1 (We assume an int here as a partial occupant would be odd.) the foo method (being called on the array or similar in question.) would then return an array or similar as defined in the foo method for this container array or what have you. The elements of the returned array would be the 'x' objects in the first array thingie that fit the criteria of "Greater than 1".
Python has built-in methods via list comprehension to deal with this in a much more succinct and vastly simplified way. Rather than implementing two full classes and several methods, I write that one line of code.
I'm confused about "x" in the python code below.
>>> # Grocery list
... grocery_list = ['apples', 'bananas', 'oranges', 'milk']
>>> for x in grocery_list:
... print(x, len(x))
I am confused about x's role in the for statement above. Is "x" a variable that is being defined within the for statement, or is it something else? It just seems different than how I am used to defining a variable, but I noticed "x" can be anything, which makes me think it is, in fact, a user-defined variable.
Please help.
Yes it's defined within the for statement. It's just a placeholder for an element in the list and can be called anything, e.g.
grocery_list = ['apples', 'bananas', 'oranges', 'milk']
for grocery in grocery_list:
print(grocery, len(grocery))
Python is a late-binding dynamic language. While Python programmers and the Python documentation frequently use the terms "variable" and "assignment" the more precise terms are "name" and "binding."
In your example x is a name of an object. At each iteration over the loop it's bound to the next object from your list. Python lists, and most other Python containers as well as many other Python object classes, feature iteration functions. That is to say that they define functions following a protocol which allows them to be used in for loops.
As you've noted a Python name (analogous to a "variable" in other languages) can be bound to any object of any type. A list can contain any mixture of object references. Thus, when you iterate over a list your "variable" (or loop name(s)) can be bound to objects of different types, potentially different types on each pass through the loop.
You can also have multiple names bound through "tuple unpacking" at each step through the iteration. For example the following snippet of code is a commonly used way to deal with dictionaries:
for key, value in some_dictionary.items():
# do something with each key and its associated value
This form isn't specific to dictionaries. The .items() method of dictionaries returns a sequence of two item tuples and this form of for loop could be used with any list or sequence which returned two-item tuples (or even two-item lists). Similarly you could see tuple unpacking used on sequence consisting of items which contain a uniform number of items:
for category, item, quantity, price in some_invoice_query():
# do something with these items, etc.
Conceptually a "variable" in most programming languages is a placeholder for data of a certain type (as well as a name by which that placeholder is referred throughout a program's source code). However, in Python (and other late-binding dynamic languages) a name is a reference to an object and conveys no constraint regarding the type or class of object to which the reference is made.
You can, rather crudely, think of Python names as if they were all void pointers in C.
x is a name in your current namespace, and the objects in grocery_list are assigned to this name one after another.
I think it is okay for you to treat x as a variable that is being defined within the for statement. Anything else is okay too. Python does not require a seperated "define" process, any variable is "defined" the first time it has been given a value.
The variable will be assigned behind the scenes each of the values of the list, in order. If the list holds references, then the reference will be assigned to the loop variable, of course.
It's almost equivalent to:
for i in xrange(len(grocery_list)):
x = grocery_list[i]
# rest of code here
But much much cleaner and (potentially) faster. The name (x) is not signicifant, it can be whatever you please.
After the loop has executed, the variable remains in scope, with the value of the last iteration that ran. So if you use a break to get out of the loop, that will show:
>>> for i in xrange(100):
... if i == 10: break
...
>>> i
10
x is a temporary variable that steps through a sequence. In lists, x will step through each item in the list.
>>> grocery_list = ['apples', 'bananas', 'oranges', 'milk']
>>> for x in grocery_list:
... print(x, len(x))
...
apples 6
bananas 7
oranges 7
milk 4
>>> print(x)
milk
EDIT: Apparently x remains defined even after the for loop exits.
A few more examples should clear things up:
>>> for x in 'some string': # x is bound to each character in the string
... print(x)
...
s
o
m
e
s
t
r
i
n
g
>>> for x in (0, 1, 2, 3): # x is bound to each number in the tuple
... print(x)
...
0
1
2
3
>>> for x in [(0,0), (1,1), (2,2)]: # x is bound to each tuple in the list
... print(x)
...
(0, 0)
(1, 1)
(2, 2)
In your example, x is the user-defined variable to which each value in grocery_list will be assigned in turn.
One must remember what a Python variable stores. It stores a location in memory where the object is pointing at is present. In other words, Python variables are basically pointers (void*s). They "know how to find their objects."
If you have
x = 5
y = 3
the assignment y = x actually hands a copy of the memory address where 5 is stored to y. Now x and y point at the copy of 3 in memory. Suppose you attempt this
x = [1,2,3]
for k in x:
k = 0
What happens? You hand k a copy of the memory address where each item is
stored. You then assign k to point a 0. The items in x are left undisturbed.
Now do this
x = [[1,2,3], [4,5,6], [7,8,9]]
for k in x:
k[0] = 0
Then x holds the list
[[0, 2, 3], [0, 5, 6], [0, 8, 9]]
You can change state of a mutable object via its memory address.
The moral: Python variables know WHERE to find their objects because they know where they are located in memory. When you assign one variable to another, you hand the recipient variable a copy of the donor variable's address. The loop variable in a for loop is just another variable.
I'm trying to understand Python's approach to variable scope. In this example, why is f() able to alter the value of x, as perceived within main(), but not the value of n?
def f(n, x):
n = 2
x.append(4)
print('In f():', n, x)
def main():
n = 1
x = [0,1,2,3]
print('Before:', n, x)
f(n, x)
print('After: ', n, x)
main()
Output:
Before: 1 [0, 1, 2, 3]
In f(): 2 [0, 1, 2, 3, 4]
After: 1 [0, 1, 2, 3, 4]
See also: How do I pass a variable by reference?
Some answers contain the word "copy" in the context of a function call. I find it confusing.
Python doesn't copy objects you pass during a function call ever.
Function parameters are names. When you call a function, Python binds these parameters to whatever objects you pass (via names in a caller scope).
Objects can be mutable (like lists) or immutable (like integers and strings in Python). A mutable object you can change. You can't change a name, you just can bind it to another object.
Your example is not about scopes or namespaces, it is about naming and binding and mutability of an object in Python.
def f(n, x): # these `n`, `x` have nothing to do with `n` and `x` from main()
n = 2 # put `n` label on `2` balloon
x.append(4) # call `append` method of whatever object `x` is referring to.
print('In f():', n, x)
x = [] # put `x` label on `[]` ballon
# x = [] has no effect on the original list that is passed into the function
Here are nice pictures on the difference between variables in other languages and names in Python.
You've got a number of answers already, and I broadly agree with J.F. Sebastian, but you might find this useful as a shortcut:
Any time you see varname =, you're creating a new name binding within the function's scope. Whatever value varname was bound to before is lost within this scope.
Any time you see varname.foo() you're calling a method on varname. The method may alter varname (e.g. list.append). varname (or, rather, the object that varname names) may exist in more than one scope, and since it's the same object, any changes will be visible in all scopes.
[note that the global keyword creates an exception to the first case]
f doesn't actually alter the value of x (which is always the same reference to an instance of a list). Rather, it alters the contents of this list.
In both cases, a copy of a reference is passed to the function. Inside the function,
n gets assigned a new value. Only the reference inside the function is modified, not the one outside it.
x does not get assigned a new value: neither the reference inside nor outside the function are modified. Instead, x’s value is modified.
Since both the x inside the function and outside it refer to the same value, both see the modification. By contrast, the n inside the function and outside it refer to different values after n was reassigned inside the function.
I will rename variables to reduce confusion. n -> nf or nmain. x -> xf or xmain:
def f(nf, xf):
nf = 2
xf.append(4)
print 'In f():', nf, xf
def main():
nmain = 1
xmain = [0,1,2,3]
print 'Before:', nmain, xmain
f(nmain, xmain)
print 'After: ', nmain, xmain
main()
When you call the function f, the Python runtime makes a copy of xmain and assigns it to xf, and similarly assigns a copy of nmain to nf.
In the case of n, the value that is copied is 1.
In the case of x the value that is copied is not the literal list [0, 1, 2, 3]. It is a reference to that list. xf and xmain are pointing at the same list, so when you modify xf you are also modifying xmain.
If, however, you were to write something like:
xf = ["foo", "bar"]
xf.append(4)
you would find that xmain has not changed. This is because, in the line xf = ["foo", "bar"] you have change xf to point to a new list. Any changes you make to this new list will have no effects on the list that xmain still points to.
Hope that helps. :-)
If the functions are re-written with completely different variables and we call id on them, it then illustrates the point well. I didn't get this at first and read jfs' post with the great explanation, so I tried to understand/convince myself:
def f(y, z):
y = 2
z.append(4)
print ('In f(): ', id(y), id(z))
def main():
n = 1
x = [0,1,2,3]
print ('Before in main:', n, x,id(n),id(x))
f(n, x)
print ('After in main:', n, x,id(n),id(x))
main()
Before in main: 1 [0, 1, 2, 3] 94635800628352 139808499830024
In f(): 94635800628384 139808499830024
After in main: 1 [0, 1, 2, 3, 4] 94635800628352 139808499830024
z and x have the same id. Just different tags for the same underlying structure as the article says.
My general understanding is that any object variable (such as a list or a dict, among others) can be modified through its functions. What I believe you are not able to do is reassign the parameter - i.e., assign it by reference within a callable function.
That is consistent with many other languages.
Run the following short script to see how it works:
def func1(x, l1):
x = 5
l1.append("nonsense")
y = 10
list1 = ["meaning"]
func1(y, list1)
print(y)
print(list1)
It´s because a list is a mutable object. You´re not setting x to the value of [0,1,2,3], you´re defining a label to the object [0,1,2,3].
You should declare your function f() like this:
def f(n, x=None):
if x is None:
x = []
...
n is an int (immutable), and a copy is passed to the function, so in the function you are changing the copy.
X is a list (mutable), and a copy of the pointer is passed o the function so x.append(4) changes the contents of the list. However, you you said x = [0,1,2,3,4] in your function, you would not change the contents of x in main().
Python is copy by value of reference. An object occupies a field in memory, and a reference is associated with that object, but itself occupies a field in memory. And name/value is associated with a reference. In python function, it always copy the value of the reference, so in your code, n is copied to be a new name, when you assign that, it has a new space in caller stack. But for the list, the name also got copied, but it refer to the same memory(since you never assign the list a new value). That is a magic in python!
When you are passing the command n = 2 inside the function, it finds a memory space and label it as 2. But if you call the method append, you are basically refrencing to location x (whatever the value is) and do some operation on that.
Python is a pure pass-by-value language if you think about it the right way. A python variable stores the location of an object in memory. The Python variable does not store the object itself. When you pass a variable to a function, you are passing a copy of the address of the object being pointed to by the variable.
Contrast these two functions
def foo(x):
x[0] = 5
def goo(x):
x = []
Now, when you type into the shell
>>> cow = [3,4,5]
>>> foo(cow)
>>> cow
[5,4,5]
Compare this to goo.
>>> cow = [3,4,5]
>>> goo(cow)
>>> goo
[3,4,5]
In the first case, we pass a copy the address of cow to foo and foo modified the state of the object residing there. The object gets modified.
In the second case you pass a copy of the address of cow to goo. Then goo proceeds to change that copy. Effect: none.
I call this the pink house principle. If you make a copy of your address and tell a
painter to paint the house at that address pink, you will wind up with a pink house.
If you give the painter a copy of your address and tell him to change it to a new address,
the address of your house does not change.
The explanation eliminates a lot of confusion. Python passes the addresses variables store by value.
As jouell said. It's a matter of what points to what and i'd add that it's also a matter of the difference between what = does and what the .append method does.
When you define n and x in main, you tell them to point at 2 objects, namely 1 and [1,2,3]. That is what = does : it tells what your variable should point to.
When you call the function f(n,x), you tell two new local variables nf and xf to point at the same two objects as n and x.
When you use "something"="anything_new", you change what "something" points to. When you use .append, you change the object itself.
Somehow, even though you gave them the same names, n in the main() and the n in f() are not the same entity, they only originally point to the same object (same goes for x actually). A change to what one of them points to won't affect the other. However, if you instead make a change to the object itself, that will affect both variables as they both point to this same, now modified, object.
Lets illustrate the difference between the method .append and the = without defining a new function :
compare
m = [1,2,3]
n = m # this tells n to point at the same object as m does at the moment
m = [1,2,3,4] # writing m = m + [4] would also do the same
print('n = ', n,'m = ',m)
to
m = [1,2,3]
n = m
m.append(4)
print('n = ', n,'m = ',m)
In the first code, it will print n = [1, 2, 3] m = [1, 2, 3, 4], since in the 3rd line, you didnt change the object [1,2,3], but rather you told m to point to a new, different, object (using '='), while n still pointed at the original object.
In the second code, it will print n = [1, 2, 3, 4] m = [1, 2, 3, 4]. This is because here both m and n still point to the same object throughout the code, but you modified the object itself (that m is pointing to) using the .append method... Note that the result of the second code will be the same regardless of wether you write m.append(4) or n.append(4) on the 3rd line.
Once you understand that, the only confusion that remains is really to understand that, as I said, the n and x inside your f() function and the ones in your main() are NOT the same, they only initially point to the same object when you call f().
Please allow me to edit again. These concepts are my experience from learning python by try error and internet, mostly stackoverflow. There are mistakes and there are helps.
Python variables use references, I think reference as relation links from name, memory adress and value.
When we do B = A, we actually create a nickname of A, and now the A has 2 names, A and B. When we call B, we actually are calling the A. we create a ink to the value of other variable, instead of create a new same value, this is what we call reference. And this thought would lead to 2 porblems.
when we do
A = [1]
B = A # Now B is an alias of A
A.append(2) # Now the value of A had been changes
print(B)
>>> [1, 2]
# B is still an alias of A
# Which means when we call B, the real name we are calling is A
# When we do something to B, the real name of our object is A
B.append(3)
print(A)
>>> [1, 2, 3]
This is what happens when we pass arguments to functions
def test(B):
print('My name is B')
print(f'My value is {B}')
print(' I am just a nickname, My real name is A')
B.append(2)
A = [1]
test(A)
print(A)
>>> [1, 2]
We pass A as an argument of a function, but the name of this argument in that function is B.
Same one with different names.
So when we do B.append, we are doing A.append
When we pass an argument to a function, we are not passing a variable , we are passing an alias.
And here comes the 2 problems.
the equal sign always creates a new name
A = [1]
B = A
B.append(2)
A = A[0] # Now the A is a brand new name, and has nothing todo with the old A from now on.
B.append(3)
print(A)
>>> 1
# the relation of A and B is removed when we assign the name A to something else
# Now B is a independent variable of hisown.
the Equal sign is a statesment of clear brand new name,
this was the concused part of mine
A = [1, 2, 3]
# No equal sign, we are working on the origial object,
A.append(4)
>>> [1, 2, 3, 4]
# This would create a new A
A = A + [4]
>>> [1, 2, 3, 4]
and the function
def test(B):
B = [1, 2, 3] # B is a new name now, not an alias of A anymore
B.append(4) # so this operation won't effect A
A = [1, 2, 3]
test(A)
print(A)
>>> [1, 2, 3]
# ---------------------------
def test(B):
B.append(4) # B is a nickname of A, we are doing A
A = [1, 2, 3]
test(A)
print(A)
>>> [1, 2, 3, 4]
the first problem is
the left side of and equation is always a brand new name, new variable,
unless the right side is a name, like B = A, this create an alias only
The second problem, there are something would never be changed, we cannot modify the original, can only create a new one.
This is what we call immutable.
When we do A= 123 , we create a dict which contains name, value, and adress.
When we do B = A, we copy the adress and value from A to B, all operation to B effect the same adress of the value of A.
When it comes to string, numbers, and tuple. the pair of value and adress could never be change. When we put a str to some adress, it was locked right away, the result of all modifications would be put into other adress.
A = 'string' would create a protected value and adess to storage the string 'string' . Currently, there is no built-in functions or method cound modify a string with the syntax like list.append, because this code modify the original value of a adress.
the value and adress of a string, a number, or a tuple is protected, locked, immutable.
All we can work on a string is by the syntax of A = B.method , we have to create a new name to storage the new string value.
please extend this discussion if you still get confused.
this discussion help me to figure out mutable / immutable / refetence / argument / variable / name once for all, hopely this could do some help to someone too.
##############################
had modified my answer tons of times and realized i don't have to say anything, python had explained itself already.
a = 'string'
a.replace('t', '_')
print(a)
>>> 'string'
a = a.replace('t', '_')
print(a)
>>> 's_ring'
b = 100
b + 1
print(b)
>>> 100
b = b + 1
print(b)
>>> 101
def test_id(arg):
c = id(arg)
arg = 123
d = id(arg)
return
a = 'test ids'
b = id(a)
test_id(a)
e = id(a)
# b = c = e != d
# this function do change original value
del change_like_mutable(arg):
arg.append(1)
arg.insert(0, 9)
arg.remove(2)
return
test_1 = [1, 2, 3]
change_like_mutable(test_1)
# this function doesn't
def wont_change_like_str(arg):
arg = [1, 2, 3]
return
test_2 = [1, 1, 1]
wont_change_like_str(test_2)
print("Doesn't change like a imutable", test_2)
This devil is not the reference / value / mutable or not / instance, name space or variable / list or str, IT IS THE SYNTAX, EQUAL SIGN.