I was told that += can have different effects than the standard notation of i = i +. Is there a case in which i += 1 would be different from i = i + 1?
This depends entirely on the object i.
+= calls the __iadd__ method (if it exists -- falling back on __add__ if it doesn't exist) whereas + calls the __add__ method1 or the __radd__ method in a few cases2.
From an API perspective, __iadd__ is supposed to be used for modifying mutable objects in place (returning the object which was mutated) whereas __add__ should return a new instance of something. For immutable objects, both methods return a new instance, but __iadd__ will put the new instance in the current namespace with the same name that the old instance had. This is why
i = 1
i += 1
seems to increment i. In reality, you get a new integer and assign it "on top of" i -- losing one reference to the old integer. In this case, i += 1 is exactly the same as i = i + 1. But, with most mutable objects, it's a different story:
As a concrete example:
a = [1, 2, 3]
b = a
b += [1, 2, 3]
print(a) # [1, 2, 3, 1, 2, 3]
print(b) # [1, 2, 3, 1, 2, 3]
compared to:
a = [1, 2, 3]
b = a
b = b + [1, 2, 3]
print(a) # [1, 2, 3]
print(b) # [1, 2, 3, 1, 2, 3]
notice how in the first example, since b and a reference the same object, when I use += on b, it actually changes b (and a sees that change too -- After all, it's referencing the same list). In the second case however, when I do b = b + [1, 2, 3], this takes the list that b is referencing and concatenates it with a new list [1, 2, 3]. It then stores the concatenated list in the current namespace as b -- With no regard for what b was the line before.
1In the expression x + y, if x.__add__ isn't implemented or if x.__add__(y) returns NotImplemented and x and y have different types, then x + y tries to call y.__radd__(x). So, in the case where you have
foo_instance += bar_instance
if Foo doesn't implement __add__ or __iadd__ then the result here is the same as
foo_instance = bar_instance.__radd__(bar_instance, foo_instance)
2In the expression foo_instance + bar_instance, bar_instance.__radd__ will be tried before foo_instance.__add__ if the type of bar_instance is a subclass of the type of foo_instance (e.g. issubclass(Bar, Foo)). The rationale for this is that Bar is in some sense a "higher-level" object than Foo so Bar should get the option of overriding Foo's behavior.
Under the covers, i += 1 does something like this:
try:
i = i.__iadd__(1)
except AttributeError:
i = i.__add__(1)
While i = i + 1 does something like this:
i = i.__add__(1)
This is a slight oversimplification, but you get the idea: Python gives types a way to handle += specially, by creating an __iadd__ method as well as an __add__.
The intention is that mutable types, like list, will mutate themselves in __iadd__ (and then return self, unless you're doing something very tricky), while immutable types, like int, will just not implement it.
For example:
>>> l1 = []
>>> l2 = l1
>>> l1 += [3]
>>> l2
[3]
Because l2 is the same object as l1, and you mutated l1, you also mutated l2.
But:
>>> l1 = []
>>> l2 = l1
>>> l1 = l1 + [3]
>>> l2
[]
Here, you didn't mutate l1; instead, you created a new list, l1 + [3], and rebound the name l1 to point at it, leaving l2 pointing at the original list.
(In the += version, you were also rebinding l1, it's just that in that case you were rebinding it to the same list it was already bound to, so you can usually ignore that part.)
Here is an example that directly compares i += x with i = i + x:
def foo(x):
x = x + [42]
def bar(x):
x += [42]
c = [27]
foo(c); # c is not changed
bar(c); # c is changed to [27, 42]
Related
This question already has answers here:
Why does += behave unexpectedly on lists?
(9 answers)
Closed last month.
I had learned that n = n + v and n += v are the same. Until this;
def assign_value(n, v):
n += v
print(n)
l1 = [1, 2, 3]
l2 = [4, 5, 6]
assign_value(l1, l2)
print(l1)
The output will be:
[1, 2, 3, 4, 5, 6]
[1, 2, 3, 4, 5, 6]
Now when I use the expanded version:
def assign_value(n, v):
n = n + v
print(n)
l1 = [1, 2, 3]
l2 = [4, 5, 6]
assign_value(l1, l2)
print(l1)
The output will be:
[1, 2, 3, 4, 5, 6]
[1, 2, 3]
Using the += has a different result with the fully expanded operation. What is causing this?
Thats because in the first implementation you are editing the list n itself (and therefore the changes still apply when leaving the function), while on the other implementation you are creating a new temporary list with the same name, so when you leave the function the new list disappears and the variable n is linked to the original list.
the += operator works similarly to x=x+y for immutable objects (since they always create new objects), but for mutable objects such as lists they work differently. x=x+y creats a new object x while x+=y edits the current object.
It may seem counter-intuitive, but they are not always the same. In fact,
a = a + b means a = a.__add__(b), creating a new object
a += b means a = a.__iadd__(b), mutating the object
__iadd__, if absent, defaults to the __add__, but it also can (and it does, in the case of lists) mutate the original object in-place.
This works on how python treats objects and passes variables into functions.
Basically - in first example (with += )
You are passing n and v into function by "pass-by-assignment"
So n gets modified and it will be also modified out of function scope.
In second example - n is reassigned inside of the function to a new list. Which is not seen outside of the function.
In your 1st code. You changes list n itself see the below image..!
In your 2nd code. you just created a temporary list which is cleared when function call ends.. see the below images..!
In the next step when function ends the temporary list clear!!
I was going through the topic about list in Learning Python 5E book.
I notice that if we do concatenation on list, it creates new object. Extend method do not create new object i.e. In place change.
What actually happens in case of Concatenation?
For example
l = [1,2,3,4]
m = l
l = l + [5,6]
print l,m
#output
([1,2,3,4,5,6], [1,2,3,4])
And if I use Augmented assignment as follows,
l = [1,2,3,4]
m = l
l += [5,6]
print l,m
#output
([1,2,3,4,5,6], [1,2,3,4,5,6])
What is happening in background in case of both operations?
There are two methods being used there: __add__ and __iadd__. l + [5, 6] is a shortcut for l.__add__([5, 6]). l's __add__ method returns the result of addition to something else. Therefore, l = l + [5, 6] is reassigning l to the addition of l and [5, 6]. It doesn't affect m because you aren't changing the object, you are redefining the name. l += [5, 6] is a shortcut for l.__iadd__([5, 6]). In this case, __iadd__ changes the list. Since m refers to the same object, m is also affected.
Edit: If __iadd__ is not implemented, for example with immutable types like tuple and str, then Python uses __add__ instead. For example, x += y would be converted to x = x + y. Also, in x + y if x does not implement __add__, then y.__radd__(x), if available, will be used instead. Therefore x += y could actually be x = y.__radd__(x) in the background.
You can understand this better by inspecting the objects referenced in memory. Lets use id for the inspection
In the first case:
>>> l = [1,2,3,4]
>>> id(l)
4497052232
>>> m = l
>>> id(m)
4497052232
>>> l = l + [5,6]
>>> id(l)
4497052448
>>> print l, id(l), m, id(m)
[1, 2, 3, 4, 5, 6] 4497052448 [1, 2, 3, 4] 4497052232
>>>
Notice that l = l + [5,6] creates a new object in memory, and l references that.
In the second case:
>>> l = [1,2,3,4]
>>> id(l)
4497052520
>>> m = l
>>> id(m)
4497052520
>>> l += [5,6]
>>> id(l)
4497052520
>>> print l, id(l), m, id(m)
[1, 2, 3, 4, 5, 6] 4497052520 [1, 2, 3, 4, 5, 6] 4497052520
>>>
l += [5,6] references to the same object in memory. Hence the result.
So, basically += is the equivalent of inplace add.. More on this can be read here
I was told that += can have different effects than the standard notation of i = i +. Is there a case in which i += 1 would be different from i = i + 1?
This depends entirely on the object i.
+= calls the __iadd__ method (if it exists -- falling back on __add__ if it doesn't exist) whereas + calls the __add__ method1 or the __radd__ method in a few cases2.
From an API perspective, __iadd__ is supposed to be used for modifying mutable objects in place (returning the object which was mutated) whereas __add__ should return a new instance of something. For immutable objects, both methods return a new instance, but __iadd__ will put the new instance in the current namespace with the same name that the old instance had. This is why
i = 1
i += 1
seems to increment i. In reality, you get a new integer and assign it "on top of" i -- losing one reference to the old integer. In this case, i += 1 is exactly the same as i = i + 1. But, with most mutable objects, it's a different story:
As a concrete example:
a = [1, 2, 3]
b = a
b += [1, 2, 3]
print(a) # [1, 2, 3, 1, 2, 3]
print(b) # [1, 2, 3, 1, 2, 3]
compared to:
a = [1, 2, 3]
b = a
b = b + [1, 2, 3]
print(a) # [1, 2, 3]
print(b) # [1, 2, 3, 1, 2, 3]
notice how in the first example, since b and a reference the same object, when I use += on b, it actually changes b (and a sees that change too -- After all, it's referencing the same list). In the second case however, when I do b = b + [1, 2, 3], this takes the list that b is referencing and concatenates it with a new list [1, 2, 3]. It then stores the concatenated list in the current namespace as b -- With no regard for what b was the line before.
1In the expression x + y, if x.__add__ isn't implemented or if x.__add__(y) returns NotImplemented and x and y have different types, then x + y tries to call y.__radd__(x). So, in the case where you have
foo_instance += bar_instance
if Foo doesn't implement __add__ or __iadd__ then the result here is the same as
foo_instance = bar_instance.__radd__(bar_instance, foo_instance)
2In the expression foo_instance + bar_instance, bar_instance.__radd__ will be tried before foo_instance.__add__ if the type of bar_instance is a subclass of the type of foo_instance (e.g. issubclass(Bar, Foo)). The rationale for this is that Bar is in some sense a "higher-level" object than Foo so Bar should get the option of overriding Foo's behavior.
Under the covers, i += 1 does something like this:
try:
i = i.__iadd__(1)
except AttributeError:
i = i.__add__(1)
While i = i + 1 does something like this:
i = i.__add__(1)
This is a slight oversimplification, but you get the idea: Python gives types a way to handle += specially, by creating an __iadd__ method as well as an __add__.
The intention is that mutable types, like list, will mutate themselves in __iadd__ (and then return self, unless you're doing something very tricky), while immutable types, like int, will just not implement it.
For example:
>>> l1 = []
>>> l2 = l1
>>> l1 += [3]
>>> l2
[3]
Because l2 is the same object as l1, and you mutated l1, you also mutated l2.
But:
>>> l1 = []
>>> l2 = l1
>>> l1 = l1 + [3]
>>> l2
[]
Here, you didn't mutate l1; instead, you created a new list, l1 + [3], and rebound the name l1 to point at it, leaving l2 pointing at the original list.
(In the += version, you were also rebinding l1, it's just that in that case you were rebinding it to the same list it was already bound to, so you can usually ignore that part.)
Here is an example that directly compares i += x with i = i + x:
def foo(x):
x = x + [42]
def bar(x):
x += [42]
c = [27]
foo(c); # c is not changed
bar(c); # c is changed to [27, 42]
I understand that in Python regular c++ style variable assignment is replaced by references to stuff ie
a=[1,2,3]
b=a
a.append(4)
print(b) #gives [1,2,3,4]
print(a) #gives [1,2,3,4]
but I'm still confused why an analogous situation with basic types eg. integers works differently?
a=1
b=a
a+=1
print(b) # gives 1
print(a) # gives 2
But wait, it gets even more confusing when we consider loops!
li=[1,2,3]
for x in li:
x+=1
print(li) #gives [1,2,3]
Which is what I expected, but what happens if we do:
a,b,c=1,2,3
li=[a,b,c]
for x in li:
x+=1
print(li) #gives [1,2,3]
Maybe my question should be how to loop over a list of integers and change them without map() as i need a if statement in there. The only thing I can come up short of using
for x in range(len(li)):
Do stuff to li[x]
is packaging the integers in one element list. But there must be a better way.
Well, you need to think of mutable and immutable type.
For a list, it's mutable.
For a integer, it's immutable, which means you will refer to a new object if you change it. When a+=1 is executed, a will be assigned a new object, but b is still refer to the same one.
a=[1,2,3]
b=a
a.append(4)
print(b) #[1,2,3,4]
print(a) #[1,2,3,4]
Here you are modifying the list. The list content changes, but the list identity remains.
a=1
b=a
a+=1
This, however, is a reassignment. You assign a different object to a.
Note that if you did a += [4] in the 1st example, you would have seen the same result. This comes from the fact that a += something is the same as a = a.__iadd__(something), with a fallback to a = a.__add__(something) if __iadd__() doesn't exist.
The difference is that __iadd__() tries to do its job "inplace", by modifying the object it works on and returning it. So a refers to the same as before. This only works with mutable objects such as lists.
On immutable objects such as ints __add__() is called. It returns a different object, which leads to a pointing to another object than before. There is no other choice, as ints are immutable.
a,b,c=1,2,3
li=[a,b,c]
for x in li:
x+=1
print(li) #[1,2,3]
Here x += 1 means the same as x = x + 1. It changes where x refers to, but not the list contents.
Maybe my question should be how to loop over a list of integers and change them without >map() as i need a if statement in there.
for i, x in enumerate(li):
li[i] = x + 1
assigns to every list position the old value + 1.
The important thing here are the variable names. They really are just keys to a dictionary. They are resolved at runtime, depending on the current scope.
Let's have a look what names you access in your code. The locals function helps us: It shows the names in the local scope (and their value). Here's your code, with some debugging output:
a = [1, 2, 3] # a is bound
print(locals())
for x in a: # a is read, and for each iteration x is bound
x = x + 3 # x is read, the value increased and then bound to x again
print(locals())
print(locals())
print(x)
(Note I expanded x += 3 to x = x + 3 to increase visibility for the name accesses - read and write.)
First, you bind the list [1, 2, 3]to the name a. Then, you iterate over the list. During each iteration, the value is bound to the name x in the current scope. Your assignment then assigns another value to x.
Here's the output
{'a': [1, 2, 3]}
{'a': [1, 2, 3], 'x': 4}
{'a': [1, 2, 3], 'x': 5}
{'a': [1, 2, 3], 'x': 6}
{'a': [1, 2, 3], 'x': 6}
6
At no point you're accessing a, the list, and thus will never modify it.
To fix your problem, I'd use the enumerate function to get the index along with the value and then access the list using the name a to change it.
for idx, x in enumerate(a):
a[idx] = x + 3
print(a)
Output:
[4, 5, 6]
Note you might want to wrap those examples in a function, to avoid the cluttered global namespace.
For more about scopes, read the chapter in the Python tutorial. To further investigate that, use the globals function to see the names of the global namespace. (Not to be confused with the global keyword, note the missing 's'.)
Have fun!
For a C++-head it easiest tho think that every Python object is a pointer. When you write a = [1, 2, 3] you essentially write List * a = new List(1, 2, 3). When you write a = b, you essentially write List * b = a.
But when you take out actual items from the lists, these items happen to be numbers. Numbers are immutable; holding a pointer to an immutable object is about as good as holding this object by value.
So your for x in a: x += 1 is essentially
for (int x, it = a.iterator(); it->hasMore(); x=it.next()) {
x+=1; // the generated sum is silently discarded
}
which obviously has no effect.
If list elements were mutable objects you could mutate them exactly the way you wrote. See:
a = [[1], [2], [3]] # list of lists
for x in a: # x iterates over each sub-list
x.append(10)
print a # prints [[1, 10], [2, 10], [3, 10]]
But unless you have a compelling reason (e.g. a list of millions of objects under heavy memory load) you are better off making a copy of the list, applying a transformation and optionally a filter. This is easily done with a list comprehension:
a = [1, 2, 3, 0]
b = [n + 1 for n in a] # [2, 3, 4, 1]
c = [n * 10 for n in a if n < 3] # [10, 20, 0]
Either that, or you can write an explicit loop that creates another list:
source = [1, 2, 3]
target = []
for n in source:
n1 = <many lines of code involving n>
target.append(n1)
Your question has multiple parts, so it's going to be hard for one answer to cover all of them. glglgl has done a great job on most of it, but your final question is still unexplained:
Maybe my question should be how to loop over a list of integers and change them without map() as i need a if statement in there
"I need an if statement in there" doesn't mean you can't use map.
First, if you want the if to select which values you want to keep, map has a good friend named filter that does exactly that. For example, to keep only the odd numbers, but add one to each of them, you could do this:
>>> a = [1, 2, 3, 4, 5]
>>> b = []
>>> for x in a:
... if x%2:
... b.append(x+1)
Or just this:
>>> b = map(lambda x: x+1, filter(lambda x: x%2, a))
If, on the other hand, you want the if to control the expression itself—e.g., to add 1 to the odd numbers but leave the even ones alone, you can use an if expression the same way you'd use an if statement:
>>> for x in a:
... if x%2:
... b.append(x+1)
... else:
... b.append(x)
>>> b = map(lambda x: x+1 if x%2 else x, a)
Second, comprehensions are basically equivalent to map and filter, but with expressions instead of functions. If your expression would just be "call this function", then use map or filter. If your function would just be a lambda to "evaluate this expression", then use a comprehension. The above two examples get more readable this way:
>>> b = [x+1 for x in a if x%2]
>>> b = [x+1 if x%2 else x for x in a]
You can do something like this: li = [x+1 for x in li]
This question already has answers here:
Why does += behave unexpectedly on lists?
(9 answers)
Closed 9 years ago.
>>> c = [1, 2, 3]
>>> print(c, id(c))
[1, 2, 3] 43955984
>>> c += c
>>> print(c, id(c))
[1, 2, 3, 1, 2, 3] 43955984
>>> del c
>>> c = [1, 2, 3]
>>> print(c, id(c))
[1, 2, 3] 44023976
>>> c = c + c
>>> print(c, id(c))
[1, 2, 3, 1, 2, 3] 26564048
What's the difference? are += and + not supposed to be merely syntactic sugar?
docs explain it very well, I think:
__iadd__(), etc.
These methods are called to implement the augmented arithmetic assignments (+=, -=, *=, /=, //=, %=, **=, <<=, >>=, &=, ^=, |=). These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self). If a specific method is not defined, the augmented assignment falls back to the normal methods. For instance, to execute the statement x += y, where x is an instance of a class that has an __iadd__() method, x.__iadd__(y) is called.
+= are designed to implement in-place modification. in case of simple addition, new object created and it's labelled using already-used name (c).
Also, you'd notice that such behaviour of += operator only possible because of mutable nature of lists. Integers - an immutable type - won't produce the same result:
>>> c = 3
>>> print(c, id(c))
3 505389080
>>> c += c
>>> print(c, id(c))
6 505389128
They are not same
c += c append a copy of content of c to c itself
c = c + c create new object with c + c
For
foo = []
foo+=foo is syntactic sugar for foo.extend(foo) (and not foo = foo + foo)
In the first case, you're just appending members of a list into another (and not creating a new one).
The id changes in the second case because a new list is created by adding two lists. It's incidental that both are the same and the result is being bound to the same identifier than once pointed to them.
If you rephrase this question with different lists (and not c itself), it will probably become clearer.
The += operator appends the second list to the first, but the modification is in-place, so the ID remains the same.
When you use + , a new list is created, and the final "c" is a new list, so it has a different ID.
The end result is the same for both operations though.