We Keep Coding

Modifying Python lists via slices and for-loops?

I was trying to modify the values in lists via slices and for-loops, and ran into some pretty interesting behavior. I would appreciate if someone could explain what's happening internally here.
>>> x = [1,2,3,4,5]
>>> x[:2] = [6,7] #slices can be modified
>>> x
[6, 7, 3, 4, 5]
>>> x[:2][0] = 8 #indices of slices cannot be modified
>>> x
[6, 7, 3, 4, 5]
>>> x[:2][:1] = [8] #slices of slices cannot be modified
>>> x
[6, 7, 3, 4, 5]
>>> for z in x: #this version of a for-loop cannot modify lists
... z += 1
...
>>> x
[6, 7, 3, 4, 5]
>>> for i in range(len(x)): #this version of a for-loop can modify lists
... x[i] += 1
...
>>> x
[7, 8, 4, 5, 6]
>>> y = x[:2] #if I assign a slice to a var, it can be modified...
>>> y[0] = 1
>>> y
[1, 8]
>>> x #...but it has no impact on the original list
[7, 8, 4, 5, 6]

Let's break down your comments 1 by 1:
1.) x[:2] = [6, 7] slices can be modified:
See these answers here. It's calling the __setitem__ method from the list object and assigning the slice to it. Each time you reference x[:2] a new slice object is created (you can simple do id(x[:2]) and it's apparent, not once will it be the same id).
2.) indices of slices cannot be modified:
That's not true. It couldn't be modified because you're performing the assignment on the slice instance, not the list, so it doesn't trigger the __setitem__ to be performed on the list. Also, int are immutable so it cannot be changed either way.
3.) slices of slices cannot be modified:
See above. Same reason - you are assigning to an instance of the slice and not modifying the list directly.
4.) this version of a for-loop cannot modify lists:
z being referenced here is the actual objects in the elements of x. If you ran the for loop with id(z) you'll note that they're identical to id(6), id(7), id(3), id(4), id(5). Even though list contains all 5 identical references, when you do z = ... you are only assigning the new value to the object z, not the object that is stored in list. If you want to modify the list, you'll need to assign it by index, for the same reason you can't expect 1 = 6 will turn x into [6, 2, 3, 4, 5].
5.) this version of a for-loop can modify lists:
See my answer above. Now you are directly performing item assignment on the list instead of its representation.
6.) if I assign a slice to a var, it can be modified:
If you've been following so far, you'll realize now you are assigning the instance of x[:2] to the object y, which is now a list. The story follows - you perform an item assignment by index on y, of course it will be updated.
7.) ...but it has no impact on the original list:
Of course. x and y are two different objects. id(x) != id(y), therefore any operation performed on x will not affect y whatsoever. if you however assigned y = x and then made a change to y, then yes, x will be affected as well.
To expand a bit on for z in x:, say you have a class foo() and assigned two instances of such to the list f:
f1 = foo()
f2 = foo()
f = [f1, f2]
f
# [<__main__.foo at 0x15f4b898>, <__main__.foo at 0x15f4d3c8>]
Note that the reference in question is the actual foo instance, not the object f1 and f2. So even if I did the following:
f1 = 'hello'
f
# [<__main__.foo at 0x15f4b898>, <__main__.foo at 0x15f4d3c8>]
f still remains unchanged since the foo instances remains the same even though object f1 now is assigned to a different value. For the same reason, whenever you make changes to z in for z in x:, you are only affecting the object z, but nothing in the list is changed until you update x by index.
If however the object have attribute or is mutable, you can directly update the referenced object in the loop:
x = ['foo']
y = ['foo']
lst = [x,y]
lst
# [['foo'], ['foo']]
for z in lst:
z.append('bar')
lst
# [['foo', 'bar'], ['foo', 'bar']]
x.append('something')
lst
# [['foo', 'bar', 'something'], ['foo', 'bar']]
That is because you are directly updating the object in reference instead of assigning to object z. If you however assigned x or y to a new object, lst will not be affected.

There is nothing odd happening here. Any slice that you obtain from a list is a new object containing copies of your original list. The same is true for tuples.
When you iterate through your list, you get the object which the iteration yields. Since ints are immutable in Python you can't change the state of int objects. Each time you add two ints a new int object is created. So your "version of a for-loop [which] cannot modify lists" is not really trying to modify anything because it will not assign the result of the addition back to the list.
Maybe you can guess now why your second approach is different. It uses a special slicing syntax which is not really creating a slice of your list and allows you to assign to the list (documentation). The newly created object created by the addition operation is stored in the list through this method.
For understanding your last (and your first) examples, it is important to know that slicing creates (at least for lists and tuples, technically you could override this in your own classes) a partial copy of your list. Any change to this new object will, as you already found out, not change anything in your original list.

Why does concatenating a list to another one creates another object in memory, whereas other manipulations induce mutation? [duplicate]

This question already has answers here:
Object id in Python
(4 answers)
Closed 5 years ago.
Case A:
list1=[0, 1, 2, 3]
list2=list1
list1=list1+[4]
print(list1)
print(list2)
Output:
[0, 1, 2, 3, 4]
[0, 1, 2, 3]
(This non-mutating behavior also happens when concatenating a list of more than a single entry, and when 'multiplying' the list, e.g. list1=list1*2, in fact any type of "re-assignment" that performs an operation with an infix operator to the list and then assigns the result of that operation to the same list name using "=" )
In this case the original list object that list1 pointed to has not been altered in memory and list2 still points to it, another object has simply been created in memory for the result of the concatenation that list1 now points to (there are now two distinct, different list objects in memory)
Case B:
list1=[0, 1, 2, 3]
list2=list1
list1.append(4)
print(list1)
print(list2)
---
Output :
[0, 1, 2, 3, 4]
[0, 1, 2, 3, 4]
Case C:
list1=[0, 1, 2, 3]
list2=list1
list1[-1]="foo"
print(list1)
print(list2)
Outputs:
[0, 1, 2, 'foo']
[0, 1, 2, 'foo']
In case B and C the original list object that list1 points to has mutated, list2 still points to that same object, and as a result the value of list2 has changed. (there is still one single list object in memory and it has mutated).
This behavior seems inconsistent to me as a noob. Is there a good reason/utility for this?
EDIT :
I changed the list variables names from "list" and "list_copy" to "list1" and "list2" as this was clearly a very poor and confusing choice of names.
I chose Kabanus' answer as I liked how he pointed out that mutating operations are always(?) explicit in python.
In fact a short and simple answer to my question can be done summarizing Kabanus' answer into two of his statements :
-"In python mutating operations are explicit"
-"The addition[or multiplication] operator [performed on list objects] creates a new object, and doesn't change x[the list object] implicitly."
I could also add:
-"Every time you use square brackets to describe a list, that creates a new list object"
[this is from : http://www-inst.eecs.berkeley.edu/~selfpace/cs9honline/Q2/mutation.html , great explanations there on this topic]
Also I realized after Kabanus' answer how careful one must be tracking mutations in a program :
l=[1,2]
z=l
l+=[3]
z=z+[3]
and
l=[1,2]
z=l
z=z+[3]
l+=[3]
Will yield completely different values for z. This must be a frequent source of errors isn't it?
I'm only in the beginning of my learning and haven't delved deeply into OOP concepts just yet, but I think I'm already starting to understand what the fuss around functional paradigm is about...

l += [4]
is equivalent to:
l.append(4)
and won't create a copy.
l = l + [4]
is an assignment to l, it first evaluates the right side of the assignment, then assigns the resulting object to name l. There is no way for this operation to be mutating l.
Update: I guess I haven't made myself clear enough. Of course operations on the RHS of the assignment may involve mutating the object that is the current value of LHS; but finally, the result of computing RHS is assigned to the LHS, thus overwriting any previous mutations. Example:
def increment_first(x):
x[0] += 1
return []
l = [ 1 ]
l = increment_first(l)
While the call to increment_first will increment l[0] as its side effect, the mutated list object will be lost anyway as soon as the value of RHS (in this case - an empty list) is assigned to l.

This is by design. The point is python does not like non explicit side effects. Suppose this valid line in your file:
x=[1,2]
print(x+[3,4])
Note there is no assignment, but it's still a valid line. Do you expect x to have changed after that second line? For me it doesn't make sense.
That's what you're seeing - the addition operator creates a new object, and doesn't change x implicitly. If you feel it should, then what about:
[3,4]+x
Of course, addition does not change behavior in assignment, to avoid confusion.
In python mutating operations are explicit:
x+=[3,4]
Or your example:
x[0]=1
Here you are explicitly asking to change a cell, i.e. explicit mutation. These things are consistent - an operation is always a mutation or it isn't, and it won't be both. Usually it makes sense as well, such as concatenating lists on the fly.

Explanation and preferred method for appending a list in Python [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I'm learning Python, coming from a C#/Java background, and was playing around with list behavior. I've read some of the documentation, but I don't understand how, or why, slices with indices larger than length-1 make is possible to append items.
ls = ["a", "b", "c", "d"]
n = len(ls) # n = 4
letter = ls[4] # index out of range (list indexes are 0 - 3)
# the following all do the same thing
ls += ["echo"]
ls.append("foxtrot")
ls[len(ls):] = ["golf"] # equivalent to ls[len(ls): len(ls)]
print(ls)
Although it seems odd to me, I understand how slices can modify the list they operate on. What I don't get is why list[len(list)] results in the expected out of bounds error, but list[len(list):] doesn't. I understand that slices are fundamentally different than indexes, just not what happens internally when a slice begins with an index outside of the list values.
Why can I return a slice from a list that starts with a non-existent element (len(list))? And why does this allow me to expand the list?
Also, of the first three above methods for appending an item, which one is preferred from a convention or performance perspective? Are there performance disadvantages to any of them?

While a.append(x) and a[len(a):] = [x] do the same thing, the first is clearer and should almost always be used. Slice replacement may have a place, but you shouldn't look for reasons to do it. If the situation warrants it, it should be obvious when to do it.
Start with the fact that indexing returns an element of the list, and slicing returns a "sublist" of a list. While an element at a particular either exists or does not exist, it is mathematically convenient to think of every list have the empty list as a "sublist" between any two adjacent positions. Consider
>>> a = [1, 2, 3]
>>> a[2:2]
[]
Now consider the following sequence. At each step, we increment the starting index of the slice
>>> a[0:] # A copy of the list
[1, 2, 3]
>>> a[1:]
[2, 3]
>>> a[2:]
[3]
Each time we do so, we remove one more element from the beginning of the list for the resulting slice. So what should happen when the starting index finally equals the final index (which is implicitly the length of the list)?
>>> a[3:]
[]
You get the empty list, which is consistent with the following identities
a + [] = a
a[:k] + a[k:] = a
(The first is a special case of the second where k >= len(a).
Now, once we see why it makes sense for the slice to exist, what does it mean to replace that slice with another list? In other words, what list do we get if we replace the empty list at the end of [1, 2, 3] with the list [4]? You get [1, 2, 3, 4], which is consistent with
[1, 2, 3] + [4] = [1, 2, 3, 4]
Consider another sequence, this time involving slice replacements.
>>> a = [1,2,3]; a[0:] = ['a']; a
['a']
>>> a = [1,2,3]; a[1:] = ['a']; a
[1, 'a']
>>> a = [1,2,3]; a[2:] = ['a']; a
[1, 2, 'a']
>>> a = [1,2,3]; a[3:] = ['a']; a
[1, 2, 3, 'a']
Another way of looking at slice replacement is to treat it as appending a list to a sublist of the original. When that sublist is the list itself, then it is equal to appending to the original.

Why can I return a slice from a list that starts with a non-existent element?
I can't tell for sure, but my guess would be that it's because a reasonable value can be returned (there are no elements this refers to, so return an empty list).
And why does this allow me to expand the list?
Your slice selects the last 0 elements of the list. You then replace those elements with the elements of another list.
Also, of the first three above methods for appending an item, which one is preferred
Use .append() if you're adding a single element, and .extend() if you have elements inside another list.
I'm not sure about performance, but my guess would be that those are pretty well optimized.

How Does Python Swap Elements inside a List?

I am new to Python, and just learned about mutable and immutable objects. It appears that when we are swapping elements inside a list, Python creates copies of the individual elements and then copy them back into the list.
In the above example, we begin with n = [[1, 2], [3, 4]]. The first sublist, [1, 2] occupies ID# 24, and the second sublist [3, 4] occupies ID# 96. At first, I thought that since lists are mutable, after swapping, ID# 24 holds [3, 4] and ID# 96 holds [1, 2]. But as we can see in the test above, that is not the case. Rather, Python is pointing to the objects in a manner determined by our swap; the first slot of the list points to 96 and the second slot points to 24.
Certainly, the ID# of n did not change, so n did not violate the definition of a mutable object. But the way the sublists got swapped seems to be a caveat. Can you kindly confirm or explain in more precise language?

You are mistaken about IDs. The ID#24 is the ID of the list [1,2]. It is NOT the memory address or ID of index 0 in n.
You might want to try an example like this to confirm:
>>> x = [1, 2]
>>> id(x)
4332120256
>>> l = [x, 3]
>>> id(l[0])
4332120256
>>>

From the documentation:
Assignment of an object to a target list is recursively defined as follows.
If the target list is a single target: The object is assigned to that target.
If the target list is a comma-separated list of targets: The object must be an iterable with the same number of items as there are targets in the target list, and the items are assigned, from left to right, to the corresponding targets.
If you read that closely, you'll see that you have the scenario:
target_list = expression_list
The expression list is evaluated first, and evaluates into a tuple:
An assignment statement evaluates the expression list (remember that this can be a single expression or a comma-separated list, the latter yielding a tuple)
So if you have:
a = [1, 2]
a[1], a[0] = a[0], a[1]
It is effectively like writing the following:
a = [1, 2]
temp = (a[0], a[1]) # Expression list evaluates to tuple.
# target_list assignment to corresponding elements from expression list.
a[1] = temp[0]
a[0] = temp[1]

Python: .append(0)

I would like to ask what the following does in Python.
It was taken from http://danieljlewis.org/files/2010/06/Jenks.pdf
I have entered comments telling what I think is happening there.
# Seems to be a function that returns a float vector
# dataList seems to be a vector of flat.
# numClass seems to an int
def getJenksBreaks( dataList, numClass ):
# dataList seems to be a vector of float. "Sort" seems to sort it ascendingly
dataList.sort()
# create a 1-dimensional vector
mat1 = []
# "in range" seems to be something like "for i = 0 to len(dataList)+1)
for i in range(0,len(dataList)+1):
# create a 1-dimensional-vector?
temp = []
for j in range(0,numClass+1):
# append a zero to the vector?
temp.append(0)
# append the vector to a vector??
mat1.append(temp)
(...)
I am a little confused because in the pdf there are no explicit variable declarations. However I think and hope I could guess the variables.

Yes, the method append() adds elements to the end of the list. I think your interpretation of the code is correct.
But note the following:
x =[1,2,3,4]
x.append(5)
print(x)
[1, 2, 3, 4, 5]
while
x.append([6,7])
print(x)
[1, 2, 3, 4, 5, [6, 7]]
If you want something like
[1, 2, 3, 4, 5, 6, 7]
you may use extend()
x.extend([6,7])
print(x)
[1, 2, 3, 4, 5, 6, 7]

Python doesn't have explicit variable declarations. It's dynamically typed, variables are whatever type they get assigned to.
Your assessment of the code is pretty much correct.
One detail: The range function goes up to, but does not include, the last element. So the +1 in the second argument to range causes the last iterated value to be len(dataList) and numClass, respectively. This looks suspicious, because the range is zero-indexed, which means it will perform a total of len(dataList) + 1 iterations (which seems suspicious).
Presumably dataList.sort() modifies the original value of dataList, which is the traditional behavior of the .sort() method.
It is indeed appending the new vector to the initial one, if you look at the full source code there are several blocks that continue to concatenate more vectors to mat1.

append is a list function used to append a value at the end of the list
mat1 and temp together are creating a 2D array (eg = [[], [], []]) or matrix of (m x n)
where m = len(dataList)+1 and n = numClass
the resultant matrix is a zero martix as all its value is 0.

In Python, variables are implicitely declared. When you type this:
i = 1
i is set to a value of 1, which happens to be an integer. So we will talk of i as being an integer, although i is only a reference to an integer value. The consequence of that is that you don't need type declarations as in C++ or Java.
Your understanding is mostly correct, as for the comments. [] refers to a list. You can think of it as a linked-list (although its actual implementation is closer to std::vectors for instance).
As Python variables are only references to objects in general, lists are effectively lists of references, and can potentially hold any kind of values. This is valid Python:
# A vector of numbers
vect = [1.0, 2.0, 3.0, 4.0]
But this is perfectly valid code as well:
# The list of my objects:
list = [1, [2,"a"], True, 'foo', object()]
This list contains an integer, another list, a boolean... In Python, you usually rely on duck typing for your variable types, so this is not a problem.
Finally, one of the methods of list is sort, which sorts it in-place, as you correctly guessed, and the range function generates a range of numbers.
The syntax for x in L: ... iterates over the content of L (assuming it is iterable) and sets the variable x to each of the successive values in that context. For example:
>>> for x in ['a', 'b', 'c']:
... print x
a
b
c
Since range generates a range of numbers, this is effectively the idiomatic way to generate a for i = 0; i < N; i += 1 type of loop:
>>> for i in range(4): # range(4) == [0,1,2,3]
... print i
0
1
2
3