Given the below examples:
array = [1,2,3,4,0]
In: array[0] += 2
Out: 3
In: array[1:3] += 2
Out: TypeError: 'int' object is not iterable
In: array[1:3] += [100, 100]
Out: [1, 2, 3, 100, 100, 4, 5]
Can someone explain me why the two last examples wont return something like [1,102,103,4,0] AND if it is possible doing this with a simple slice, not using a for loop...
When using the slice operator it refers to sub-part of the list, so operating on it requires a list too (we can use the "add" operator on list with list, and not with an int, unlike in some other languages).
Therefore the following:
array[1:3] += 2
Throws:
TypeError: 'int' object is not iterable
Because 2 is not a list (actually an iterable, which is more general than list).
But:
array[1:3] += [100, 100]
Works and adds (actually appends) the two elements in the middle (index 3) of array according to indexes:
[3, 2, 3, 100, 100, 4, 0]
Without using a for loop, as requested
If you want to add to the values in the slice:
array = [1,2,3,4,0]
array.__setitem__(slice(1, 3), [x+2 for x in array[1:3]])
# [1, 102, 103, 4, 0]
print(array)
Which can be written also as:
array = [1,2,3,4,0]
def apply_on_slice(lst, start, end, callable):
array.__setitem__(slice(start, end), [callable(x) for x in array[start:end]])
apply_on_slice(array, 1, 3, lambda x : x + 100)
# [1, 102, 103, 4, 0]
print(array)
Using a for loop
Here are some other options to do so, elegantly:
array[1:3] = (x+2 for x in array[1:3])
Or of course, using a regular for loop, which is more efficient than using slicing twice:
for i in range(1, 3):
array[i] += 2
You are clearly expecting the operation to be applied element-wise, as in R and other languages (and within Python, in numpy arrays and the like). E.g., adding 2 to a list would add 2 to each of the list's elements. This is not how Python lists work: Each of the statements you ask about constructs one object on each side of the operator (a list or list slice, a list element, an integer), then applies the operation (just once) to these two objects. So if you "add" two lists you get concatenation, if you try to add a list and an int you get a TypeError, etc. The details you can read in #Aviv's answer.
Related
a_list = [1,2,4]
How to change a_list to
a_list = [1,2,3]
without just reassigning it by doing
a_list = [1,2,3]
I just want the last element changed. Do I have to use a for loop or is it just not possible?
Python datastructures are introduced in the 3rd chapter of the python.org documentation / tutorial.
Lists are introduced here.
squares = [1, 4, 9, 16, 25]
Like strings (and all other built-in sequence types), lists can be
indexed and sliced:
squares[0] # indexing returns the item
> 1
squares[-1] # indexing from the back
> 25
[ ... snipp ...]
Unlike strings, which are immutable, lists are a mutable type, i.e. it
is possible to change their content:
cubes = [1, 8, 27, 65, 125] # something's wrong here: 4 cubed is 64
cubes[3] = 64 # replace the wrong value
More methods applicable to lists are handled in chapter 5 of the python documentation
Consider this:
numbers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
These are correct statements in Python to remove elements:
numbers[0:2] = []
numbers[3:5] = []
However the statement below is not allowed:
numbers[::2] = []
ValueError: attempt to assign sequence of size 0 to extended slice of size 5
What prevents such a statement in Python?
ValueError: attempt to assign sequence of size 0 to extended slice of size 5
What prevents such a statement in Python?
It is noted in the documentation that the replacement must have the same length for the case where there is an explicit step (which is 2 in your case).
Operation
Result
Notes
s[i:j] = t
slice of s from i to j is replaced by the contents of the iterable t
s[i:j:k] = t
the elements of s[i:j:k] are replaced by those of t
(1) t must have the same length as the slice it is replacing.
The correct way is also documented there.
Operation
Result
Notes
del s[i:j]
same as s[i:j] = []
del s[i:j:k]
removes the elements of s[i:j:k] from the list
Code:
numbers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
del numbers[::2]
print(numbers)
Output:
[1, 3, 5, 7, 9]
There is no need for numbers[::2] = [] to have the behaviour of deleting every second element, because you can already do that by writing numbers[:] = numbers[1::2]. If instead you want to (e.g.) replace every second element with the value 1, you can write one of the below, which are explicit about their behaviour.
for i in range(0, len(numbers), 2):
numbers[i] = 1
# or:
numbers[::2] = [1] * len(numbers[::2])
It is not obvious what the correct behaviour should be for assigning m elements to a non-contiguous slice of n list locations when m != n. In the comments you propose a possible behaviour, but your proposal is not consistent with how slice assignment works in other cases (normally, each element on the right-hand side gets used once on the left-hand side) and certainly doesn't fulfil the principle of least astonishment. In cases like this, I think there is no (non-raising) behaviour that most people would expect, so raising an error is the best option.
You have to create a new list with the slicing already mentioned in other answers:
numbers = numbers[1::2]
if you work on the same list you incur into heavy performance loss because inserting or deleting (not appending!) an element to a list is O(N). Since you have O(N) for each insertion and you have N/2 insertions your total cost is O(N**2). You really don't want that cost.
Creating a new list with the output of the slicing, on the other hand, has just O(N) total cost.
I have an array that I would like to iterate over and modify the array itself, through inserts or deletions.
for idx, ele in enumerate(np.nditer(array)):
if idx + 1 < array.shape[0] and ele > array[idx+1]:
array = np.delete(array, idx+1)
print(ele)
Given [5, 4, 3, 2, 1] I want the loop to print out 5 3 1 because 4 and 2 are smaller than their previous elements. But because python creates an iterator based on the first instance of array so it prints 5 4 3 2 1. I want to know if I can get
Generally speaking I want the iterator to be modified if I modify the array within the body of my loop.
You cannot mutate the length of a numpy array, because numpy assigns the required memory for an array upon its creation.
With
array = np.delete(array, idx+1)
You are creating a new array on the right hand side of the = and reassign the name array.
The return value for enumerate(np.nditer(array)) has already been created at that point and won't recognize that the name array has been rebound.
In principle, you can iterate over a sequence and mutate its length at the same time (generally not a good idea). The object just needs to have methods that let you mutate its length (like lists, for example).
Consider:
>>> l = [5, 4, 3, 2, 1]
>>> for idx, ele in enumerate(l):
...: if ele == 3:
...: l.pop(idx) # mutates l
...: print(ele)
...:
5
4
3
1
>>> l
[5, 4, 2, 1]
Notice that
l is mutated.
The loop does not print 2 because popping an element reduces the indexes of all the remaining elements by one. Now l[2] == 2, but index 2 has already been visited by the iterator, so the next print-call prints l[3] which is 1.
This proves that mutations to l have effect on subsequent iterations.
Instead of looping over an array, you can use where method to find
indices of elements meeting some condition.
Then to delete the selected element (or elements), you can use
delete method, passing the source array, and a list of indices.
Then save the result, e.g. under the same variable.
To add an element, you can use append or insert methods
(for details see Numpy documentation).
I have also found a SO post concerning how to loop and delete over an array.
See Deleting elements from numpy array with iteration
I am learning trees in python, which I was showed with these functions to build a tree:
def tree(label, branches=[]):
return [label] + list(branches)
def label(tree):
return tree[0]
def branches(tree):
return tree[1:]
There is a function that can extract all the nodes of a tree into a list:
def all_nodes(tree):
return [label(tree)] + sum([all_nodes(b) for b in branches(tree)], [])
T = tree(1, [tree(2, [tree(4), tree(5)]), tree(3, [tree(6), tree(7)])])
print(all_nodes(T))
# >>> [1, 2, 4, 5, 3, 6, 7]
You can see that this worked very well, but I got confused how sum() is used here.
I know that a list can be added to another list:
print([1] + [2]) # >>> [1, 2]
But I can't make it work by using sum():
a, b = [1], [2]
print(sum(a, b))
# >>> TypeError: can only concatenate list (not "int") to list
print(sum([a, b]))
# >>> TypeError: unsupported operand type(s) for +: 'int' and 'list
In the tree function, how did sum() work to merge all the lists?
The builtin method sum will use the + operation to sum a list of elements. Its second argument is the starting value.
By default the starting value is 0, meaning sum([[1], [2]]) is equivalent to 0 + [1] + [2] which raises a TypeError.
To concatenate lists you want the initial value to be [], an empty list. Then, sum([[1], [2], [3]], []) is equivalent to [] + [1] + [2] + [3] as desired.
Performance
It is not recommended to use sum to concatenate a list of lists. Indeed, on every addition a new list is created. Instead you want to use a solution that traverses all lists and append the items to a new list.
def concat_lists(lists):
new_list = []
for l in lists:
new_list.extend(l)
Or alternatively using itertools.
from itertools import chain
new_list = list(chain(*lists))
sum operates on a sequence of elements, such as sum([1, 2, 3]) (producing 6) or sum([ [1], [2] ], []) (producing [1, 2]). There is an optional second argument, the start value. For instance, sum([1, 2, 3], 10) would start the summation at 10, providing 16. start defaults to 0: if you're summing non-numeric objects, you have to provide a compatible start value.
When you give it sum(a, b), the list a becomes the list of arguments. What sum did was to (correctly) iterate through the items of that list, adding them to the start value you provided. The logic is something like this:
result = b
for element in a:
result = result + element
Thus, the first thing you tried to do was result = [2] + 1. Remember, that first argument is a sequence of the things you want to add up. The most trivial change (although not the most readable) that would let your attempt work is
sum([a], b)
which produces [2, 1], since b is the starting value.
Does that explain what happened?
I am trying to insert values, one at at time, from several python lists of lists (i.e. 2D lists) into another 2D list. (I know numpy is better at this, but I am trying to compare the performance of lists to numpy, so please don't just suggest numpy.) I want to insert the values at specific locations, hence the indexing on the left hand side.
resampled_pix_spot_list is a 240 by 240 list of lists, and pix_spot_list is a 225 by 225 list of lists.
The error I am getting, from the final four lines in the example, is "TypeError: 'float' object is not subscriptable". I get that pix_prod_bl[0][0], for example, is a float, but I don't understand why I can't insert it into a particular set of indices in resampled_pix_spot_list.
Edit 1- added minimal working example.
Edit 2- in adding the working example, I found that I accidentally had the line commented where I convert the lists back to numpy, and somehow I misinterpreted the Spyder console about where the error was originating. Anyway it works now, thank you very much for the quick feedback. I guess I'll leave this here in case it's helpful to anyone else.
Edit 3- pix_spot_values is an array of data, so just a random array of floats between 0 and 1 will suffice.
xc=57
yc=189
rebin=15
# fraction pixel offset requiring interpolation
dx=xc*rebin-int(np.floor(xc*rebin)) # positive value between 0 and 1
dy=yc*rebin-int(np.floor(yc*rebin)) # positive value between 0 and 1
# weights for interpolation
w00=(1-dy)*(1-dx)
w10=dy*(1-dx)
w01=(1-dy)*dx
w11=dy*dx
# now the rest of the offset is an integer shift
dx=int(np.floor(xc*rebin))-int(np.floor(xc))*rebin # positive integer between 0 and 14
dy=int(np.floor(yc*rebin))-int(np.floor(yc))*rebin # positive integer between 0 and 14
def new_pix_spot(w00, w10, w01, w11, pix_spot_list, ny_spot, nx_spot, rebin, dy, dx):
#first change numpy array to list
pix_spot_list=pix_spot_values.tolist()
#preallocate array of zeros
resampled_pix_spot_list=[[0 for x in range (ny_spot + rebin)] for y in range(nx_spot+rebin)]
#create 2D lists
pix_prod_bl = [[x*w00 for x in y] for y in pix_spot_list]#bottom left
pix_prod_br = [[x*w10 for x in y] for y in pix_spot_list]#bottom right
pix_prod_tl = [[x*w01 for x in y] for y in pix_spot_list]#top left
pix_prod_tr = [[x*w11 for x in y] for y in pix_spot_list]#top right
for i in range (len(pix_spot_list)):
for j in range (len(pix_spot_list)):
k=dy + i
m=dx + j
n=dy + 1 + i
p=dx + 1 + i
resampled_pix_spot_list[k][m] += pix_prod_bl[i][j] #bottom left
resampled_pix_spot_list[n][m] += pix_prod_br[i][j] #bottom right
resampled_pix_spot_list[k][p] += pix_prod_tl[i][j] #top left
resampled_pix_spot_list[n][p] += pix_prod_tr[i][j] #top right
resampled_pix_spot_values = np.array(resampled_pix_spot_list)
return resampled_pix_spot_values
Inserting and Replacing
To insert values into a list in Python, you must work with the list object (for example, resampled_pix_spot_list[0]) rather than the elements within it (resampled_pix_spot_list[0][0], as you tried).
In both Python 2 and 3, you can insert into a list with your_list.insert(<index>, <element>) (list insertion docs here).
So to insert a number to the left of your chosen coordinate, the code would be:
resampled_pix_spot_list[k].insert(m, pix_prod_bl[i][j])
If you wanted to replace the pixel at that position, you would write:
resampled_pix_spot_list[k][m] = pix_prod_bl[i][j]
(Notice the [k] vs [k][m].) In short: To insert, talk to the list; to replace, talk to the element.
Pitfalls of Repeated Inserts
Just a tip: if you're planning on repeatedly inserting values into specific places in a list, try to iterate from the end of the list, backwards. If you don't, you'll have to adjust your indices, since each .insert() call will shift part of your list to the right.
To see what I mean, let's imagine I have the list [1, 2, 3] and want to end up with [1, 88, 2, 99, 3] via insertions. The order we insert matters. Compare the wrong order (iterating forwards):
data = [1, 2, 3]
>>> data.insert(1, 88)
>>> print(data)
[1, 88, 2, 3] # so far so good
>>> data.insert(2, 99)
>>> print(data)
[1, 88, 99, 2, 3] # oops! the first insert changed my indices, so index "2" was wrong!
with the right order (iterating backwards):
data = [1, 2, 3]
>>> data.insert(2, 99)
>>> print(data)
[1, 2, 99, 3] # so far so good
>>> data.insert(1, 88)
>>> print(data)
[1, 88, 2, 99, 3] # same insertions + different order = different results!
Slices
Some food for thought: Python 2.7 and 3 both allow you to replace whole "slices" of lists with a very clean syntax, which would also help you avoid "off-by-one" errors (slice notation docs here). For example:
>>> data = [1, 2, 3]
>>> data[1:2] = [88, data[1], 99] # safer, shorter, faster, and clearer
>>> print(data)
[1, 88, 2, 99, 3]
Working with slices might be a bit more declarative and clear. Hope this helps!