We Keep Coding

Take the mean values of a number of integers in one list, based on the integers in another list

I have one list containing a large number of integers, and then another list also containing integers that (if added together) sums up to the total amount of integers in the first list. I wish to create a function which iterates over the second list, and for each number in the second list, takes the mean of the values in the first list and repeats for all integers in the second list...making one final third list containing the desired mean-values.
For example: A small portion of my two lists look like this: [20, 15, 20, 30, 40, 20, 10, 8], [2, 3, 1, 2]
So since 2 is the first number in my second list, I want to take the mean of the first two integers in my first list, and then the 3 next, and so on, and add these into a third list.
Here is a brief idea of what I am thinking, but its obviously not complete.
def mean_values(list_of_numbers, numbers_for_meanvalue):
list_of_means = []
for i in numbers_for_meanvalue:
mean = sum(list_of_numbers[0]+...+list_of_numbers[i])/numbers_for_meanvalue[i]
list_of_means.append[mean]
return list_of_means

you can take a slice of a list: list[start : end].
def mean_values(list_of_numbers, numbers_for_meanvalue):
list_of_means = []
last_start = 0
for num in numbers_for_meanvalue:
mean = sum(list_of_numbers[last_start:last_start + num]) / len(list_of_numbers[last_start:last_start + num])
last_start += num
list_of_means.append[mean]
return list_of_means

I don't if I understand you correctly, but:
>>> a = [20, 15, 20, 30, 40, 20, 10, 8]
>>> b = [2, 3, 1, 2]
>>> n = 0
>>> for e in b:
... sl = a[n:n+e]
... print(sl,(sum(sl) / e))
... n += e
...
[20, 15] 17.5
[20, 30, 40] 30.0
[20] 20.0
[10, 8] 9.0

Reusable code to iterate along different array dimensions

Say, I have an N dimensional array my_array[D1][D2]...[DN]
For a certain application, like sensitivity analysis, I need to fix a point p=(d1, d2, ..., dN) and iterate along each dimension at a time.
The resulting behavior is
for x1 in range(0, D1):
do_something(my_array[x1][d2][d3]...[dN])
for x2 in range(0, D2):
do_something(my_array[d1][x2][d3]...[dN])
.
.
.
for xN in range(0, DN):
do_something(my_array[d1][d2][d3]...[xN])
As you can see, there are many duplicated code here. How can I reduce the work and write some elegant code instead?
For example, is there any approach to the generation of code similar to the below?
for d in range(0, N):
iterate along the (d+1)th dimension of my_array, denoting the element as x:
do_something(x)

You can use numpy.take and do something like the following. Go through the documentation for reference.
https://docs.scipy.org/doc/numpy/reference/generated/numpy.take.html
N = len(my_array)
for i in range(N):
n = len(my_array(i))
indices = p
indices[i] = x[i]
for j in range(n):
do_something(np.take(my_array,indices))

I don't understand what are d1 d2 d3, but I guess you can do something like this:
def get_list_item_by_indexes_list(in_list, indexes_list):
if len(indexes_list) <= 1:
return in_list[indexes_list[0]]
else:
return get_list_item_by_indexes_list(in_list[indexes_list[0]], indexes_list[1:])
def do_to_each_dimension(multi_list, func, dimensions_lens):
d0_to_dN_list = [l - 1 for l in dimensions_lens] # I dont know what is it
for dimension_index in range(0, len(dimensions_lens)):
dimension_len = dimensions_lens[dimension_index]
for x in range(0, dimension_len):
curr_d0_to_dN_list = d0_to_dN_list.copy()
curr_d0_to_dN_list[dimension_index] = x
func(get_list_item_by_indexes_list(multi_list, curr_d0_to_dN_list))
def do_something(n):
print(n)
dimensions_lens = [3, 5]
my_array = [
[1, 2, 3, 4, 5],
[6, 7, 8, 9, 10],
[11, 12, 13, 14, 15]
]
do_to_each_dimension(my_array, do_something, dimensions_lens)
Output:
5 10 15 11 12 13 14 15
This code iterates through the last column and the last row of a 2d array.
Now, to iterate through the last line of each dimension of 3d array:
dimensions_lens = [2, 4, 3]
my_array = [
[
[1, 2, 3],
[4, 5, 6],
[7, 8, 9],
[10, 11, 12]
],
[
[13, 14, 15],
[16, 17, 18],
[19, 20, 21],
[22, 23, 24]
],
]
do_to_each_dimension(my_array, do_something, dimensions_lens)
Output:
12 24 15 18 21 24 22 23 24
(Note: don't use zero-length dimensions with this code)

You could mess with the string representation of your array access (my_arr[d1][d2]...[dN]) and eval that afterwards to get the values you want. This is fairly "hacky", but it will work on arrays with arbitrary dimensions and allows you to supply the indices as a list while handling the nested array access under the hood, allowing for a clean double for loop .
def access_at(arr, point):
# build 'arr[p1][p2]...[pN]'
access_str = 'arr' + ''.join([f'[{p}]' for p in point])
return eval(access_str)
Using this access method is pretty straight forward:
p = [p1, ..., pN]
D = [D1, ..., DN]
for i in range(N):
# deep copy p
pt = p[:]
for x in range(D[i]):
pt[i] = x
do_something(access_at(my_array, pt))

Splitting arrays depending on unique values in an array

I currently have two arrays, one of which has several repeated values and another with unique values.
Eg array 1 : a = [1, 1, 2, 2, 3, 3]
Eg array 2 : b = [10, 11, 12, 13, 14, 15]
I was developing a code in python that looks at the first array and distinguishes the elements that are all the same and remembers the indices. A new array is created that contains the elements of array b at those indices.
Eg: As array 'a' has three unique values at positions 1,2... 3,4... 5,6, then three new arrays would be created such that it contains the elements of array b at positions 1,2... 3,4... 5,6. Thus, the result would be three new arrays:
b1 = [10, 11]
b2 = [12, 13]
b3 = [14, 15]
I have managed to develop a code, however, it only works for when there are three unique values in array 'a'. In the case there are more or less unique values in array 'a', the code has to be physically modified.
import itertools
import numpy as np
import matplotlib.tri as tri
import sys
a = [1, 1, 2, 2, 3, 3]
b = [10, 10, 20, 20, 30, 30]
b_1 = []
b_2 = []
b_3 = []
unique = []
for vals in a:
if vals not in unique:
unique.append(vals)
if len(unique) != 3:
sys.exit("More than 3 'a' values - check dimension")
for j in range(0,len(a)):
if a[j] == unique[0]:
b_1.append(c[j])
elif a[j] == unique[1]:
b_2.append(c[j])
elif a[j] == unique[2]:
b_3.append(c[j])
else:
sys.exit("More than 3 'a' values - check dimension")
print (b_1)
print (b_2)
print (b_3)
I was wondering if there is perhaps a more elegant way to perform this task such that the code is able to cope with an n number of unique values.

Well given that you are also using numpy, here's one way using np.unique. You can set return_index=True to get the indices of the unique values, and use them to split the array b with np.split:
a = np.array([1, 1, 2, 2, 3, 3])
b = np.array([10, 11, 12, 13, 14, 15])
u, s = np.unique(a, return_index=True)
np.split(b,s[1:])
Output
[array([10, 11]), array([12, 13]), array([14, 15])]

You can use the function groupby():
from itertools import groupby
from operator import itemgetter
a = [1, 1, 2, 2, 3, 3]
b = [10, 11, 12, 13, 14, 15]
[[i[1] for i in g] for _, g in groupby(zip(a, b), key=itemgetter(0))]
# [[10, 11], [12, 13], [14, 15]]

Intersect multiple 2D np arrays for determining zones

Using this small reproducible example, I've so far been unable to generate a new integer array from 3 arrays that contains unique groupings across all three input arrays.
The arrays are related to topographic properties:
import numpy as np
asp = np.array([8,1,1,2,7,8,2,3,7,6,4,3,6,5,5,4]).reshape((4,4)) #aspect
slp = np.array([9,10,10,9,9,12,12,9,10,11,11,9,9,9,9,9]).reshape((4,4)) #slope
elv = np.array([13,14,14,13,14,15,16,14,14,15,16,14,13,14,14,13]).reshape((4,4)) #elevation
The idea is that the geographic contours are broken into 3 different properties using GIS routines:
1-8 for aspect (1=north facing, 2=northeast facing, etc.)
9-12 for slope (9=gentle slope...12=steepest slope)
13-16 for elevation (13=lowest elevations...16=highest elevations)
The small graphic below attempts to depict the kind of result I'm after (array shown in lower left). Note, the "answer" given in the graphic is but one possible answer. I'm not concerned about the final arrangement of integers in the resulting array so long as the final array contains an integer at each row/column index that identifies unique groupings.
For example, the array indexes at [0,1] and [0,2] have the same aspect, slope, and elevation and therefore receive the same integer identifier in the resulting array.
Does numpy have a built in routine for this kind of thing?

Each location in the grid is associated with a tuple composed of one value from
asp, slp and elv. For example, the upper left corner has tuple (8,9,13).
We would like to map this tuple to a number which uniquely identifies this tuple.
One way to do that would be to think of (8,9,13) as the index into the 3D array
np.arange(9*13*17).reshape(9,13,17). This particular array was chosen
to accommodate the largest values in asp, slp and elv:
In [107]: asp.max()+1
Out[107]: 9
In [108]: slp.max()+1
Out[108]: 13
In [110]: elv.max()+1
Out[110]: 17
Now we can map the tuple (8,9,13) to the number 1934:
In [113]: x = np.arange(9*13*17).reshape(9,13,17)
In [114]: x[8,9,13]
Out[114]: 1934
If we do this for each location in the grid, then we get a unique number for each location.
We could end right here, letting these unique numbers serve as labels.
Or, we can generate smaller integer labels (starting at 0 and increasing by 1)
by using np.unique with
return_inverse=True:
uniqs, labels = np.unique(vals, return_inverse=True)
labels = labels.reshape(vals.shape)
So, for example,
import numpy as np
asp = np.array([8,1,1,2,7,8,2,3,7,6,4,3,6,5,5,4]).reshape((4,4)) #aspect
slp = np.array([9,10,10,9,9,12,12,9,10,11,11,9,9,9,9,9]).reshape((4,4)) #slope
elv = np.array([13,14,14,13,14,15,16,14,14,15,16,14,13,14,14,13]).reshape((4,4)) #elevation
x = np.arange(9*13*17).reshape(9,13,17)
vals = x[asp, slp, elv]
uniqs, labels = np.unique(vals, return_inverse=True)
labels = labels.reshape(vals.shape)
yields
array([[11, 0, 0, 1],
[ 9, 12, 2, 3],
[10, 8, 5, 3],
[ 7, 6, 6, 4]])
The above method works fine as long as the values in asp, slp and elv are small integers. If the integers were too large, the product of their maximums could overflow the maximum allowable value one can pass to np.arange. Moreover, generating such a large array would be inefficient.
If the values were floats, then they could not be interpreted as indices into the 3D array x.
So to address these problems, use np.unique to convert the values in asp, slp and elv to unique integer labels first:
indices = [ np.unique(arr, return_inverse=True)[1].reshape(arr.shape) for arr in [asp, slp, elv] ]
M = np.array([item.max()+1 for item in indices])
x = np.arange(M.prod()).reshape(M)
vals = x[indices]
uniqs, labels = np.unique(vals, return_inverse=True)
labels = labels.reshape(vals.shape)
which yields the same result as shown above, but works even if asp, slp, elv were floats and/or large integers.
Finally, we can avoid the generation of np.arange:
x = np.arange(M.prod()).reshape(M)
vals = x[indices]
by computing vals as a product of indices and strides:
M = np.r_[1, M[:-1]]
strides = M.cumprod()
indices = np.stack(indices, axis=-1)
vals = (indices * strides).sum(axis=-1)
So putting it all together:
import numpy as np
asp = np.array([8,1,1,2,7,8,2,3,7,6,4,3,6,5,5,4]).reshape((4,4)) #aspect
slp = np.array([9,10,10,9,9,12,12,9,10,11,11,9,9,9,9,9]).reshape((4,4)) #slope
elv = np.array([13,14,14,13,14,15,16,14,14,15,16,14,13,14,14,13]).reshape((4,4)) #elevation
def find_labels(*arrs):
indices = [np.unique(arr, return_inverse=True)[1] for arr in arrs]
M = np.array([item.max()+1 for item in indices])
M = np.r_[1, M[:-1]]
strides = M.cumprod()
indices = np.stack(indices, axis=-1)
vals = (indices * strides).sum(axis=-1)
uniqs, labels = np.unique(vals, return_inverse=True)
labels = labels.reshape(arrs[0].shape)
return labels
print(find_labels(asp, slp, elv))
# [[ 3 7 7 0]
# [ 6 10 12 4]
# [ 8 9 11 4]
# [ 2 5 5 1]]

This can be done using numpy.unique() and then a mapping like:
Code:
combined = 10000 * asp + 100 * slp + elv
unique = dict(((v, i + 1) for i, v in enumerate(np.unique(combined))))
combined_unique = np.vectorize(unique.get)(combined)
Test Code:
import numpy as np
asp = np.array([8, 1, 1, 2, 7, 8, 2, 3, 7, 6, 4, 3, 6, 5, 5, 4]).reshape((4, 4)) # aspect
slp = np.array([9, 10, 10, 9, 9, 12, 12, 9, 10, 11, 11, 9, 9, 9, 9, 9]).reshape((4, 4)) # slope
elv = np.array([13, 14, 14, 13, 14, 15, 16, 14, 14, 15, 16, 14, 13, 14, 14, 13]).reshape((4, 4))
combined = 10000 * asp + 100 * slp + elv
unique = dict(((v, i + 1) for i, v in enumerate(np.unique(combined))))
combined_unique = np.vectorize(unique.get)(combined)
print(combined_unique)
Results:
[[12 1 1 2]
[10 13 3 4]
[11 9 6 4]
[ 8 7 7 5]]

This seems like a similar problem to labeling unique regions in an image. This is a function I've written to do this, though you would first need to concatenate your 3 arrays to 1 3D array.
def labelPix(pix):
height, width, _ = pix.shape
pixRows = numpy.reshape(pix, (height * width, 3))
unique, counts = numpy.unique(pixRows, return_counts = True, axis = 0)
unique = [list(elem) for elem in unique]
labeledPix = numpy.zeros((height, width), dtype = int)
offset = 0
for index, zoneArray in enumerate(unique):
index += offset
zone = list(zoneArray)
zoneArea = (pix == zone).all(-1)
elementsArray, numElements = scipy.ndimage.label(zoneArea)
elementsArray[elementsArray!=0] += offset
labeledPix[elementsArray!=0] = elementsArray[elementsArray!=0]
offset += numElements
return labeledPix
This will label unique 3-value combinations, while also assigning separate labels to zones which have the same 3-value combination, but are not in contact with one another.
asp = numpy.array([8,1,1,2,7,8,2,3,7,6,4,3,6,5,5,4]).reshape((4,4)) #aspect
slp = numpy.array([9,10,10,9,9,12,12,9,10,11,11,9,9,9,9,9]).reshape((4,4)) #slope
elv = numpy.array([13,14,14,13,14,15,16,14,14,15,16,14,13,14,14,13]).reshape((4,4)) #elevation
pix = numpy.zeros((4,4,3))
pix[:,:,0] = asp
pix[:,:,1] = slp
pix[:,:,2] = elv
print(labelPix(pix))
returns:
[[ 0 1 1 2]
[10 12 3 4]
[11 9 6 4]
[ 8 7 7 5]]

Here's a plain Python technique using itertools.groupby. It requires the input to be 1D lists, but that shouldn't be a major issue. The strategy is to zip the lists together, along with an index number, then sort the resulting columns. We then group identical columns together, ignoring the index number when comparing columns. Then we gather the index numbers from each group, and use them to build the final output list.
from itertools import groupby
def show(label, seq):
print(label, ' '.join(['{:2}'.format(u) for u in seq]))
asp = [8, 1, 1, 2, 7, 8, 2, 3, 7, 6, 4, 3, 6, 5, 5, 4]
slp = [9, 10, 10, 9, 9, 12, 12, 9, 10, 11, 11, 9, 9, 9, 9, 9]
elv = [13, 14, 14, 13, 14, 15, 16, 14, 14, 15, 16, 14, 13, 14, 14, 13]
size = len(asp)
a = sorted(zip(asp, slp, elv, range(size)))
groups = sorted([u[-1] for u in g] for _, g in groupby(a, key=lambda t:t[:-1]))
final = [0] * size
for i, g in enumerate(groups, 1):
for j in g:
final[j] = i
show('asp', asp)
show('slp', slp)
show('elv', elv)
show('out', final)
output
asp 8 1 1 2 7 8 2 3 7 6 4 3 6 5 5 4
slp 9 10 10 9 9 12 12 9 10 11 11 9 9 9 9 9
elv 13 14 14 13 14 15 16 14 14 15 16 14 13 14 14 13
out 1 2 2 3 4 5 6 7 8 9 10 7 11 12 12 13
There's no need to do that second sort, we could just use a plain list comp
groups = [[u[-1] for u in g] for _, g in groupby(a, key=lambda t:t[:-1])]
or generator expression
groups = ([u[-1] for u in g] for _, g in groupby(a, key=lambda t:t[:-1]))
I only did it so that my output matches the output in the question.

Here's one way to solve this problem using a dictionary based lookup.
from collections import defaultdict
import itertools
group_dict = defaultdict(list)
idx_count = 0
for a, s, e in np.nditer((asp, slp, elv)):
asp_tuple = (a.tolist(), s.tolist(), e.tolist())
if asp_tuple not in group_dict:
group_dict[asp_tuple] = [idx_count+1]
idx_count += 1
else:
group_dict[asp_tuple].append(group_dict[asp_tuple][-1])
list1d = list(itertools.chain(*list(group_dict.values())))
np.array(list1d).reshape(4, 4)
# result
array([[ 1, 2, 2, 3],
[ 4, 5, 6, 7],
[ 7, 8, 9, 10],
[11, 12, 12, 13]])

How to keep specific element indexs in an array

Suppose I have two arrays:
a = [1, 4, 9, 16, 25, 36, 49, 64, 81]
b = [2,5,7]
And I want to keep the elements indexed in a that are listed in b so the 2nd, 5th and 7th index of a:
a_new = [4, 25, 49]
I will then plot b against a_new / perform analysis on it.
For my application, a is a long series of simulated data, and b is a time series of when I want to sample from this data.
Thanks

First remember that the first element of an array (or in this case a list) is number 0 and not number 1:
a = [1, 4, 9, 16, 25, 36, 49, 64, 81]
a[0] = 1 # How to adress the first element
a[1] = 4 # How to adress the second element
a[2] = 9 # ...
So the elements you want (as specified in the array b) are:
a[1] = 4 # a[1] is the same as a[2 - 1] (note that b[0] = 2)
a[4] = 25 # a[4] is the same as a[5 - 1] (note that b[1] = 5)
a[6] = 49 # a[6] is the same as a[7 - 1] (note that b[2] = 7)
So you can also access the elements this way:
a[ b[0] - 1 ] = 4 # Same as a[1] which is the second element
a[ b[1] - 1 ] = 25 # Same as a[4] which is the fifth element
a[ b[2] - 1 ] = 49 # Same as a[6] which is the seventh element
This can be wrapped up in a for-loop:
a_new = [] # Start with an empty list
for index in b: # index in b are all elements in b, thus: b[0] = 2, b[1] = 5 and b[2] = 7
a_new.append(a[ index - 1])
This loop will put the elements a[2 - 1] (4), a[5 - 1] (25) and a[7 - 1] (49) into the lista_new.
But there is a shorter way to write that loop:
a_new = [ a[ index - 1] for index in b ]
Basically, you say a_new = [ ... ], so a_new is a list, and the ... inside will specify, what the list will be filled with. In this case, it will be the elements that the for-loop produces, note that a[ index - 1] for index in b is the same for-loop as in the first example, written in a compact way.
What if you get an list index out of range error?
Your lista contains 9 elements, so the first element is a[0] and the last is a[8]. If you try to access any other element in a list, for example a[12], you will get a "list index out of range" error.
That means: the list b should only contain numbers between 1 and 9 (the length of the list, which you can find out this way len[a] = 9).
I would recommend, that you change your list b to b = [1, 4, 6], since the fifth element of an array is actually adressed like a[4] and not a[5].
The code will be a bit easier:
a_new = [ a[index] for index in b ]
If you don't want errors to happen, the values in b should then be between 0 and 8 (which is len(a) - 1), since a[0] is the first and a[8] is the last element, and only elements between that exist!

There are two possible problems that you may be encountering, both of which have been somewhat mentioned in the comments. From what I see, you either have a problem reading the or you have an invalid index in b.
For the former, you may actually want
a = [1, 4, 9, 16, 25, 36, 49, 64, 81]
b = [2,5,7]
To produce:
a_new = [9, 36, 64]
Since you always count starting from zero:
[1, 4, 9, 16, 25, 36, 49, 64, 81]
0 1 2 3 4 5 6 7 8
Hence, leading to an XY problem, where you try to solve a problem your way which is the wrong way. Therefore, it wastes our time to try to fix a problem that doesn't work since it is actually something else.
However, for the latter, you may have an anomaly in your b list. The way to index the list (given in the comments) as you wanted is using list comprehension:
a_new = [a[i-1] for i in b]
What this does is:
a_new = []
for i in b:
a_new.append(a[i-1])
Hence, when i is larger than or equal to len(a), it evaluates to an invalid index:
>>> a = [1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> len(a)
9
>>> a[9]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range