Related
I currently have some code that uses a mask to calculate the mean of values that are overloads, and values that are baseline values. It does this over the entire length of the dataframe. However, now I want to only apply this to a certain range in the dataframe column, between first and last values (ie, a specified region in the column, dictated by user input). Here is my code as it stands:
mask_number = 5
no_overload_cycles = 1
hyst = pd.DataFrame({"test":[12, 4, 5, 4, 1, 3, 2, 5, 10, 9, 7, 5, 3, 6, 3, 2 ,1, 5, 2]})
list_test = []
for i in range(0,len(hyst)-1,mask_number):
for x in range(no_overload_cycles):
list_test.append(i+x)
mask = np.array(list_test)
print(mask)
[0 1 5 10 15 20]
first = 4
last = 17
regression_area = hyst.iloc[first:last]
mean_range_overload = regression_area.loc[np.where(mask == regression area.index)]['test'].mean()
mean_range_baseline = regression_area.drop(mask[first:last])['test'].mean()
So the overload mean would be be cycles, 5, 10, and 15 in test, and the baseline mean would be from positions 4 to 17, excluding 5, 10 and 15. This would be my expected output from this:
print (mean_range_overload)
4
print(mean_range_baseline)
4.545454
However, the no_overload_cycles value can change, and may for example, be 3, which would then create a mask of this:
mask_number = 5
no_overload_cycles = 3
hyst = pd.DataFrame({"test":[12, 4, 5, 4, 1, 3, 2, 5, 10, 9, 7, 5, 3, 6, 3, 2 ,1, 5, 2]})
list_test = []
for i in range(0,len(hyst)-1,mask_number):
for x in range(no_overload_cycles):
list_test.append(i+x)
mask = np.array(list_test)
print(mask)
[0 1 2 5 6 7 10 11 12 15 16 17 20]
So the mean_range_overload would be mean of the values at 5,6,7,10,11,12,15,16,17, and the mean_range_baseline would be the values inbetween these, in the range of first and last in the dataframe column.
Any help on this would be greatly appreciated!
Assuming no_overload_cycles == 1 always, you can simply use slice objects to index the DataFrame.
Say you wish to, in your example, specifically pick cycles 5, 10 and 15 and use them as overload. Then you can get them by doing df.loc[5:15:5].
On the other hand, if you wish to pick the 5th, 10th and 15th cycles from the range you selected, you can get them by doing df.iloc[5:15+1:5] (iloc does not include the right index, so we add one). No loops required.
As mentioned in the comments, your question is slightly confusing, and it'd be helpful if you gave a better description and some expected results; in general I'd also advise you to decouple the domain-specific part of your problem before asking it in a forum, since not everyone knows what you mean by "overload", "baseline", "cycles" etc. I'm not commenting that since I still don't have enough reputation to do so.
I renamed a few of the variables, so what I called a "mask" is not exactly what you called a mask, but I reckon this is what you were trying to make:
mask_length = 5
overload_cycles_per_mask = 3
df = pd.DataFrame({"test": [12, 4, 5, 4, 1, 3, 2, 5, 10, 9, 7, 5, 3, 6, 3, 2 ,1, 5, 2]})
selected_range = (4, 17)
overload_indices = []
baseline_indices = []
# `range` does not include the right hand side so we add one
# ideally you would specify the range as (4, 18) instead
for i in range(selected_range[0], selected_range[1]+1):
if i % mask_length < overload_cycles_per_mask:
overload_indices.append(i)
else:
baseline_indices.append(i)
print(overload_indices)
print(df.iloc[overload_indices].test.mean())
print(baseline_indices)
print(df.iloc[baseline_indices].test.mean())
Basically, the DataFrame rows inside selected_range are divided into segments of length mask_length, each of which has their first overload_cycles_per_mask elements marked as overload, and any others, as baseline.
With that, you get two lists of indices, which you can directly pass to df.iloc, as according to the documentation it supports a list of integers.
Here is the output for mask_length = 5 and overload_cycles_per_mask = 1:
[5, 10, 15]
4.0
[4, 6, 7, 8, 9, 11, 12, 13, 14, 16, 17]
4.545454545454546
And here is for mask_length = 5 and overload_cycles_per_mask = 3:
[5, 6, 7, 10, 11, 12, 15, 16, 17]
3.6666666666666665
[4, 8, 9, 13, 14]
5.8
I do believe calling this a single mask makes things more confusing. In any case, I would tuck the logic for getting the indices away in some separate function to the one which calculates the mean.
My data
List = [[[12,1,6],[12,1,6],15],[[12,2,6],[12,2,6],18]],[[12,3,6],[12,3,6],24]]
I have a data containing
number of rows having a transition from 12,1,6 to 12,1,6 is 15
number of rows having a transition from 12,2,6 to 12,2,6 is 18
number of rows having a transition from 12,3,6 to 12,3,6 is 24
as list
This data is not generated.There are many other possible combinations are there in my data.The above said is the sample
I want my output to be a list having probabilities of this transition
for example
P1 = the probability of transition from 12,1,6 to 12,1,6
= 15/total length of rows/elements in the list.(In this case it is 3)
P2 = the probability of transition from 12,2,6 to 12,2,6
= 18/total length of rows in the list
my output needs to be
List =[[[12,1,6],[12,1,6],15,P1=(15/3)*100],[[12,2,6],[12,2,6],18,P2]],[[12,3,6],[12,3,6],24,P3]]
Have tried a lot and would be helpful if i get suggestions.
def Sort(sub_li):
sub_li.sort(reverse = True, key = lambda x: x[1])
return sub_li
print(Sort())
List had an extra ] after 18, so I removed it before writing this piece of code, which assumes that the length is the same for all the rows.
List = [[[12,1,6],[12,1,6],15],[[12,2,6],[12,2,6],18],[[12,3,6],[12,3,6],24]]
for i, value in enumerate(List):
value.append(("P%d=%f" % ((i + 1), value[2] / len(value[0]) * 100)))
print(List)
Output:
[[[12, 1, 6], [12, 1, 6], 15, 'P1=500.000000'], [[12, 2, 6], [12, 2, 6], 18, 'P2=600.000000'], [[12, 3, 6], [12, 3, 6], 24, 'P3=800.000000']]
Using this small reproducible example, I've so far been unable to generate a new integer array from 3 arrays that contains unique groupings across all three input arrays.
The arrays are related to topographic properties:
import numpy as np
asp = np.array([8,1,1,2,7,8,2,3,7,6,4,3,6,5,5,4]).reshape((4,4)) #aspect
slp = np.array([9,10,10,9,9,12,12,9,10,11,11,9,9,9,9,9]).reshape((4,4)) #slope
elv = np.array([13,14,14,13,14,15,16,14,14,15,16,14,13,14,14,13]).reshape((4,4)) #elevation
The idea is that the geographic contours are broken into 3 different properties using GIS routines:
1-8 for aspect (1=north facing, 2=northeast facing, etc.)
9-12 for slope (9=gentle slope...12=steepest slope)
13-16 for elevation (13=lowest elevations...16=highest elevations)
The small graphic below attempts to depict the kind of result I'm after (array shown in lower left). Note, the "answer" given in the graphic is but one possible answer. I'm not concerned about the final arrangement of integers in the resulting array so long as the final array contains an integer at each row/column index that identifies unique groupings.
For example, the array indexes at [0,1] and [0,2] have the same aspect, slope, and elevation and therefore receive the same integer identifier in the resulting array.
Does numpy have a built in routine for this kind of thing?
Each location in the grid is associated with a tuple composed of one value from
asp, slp and elv. For example, the upper left corner has tuple (8,9,13).
We would like to map this tuple to a number which uniquely identifies this tuple.
One way to do that would be to think of (8,9,13) as the index into the 3D array
np.arange(9*13*17).reshape(9,13,17). This particular array was chosen
to accommodate the largest values in asp, slp and elv:
In [107]: asp.max()+1
Out[107]: 9
In [108]: slp.max()+1
Out[108]: 13
In [110]: elv.max()+1
Out[110]: 17
Now we can map the tuple (8,9,13) to the number 1934:
In [113]: x = np.arange(9*13*17).reshape(9,13,17)
In [114]: x[8,9,13]
Out[114]: 1934
If we do this for each location in the grid, then we get a unique number for each location.
We could end right here, letting these unique numbers serve as labels.
Or, we can generate smaller integer labels (starting at 0 and increasing by 1)
by using np.unique with
return_inverse=True:
uniqs, labels = np.unique(vals, return_inverse=True)
labels = labels.reshape(vals.shape)
So, for example,
import numpy as np
asp = np.array([8,1,1,2,7,8,2,3,7,6,4,3,6,5,5,4]).reshape((4,4)) #aspect
slp = np.array([9,10,10,9,9,12,12,9,10,11,11,9,9,9,9,9]).reshape((4,4)) #slope
elv = np.array([13,14,14,13,14,15,16,14,14,15,16,14,13,14,14,13]).reshape((4,4)) #elevation
x = np.arange(9*13*17).reshape(9,13,17)
vals = x[asp, slp, elv]
uniqs, labels = np.unique(vals, return_inverse=True)
labels = labels.reshape(vals.shape)
yields
array([[11, 0, 0, 1],
[ 9, 12, 2, 3],
[10, 8, 5, 3],
[ 7, 6, 6, 4]])
The above method works fine as long as the values in asp, slp and elv are small integers. If the integers were too large, the product of their maximums could overflow the maximum allowable value one can pass to np.arange. Moreover, generating such a large array would be inefficient.
If the values were floats, then they could not be interpreted as indices into the 3D array x.
So to address these problems, use np.unique to convert the values in asp, slp and elv to unique integer labels first:
indices = [ np.unique(arr, return_inverse=True)[1].reshape(arr.shape) for arr in [asp, slp, elv] ]
M = np.array([item.max()+1 for item in indices])
x = np.arange(M.prod()).reshape(M)
vals = x[indices]
uniqs, labels = np.unique(vals, return_inverse=True)
labels = labels.reshape(vals.shape)
which yields the same result as shown above, but works even if asp, slp, elv were floats and/or large integers.
Finally, we can avoid the generation of np.arange:
x = np.arange(M.prod()).reshape(M)
vals = x[indices]
by computing vals as a product of indices and strides:
M = np.r_[1, M[:-1]]
strides = M.cumprod()
indices = np.stack(indices, axis=-1)
vals = (indices * strides).sum(axis=-1)
So putting it all together:
import numpy as np
asp = np.array([8,1,1,2,7,8,2,3,7,6,4,3,6,5,5,4]).reshape((4,4)) #aspect
slp = np.array([9,10,10,9,9,12,12,9,10,11,11,9,9,9,9,9]).reshape((4,4)) #slope
elv = np.array([13,14,14,13,14,15,16,14,14,15,16,14,13,14,14,13]).reshape((4,4)) #elevation
def find_labels(*arrs):
indices = [np.unique(arr, return_inverse=True)[1] for arr in arrs]
M = np.array([item.max()+1 for item in indices])
M = np.r_[1, M[:-1]]
strides = M.cumprod()
indices = np.stack(indices, axis=-1)
vals = (indices * strides).sum(axis=-1)
uniqs, labels = np.unique(vals, return_inverse=True)
labels = labels.reshape(arrs[0].shape)
return labels
print(find_labels(asp, slp, elv))
# [[ 3 7 7 0]
# [ 6 10 12 4]
# [ 8 9 11 4]
# [ 2 5 5 1]]
This can be done using numpy.unique() and then a mapping like:
Code:
combined = 10000 * asp + 100 * slp + elv
unique = dict(((v, i + 1) for i, v in enumerate(np.unique(combined))))
combined_unique = np.vectorize(unique.get)(combined)
Test Code:
import numpy as np
asp = np.array([8, 1, 1, 2, 7, 8, 2, 3, 7, 6, 4, 3, 6, 5, 5, 4]).reshape((4, 4)) # aspect
slp = np.array([9, 10, 10, 9, 9, 12, 12, 9, 10, 11, 11, 9, 9, 9, 9, 9]).reshape((4, 4)) # slope
elv = np.array([13, 14, 14, 13, 14, 15, 16, 14, 14, 15, 16, 14, 13, 14, 14, 13]).reshape((4, 4))
combined = 10000 * asp + 100 * slp + elv
unique = dict(((v, i + 1) for i, v in enumerate(np.unique(combined))))
combined_unique = np.vectorize(unique.get)(combined)
print(combined_unique)
Results:
[[12 1 1 2]
[10 13 3 4]
[11 9 6 4]
[ 8 7 7 5]]
This seems like a similar problem to labeling unique regions in an image. This is a function I've written to do this, though you would first need to concatenate your 3 arrays to 1 3D array.
def labelPix(pix):
height, width, _ = pix.shape
pixRows = numpy.reshape(pix, (height * width, 3))
unique, counts = numpy.unique(pixRows, return_counts = True, axis = 0)
unique = [list(elem) for elem in unique]
labeledPix = numpy.zeros((height, width), dtype = int)
offset = 0
for index, zoneArray in enumerate(unique):
index += offset
zone = list(zoneArray)
zoneArea = (pix == zone).all(-1)
elementsArray, numElements = scipy.ndimage.label(zoneArea)
elementsArray[elementsArray!=0] += offset
labeledPix[elementsArray!=0] = elementsArray[elementsArray!=0]
offset += numElements
return labeledPix
This will label unique 3-value combinations, while also assigning separate labels to zones which have the same 3-value combination, but are not in contact with one another.
asp = numpy.array([8,1,1,2,7,8,2,3,7,6,4,3,6,5,5,4]).reshape((4,4)) #aspect
slp = numpy.array([9,10,10,9,9,12,12,9,10,11,11,9,9,9,9,9]).reshape((4,4)) #slope
elv = numpy.array([13,14,14,13,14,15,16,14,14,15,16,14,13,14,14,13]).reshape((4,4)) #elevation
pix = numpy.zeros((4,4,3))
pix[:,:,0] = asp
pix[:,:,1] = slp
pix[:,:,2] = elv
print(labelPix(pix))
returns:
[[ 0 1 1 2]
[10 12 3 4]
[11 9 6 4]
[ 8 7 7 5]]
Here's a plain Python technique using itertools.groupby. It requires the input to be 1D lists, but that shouldn't be a major issue. The strategy is to zip the lists together, along with an index number, then sort the resulting columns. We then group identical columns together, ignoring the index number when comparing columns. Then we gather the index numbers from each group, and use them to build the final output list.
from itertools import groupby
def show(label, seq):
print(label, ' '.join(['{:2}'.format(u) for u in seq]))
asp = [8, 1, 1, 2, 7, 8, 2, 3, 7, 6, 4, 3, 6, 5, 5, 4]
slp = [9, 10, 10, 9, 9, 12, 12, 9, 10, 11, 11, 9, 9, 9, 9, 9]
elv = [13, 14, 14, 13, 14, 15, 16, 14, 14, 15, 16, 14, 13, 14, 14, 13]
size = len(asp)
a = sorted(zip(asp, slp, elv, range(size)))
groups = sorted([u[-1] for u in g] for _, g in groupby(a, key=lambda t:t[:-1]))
final = [0] * size
for i, g in enumerate(groups, 1):
for j in g:
final[j] = i
show('asp', asp)
show('slp', slp)
show('elv', elv)
show('out', final)
output
asp 8 1 1 2 7 8 2 3 7 6 4 3 6 5 5 4
slp 9 10 10 9 9 12 12 9 10 11 11 9 9 9 9 9
elv 13 14 14 13 14 15 16 14 14 15 16 14 13 14 14 13
out 1 2 2 3 4 5 6 7 8 9 10 7 11 12 12 13
There's no need to do that second sort, we could just use a plain list comp
groups = [[u[-1] for u in g] for _, g in groupby(a, key=lambda t:t[:-1])]
or generator expression
groups = ([u[-1] for u in g] for _, g in groupby(a, key=lambda t:t[:-1]))
I only did it so that my output matches the output in the question.
Here's one way to solve this problem using a dictionary based lookup.
from collections import defaultdict
import itertools
group_dict = defaultdict(list)
idx_count = 0
for a, s, e in np.nditer((asp, slp, elv)):
asp_tuple = (a.tolist(), s.tolist(), e.tolist())
if asp_tuple not in group_dict:
group_dict[asp_tuple] = [idx_count+1]
idx_count += 1
else:
group_dict[asp_tuple].append(group_dict[asp_tuple][-1])
list1d = list(itertools.chain(*list(group_dict.values())))
np.array(list1d).reshape(4, 4)
# result
array([[ 1, 2, 2, 3],
[ 4, 5, 6, 7],
[ 7, 8, 9, 10],
[11, 12, 12, 13]])
What is the best way to create a 2D list (or numpy array) in python, in which the diagonal is set to -1 and the remaining values are increasing from 0 by 1, for different values of n. For example, if n = 3 the array would look like:
[[-1,0,1]
[2,-1,3]
[4,5,-1]]
or for n = 4:
[[-1,0,1,2]
[3,-1,4,5]
[6,7,-1,8]
[9,10,11,-1]]
etc.
I know I can create an array with zeros and with the diagonal set to -1 with:
a = numpy.zeros((n,n))
numpy.fill_diagonal(a,-1)
And so if n = 3 this would give:
[[-1,0,0]
[0,-1,0]
[0,0,-1]]
But how would I then set the 0's to be increasing numbers, as shown in the example above? Would I need to iterate through and set the values through a loop? Or is there a better way to approach this?
Thanks in advance.
One approach -
def set_matrix(n):
out = np.full((n,n),-1)
off_diag_mask = ~np.eye(n,dtype=bool)
out[off_diag_mask] = np.arange(n*n-n)
return out
Sample runs -
In [23]: set_matrix(3)
Out[23]:
array([[-1, 0, 1],
[ 2, -1, 3],
[ 4, 5, -1]])
In [24]: set_matrix(4)
Out[24]:
array([[-1, 0, 1, 2],
[ 3, -1, 4, 5],
[ 6, 7, -1, 8],
[ 9, 10, 11, -1]])
Here is an arithmetic way:
m=np.arange(n*n).reshape(n,n)*n//(n+1)
m.flat[::n+1]=-1
for n=5 :
[[-1 0 1 2 3]
[ 4 -1 5 6 7]
[ 8 9 -1 10 11]
[12 13 14 -1 15]
[16 17 18 19 -1]]
Suppose I have the following matrix in python:
[[1,2,3,4],
[5,6,7,8],
[9,10,11,12],
[13,14,15,16]]
I want to slice it into the following matrices (or quadrants/corners):
[[1,2], [5,6]]
[[3,4], [7,8]]
[[9,10], [13,14]]
[[11,12], [15,16]]
Is this supported with standard slicing operators in python or is it necessary to use an extended library like numpy?
If you are always working with a 4x4 matrix:
a = [[1 ,2 , 3, 4],
[5 ,6 , 7, 8],
[9 ,10,11,12],
[13,14,15,16]]
top_left = [a[0][:2], a[1][:2]]
top_right = [a[0][2:], a[1][2:]]
bot_left = [a[2][:2], a[3][:2]]
bot_right = [a[2][2:], a[3][2:]]
You could also do the same for an arbitrary size matrix:
h = len(a)
w = len(a[1])
top_left = [a[i][:w // 2] for i in range(h // 2)]
top_right = [a[i][w // 2:] for i in range(h // 2)]
bot_left = [a[i][:w // 2] for i in range(h // 2, h)]
bot_right = [a[i][w // 2:] for i in range(h // 2, h)]
The question is already answered, but I think this solution is more general.
It can also be used numpy.split and list comprehension in the following way:
import numpy as np
A = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]])
B = [M for SubA in np.split(A,2, axis = 0) for M in np.split(SubA,2, axis = 1)]
Getting:
>>>[array([[1, 2],[5, 6]]),
array([[3, 4],[7, 8]]),
array([[ 9, 10],[13, 14]]),
array([[11, 12],[15, 16]])]
Now if you want to have them assigned into different variables, just:
C1,C2,C3,C4 = B
Have a look to numpy.split doc.
Changing the parameter indices_or_sections you can get a higher number of splits.
>>> a = [[1,2,3,4], [5,6,7,8], [9,10,11,12], [13,14,15,16]]
>>> x = map(lambda x:x[:2], a)
>>> x
[[1, 2], [5, 6], [9, 10], [13, 14]]
>>> y = map(lambda x: x[2:], a)
>>> y
[[3, 4], [7, 8], [11, 12], [15, 16]]
>>> x[:2] + y[:2] + x[2:] + y[2:]
[[1, 2], [5, 6], [3, 4], [7, 8], [9, 10], [13, 14], [11, 12], [15, 16]]
Although the answers can provide the required solution. These are not applicable for the arrays in different sizes. If you have a NumPy array in size of (6x7), then these methods will not create a solution. I have prepared a solution for myself and want to share it here.
Note: In my solution, there will be overlaps due to the different axis sizes.
I have created this solution to divide an astronomical image into four quadrants. I, then, use these quadrants to calculate the mean and median in an annulus.
import numpy as np
def quadrant_split2d(array):
"""Example function for identifiying the elements of quadrants in an array.
array:
A 2D NumPy array.
Returns:
The quadrants. True for the members of the quadrants, False otherwise.
"""
Ysize = array.shape[0]
Xsize = array.shape[1]
y, x = np.indices((Ysize,Xsize))
if not (Xsize==Ysize)&(Xsize % 2 == 0): print ('There will be overlaps')
sector1=(x<Xsize/2)&(y<Ysize/2)
sector2=(x>Xsize/2-1)&(y<Ysize/2)
sector3=(x<Xsize/2)&(y>Ysize/2-1)
sector4=(x>Xsize/2-1)&(y>Ysize/2-1)
sectors=(sector1,sector2,sector3,sector4)
return sectors
You can test the function with the different type of arrays.
For example:
test = np.arange(42).reshape(6,7)
print ('Original array:\n', test)
sectors = quadrant_split2d(test)
print ('Sectors:')
for ss in sectors: print (test[ss])
This will give us the following sectors:
[ 0 1 2 3 7 8 9 10 14 15 16 17]
[ 3 4 5 6 10 11 12 13 17 18 19 20]
[21 22 23 24 28 29 30 31 35 36 37 38]
[24 25 26 27 31 32 33 34 38 39 40 41]