Related
I am looking for a way to reshape the following 1d-numpy array:
# dimensions
n = 2 # int : 1 ... N
h = 2 # int : 1 ... N
m = n*(2*h+1)
input_data = np.arange(0,(n*(2*h+1))**2)
The expected output should be reshaped into (2*h+1)**2 blocks of shape (n,n) such as:
input_data.reshape(((2*h+1)**2,n,n))
>>> array([[[ 0 1]
[ 2 3]]
[[ 4 5]
[ 6 7]]
...
[[92 93]
[94 95]]
[[96 97]
[98 99]]]
These blocks finally need to be reshaped into a (m,m) matrix so that they are stacked in rows of 2*h+1 blocks:
>>> array([[ 0, 1, 4, 5, 8, 9, 12, 13, 16, 17],
[ 2, 3, 6, 7, 10, 11, 14, 15, 18, 19],
...
[80, 81, 84, 85, 88, 89, 92, 93, 96, 97],
[82, 83, 86, 87, 90, 91, 94, 95, 98, 99]])
My problem is that I can't seem to find proper axis permutations after the first reshape into (n,n) blocks. I have looked at several answers such as this one but in vain.
As the real dimensions n and h are quite bigger and this operation takes place in an iterative process, I am looking for an efficient reshaping operation.
I don't think you can do this with reshape and transpose alone (although I'd love to be proven wrong). Using np.block works, but it's a bit messy:
np.block([list(i) for i in input_data.reshape( (2*h+1), (2*h+1), n, n )])
array([[ 0, 1, 4, 5, 8, 9, 12, 13, 16, 17],
[ 2, 3, 6, 7, 10, 11, 14, 15, 18, 19],
[20, 21, 24, 25, 28, 29, 32, 33, 36, 37],
[22, 23, 26, 27, 30, 31, 34, 35, 38, 39],
[40, 41, 44, 45, 48, 49, 52, 53, 56, 57],
[42, 43, 46, 47, 50, 51, 54, 55, 58, 59],
[60, 61, 64, 65, 68, 69, 72, 73, 76, 77],
[62, 63, 66, 67, 70, 71, 74, 75, 78, 79],
[80, 81, 84, 85, 88, 89, 92, 93, 96, 97],
[82, 83, 86, 87, 90, 91, 94, 95, 98, 99]])
EDIT: Never mind, you can do without np.block:
input_data.reshape( (2*h+1), (2*h+1), n, n).transpose(0, 2, 1, 3).reshape(10, 10)
array([[ 0, 1, 4, 5, 8, 9, 12, 13, 16, 17],
[ 2, 3, 6, 7, 10, 11, 14, 15, 18, 19],
[20, 21, 24, 25, 28, 29, 32, 33, 36, 37],
[22, 23, 26, 27, 30, 31, 34, 35, 38, 39],
[40, 41, 44, 45, 48, 49, 52, 53, 56, 57],
[42, 43, 46, 47, 50, 51, 54, 55, 58, 59],
[60, 61, 64, 65, 68, 69, 72, 73, 76, 77],
[62, 63, 66, 67, 70, 71, 74, 75, 78, 79],
[80, 81, 84, 85, 88, 89, 92, 93, 96, 97],
[82, 83, 86, 87, 90, 91, 94, 95, 98, 99]])
I want to create multiple list by using a single loop with some condition.
I know how to create one list which is done by appending, but here all the results of loop goes in one single list which is not what I want.
So lets say we run a loop on first 100 numbers and I want to create multiple list where first list contains numbers till 5 , second from 6 to 10, third from 11 to 15 and so.
This code is just for one list created by me.
number = range(100)
first = []
for i in number:
first.append(i)
first
Something like that:
l = []
for x in range(6, 102, 5):
l.append([y for y in range(x-5, x)])
Output:
[[1, 2, 3, 4, 5],
[6, 7, 8, 9, 10],
[11, 12, 13, 14, 15],
[16, 17, 18, 19, 20],
[21, 22, 23, 24, 25],
[26, 27, 28, 29, 30],
[31, 32, 33, 34, 35],
[36, 37, 38, 39, 40],
[41, 42, 43, 44, 45],
[46, 47, 48, 49, 50],
[51, 52, 53, 54, 55],
[56, 57, 58, 59, 60],
[61, 62, 63, 64, 65],
[66, 67, 68, 69, 70],
[71, 72, 73, 74, 75],
[76, 77, 78, 79, 80],
[81, 82, 83, 84, 85],
[86, 87, 88, 89, 90],
[91, 92, 93, 94, 95],
[96, 97, 98, 99, 100]]
The range function takes three parameters start, stop and step.
Try this, This will create what you want:
lists = []
for i in range(1, 100, 5): # range(start, end, size_of_each_list)
lists.append(list(range(i,i + 5)))
The lists will be:
[[1, 2, 3, 4, 5],
[6, 7, 8, 9, 10],
[11, 12, 13, 14, 15],
[16, 17, 18, 19, 20],
[21, 22, 23, 24, 25],
[26, 27, 28, 29, 30],
[31, 32, 33, 34, 35],
[36, 37, 38, 39, 40],
[41, 42, 43, 44, 45],
[46, 47, 48, 49, 50],
[51, 52, 53, 54, 55],
[56, 57, 58, 59, 60],
[61, 62, 63, 64, 65],
[66, 67, 68, 69, 70],
[71, 72, 73, 74, 75],
[76, 77, 78, 79, 80],
[81, 82, 83, 84, 85],
[86, 87, 88, 89, 90],
[91, 92, 93, 94, 95],
[96, 97, 98, 99, 100]]
This is one way to do it. Basically you create an iterator from your list and then get all the lists you want from it.
number = range(100)
number_iter = iter(number)
lists = []
while True:
try:
lists.append([next(number_iter) for _ in range(5)])
except StopIteration as e:
break
lists
this has the advantage that your initial `number' list can be anything...
I would use second list inside the first.
number = range(100)
cnt = 0
first = []
second = []
for i in number:
cnt += 1
second.append(i)
if cnt == 5:
first.append(second)
cnt = 0
second = []
first
output:
[[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9],
...
[95, 96, 97, 98, 99]]
I would do a list comprehension similar to this:
[[i, i+1, i+2, i+3, i+4] for i in range (1, 100, 5)]
The output would look like this:
[[1, 2, 3, 4, 5],
...,
[96, 97, 98, 99, 100]]
https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions
It is possible. We can do this by directly creating variables in our globals() environment.
(Use locals() if it should exist only in the enclosing function).
You can do this with the following code:
# Run the loop
for i in range(100):
# for 0-4, list_name = 'list_1'
list_name = 'list_' + str(i//5 + 1)
# Try to append to that list
try:
globals()[list_name].append(i)
# If if doesn't exist, create it on the run!
except KeyError:
globals()[list_name] = [i]
This globals()[list_name] = [i] first:
Gets the module environment dictionary.
Creates list_name variable.
Initialises it to a list containing i.
Let us print them all:
for i in range(20):
# Print list_1 through list_20
list_name = 'list_' + str(i+1)
print(list_name + ':', globals()[list_name])
You get:
list_1: [0, 1, 2, 3, 4]
list_2: [5, 6, 7, 8, 9]
list_3: [10, 11, 12, 13, 14]
list_4: [15, 16, 17, 18, 19]
list_5: [20, 21, 22, 23, 24]
list_6: [25, 26, 27, 28, 29]
list_7: [30, 31, 32, 33, 34]
list_8: [35, 36, 37, 38, 39]
list_9: [40, 41, 42, 43, 44]
list_10: [45, 46, 47, 48, 49]
list_11: [50, 51, 52, 53, 54]
list_12: [55, 56, 57, 58, 59]
list_13: [60, 61, 62, 63, 64]
list_14: [65, 66, 67, 68, 69]
list_15: [70, 71, 72, 73, 74]
list_16: [75, 76, 77, 78, 79]
list_17: [80, 81, 82, 83, 84]
list_18: [85, 86, 87, 88, 89]
list_19: [90, 91, 92, 93, 94]
list_20: [95, 96, 97, 98, 99]
Note: Why not have some number fun! See python inflect package.
I have a 3D tensor A x B x C. For each matrix B x C, I want to extract the leading diagonal.
Is there a vectorized way of doing this in numpy or pytorch instead of looping over A?
You can use numpy.diagonal()
np.diagonal(a, axis1=1, axis2=2)
Example:
In [10]: a = np.arange(3*4*5).reshape(3,4,5)
In [11]: a
Out[11]:
array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39]],
[[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
In [12]: np.diagonal(a, axis1=1, axis2=2)
Out[12]:
array([[ 0, 6, 12, 18],
[20, 26, 32, 38],
[40, 46, 52, 58]])
Assuming that the leading diagonal for a generic non-squared (BxC) slice starts off from the top-left corner, we can reshape and slice -
a.reshape(a.shape[0],-1)[:,::a.shape[-1]+1]
Sample run -
In [193]: np.random.seed(0)
In [194]: a = np.random.randint(11,99,(3,4,5))
In [195]: a
Out[195]:
array([[[55, 58, 75, 78, 78],
[20, 94, 32, 47, 98],
[81, 23, 69, 76, 50],
[98, 57, 92, 48, 36]],
[[88, 83, 20, 31, 91],
[80, 90, 58, 75, 93],
[60, 40, 30, 30, 25],
[50, 43, 76, 20, 68]],
[[43, 42, 85, 34, 46],
[86, 66, 39, 45, 11],
[11, 47, 64, 16, 49],
[28, 90, 15, 53, 69]]])
In [196]: a.reshape(a.shape[0],-1)[:,::a.shape[-1]+1]
Out[196]:
array([[55, 94, 69, 48],
[88, 90, 30, 20],
[43, 66, 64, 53]])
In PyTorch, use torch.diagonal():
t.diagonal(dim0=-2, dim1=-1)
I'd like to create a function, that returns the nearest value in the array along a specified axis to a given value.
To get the index of the nearest value I use the following code where arr is a multidimensional array and value is the value to look for:
def nearest_index( arr, value, axis=None ):
return ( np.abs( arr - value ) ).argmin( axis=axis )
But I struggle with using the result of this function to get the values from the array.
It is easy with 1D-arrays:
In [14]: arr_1 = np.random.randint( 10, 100, size=( 10, ) )
In [15]: arr_1
Out[15]: array([67, 49, 90, 29, 60, 80, 31, 55, 29, 10])
In [16]: nearest_index( arr_1, 50 )
Out[16]: 1
In [17]: arr_1[nearest_index( arr_1, 50 )]
Out[17]: 49
or with flattened arrays:
In [25]: arr_3 = np.random.randint( 10, 100, size=( 2, 3, 4, ) )
In [26]: arr_3
Out[26]:
array([[[85, 51, 74, 79],
[63, 42, 27, 75],
[89, 68, 80, 63]],
[[85, 72, 74, 16],
[85, 22, 47, 78],
[44, 70, 98, 34]]])
In [27]: idx_flat = nearest_index( arr_3, 50, axis=None )
In [28]: idx_flat
Out[28]: 1
In [29]: idx = np.unravel_index( idx_flat, arr_3.shape )
In [30]: idx
Out[30]: (0, 0, 1)
In [31]: arr_3[idx]
Out[31]: 51
How can I create a function, that returns the values along the defined axis?
I tried the solution for this Question, but I only got it working for axis=-1.
Note that it is not an issue to me, if only the first occurance of the result is found if multiple elements in the array are equally near the expected value.
For a multi-dimensional array, we need to use advanced-indexing. So, for a generic n-dim array and with a specified axis, we could do something like this -
def argmin_values_along_axis(arr, value, axis):
argmin_idx = np.abs(arr - value).argmin(axis=axis)
shp = arr.shape
indx = list(np.ix_(*[np.arange(i) for i in shp]))
indx[axis] = np.expand_dims(argmin_idx, axis=axis)
return np.squeeze(arr[indx])
Sample runs -
In [203]: arr_3 = np.random.randint( 10, 100, size=( 2, 3, 4, ) )
In [204]: arr_3
Out[204]:
array([[[94, 55, 26, 51],
[82, 66, 80, 66],
[96, 54, 93, 57]],
[[59, 28, 95, 56],
[47, 48, 17, 77],
[15, 57, 57, 25]]])
In [205]: argmin_values_along_axis(arr_3, value=50, axis=0)
Out[205]:
array([[59, 55, 26, 51],
[47, 48, 80, 66],
[15, 54, 57, 57]])
In [206]: argmin_values_along_axis(arr_3, value=50, axis=1)
Out[206]:
array([[82, 54, 26, 51],
[47, 48, 57, 56]])
In [207]: argmin_values_along_axis(arr_3, value=50, axis=2)
Out[207]:
array([[51, 66, 54],
[56, 48, 57]])
Well it works for me.
def nearest_index(arr, value, axis=None):
return np.argmin(np.abs( arr - value ), axis=axis)
>>> X
array([[76, 94, 56, 93, 28, 0, 44, 50, 89, 93],
[80, 99, 29, 98, 39, 27, 55, 70, 19, 76],
[87, 7, 28, 78, 47, 95, 34, 97, 66, 27],
[75, 78, 82, 30, 15, 0, 2, 25, 58, 69],
[31, 2, 34, 1, 56, 7, 87, 78, 32, 77],
[89, 80, 76, 97, 49, 18, 62, 35, 94, 41],
[ 2, 44, 83, 3, 64, 4, 49, 93, 46, 8],
[51, 63, 45, 57, 77, 90, 93, 4, 26, 81],
[43, 92, 22, 98, 93, 36, 46, 25, 35, 36],
[30, 14, 42, 91, 86, 14, 78, 9, 37, 19]])
>>> X[nearest_index(X, 2, axis=0), np.arange(10)]
array([ 2, 2, 22, 1, 15, 0, 2, 4, 19, 8])
>>> X[np.arange(10), nearest_index(X, 2, axis=1)]
array([ 0, 19, 7, 2, 2, 18, 2, 4, 22, 9])
I have data in a numpy array:
a = np.arange(100)
a = a.reshape((20,5))
When I type
a[:10]
it returns
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39],
[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49]])
Now i decided to reshape the array into 3d array.
b = a.reshape((5,4,5))
array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39]],
[[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]],
[[60, 61, 62, 63, 64],
[65, 66, 67, 68, 69],
[70, 71, 72, 73, 74],
[75, 76, 77, 78, 79]],
[[80, 81, 82, 83, 84],
[85, 86, 87, 88, 89],
[90, 91, 92, 93, 94],
[95, 96, 97, 98, 99]]])
How do I slice b to that I obtain the values like a[:10]?
I tried
b[:10,0,:5]
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[20, 21, 22, 23, 24],
[30, 31, 32, 33, 34],
[40, 41, 42, 43, 44],
[50, 51, 52, 53, 54],
[60, 61, 62, 63, 64],
[70, 71, 72, 73, 74],
[80, 81, 82, 83, 84],
[90, 91, 92, 93, 94]])
But its not correct.
Thank you in advance!
When you use b = a.reshape((5,4,5)) you just create a different view on the same data used by the array a. (ie changes to the elements of a will appear in b). reshape() does not copy data in this case, so it is a very fast operation. Slicing b and slicing a accesses the same memory, so there shouldn't be any need for a different syntax for the b array (just use a[:10]). If you have created a copy of the data, perhaps with np.resize(), and discarded a, just reshape b: b.reshape((20,5))[:10].
By reshaping (20,5) to (5,4,5), there's no way you can pull out the 1st half of the values. You can't split those 5 rows into 2 even groups:
In [9]: b[:2]
Out[9]:
array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39]]])
In [10]: b[:3]
Out[10]:
array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39]],
[[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
The last row of a[:10] is in the middle of b[3,:,:].
Note that b[:2] is (2,4,5), 8 rows of a, grouped into 2 sets of 4.
Now if you'd done c=a.reshape(4,5,5), then c[:2] would have those same 10 rows - in 2 sets of 5. And c[:2].reshape(10,-1) will look just like a[:10].
There could be a programmatic way to get what you want, but not a python slice.
It is important to understand what every component in the shape tells us about the arrangement. I like to think in terms of vectors.
Let's talk about the shape (20, 5) - this would mean, I have 20 vectors where every vector has 5 elements.
For the shape (5, 4, 5) - this would mean, I have 5 vectors, where each vector again has 4 vectors where every vector within has 5 elements.
This might sound complicated but with some deliberation, this could be understood.
Coming to your question, by a[:10] you want to retrieve the first 10 rows where each row should be a vector containing 5 elements but using a shape of (5, 4, 5).
This is only possible if you retrieve the first 4 vectors from 1st vector of the leftmost dimension (5), next 4 vectors from the next vector and next 2 from the 3rd.
Python slicing might not be the best tool to achieve this.