Related
I have the following numpy 3d array, in which I need to duplicate the last column
array([[[ 7, 5, 93],
[19, 4, 69],
[62, 2, 52]],
[[ 6, 1, 65],
[41, 9, 94],
[39, 4, 49]]])
The desired output is:
array([[[ 7, 5, 93, 93],
[19, 4, 69, 69],
[62, 2, 52, 52]],
[[ 6, 1, 65, 65],
[41, 9, 94, 94],
[39, 4, 49, 49]]])
Is there a clever way of doing this?
You could concatenate along the last axis as follows-
numpy.concatenate([a, numpy.expand_dims(a[:, :, -1], axis=2)], axis=2)
There is a built-in numpy function for this purpose:
np.insert(x,-1,x[...,-1],-1)
output:
array([[[ 7, 5, 93, 93],
[19, 4, 69, 69],
[62, 2, 52, 52]],
[[ 6, 1, 65, 65],
[41, 9, 94, 94],
[39, 4, 49, 49]]])
I am trying to achieve something like this in torch.
I have two tensors of shapes [X, 4] and [Y, 4]. this 4s are basically 4 coordinates of something. So for each combination of X and Y (2 vectors of length 4), I want to apply some function (elementwise average for example), and form result of the shape [X, Y, 4].
How to do this?
By elementwise average, I mean this operation,
[2 4 6 8] OP [8 6 4 2] = [5 5 5 5]
But it can be any arbitrary operation.
N.B. I was able to solve it using loops, but searching for a vectorized solution.
Your answer depends on the function. For example, for element wise mean:
np.mean([x[:,None,:],y[None,...]], axis=0)
or for einsum:
np.einsum('ij,kj->ikj',x,y)
or for summation:
x[:,None,:]+y[None,...]
And if you implement your CUSTOM function properly (elementwise), you can use broadcasting to do the job using this:
np.frompyfunc(myfunc,2,1)(x[:,None,:],y[None,...])
sample:
x = array([[0, 1, 2, 3],
[4, 5, 6, 7]])
y = array([[100, 101, 102, 103],
[104, 105, 106, 107],
[108, 109, 110, 111]])
#np.mean([x[:,None,:],y[None,...]], axis=0)
array([[[50, 51, 52, 53],
[52, 53, 54, 55],
[54, 55, 56, 57]],
[[52, 53, 54, 55],
[54, 55, 56, 57],
[56, 57, 58, 59]]])
#np.einsum('ij,kj->ikj',x,y)
array([[[ 0, 101, 204, 309],
[ 0, 105, 212, 321],
[ 0, 109, 220, 333]],
[[400, 505, 612, 721],
[416, 525, 636, 749],
[432, 545, 660, 777]]])
#x[:,None,:]+y[None,...]
array([[[100, 102, 104, 106],
[104, 106, 108, 110],
[108, 110, 112, 114]],
[[104, 106, 108, 110],
[108, 110, 112, 114],
[112, 114, 116, 118]]])
def myfunc(x, y):
return x+y
#np.frompyfunc(myfunc,2,1)(x[:,None,:],y[None,...])
array([[[100, 102, 104, 106],
[104, 106, 108, 110],
[108, 110, 112, 114]],
[[104, 106, 108, 110],
[108, 110, 112, 114],
[112, 114, 116, 118]]], dtype=object)
What is the most pythonic way of splitting a NumPy matrix (a 2-D array) into equal chunks both vertically and horizontally?
For example :
aa = np.reshape(np.arange(270),(18,15)) # a 18x15 matrix
then a "function" like
ab = np.split2d(aa,(2,3))
would result in a list of 6 matrices shaped (9,5) each. The first guess is combine hsplit, map and vsplit, but how the mar has to be applied if there are two parameters to define for it, like :
map(np.vsplit(#,3),np.hsplit(aa,2))
Here's one approach staying within NumPy environment -
def view_as_blocks(arr, BSZ):
# arr is input array, BSZ is block-size
m,n = arr.shape
M,N = BSZ
return arr.reshape(m//M, M, n//N, N).swapaxes(1,2).reshape(-1,M,N)
Sample runs
1) Actual big case to verify shapes :
In [41]: aa = np.reshape(np.arange(270),(18,15))
In [42]: view_as_blocks(aa, (9,5)).shape
Out[42]: (6, 9, 5)
2) Small case to manually verify values:
In [43]: aa = np.reshape(np.arange(36),(6,6))
In [44]: aa
Out[44]:
array([[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35]])
In [45]: view_as_blocks(aa, (2,3)) # Blocks of shape (2,3)
Out[45]:
array([[[ 0, 1, 2],
[ 6, 7, 8]],
[[ 3, 4, 5],
[ 9, 10, 11]],
[[12, 13, 14],
[18, 19, 20]],
[[15, 16, 17],
[21, 22, 23]],
[[24, 25, 26],
[30, 31, 32]],
[[27, 28, 29],
[33, 34, 35]]])
If you are willing to work with other libraries, scikit-image could be of use here, like so -
from skimage.util import view_as_blocks as viewB
out = viewB(aa, tuple(BSZ)).reshape(-1,*BSZ)
Runtime test -
In [103]: aa = np.reshape(np.arange(270),(18,15))
# #EFT's soln
In [99]: %timeit split_2d(aa, (2,3))
10000 loops, best of 3: 23.3 µs per loop
# #glegoux's soln-1
In [100]: %timeit list(get_chunks(aa, 2,3))
100000 loops, best of 3: 3.7 µs per loop
# #glegoux's soln-2
In [111]: %timeit list(get_chunks2(aa, 9, 5))
100000 loops, best of 3: 3.39 µs per loop
# Proposed in this post
In [101]: %timeit view_as_blocks(aa, (9,5))
1000000 loops, best of 3: 1.86 µs per loop
Please note that I have used (2,3) for split_2d and get_chunks as by their definitions, they are using that as the number of blocks. In my case with view_as_blocks, I have the parameter BSZ indicating the block size. So, I have (9,5) there. get_chunks2 follows the same format as view_as_blocks. The outputs should represent the same there.
You could use np.split & np.concatenate, the latter to allow the second split to be conducted in a single step:
def split_2d(array, splits):
x, y = splits
return np.split(np.concatenate(np.split(array, y, axis=1)), x*y)
ab = split_2d(aa,(2,3))
ab[0].shape
Out[95]: (9, 5)
len(ab)
Out[96]: 6
This also seems like it should be relatively straightforward to generalize to the n-dim case, though I haven't followed that thought all the way through just yet.
Edit:
For a single array as output, just add np.stack:
np.stack(ab).shape
Out[99]: (6, 9, 5)
To cut, this matrix (18,15) :
+-+-+-+
+ +
+-+-+-+
in 2x3 blocks (9,5) like it :
+-+-+-+
+-+-+-+
+-+-+-+
Do:
from pprint import pprint
import numpy as np
M = np.reshape(np.arange(18*15),(18,15))
def get_chunks(M, n, p):
n = len(M)//n
p = len(M[0])//p
for i in range(0, len(M), n):
for j in range(0, len(M[0]), p):
yield M[i:i+n,j:j+p]
def get_chunks2(M, n, p):
for i in range(0, len(M), n):
for j in range(0, len(M[0]), p):
yield M[i:i+n,j:j+p]
# list(get_chunks2(M, 9, 5)) same result more faster
chunks = list(get_chunks(M, 2, 3))
pprint(chunks)
Output:
[array([[ 0, 1, 2, 3, 4],
[ 15, 16, 17, 18, 19],
[ 30, 31, 32, 33, 34],
[ 45, 46, 47, 48, 49],
[ 60, 61, 62, 63, 64],
[ 75, 76, 77, 78, 79],
[ 90, 91, 92, 93, 94],
[105, 106, 107, 108, 109],
[120, 121, 122, 123, 124]]),
array([[ 5, 6, 7, 8, 9],
[ 20, 21, 22, 23, 24],
[ 35, 36, 37, 38, 39],
[ 50, 51, 52, 53, 54],
[ 65, 66, 67, 68, 69],
[ 80, 81, 82, 83, 84],
[ 95, 96, 97, 98, 99],
[110, 111, 112, 113, 114],
[125, 126, 127, 128, 129]]),
array([[ 10, 11, 12, 13, 14],
[ 25, 26, 27, 28, 29],
[ 40, 41, 42, 43, 44],
[ 55, 56, 57, 58, 59],
[ 70, 71, 72, 73, 74],
[ 85, 86, 87, 88, 89],
[100, 101, 102, 103, 104],
[115, 116, 117, 118, 119],
[130, 131, 132, 133, 134]]),
array([[135, 136, 137, 138, 139],
[150, 151, 152, 153, 154],
[165, 166, 167, 168, 169],
[180, 181, 182, 183, 184],
[195, 196, 197, 198, 199],
[210, 211, 212, 213, 214],
[225, 226, 227, 228, 229],
[240, 241, 242, 243, 244],
[255, 256, 257, 258, 259]]),
array([[140, 141, 142, 143, 144],
[155, 156, 157, 158, 159],
[170, 171, 172, 173, 174],
[185, 186, 187, 188, 189],
[200, 201, 202, 203, 204],
[215, 216, 217, 218, 219],
[230, 231, 232, 233, 234],
[245, 246, 247, 248, 249],
[260, 261, 262, 263, 264]]),
array([[145, 146, 147, 148, 149],
[160, 161, 162, 163, 164],
[175, 176, 177, 178, 179],
[190, 191, 192, 193, 194],
[205, 206, 207, 208, 209],
[220, 221, 222, 223, 224],
[235, 236, 237, 238, 239],
[250, 251, 252, 253, 254],
[265, 266, 267, 268, 269]])]
For a simpler solution, I used np.array_split together with transforming the matrices. So let's say that I want it split into 3 equal chunks vertically and 2 equal chunks horizontally, then:
# Create your matrix
matrix = np.reshape(np.arange(270),(18,15)) # a 18x15 matrix
# Container for your final matrices
final_matrices = []
# Then split into 3 equal chunks vertically
vertically_split_matrices = np.array_split(matrix)
for v_m in vertically_split_matrices:
# Then split the transformed matrices equally
m1, m2 = np.array_split(v_m.T, 2)
# And transform the matrices back
final_matrices.append(m1.T)
final_matrices.append(m2.T)
So I end up with 6 chunks, all of which are the same height and the same width.
I have a numpy array that looks like the following:
np.array([
[23, 12, 4, 103, 87, 0.6],
[32, 18, 3, 120, 70, 0.6],
[43, 12, 8, 109, 89, 0.4],
[20, 13, 7, 111, 77, 0.8]
])
I want to transform this array where the last column becomes its own array, such that it will look like this:
np.array([
[[23, 12, 4, 103, 87], [0.6]],
[[32, 18, 3, 120, 70], [0.6]],
[[43, 12, 8, 109, 89], [0.4]],
[[20, 13, 7, 111, 77], [0.8]]
])
What would be the best way to go about this? I am relatively new to Python and have tried out some loops but to no avail. Thanks!
numpy requires consistent dimensions in its array; that would give two different sizes. You can either use two separate variables (i.e. parallel arrays):
X = data[:, :-1]
y = data[:, -1]
X = np.array([
[23, 12, 4, 103, 87],
[32, 18, 3, 120, 70],
[43, 12, 8, 109, 89],
[20, 13, 7, 111, 77],
])
y = np.array([
0.6, 0.6, 0.4, 0.8
])
Or you can store a list of pairs:
my_list = [(row[:-1], [row[-1]]) for row in data]
my_list = [
([23, 12, 4, 103, 87], [0.6]),
([32, 18, 3, 120, 70], [0.6]),
([43, 12, 8, 109, 89], [0.4]),
([20, 13, 7, 111, 77], [0.8])
]
The best strategy depends on your use case.
I am a little confused with Python's advanced slicing. I basically had a dictionary and with help from SO, I made it into an array.
array1 =
([[[36, 16],
[48, 24],
[12, 4],
[12, 4]],
[[48, 24],
[64, 36],
[16, 6],
[16, 6]],
[[12, 4],
[16, 6],
[ 4, 1],
[ 4, 1]],
[[12, 4],
[16, 6],
[ 4, 1],
[ 4, 1]]])
To practice using matrix solver, the array was turned into a square matrix (4 x 4) using:
array_matrix_sized = array[:, :, 0]
I read that this means [number of indices, rows, columns]. I am a little clueless as to why [:,:,0] returns a 4 x 4 matrix. To try to help, I made an array that has a length 100, and I have been trying to turn it into a 10 x 10 matrix in a similar manner with no success. What throws me off is the number of rows is ":" and the number of columns is "0", if I read this concept correctly. For a 4 x 4 matrix, why isn't it array[:, 4, 4]? I am also assuming the : is because I am interested in all the values.
Thank you in advance for any help/advice. I do apologize if this is a simple question, but I really could use the clarification on how this works.
Still not quite understanding.
If I have
array2 = array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38,
39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77,
78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103,
104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116,
117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129,
130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,
143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155,
156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168,
169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181,
182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194,
195, 196, 197, 198, 199])
To get it into a 10 X 10 matrix, I tried using array2[:,:,0] and get the error IndexError: too many indices for array. Isn't this similar to my first example?
I read that this means [number of indices, rows, columns]. [...] What throws me off is the number of rows is ":" and the number of columns is "0", if I read this concept correctly.
No. It means [which parts I want on dimension 1, which parts I want on dimension 2, which parts I want on dimension 3]. The indices are not how many rows/columns you want, they are which ones you want. And, as you said : means "all" in this context.
For a 4 x 4 matrix, why isn't it array[:, 4, 4]?
You don't specify the shape of the result. The shape of the result depends on the shape of the original array. Since your array is 4x4x2, getting one element on the last dimension gives you 4x4. If the array was 8x7x2, then [:, :, 0] would give you an 8x7 result.
So [:, :, 0] means "give me everything on the first two dimensions, but only the first item on the last dimension. This amounts to getting the first element of each "row" (or the first "column" as it appears in the display) which is why you get the result you get:
>>> array1[:, :, 0]
array([[36, 48, 12, 12],
[48, 64, 16, 16],
[12, 16, 4, 4],
[12, 16, 4, 4]])
Likewise, doing [0, :, :] gives you the first "chunk":
>>> array1[0, :, :]
array([[36, 16],
[48, 24],
[12, 4],
[12, 4]])
And doing [:, 0, :] gives you the first row of each chunk:
>>> x[:, 0, :]
array([[36, 16],
[48, 24],
[12, 4],
[12, 4]])
I just wanted to add a clarifying example:
>>> np.arange(4*4*2).reshape(4,4,2)
array([[[ 0, 1],
[ 2, 3],
[ 4, 5],
[ 6, 7]],
[[ 8, 9],
[10, 11],
[12, 13],
[14, 15]],
[[16, 17],
[18, 19],
[20, 21],
[22, 23]],
[[24, 25],
[26, 27],
[28, 29],
[30, 31]]])
Since we are in three dimensions, we can still maintain a spatial metaphor. Imagine these 4X2 slices were all stacked up against each other, in front of you, in order (like if they were books). That is, we take the first one and prop it up like a book, the second one behind it and so forth. We choose the first chunk from the first dimension, and it merely gives us back the first "book":
>>> a[0,:,:]
array([[0, 1],
[2, 3],
[4, 5],
[6, 7]])
Now look at the difference between that and the first chunk of the second dimension:
>>> a[:,0,:]
array([[ 0, 1],
[ 8, 9],
[16, 17],
[24, 25]])
This is like slicing off the top. Imagine shaving off the top. It just so happens that with the array you posted, these are the same!
Now finally, the first chunk of the third dimension:
>>> a[:,:,0]
array([[ 0, 2, 4, 6],
[ 8, 10, 12, 14],
[16, 18, 20, 22],
[24, 26, 28, 30]])
This is like slicing what you have in front of you in half - imagine a karate-chop.
Here's a (very crude) image (drawn on my laptop...sorry).
The book analogy is good, but I tend to arrange my arrays in a slightly different order. If we consider the following data array...
a = np.arange(2*3*4).reshape(2,3,4)
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
To me, the above reads as 2 pages, one above the other. Each page has 3 rows and there are 4 words in each row.
I wrote a function to take the same information and arrange it side-by-side since this is the way I tend to arrange things I am working on. (details aren't relevant here...). Here is the rearrangement for visual purposes...
a = np.arange(2*3*4).reshape(2,3,4)
Array... shape (2, 3, 4), ndim 3, not masked
0, 1, 2, 3 12, 13, 14, 15
4, 5, 6, 7 16, 17, 18, 19
8, 9, 10, 11 20, 21, 22, 23
sub (0) sub (1)
a[0,:,:]
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
a[:,0,:]
array([[ 0, 1, 2, 3],
[12, 13, 14, 15]])
>>> a[:,:,0]
array([[ 0, 4, 8],
[12, 16, 20]])
So in my case the sequence from a[0,:,:], a[:,0,:] to a[:,:,0] follows the sequence page, row and word.
People can argue from a different perspective, but I think it is important to realize that not all people view things the same way. I often work with images I prefer the above arrangement of (image, row, column) which is equivalent to the (page, row, word) notation.
Do note... if you don't like the way an array looks, or it doesn't work for you... just swap axes.
a.swapaxes(2,0)
array([[[ 0, 12],
[ 4, 16],
[ 8, 20]],
[[ 1, 13],
[ 5, 17],
[ 9, 21]],
[[ 2, 14],
[ 6, 18],
[10, 22]],
[[ 3, 15],
[ 7, 19],
[11, 23]]])
Still not feeling it?... try a different arrangement until it clicks or simplifies your calculations.
code s=np.arange(Total no. of matrices * number of rows * number of columns).reshape(Total no. of matrices * number of rows * number of columns).Example if code is rameez=np.arange(5*4*4).reshape(5,4,4) [[5=Total no of matrices to be generated]].[[4*4 is the 4*4 dimensional matrix]].Here we will get 5 matrices each of which is 4*4 dimensional matrix.Code rameez=np.arange(10*3*2).reshape(10,3,2) will generate 10 matrices as a whole each with the ((3 * 2)) dimensions.