We Keep Coding

numpy select members with a certain shifted window

I want to extract some members from many large numpy arrays. A simple example is
A = np.arange(36).reshape(6,6)
array([[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35]])
I want to extract the members in a shifted windows in each row with minimum stride 2 and maximum stride 4. For example, in the first row, I would like to have
[2, 3, 4] # A[i,i+2:i+4+1] where i == 0
In the second row, I want to have
[9, 10, 11] # A[i,i+2:i+4+1] where i == 1
In the third row, I want to have
[16, 17, 0]
[[2, 3, 4],
[9, 10, 11],
[16 17, 0],
[23, 0, 0]]
I want to know efficient ways to do this. Thanks.

you can extract values in an array by providing a list of indices for each dimension. For example, if you want the second diagonal, you can use arr[np.arange(0, len(arr)-1), np.arange(1, len(arr))]
For your example, I would do smth like what's in the code below. although, I did not account for different strides. If you want to account for strides, you can change the behaviour of the index list creation. If you struggle with adding the stride functionality, comment and I'll edit this answer.
import numpy as np
def slide(arr, window_len = 3, start = 0):
# pad array to get zeros out of bounds
arr_padded = np.zeros((arr.shape[0], arr.shape[1]+window_len-1))
arr_padded[:arr.shape[0], :arr.shape[1]] = arr
# compute the number of window moves
repeats = min(arr.shape[0], arr.shape[1]-start)
# create index lists
idx0 = np.arange(repeats).repeat(window_len)
idx1 = np.concatenate(
[np.arange(start+i, start+i+window_len)
for i in range(repeats)])
return arr_padded[idx0, idx1].reshape(-1, window_len)
A = np.arange(36).reshape(6,6)
print(f'A =\n{A}')
print(f'B =\n{slide(A, start = 2)}')
output:
A =
[[ 0 1 2 3 4 5]
[ 6 7 8 9 10 11]
[12 13 14 15 16 17]
[18 19 20 21 22 23]
[24 25 26 27 28 29]
[30 31 32 33 34 35]]
B =
[[ 2. 3. 4.]
[ 9. 10. 11.]
[16. 17. 0.]
[23. 0. 0.]]

Tensorflow: Pick a list of non-overlapping slices from a tensor based on an info from another tensor

I have a output of a conv layer, conv, of shape Batch_size x H x W x C.
I also have another tensort with the shape Batch_size x None x 2. The latter tensor supplies a list of points (Height and Width coordinates) for each example in a bach (the list is of different length for each example). And I want to extract the Channel dimension for each of those points.
I tried using tf.gather, and tf.batch_gather, but both doesn't seem to be the right thing to use here.
Basically what I want is for each batch b to loop through the points: for each point i having its h_i (height coord) and w_i (coord) and return conv[b, h_i, w_j, :]. And then stack these results.

Here is how you can do that:
import tensorflow as tf
def pick_points(images, coords):
coords = tf.convert_to_tensor(coords)
s = tf.shape(coords)
batch_size, num_coords = s[0], s[1]
# Make batch indices
r = tf.range(batch_size, dtype=coords.dtype)
idx_batch = tf.tile(tf.expand_dims(r, 1), [1, num_coords])
# Full index
idx = tf.concat([tf.expand_dims(idx_batch, 2), coords], axis=2)
# Gather pixels
pixels = tf.gather_nd(images, idx)
# Output has shape [batch_size, num_coords, num_channels]
return pixels
# Test
with tf.Graph().as_default(), tf.Session() as sess:
# 2 x 2 x 3 x 3
images = [
[
[[ 1, 2, 3], [ 4, 5, 6], [ 7, 8, 9]],
[[10, 11, 12], [13, 14, 15], [16, 17, 18]],
],
[
[[19, 20, 21], [22, 23, 24], [25, 26, 27]],
[[28, 29, 30], [31, 32, 33], [34, 35, 36]],
],
]
# 2 x 2 x 2
coords = [
[[0, 1], [1, 2]],
[[1, 0], [1, 1]],
]
pixels = pick_points(images, coords)
print(sess.run(pixels))
# [[[ 4 5 6]
# [16 17 18]]
#
# [[28 29 30]
# [31 32 33]]]

How can I iterate through numpy 3d array

So I have an array:
array([[[27, 27, 28],
[27, 14, 28]],
[[14, 5, 4],
[ 5, 6, 14]]])
How can I iterate through it and on each iteration get the [a, b, c] values, I try like that:
for v in np.nditer(a):
print(v)
but it just prints
27
27
28
27
14
28
14
5
4
5
6
I need:
[27 27 28]
[27 14 28]...

b = a.reshape(-1, 3)
for triplet in b:
...

Apparently you want to iterate on the first 2 dimensions of the array, returning the 3rd (as 1d array).
In [242]: y = np.array([[[27, 27, 28],
...: [27, 14, 28]],
...:
...: [[14, 5, 4],
...: [ 5, 6, 14]]])
Double loops are fine, as is reshaping to a (4,2) and iterating.
nditer isn't usually needed, or encouraged as an iteration mechanism (its documentation needs a stronger disclaimer). It's really meant for C level code. It isn't used much in Python level code. One exception is the np.ndindex function, which can be useful in this case:
In [244]: for ij in np.ndindex(y.shape[:2]):
...: print(ij, y[ij])
...:
(0, 0) [27 27 28]
(0, 1) [27 14 28]
(1, 0) [14 5 4]
(1, 1) [ 5 6 14]
ndindex uses nditer in multi_index mode on a temp array of the specified shape.
Where possible try to work without iteration. Iteration, with any of these tricks, is relatively slow.

You could do something ugly as
for i in range(len(your_array)):
for j in range(len(your_array[i])):
print(your_array[i][j])

Think of it as having arrays within an array. So within array v you have array a which in turn contains the triplets b
import numpy as np
na = np.array
v=na([[[27, 27, 28], [27, 14, 28]], [[14, 5, 4],[ 5, 6, 14]]])
for a in v:
for b in a:
print b
Output:
[27 27 28]
[27 14 28]
[14 5 4]
[ 5 6 14]
Alternatively you could do the following,
v2 = [b for a in v for b in a]
Now all your triplets are stored in v2
[array([27, 27, 28]),
array([27, 14, 28]),
array([14, 5, 4]),
array([ 5, 6, 14])]
..and you can access them like a 1D array eg
print v2[0]
gives..
array([27, 27, 28])

Another alternative (useful for arbitrary dimensionality of the array containing the n-tuples):
a_flat = a.ravel()
n = 3
m = len(a_flat)/n
[a_flat[i:i+n] for i in range(m)]
or in one line (slower):
[a.ravel()[i:i+n] for i in range(len(a.ravel())/n)]
or for further usage within a loop:
for i in range(len(a.ravel())/n):
print a.ravel()[i:i+n]

Reshape the array A (whose shape is n1, n2, 3) to array B (whose shape is n1 * n2, 3), and iterate through B. Note that B is just A's view. A and B share the same data block in the memory, but they have different array headers information where records their shapes, and changing values in B will also change A's value. The code below:
a = np.array([[[27, 27, 28],[27, 14, 28]],
[[14, 5, 4],[ 5, 6, 14]]])
b = a.reshape((-1, 3))
for last_d in b:
a, b, c = last_d
# do something with last_d or a, b, c

Pythonic way to get both diagonals passing through a matrix entry (i,j)

What is the Pythonic way to get a list of diagonal elements in a matrix passing through entry (i,j)?
For e.g., given a matrix like:
[1 2 3 4 5]
[6 7 8 9 10]
[11 12 13 14 15]
[16 17 18 19 20]
[21 22 23 24 25]
and an entry, say, (1,3) (representing element 9) how can I get the elements in the diagonals passing through 9 in a Pythonic way? Basically, [3,9,15] and [5,9,13,17,21] both.

Using np.diagonal with a little offset logic.
import numpy as np
lst = np.array([[1, 2, 3, 4, 5],
[6, 7, 8, 9, 10],
[11, 12, 13, 14, 15],
[16, 17, 18, 19, 20],
[21, 22, 23, 24, 25]])
i, j = 1, 3
major = np.diagonal(lst, offset=(j - i))
print(major)
array([ 3, 9, 15])
minor = np.diagonal(np.rot90(lst), offset=-lst.shape[1] + (j + i) + 1)
print(minor)
array([ 5, 9, 13, 17, 21])
The indices i and j are the row and column. By specifying the offset, numpy knows from where to begin selecting elements for the diagonal.
For the major diagonal, You want to start collecting from 3 in the first row. So you need to take the current column index and subtract it by the current row index, to figure out the correct column index at the 0th row. Similarly for the minor diagonal, where the array is flipped (rotated by 90˚) and the process repeats.

As another alternative method, with raveling the array and for matrix with shape (n*n):
array = np.array([[1, 2, 3, 4, 5],
[6, 7, 8, 9, 10],
[11, 12, 13, 14, 15],
[16, 17, 18, 19, 20],
[21, 22, 23, 24, 25]])
x, y = 1, 3
a_mod = array.ravel()
size = array.shape[0]
if y >= x:
diag = a_mod[y-x:(x+size-y)*size:size+1]
else:
diag = a_mod[(x-y)*size::size+1]
if x-(size-1-y) >= 0:
reverse_diag = array[:, ::-1].ravel()[(x-(size-1-y))*size::size+1]
else:
reverse_diag = a_mod[x:x*size+1:size-1]
# diag --> [ 3 9 15]
# reverse_diag --> [ 5 9 13 17 21]
The correctness of the resulted arrays must be checked further. This can be developed to handle matrices with other shapes e.g. (n*m).

How to permutate tranposition in tensorflow?

From the docs:
Transposes a. Permutes the dimensions according to perm.
The returned tensor's dimension i will correspond to the input
dimension perm[i]. If perm is not given, it is set to (n-1...0), where
n is the rank of the input tensor. Hence by default, this operation
performs a regular matrix transpose on 2-D input Tensors.
But it's still a little unclear to me how should I be slicing the input tensor. E.g. from the docs too:
tf.transpose(x, perm=[0, 2, 1]) ==> [[[1 4]
[2 5]
[3 6]]
[[7 10]
[8 11]
[9 12]]]
Why is it that perm=[0,2,1] produces a 1x3x2 tensor?
After some trial and error:
twothreefour = np.array([ [[1,2,3,4], [5,6,7,8], [9,10,11,12]] ,
[[13,14,15,16], [17,18,19,20], [21,22,23,24]] ])
twothreefour
[out]:
array([[[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12]],
[[13, 14, 15, 16],
[17, 18, 19, 20],
[21, 22, 23, 24]]])
And if I transpose it:
fourthreetwo = tf.transpose(twothreefour)
with tf.Session() as sess:
init = tf.initialize_all_variables()
sess.run(init)
print (fourthreetwo.eval())
I get a 4x3x2 to a 2x3x4 and that sounds logical.
[out]:
[[[ 1 13]
[ 5 17]
[ 9 21]]
[[ 2 14]
[ 6 18]
[10 22]]
[[ 3 15]
[ 7 19]
[11 23]]
[[ 4 16]
[ 8 20]
[12 24]]]
But when I use the perm parameter the output, I'm not sure what I'm really getting:
twofourthree = tf.transpose(twothreefour, perm=[0,2,1])
with tf.Session() as sess:
init = tf.initialize_all_variables()
sess.run(init)
print (threetwofour.eval())
[out]:
[[[ 1 5 9]
[ 2 6 10]
[ 3 7 11]
[ 4 8 12]]
[[13 17 21]
[14 18 22]
[15 19 23]
[16 20 24]]]
Why does perm=[0,2,1] returns a 2x4x3 matrix from a 2x3x4 ?
Trying it again with perm=[1,0,2]:
threetwofour = tf.transpose(twothreefour, perm=[1,0,2])
with tf.Session() as sess:
init = tf.initialize_all_variables()
sess.run(init)
print (threetwofour.eval())
[out]:
[[[ 1 2 3 4]
[13 14 15 16]]
[[ 5 6 7 8]
[17 18 19 20]]
[[ 9 10 11 12]
[21 22 23 24]]]
Why does perm=[1,0,2] return a 3x2x4 from a 2x3x4?
Does it mean that the perm parameter is taking my np.shape and transposing the tensor based on the elements based on my array shape?
I.e. :
_size = (2, 4, 3, 5)
randarray = np.random.randint(5, size=_size)
shape_idx = {i:_s for i, _s in enumerate(_size)}
randarray_t_func = tf.transpose(randarray, perm=[3,0,2,1])
with tf.Session() as sess:
init = tf.initialize_all_variables()
sess.run(init)
tranposed_array = randarray_t_func.eval()
print (tranposed_array.shape)
print (tuple(shape_idx[_s] for _s in [3,0,2,1]))
[out]:
(5, 2, 3, 4)
(5, 2, 3, 4)

I think perm is permuting the dimensions. For example perm=[0,2,1] is short for dim_0 -> dim_0, dim_1 -> dim_2, dim_2 -> dim_1. So for a 2D tensor, perm=[1,0] is just matrix transpose. Does this answer your question?

A=[2,3,4] matrix, using perm(1,0,2) will get B=[3,2,4].
Explanation:
Index=(0,1,2)
A =[2,3,4]
Perm =(1,0,2)
B =(3,2,4) --> Perm 1 from Index 1 (3), Perm 0 from Index 0 (2), Perm 2 from Index 2 (4) --> so get (3,2,4)