Related
I have 2 tensors a and b which have the following shapes
>>K.int_shape(a)
(None, 5 , 2)
>>K.int_shape(b)
(None, 5)
What I want to get is a tensor c
>>K.int_shape(c)
(None, 2)
such that along axis 0, you pick the index of largest element in b and use that to index a along axis 1.
Example - say I have
a = np.array([[[2, 7],
[6, 5],
[9, 9],
[4, 2],
[5, 9]],
[[8, 1],
[8, 8],
[3, 9],
[9, 2],
[9, 1]],
[[3, 9],
[6, 4],
[5, 7],
[5, 2],
[5, 6]],
[[7, 5],
[9, 9],
[9, 5],
[9, 8],
[5, 7]],
[[6, 3],
[1, 7],
[3, 6],
[8, 2],
[3, 2]],
[[6, 4],
[5, 9],
[8, 6],
[5, 2],
[5, 2]],
[[2, 6],
[6, 5],
[3, 1],
[6, 2],
[6, 4]]])
and I have
b = np.array([[ 0.27, 0.25, 0.23, 0.06, 0.19],
[ 0.3 , 0.13, 0.17, 0.2 , 0.2 ],
[ 0.08, 0.04, 0.40, 0.36, 0.12],
[ 0.3 , 0.33, 0.11, 0.07, 0.19],
[ 0.15, 0.21, 0.30, 0.12, 0.22],
[ 0.3 , 0.13, 0.23, 0.1 , 0.23],
[ 0.26, 0.35 , 0.25 , 0.07, 0.07]])
What I expect c to be
c = np.zeros((7,2))
for i in range(7):
ind = np.argmax(b[i, :])
c[i, :] = a[i, ind, :]
c
array([[ 2., 7.],
[ 8., 1.],
[ 5., 7.],
[ 9., 9.],
[ 3., 6.],
[ 6., 4.],
[ 6., 5.]])
With Tensorflow for backend (I don't know much about Theano), using tf.gather_nd():
import keras.backend as K
import tensorflow as tf
# `a` and `b` the numpy arrays defined in the question
A = tf.constant(a)
B = tf.constant(b)
# Obtaining your max indices over axis 1, which will be used as indices for axis 1 of A:
col_ind = K.argmax(B, axis=1)
# Creating row range, which will be used as indices for axis 0 of A:
row_ind = K.arange(col_ind.shape[0], dtype='int64')
# Stacking the indices together:
ind = K.stack((row_ind, col_ind), axis=-1)
# Gathering the results:
c = tf.gather_nd(A, ind) # no equivalent I know in K, and no idea about theano...
with tf.Session() as sess:
print(c.eval())
# [[2 7]
# [8 1]
# [5 7]
# [9 9]
# [3 6]
# [6 4]
# [6 5]]
found a solution
A = K.constant(a)
B = K.constant(b)
mxidx = K.argmax(B, axis=1)
c = K.map_fn(lambda i: A[i, mxidx[i], :], K.arange(A.shape[0], dtype='int64'))
print K.eval(c)
array([[ 2., 7.],
[ 8., 1.],
[ 5., 7.],
[ 9., 9.],
[ 3., 6.],
[ 6., 4.],
[ 6., 5.]], dtype=float32)
EDIT: adding runtime info
%timeit K.eval(c)
The slowest run took 9.76 times longer than the fastest. This could mean
that an intermediate result is being cached.
100000 loops, best of 3: 12.2 µs per loop
Are there alternative or better ways to convert a numpy matrix to a python array than this?
>>> import numpy
>>> import array
>>> b = numpy.matrix("1.0 2.0 3.0; 4.0 5.0 6.0", dtype="float16")
>>> print(b)
[[ 1. 2. 3.]
[ 4. 5. 6.]]
>>> a = array.array("f")
>>> a.fromlist((b.flatten().tolist())[0])
>>> print(a)
array('f', [1.0, 2.0, 3.0, 4.0, 5.0, 6.0])
You could convert to a NumPy array and generate its flattened version with .ravel() or .flatten(). This could also be achieved by simply using the function np.ravel itself as it does both these takes under the hood. Finally, use array.array() on it, like so -
a = array.array('f',np.ravel(b))
Sample run -
In [107]: b
Out[107]:
matrix([[ 1., 2., 3.],
[ 4., 5., 6.]], dtype=float16)
In [108]: array.array('f',np.ravel(b))
Out[108]: array('f', [1.0, 2.0, 3.0, 4.0, 5.0, 6.0])
here is an example :
>>> x = np.matrix(np.arange(12).reshape((3,4))); x
matrix([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
>>> x.tolist()
[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]]
I have an array of matrices in which I would like to multiply each matrix by a different number. I tried it this way:
>>> import numpy as np
>>> c = np.array([[[1, 2],[3, 4]],[[1, 2],[3, 4]]])
>>> d = np.array([0.1, 0.2])
>>> d*c
array([[[ 0.1, 0.4],
[ 0.3, 0.8]],
[[ 0.1, 0.4],
[ 0.3, 0.8]]])
While my intention is to get this result -
>>> np.array([d[0]*c[0], d[1]*c[1]])
array([[[ 0.1, 0.2],
[ 0.3, 0.4]],
[[ 0.2, 0.4],
[ 0.6, 0.8]]])
What is the NumPy'iest way to do it?
You need an extra couple of axes:
In [22]: d[:,None,None] * c
Out[22]:
array([[[ 0.1, 0.2],
[ 0.3, 0.4]],
[[ 0.2, 0.4],
[ 0.6, 0.8]]])
d[:,None,None] has shape (2,1,1) which is broadcast across your c array of shape (2,2,2) to multiply each block of c by the corresponding element of d.
You can do this with normal broadcasting:
>>> c*d[:, np.newaxis, np.newaxis]
array([[[ 0.1, 0.2],
[ 0.3, 0.4]],
[[ 0.2, 0.4],
[ 0.6, 0.8]]])
I have a very a very large 2D numpy array that contains 2x2 subsets that I need to take the average of. I am looking for a way to vectorize this operation. For example, given x:
# |- col 0 -| |- col 1 -| |- col 2 -|
x = np.array( [[ 0.0, 1.0, 2.0, 3.0, 4.0, 5.0], # row 0
[ 6.0, 7.0, 8.0, 9.0, 10.0, 11.0], # row 0
[12.0, 13.0, 14.0, 15.0, 16.0, 17.0], # row 1
[18.0, 19.0, 20.0, 21.0, 22.0, 23.0]]) # row 1
I need to end up with a 2x3 array which are the averages of each 2x2 sub array, i.e.:
result = np.array( [[ 3.5, 5.5, 7.5],
[15.5, 17.5, 19.5]])
so element [0,0] is calculated as the average of x[0:2,0:2], while element [0,1] would be the average of x[2:4, 0:2]. Does numpy have vectorized/efficient ways of doing aggregates on subsets like this?
If we form the reshaped matrix y = x.reshape(2,2,3,2), then the (i,j) 2x2 submatrix is given by y[i,:,j,:]. E.g.:
In [340]: x
Out[340]:
array([[ 0., 1., 2., 3., 4., 5.],
[ 6., 7., 8., 9., 10., 11.],
[ 12., 13., 14., 15., 16., 17.],
[ 18., 19., 20., 21., 22., 23.]])
In [341]: y = x.reshape(2,2,3,2)
In [342]: y[0,:,0,:]
Out[342]:
array([[ 0., 1.],
[ 6., 7.]])
In [343]: y[1,:,2,:]
Out[343]:
array([[ 16., 17.],
[ 22., 23.]])
To get the mean of the 2x2 submatrices, use the mean method, with axis=(1,3):
In [344]: y.mean(axis=(1,3))
Out[344]:
array([[ 3.5, 5.5, 7.5],
[ 15.5, 17.5, 19.5]])
If you are using an older version of numpy that doesn't support using a tuple for the axis, you could do:
In [345]: y.mean(axis=1).mean(axis=-1)
Out[345]:
array([[ 3.5, 5.5, 7.5],
[ 15.5, 17.5, 19.5]])
See the link given by #dashesy in a comment for more background on the reshaping "trick".
To generalize this to a 2-d array with shape (m, n), where m and n are even, use
y = x.reshape(x.shape[0]/2, 2, x.shape[1], 2)
y can then be interpreted as an array of 2x2 arrays. The first and third index slots of the 4-d array act as the indices that select one of the 2x2 blocks. To get the upper left 2x2 block, use y[0, :, 0, :]; to the block in the second row and third column of blocks, use y[1, :, 2, :]; and in general, to acces block (j, k), use y[j, :, k, :].
To compute the reduced array of averages of these blocks, use the mean method, with axis=(1, 3) (i.e. average over axes 1 and 3):
avg = y.mean(axis=(1, 3))
Here's an example where x has shape (8, 10), so the array of averages of the 2x2 blocks has shape (4, 5):
In [10]: np.random.seed(123)
In [11]: x = np.random.randint(0, 4, size=(8, 10))
In [12]: x
Out[12]:
array([[2, 1, 2, 2, 0, 2, 2, 1, 3, 2],
[3, 1, 2, 1, 0, 1, 2, 3, 1, 0],
[2, 0, 3, 1, 3, 2, 1, 0, 0, 0],
[0, 1, 3, 3, 2, 0, 3, 2, 0, 3],
[0, 1, 0, 3, 1, 3, 0, 0, 0, 2],
[1, 1, 2, 2, 3, 2, 1, 0, 0, 3],
[2, 1, 0, 3, 2, 2, 2, 2, 1, 2],
[0, 3, 3, 3, 1, 0, 2, 0, 2, 1]])
In [13]: y = x.reshape(x.shape[0]/2, 2, x.shape[1]/2, 2)
Take a look at a couple of the 2x2 blocks:
In [14]: y[0, :, 0, :]
Out[14]:
array([[2, 1],
[3, 1]])
In [15]: y[1, :, 2, :]
Out[15]:
array([[3, 2],
[2, 0]])
Compute the averages of the blocks:
In [16]: avg = y.mean(axis=(1, 3))
In [17]: avg
Out[17]:
array([[ 1.75, 1.75, 0.75, 2. , 1.5 ],
[ 0.75, 2.5 , 1.75, 1.5 , 0.75],
[ 0.75, 1.75, 2.25, 0.25, 1.25],
[ 1.5 , 2.25, 1.25, 1.5 , 1.5 ]])
Right now I am doing this by iterating, but there has to be a way to accomplish this task using numpy functions. My goal is to take a 2D array and average J columns at a time, producing a new array with the same number of rows as the original, but with columns/J columns.
So I want to take this:
J = 2 // two columns averaged at a time
[[1 2 3 4]
[4 3 7 1]
[6 2 3 4]
[3 4 4 1]]
and produce this:
[[1.5 3.5]
[3.5 4.0]
[4.0 3.5]
[3.5 2.5]]
Is there a simple way to accomplish this task? I also need a way such that if I never end up with an unaveraged remainder column. So if, for example, I have an input array with 5 columns and J=2, I would average the first two columns, then the last three columns.
Any help you can provide would be great.
data.reshape(-1,j).mean(axis=1).reshape(data.shape[0],-1)
If your j divides data.shape[1], that is.
Example:
In [40]: data
Out[40]:
array([[7, 9, 7, 2],
[7, 6, 1, 5],
[8, 1, 0, 7],
[8, 3, 3, 2]])
In [41]: data.reshape(-1,j).mean(axis=1).reshape(data.shape[0],-1)
Out[41]:
array([[ 8. , 4.5],
[ 6.5, 3. ],
[ 4.5, 3.5],
[ 5.5, 2.5]])
First of all, it looks to me like you're not averaging the columns at all, you're just averaging two data points at a time. Seems to me like your best off reshaping the array, so your that you have a Nx2 data structure that you can feed directly to mean. You may have to pad it first if the number of columns isn't quite compatible. Then at the end, just do a weighted average of the padded remainder column and the one before it. Finally reshape back to the shape you want.
To play off of the example provided by TheodrosZelleke:
In [1]: data = np.concatenate((data, np.array([[5, 6, 7, 8]]).T), 1)
In [2]: data
Out[2]:
array([[7, 9, 7, 2, 5],
[7, 6, 1, 5, 6],
[8, 1, 0, 7, 7],
[8, 3, 3, 2, 8]])
In [3]: cols = data.shape[1]
In [4]: j = 2
In [5]: dataPadded = np.concatenate((data, np.zeros((data.shape[0], j - cols % j))), 1)
In [6]: dataPadded
Out[6]:
array([[ 7., 9., 7., 2., 5., 0.],
[ 7., 6., 1., 5., 6., 0.],
[ 8., 1., 0., 7., 7., 0.],
[ 8., 3., 3., 2., 8., 0.]])
In [7]: dataAvg = dataPadded.reshape((-1,j)).mean(axis=1).reshape((data.shape[0], -1))
In [8]: dataAvg
Out[8]:
array([[ 8. , 4.5, 2.5],
[ 6.5, 3. , 3. ],
[ 4.5, 3.5, 3.5],
[ 5.5, 2.5, 4. ]])
In [9]: if cols % j:
dataAvg[:, -2] = (dataAvg[:, -2] * j + dataAvg[:, -1] * (cols % j)) / (j + cols % j)
dataAvg = dataAvg[:, :-1]
....:
In [10]: dataAvg
Out[10]:
array([[ 8. , 3.83333333],
[ 6.5 , 3. ],
[ 4.5 , 3.5 ],
[ 5.5 , 3. ]])