I'm trying to write an operation in tensorflow that propagates only the top k values of each feature map.
example:
k=1, input size is [batch_size, x, y, channels] let's say it's [1,2,2,3]
The output should be the same size and if k=1 than each x,y plane will have only one nonzero.
example in numpy:
input = [[[[6.4 1.4 1.3] [2.1 6.5 4.8]][[2.3 9.2 2.8][7.9 5.1 0.6]]]]]
output should be:
[[[[6.4 0. 0.] [0. 6.5 0.]] [[0. 9.2 0.] [7.9 0. 0.]]]]
in order to do this in tensorflow I would like to use nn.top_k followed by scatter_nd.
the problem is that top_k returns the indices of the requested elements very diffrently from how scatter_nd needs it.
top_k returns array of indices[[[[0],[1]], [[1],[0]]]] in shape (1,2,2,1)
scatter_nd needs it as a list of all coordinates per value like this:
[[0, 0, 0, 0], [0, 0, 1, 1], [0, 1, 0, 1], [0, 1, 1, 0]]
Does anyone know about a way to convert between them? or even maybe diffrent approch entirly for this operation?
tf.nn.top_k() only returns the top k values in the last dimension. So you have to add back all the other dimensions. Easiest with tf.where(). Code (tested):
import tensorflow as tf
inp = tf.constant( [ [ [ [6.4, 1.4, 1.3], [2.1, 6.5, 4.8] ], [ [2.3, 9.2, 2.8], [7.9, 5.1, 10.6] ] ] ] )
t, idx = tf.nn.top_k( inp, k = 2 )
idx_one_hot = tf.one_hot( idx, depth = 3 )
idx_red = tf.reduce_sum( idx_one_hot, axis = -2 )
idx2 = tf.where( tf.not_equal( idx_red, 0 ) )
with tf.Session() as sess:
print( sess.run( idx2 ) )
Outputs (note I've changed the last number in your example to have an index of 2 as well, only 0s and 1s looked a bit misleading as if it were a boolean tensor):
[[0 0 0 0]
[0 0 0 1]
[0 0 1 1]
[0 0 1 2]
[0 1 0 1]
[0 1 0 2]
[0 1 1 0]
[0 1 1 2]]
Please note, this loses the order of indices in the last dimension as reported by top_k, it changes it to increasing order in the indices themselves.
Related
I have a numpy array that looks like this:
[[0.67058825 0.43529415 0.33725491]
[0.01568628 0.30980393 0.96862751]
[0.24705884 0.63529414 0.29411766]
[0.27843139 0.63137257 0.37647063]
[0.26274511 0.627451 0.33333334]
[0.25098041 0.61960787 0.30980393]]
I want to add a 1 to every row like this:
[[0.67058825 0.43529415 0.33725491 1]
[0.01568628 0.30980393 0.96862751 1]
[0.24705884 0.63529414 0.29411766 1]
[0.27843139 0.63137257 0.37647063 1]
[0.26274511 0.627451 0.33333334 1]
[0.25098041 0.61960787 0.30980393 1]]
Simply with numpy.insert to insert the needed value into required position along the given axis:
arr = np.insert(arr, arr.shape[1], 1, axis=1)
[[0.67058825 0.43529415 0.33725491 1. ]
[0.01568628 0.30980393 0.96862751 1. ]
[0.24705884 0.63529414 0.29411766 1. ]
[0.27843139 0.63137257 0.37647063 1. ]
[0.26274511 0.627451 0.33333334 1. ]
[0.25098041 0.61960787 0.30980393 1. ]]
Assuming a the input, you can try:
out = np.c_[a, np.ones((a.shape[0], 1))]
Or:
out = np.hstack([a, np.ones((a.shape[0], 1))])
Output:
array([[0.67058825, 0.43529415, 0.33725491, 1. ],
[0.01568628, 0.30980393, 0.96862751, 1. ],
[0.24705884, 0.63529414, 0.29411766, 1. ],
[0.27843139, 0.63137257, 0.37647063, 1. ],
[0.26274511, 0.627451 , 0.33333334, 1. ],
[0.25098041, 0.61960787, 0.30980393, 1. ]])
You can do this:
arr = [ [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0] ]
to_add = [1 for i in range(0, len(arr)) #Making array with all 1s
x = np.column_stack((arr, to_add)) #Add columns together
print(x)
Returns:
[[0 0 0 0 1]
[0 0 0 0 1]
[0 0 0 0 1]]
I'm using numpy in python , in order to create a nx1 matrix . I want the 1st element of the matrix to be 3 , the 2nd -1 , then the n-1 element -1 again and at the end the n element 3. All the in between elements , i.e. from element 3 to element n-2 should be 0. I've made a drawing of the mentioned matrix , is like this :
I'm fairly new to python and using numpy but seems like a great tool for managing matrices. What I've tried so far is creating the nx1 array (giving n some value) and initializing it to 0 .
import numpy as np
n = 100
I = np.arange(n)
matrix = np.row_stack(0*I)
print("\Matrix is \n",matrix)
Any clues to how i proceed? Or what routine to use ?
Probably the simplest way is to just do the following:
import numpy as np
n = 10
a = np.zeros(n)
a[0] = 3
a[1] = -1
a[len(a)-1] = 3
a[len(a)-2] = -1
>>print(a)
output: [ 3. -1. 0. 0. 0. 0. 0. 0. -1. 3.]
Hope this helps ;)
In [97]: n=10
In [98]: arr = np.zeros(n,int)
In [99]: arr[[0,-1]]=3; arr[[1,-2]]=-1
In [100]: arr
Out[100]: array([ 3, -1, 0, 0, 0, 0, 0, 0, -1, 3])
Easily changed to (n,1):
In [101]: arr[:,None]
Out[101]:
array([[ 3],
[-1],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[-1],
[ 3]])
I guess something that works is :
import numpy as np
n = 100
I = np.arange(n)
matrix = np.row_stack(0*I)
matrix[0]=3
matrix[1]=-1
matrix[n-2]=-1
matrix[n-1]=3
print("\Matrix is \n",matrix)
so first of all here is my code I have questions about, I feel like the answer will be simple, but I am just too blind to see it
dic = {}
for i in range(0,9):
dic['rand_num{0}'.format(i)] = np.sort(random.sample(range(0,8),5))
mat = np.zeros([8,8])
for rand_num in dic.values():
print(rand_num)
for i in range(0,5):
matrixval = rand_num[i]
#print(matrixval)
for j in range(0,8):
mat[matrixval,j] = 1
print(mat)
I created 8 different lists which will help me determine, where to set a 1 in the matrix.
so eg. if my first list looks like this:
rand_List = [0,3,5,6,7]
the matrix of the first row or column should look like this
matrix = [1, 0, 0, 1, 0, 1, 1, 1]
My position for the matrix value is set in the second loop: for i in range(0,5): matrixval = rand_num[i] . However, as soon I run my code, my matrix just gives me ones instead of ones and zeros. I feel like I have an iteration problem in my last for loop, and I don't know how to fix it.
If anyone could help me, or just give me a hint, how to actually solve this hopefully small mistake of mine, I would really appreciate it.
Hopefully this provides the expected result:
dic = {}
for i in range(0,8):
dic['rand_num{0}'.format(i)] = np.sort(random.sample(range(0,8),5))
mat = np.zeros([8,8])
for j, rand_num in enumerate(dic.values()):
print(rand_num)
for i in range(0,5):
matrixval = rand_num[i]
mat[j,matrixval] = 1
print(mat)
Output:
[1 2 3 5 7]
[0 1 2 5 6]
[0 1 2 3 5]
[1 2 3 4 6]
[0 1 3 4 7]
[1 2 4 5 7]
[0 1 3 4 7]
[1 2 3 4 6]
[[0. 1. 1. 1. 0. 1. 0. 1.]
[1. 1. 1. 0. 0. 1. 1. 0.]
[1. 1. 1. 1. 0. 1. 0. 0.]
[0. 1. 1. 1. 1. 0. 1. 0.]
[1. 1. 0. 1. 1. 0. 0. 1.]
[0. 1. 1. 0. 1. 1. 0. 1.]
[1. 1. 0. 1. 1. 0. 0. 1.]
[0. 1. 1. 1. 1. 0. 1. 0.]]
The dic you are using is not needed. Here is a solution without it.
import random
lists = []
for i in range(0, 9):
lists.append(sorted(random.sample(range(0, 8), 5)))
mat = []
for lst in lists:
arr = [0] * 8
for index in lst:
arr[index] = 1
mat.append(arr)
print(mat)
Outputs:
[[0, 1, 1, 0, 0, 1, 1, 1],
[1, 0, 1, 1, 0, 0, 1, 1],
[1, 1, 1, 1, 0, 0, 0, 1],
[0, 1, 1, 1, 0, 1, 1, 0],
[1, 1, 0, 1, 1, 0, 1, 0],
[1, 0, 1, 1, 0, 1, 1, 0],
[0, 1, 1, 1, 1, 1, 0, 0],
[0, 1, 1, 0, 1, 1, 1, 0],
[1, 0, 1, 0, 1, 0, 1, 1]]
You can also use numpy.random.randint to generate your binary 2d array.
import numpy as np
arr = np.random.randint(2, size=(8, 8))
print(arr)
Outputs:
[[1 0 1 0 1 0 1 1]
[0 0 0 0 1 0 1 1]
[1 0 1 1 0 0 1 1]
[1 1 0 1 0 1 1 0]
[0 0 1 0 0 1 1 1]
[1 0 0 0 1 1 1 1]
[1 1 0 1 0 0 1 1]
[1 0 1 0 0 1 0 0]]
How can I divide two numpy matrices A and B in python when sometimes the two matrices will have 0 on the same cell?
Basically A[i,j]>=B[i,j] for all i, j. I need to calculate C=A/B. But sometimes A[i,j]==B[i,j]==0. And when this happens I need A[i,j]/B[i,j] to be defined as 0.
Is there a simple pythonic way other than going through all the indexes?
You can use the where argument for ufuncs like np.true_divide:
np.true_divide(A, B, where=(A!=0) | (B!=0))
In case you have no negative values (as stated in the comments) and A >= B for each element (as stated in the question) you can simplify this to:
np.true_divide(A, B, where=(A!=0))
because A[i, j] == 0 implies B[i, j] == 0.
For example:
import numpy as np
A = np.random.randint(0, 3, (4, 4))
B = np.random.randint(0, 3, (4, 4))
print(A)
print(B)
print(np.true_divide(A, B, where=(A!=0) | (B!=0)))
[[1 0 2 1]
[1 0 0 0]
[2 1 0 0]
[2 2 0 2]]
[[1 0 1 1]
[2 2 1 2]
[2 1 0 1]
[2 0 1 2]]
[[ 1. 0. 2. 1. ]
[ 0.5 0. 0. 0. ]
[ 1. 1. 0. 0. ]
[ 1. inf 0. 1. ]]
As alternative: Just replace nans after the division:
C = A / B # may print warnings, suppress them with np.seterrstate if you want
C[np.isnan(C)] = 0
You could use a mask with np.where to choose between such a case of A and B being both zeros and otherwise and put out 0 or an elementwise division respectively -
from __future__ import division # For Python 2.x
mask = (A == B) & (A==0)
C = np.where(mask, 0, A/B)
About the mask creation : (A==B) would be the mask of all elements that are equal between A and B and with (A==0) we have a mask of all elements that are zero in A. Thus, with a combined mask of (A == B) & (A==0), we have mask of places where both A and B are zeros. A more simpler version to do the same task and maybe easier to understand would be to check for zeros in both A and B and it would be :
mask = (A==0) & (B==0)
About the use of np.where, its syntax is :
C = np.where(mask, array1, array2)
i.e. we would select elements for assinging into C based on the mask. If the corresponding mask element is True, we pick the corresponding element from array1, else from array2. This is done on elementwise level and thus, we have the output C.
Sample run -
In [48]: A
Out[48]:
array([[4, 1, 4, 0, 3],
[0, 4, 1, 4, 3],
[1, 0, 0, 4, 0]])
In [49]: B
Out[49]:
array([[4, 2, 2, 1, 4],
[2, 1, 2, 4, 2],
[4, 0, 2, 0, 3]])
In [50]: mask = (A == B) & (A==0)
In [51]: np.where(mask, 0, A/B)
Out[51]:
array([[ 1. , 0.5 , 2. , 0. , 0.75],
[ 0. , 4. , 0.5 , 1. , 1.5 ],
[ 0.25, 0. , 0. , inf, 0. ]])
I have 2 arrays containing zeros & ones. I want to perform hstack() on them but not getting the desired output.
Python Code..
import numpy as np
zeros = np.zeros(8)
ones = np.ones(8)
zerosThenOnes = np.hstack((zeros, ones)) # A 1 by 16 array
Current Output..
[ 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 1. 1. 1. 1. 1.]
Expected Output..
[ 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 ]
I can't understand what silly mistake I'm doing.
You must tell numpy to return the values as integers
import numpy as np
zeros = np.zeros((8,), dtype=np.int)
ones = np.ones((8,), dtype=np.int)
zerosThenOnes = np.hstack((zeros, ones))
To print out zerosThenOnes like this [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1]
Use:
print([x for x in zerosThenOnes])
Numpy Zeros
np.hstack((np.zeros(8), np.ones(8))).astype(int)
for np.array output, or
map( int, np.hstack((np.zeros(8), np.ones(8))) )
for list output