I have a 2D matrix M of shape [batch x dim], I have a vector V of shape [batch]. How can I multiply each of the columns in the matrix by the corresponding element in the V? That is:
I know an inefficient numpy implementation would look like this:
import numpy as np
M = np.random.uniform(size=(4, 10))
V = np.random.randint(4)
def tst(M, V):
rows = []
for i in range(len(M)):
col = []
for j in range(len(M[i])):
col.append(M[i][j] * V[i])
rows.append(col)
return np.array(rows)
In tensorflow, given two tensors, what is the most efficient way to achieve this?
import tensorflow as tf
sess = tf.InteractiveSession()
M = tf.constant(np.random.normal(size=(4,10)), dtype=tf.float32)
V = tf.constant([1,2,3,4], dtype=tf.float32)
In NumPy, we would need to make V 2D and then let broadcasting do the element-wise multiplication (i.e. Hadamard product). I am guessing, it should be the same on tensorflow. So, for expanding dims on tensorflow, we can use tf.newaxis (on newer versions) or tf.expand_dims or a reshape with tf.reshape -
tf.multiply(M, V[:,tf.newaxis])
tf.multiply(M, tf.expand_dims(V,1))
tf.multiply(M, tf.reshape(V, (-1, 1)))
In addition to #Divakar's answer, I would like to make a note that the order of M and V don't matter. It seems that tf.multiply also does broadcasting during multiplication.
Example:
In [55]: M.eval()
Out[55]:
array([[1, 2, 3, 4],
[2, 3, 4, 5],
[3, 4, 5, 6]], dtype=int32)
In [56]: V.eval()
Out[56]: array([10, 20, 30], dtype=int32)
In [57]: tf.multiply(M, V[:,tf.newaxis]).eval()
Out[57]:
array([[ 10, 20, 30, 40],
[ 40, 60, 80, 100],
[ 90, 120, 150, 180]], dtype=int32)
In [58]: tf.multiply(V[:, tf.newaxis], M).eval()
Out[58]:
array([[ 10, 20, 30, 40],
[ 40, 60, 80, 100],
[ 90, 120, 150, 180]], dtype=int32)
How do I sum over the columns of a tensor?
torch.Size([10, 100]) ---> torch.Size([10])
The simplest and best solution is to use torch.sum().
To sum all elements of a tensor:
torch.sum(x) # gives back a scalar
To sum over all rows (i.e. for each column):
torch.sum(x, dim=0) # size = [ncol]
To sum over all columns (i.e. for each row):
torch.sum(x, dim=1) # size = [nrow]
It should be noted that the dimension summed over is eliminated from the resulting tensor.
Alternatively, you can use tensor.sum(axis) where axis indicates 0 and 1 for summing over rows and columns respectively, for a 2D tensor.
In [210]: X
Out[210]:
tensor([[ 1, -3, 0, 10],
[ 9, 3, 2, 10],
[ 0, 3, -12, 32]])
In [211]: X.sum(1)
Out[211]: tensor([ 8, 24, 23])
In [212]: X.sum(0)
Out[212]: tensor([ 10, 3, -10, 52])
As, we can see from the above outputs, in both cases, the output is a 1D tensor. If you, on the other hand, wish to retain the dimension of the original tensor in the output as well, then you've set the boolean kwarg keepdim to True as in:
In [217]: X.sum(0, keepdim=True)
Out[217]: tensor([[ 10, 3, -10, 52]])
In [218]: X.sum(1, keepdim=True)
Out[218]:
tensor([[ 8],
[24],
[23]])
If you have tensor my_tensor, and you wish to sum across the second array dimension (that is, the one with index 1, which is the column-dimension, if the tensor is 2-dimensional, as yours is), use torch.sum(my_tensor,1) or equivalently my_tensor.sum(1) see documentation here.
One thing that is not mentioned explicitly in the documentation is: you can sum across the last array-dimension by using -1 (or the second-to last dimension, with -2, etc.)
So, in your example, you could use: outputs.sum(1) or torch.sum(outputs,1), or, equivalently, outputs.sum(-1) or torch.sum(outputs,-1). All of these would give the same result, an output tensor of size torch.Size([10]), with each entry being the sum over the all rows in a given column of the tensor outputs.
To illustrate with a 3-dimensional tensor:
In [1]: my_tensor = torch.arange(24).view(2, 3, 4)
Out[1]:
tensor([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
In [2]: my_tensor.sum(2)
Out[2]:
tensor([[ 6, 22, 38],
[54, 70, 86]])
In [3]: my_tensor.sum(-1)
Out[3]:
tensor([[ 6, 22, 38],
[54, 70, 86]])
Based on doc https://pytorch.org/docs/stable/generated/torch.sum.html
it should be
dim (int or tuple of python:ints) – the dimension or dimensions to reduce.
dim=0 means reduce row dimensions: condense all rows = sum by col
dim=1 means reduce col dimensions: condense cols= sum by row
Torch sum along multiple axis or dimensions
Just for the sake of completeness (I could not find it easily) I include how to sum along multiple dimensions with torch.sum which is heavily used in computer vision tasks where you have to reduce along H and W dimensions.
If you have an image x with shape C x H x W and want to compute the average pixel intensity value per channel you could do:
avg = torch.sum(x, dim=(1,2)) / (H*W) # Sum along (H,W) and norm
Suppose I have some Scipy sparse csr matrix, A, and a set of indices, inds=[i_1,...,i_n]. I can access the submatrix given by the rows and columns given in inds via A[inds,:][:,inds]. But I cannot figure out how to modify them. All of the following fail (i.e. do not change the matrix values):
A[inds,:][:,inds] *= 5.0
(A[inds,:][:,inds]).multiply(5.0)
A[inds,:][:,inds] = 5.0
Is there any easy way to modify a submatrix of a sparse matrix?
The rules for accessing a block, or submatrix, in sparse are the same [edit: similar] as for numpy. The 2 index arrays need to be broadcastable. The simplest way is to make the 1st one a column vector.
I'll illustrate:
In [13]: A = np.arange(24).reshape(4,6)
In [14]: M=sparse.csr_matrix(A)
In [15]: A[[[1],[2]],[1,2,3]]
Out[15]:
array([[ 7, 8, 9],
[13, 14, 15]])
In [16]: M[[[1],[2]],[1,2,3]].A
Out[16]:
array([[ 7, 8, 9],
[13, 14, 15]], dtype=int32)
In [17]: idx1=np.array([1,2])[:,None]
In [18]: idx1
Out[18]:
array([[1],
[2]])
In [19]: idx2=np.array([1,2,3])
In [20]: M[idx1, idx2].A
Out[20]:
array([[ 7, 8, 9],
[13, 14, 15]], dtype=int32)
In [21]: M[idx1, idx2] *= 2
In [22]: M.A
Out[22]:
array([[ 0, 1, 2, 3, 4, 5],
[ 6, 14, 16, 18, 10, 11],
[12, 26, 28, 30, 16, 17],
[18, 19, 20, 21, 22, 23]], dtype=int32)
M[inds,:][:,inds] has the same problem in sparse as in numpy. With a list inds, M[inds,:] is a copy of the original, not a view. I've show that with reference to the data buffers in numpy. I'm not quite sure how to demonstate it with sparse.
Roughly, A[...][...] = ... translates to A.__getitem__(...).__setitem__(...,...). If A.__getitem__(...) is a copy, then modifying it won't modify A itself.
Actually sparse matrices don't have distinction between views and copies. Most, if not all, indexing produces a copy. M[:2,:] is a copy, even though A[:2,:] is a view.
I should also add that changing the values of a sparse matrix is something you should do with caution. In place multiplications (*=) is ok.
In place addition is not supported:
In [31]: M[idx1, idx2] += 2
...
NotImplementedError:
Modification of values may produce an EfficiencyWarning - if it turns a 0 value to nonzero:
In [33]: M[:2, :2] = 3
/usr/lib/python3/dist-packages/scipy/sparse/compressed.py:690: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
SparseEfficiencyWarning)
The np.ix_ answer to your previous question works here as well.
Python - list of same columns / rows from matrix
M[np.ix_([1,2],[1,2,3])].A
I have a large NumPy.array field_array and a smaller array match_array, both consisting of int values. Using the following example, how can I check if any match_array-shaped segment of field_array contains values that exactly correspond to the ones in match_array?
import numpy
raw_field = ( 24, 25, 26, 27, 28, 29, 30, 31, 23, \
33, 34, 35, 36, 37, 38, 39, 40, 32, \
-39, -38, -37, -36, -35, -34, -33, -32, -40, \
-30, -29, -28, -27, -26, -25, -24, -23, -31, \
-21, -20, -19, -18, -17, -16, -15, -14, -22, \
-12, -11, -10, -9, -8, -7, -6, -5, -13, \
-3, -2, -1, 0, 1, 2, 3, 4, -4, \
6, 7, 8, 4, 5, 6, 7, 13, 5, \
15, 16, 17, 8, 9, 10, 11, 22, 14)
field_array = numpy.array(raw_field, int).reshape(9,9)
match_array = numpy.arange(12).reshape(3,4)
These examples ought to return True since the pattern described by match_array aligns over [6:9,3:7].
Approach #1
This approach derives from a solution to Implement Matlab's im2col 'sliding' in python that was designed to rearrange sliding blocks from a 2D array into columns. Thus, to solve our case here, those sliding blocks from field_array could be stacked as columns and compared against column vector version of match_array.
Here's a formal definition of the function for the rearrangement/stacking -
def im2col(A,BLKSZ):
# Parameters
M,N = A.shape
col_extent = N - BLKSZ[1] + 1
row_extent = M - BLKSZ[0] + 1
# Get Starting block indices
start_idx = np.arange(BLKSZ[0])[:,None]*N + np.arange(BLKSZ[1])
# Get offsetted indices across the height and width of input array
offset_idx = np.arange(row_extent)[:,None]*N + np.arange(col_extent)
# Get all actual indices & index into input array for final output
return np.take (A,start_idx.ravel()[:,None] + offset_idx.ravel())
To solve our case, here's the implementation based on im2col -
# Get sliding blocks of shape same as match_array from field_array into columns
# Then, compare them with a column vector version of match array.
col_match = im2col(field_array,match_array.shape) == match_array.ravel()[:,None]
# Shape of output array that has field_array compared against a sliding match_array
out_shape = np.asarray(field_array.shape) - np.asarray(match_array.shape) + 1
# Now, see if all elements in a column are ONES and reshape to out_shape.
# Finally, find the position of TRUE indices
R,C = np.where(col_match.all(0).reshape(out_shape))
The output for the given sample in the question would be -
In [151]: R,C
Out[151]: (array([6]), array([3]))
Approach #2
Given that opencv already has template matching function that does square of differences, you can employ that and look for zero differences, which would be your matching positions. So, if you have access to cv2 (opencv module), the implementation would look something like this -
import cv2
from cv2 import matchTemplate as cv2m
M = cv2m(field_array.astype('uint8'),match_array.astype('uint8'),cv2.TM_SQDIFF)
R,C = np.where(M==0)
giving us -
In [204]: R,C
Out[204]: (array([6]), array([3]))
Benchmarking
This section compares runtimes for all the approaches suggested to solve the question. The credit for the various methods listed in this section goes to their contributors.
Method definitions -
def seek_array(search_in, search_for, return_coords = False):
si_x, si_y = search_in.shape
sf_x, sf_y = search_for.shape
for y in xrange(si_y-sf_y+1):
for x in xrange(si_x-sf_x+1):
if numpy.array_equal(search_for, search_in[x:x+sf_x, y:y+sf_y]):
return (x,y) if return_coords else True
return None if return_coords else False
def skimage_based(field_array,match_array):
windows = view_as_windows(field_array, match_array.shape)
return (windows == match_array).all(axis=(2,3)).nonzero()
def im2col_based(field_array,match_array):
col_match = im2col(field_array,match_array.shape)==match_array.ravel()[:,None]
out_shape = np.asarray(field_array.shape) - np.asarray(match_array.shape) + 1
return np.where(col_match.all(0).reshape(out_shape))
def cv2_based(field_array,match_array):
M = cv2m(field_array.astype('uint8'),match_array.astype('uint8'),cv2.TM_SQDIFF)
return np.where(M==0)
Runtime tests -
Case # 1 (Sample data from question):
In [11]: field_array
Out[11]:
array([[ 24, 25, 26, 27, 28, 29, 30, 31, 23],
[ 33, 34, 35, 36, 37, 38, 39, 40, 32],
[-39, -38, -37, -36, -35, -34, -33, -32, -40],
[-30, -29, -28, -27, -26, -25, -24, -23, -31],
[-21, -20, -19, -18, -17, -16, -15, -14, -22],
[-12, -11, -10, -9, -8, -7, -6, -5, -13],
[ -3, -2, -1, 0, 1, 2, 3, 4, -4],
[ 6, 7, 8, 4, 5, 6, 7, 13, 5],
[ 15, 16, 17, 8, 9, 10, 11, 22, 14]])
In [12]: match_array
Out[12]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
In [13]: %timeit seek_array(field_array, match_array, return_coords = False)
1000 loops, best of 3: 465 µs per loop
In [14]: %timeit skimage_based(field_array,match_array)
10000 loops, best of 3: 97.9 µs per loop
In [15]: %timeit im2col_based(field_array,match_array)
10000 loops, best of 3: 74.3 µs per loop
In [16]: %timeit cv2_based(field_array,match_array)
10000 loops, best of 3: 30 µs per loop
Case #2 (Bigger random data):
In [17]: field_array = np.random.randint(0,4,(256,256))
In [18]: match_array = field_array[100:116,100:116].copy()
In [19]: %timeit seek_array(field_array, match_array, return_coords = False)
1 loops, best of 3: 400 ms per loop
In [20]: %timeit skimage_based(field_array,match_array)
10 loops, best of 3: 54.3 ms per loop
In [21]: %timeit im2col_based(field_array,match_array)
10 loops, best of 3: 125 ms per loop
In [22]: %timeit cv2_based(field_array,match_array)
100 loops, best of 3: 4.08 ms per loop
There's no such search function built in to NumPy, but it is certainly possible to do in NumPy
As long as your arrays are not too massive*, you could use a rolling window approach:
from skimage.util import view_as_windows
windows = view_as_windows(field_array, match_array.shape)
The function view_as_windows is written purely in NumPy so if you don't have skimage you can always copy the code from here.
Then to see if the sub-array appears in the larger array, you can write:
>>> (windows == match_array).all(axis=(2,3)).any()
True
To find the indices of where the top-left corner of the sub-array matches, you can write:
>>> (windows == match_array).all(axis=(2,3)).nonzero()
(array([6]), array([3]))
This approach should also work for arrays of higher dimensions.
*although the array windows takes up no additional memory (only the strides and shape are changed to create a new view of the data), writing windows == match_array creates a boolean array of size (7, 6, 3, 4) which is 504 bytes of memory. If you're working with very large arrays, this approach might not be feasible.
One solution is to search the entire search_in array block-at-a-time (a 'block' being a search_for-shaped slice) until either a matching segment is found or the search_for array is exhausted. I can use it to get coordinates for the matching block, or just a bool result by sending True or False for the return_coords optional argument...
def seek_array(search_in, search_for, return_coords = False):
"""Searches for a contiguous instance of a 2d array `search_for` within a larger `search_in` 2d array.
If the optional argument return_coords is True, the xy coordinates of the zeroeth value of the first matching segment of search_in will be returned, or None if there is no matching segment.
If return_coords is False, a boolean will be returned.
* Both arrays must be sent as two-dimensional!"""
si_x, si_y = search_in.shape
sf_x, sf_y = search_for.shape
for y in xrange(si_y-sf_y+1):
for x in xrange(si_x-sf_x+1):
if numpy.array_equal(search_for, search_in[x:x+sf_x, y:y+sf_y]):
return (x,y) if return_coords else True # don't forget that coordinates are transposed when viewing NumPy arrays!
return None if return_coords else False
I wonder if NumPy doesn't already have a function that can do the same thing, though...
To add to the answers already posted, I'd like to add one that takes into account errors due to floating point precision in case that matrices come from, let's say, image processing for instance, where numbers are subject to floating point operations.
You can recurse the indexes of the larger matrix, searching for the smaller matrix. Then you can extract a submatrix of the larger matrix matching the size of the smaller matrix.
You have a match if the contents of both, the submatrix of 'large' and the 'small' matrix match.
The following example shows how to return the first indexes of the location in the large matrix found to match. It would be trivial to extend this function to return an array of locations found to match if that's the intent.
import numpy as np
def find_submatrix(a, b):
""" Searches the first instance at which 'b' is a submatrix of 'a', iterates
rows first. Returns the indexes of a at which 'b' was found, or None if
'b' is not contained within 'a'"""
a_rows=a.shape[0]
a_cols=a.shape[1]
b_rows=b.shape[0]
b_cols=b.shape[1]
row_diff = a_rows - b_rows
col_diff = a_cols - b_cols
for idx_row in np.arange(row_diff):
for idx_col in np.arange(col_diff):
row_indexes = [idx + idx_row for idx in np.arange(b_rows)]
col_indexes = [idx + idx_col for idx in np.arange(b_cols)]
submatrix_indexes = np.ix_(row_indexes, col_indexes)
a_submatrix = a[submatrix_indexes]
are_equal = np.allclose(a_submatrix, b) # allclose is used for floating point numbers, if they
# are close while comparing, they are considered equal.
# Useful if your matrices come from operations that produce
# floating point numbers.
# You might want to fine tune the parameters to allclose()
if (are_equal):
return[idx_col, idx_row]
return None
Using the function above you can run the following example:
large_mtx = np.array([[1, 2, 3, 7, 4, 2, 6],
[4, 5, 6, 2, 1, 3, 11],
[10, 4, 2, 1, 3, 7, 6],
[4, 2, 1, 3, 7, 6, -3],
[5, 6, 2, 1, 3, 11, -1],
[0, 0, -1, 5, 4, -1, 2],
[10, 4, 2, 1, 3, 7, 6],
[10, 4, 2, 1, 3, 7, 6]
])
# Example 1: An intersection at column 2 and row 1 of large_mtx
small_mtx_1 = np.array([[4, 2], [2,1]])
intersect = find_submatrix(large_mtx, small_mtx_1)
print "Example 1, intersection (col,row): " + str(intersect)
# Example 2: No intersection
small_mtx_2 = np.array([[-14, 2], [2,1]])
intersect = find_submatrix(large_mtx, small_mtx_2)
print "Example 2, intersection (col,row): " + str(intersect)
Which would print:
Example 1, intersection: [1, 2]
Example 2, intersection: None
Here's a solution using the as_strided() function from stride_tricks module
import numpy as np
from numpy.lib.stride_tricks import as_strided
# field_array (I modified it to have two matching arrays)
A = np.array([[ 24, 25, 26, 27, 28, 29, 30, 31, 23],
[ 33, 0, 1, 2, 3, 38, 39, 40, 32],
[-39, 4, 5, 6, 7, -34, -33, -32, -40],
[-30, 8, 9, 10, 11, -25, -24, -23, -31],
[-21, -20, -19, -18, -17, -16, -15, -14, -22],
[-12, -11, -10, -9, -8, -7, -6, -5, -13],
[ -3, -2, -1, 0, 1, 2, 3, 4, -4],
[ 6, 7, 8, 4, 5, 6, 7, 13, 5],
[ 15, 16, 17, 8, 9, 10, 11, 22, 14]])
# match_array
B = np.arange(12).reshape(3,4)
# Window view of A
A_w = as_strided(A, shape=(A.shape[0] - B.shape[0] + 1,
A.shape[1] - B.shape[1] + 1,
B.shape[0], B.shape[1]),
strides=2*A.strides).reshape(-1, B.shape[0], B.shape[1])
match = (A_w == B).all(axis=(1,2))
We can also find the indices of the first element of each matching block in A
where = np.where(match)[0]
ind_flat = where + (B.shape[1] - 1)*(np.floor(where/(A.shape[1] - B.shape[1] + 1)).astype(int))
ind = [tuple(row) for row in np.array(np.unravel_index(ind_flat, A.shape)).T]
Result
print(match.any())
True
print(ind)
[(1, 1), (6, 3)]