We Keep Coding

How to operate torch.dot() in matrix consist of vectors in pytorch?

I have tensor like this:
arr1 = np.array([[ 1.6194, -0.6058, -0.8012], [ 1.1483, 1.6538, -0.8062]])
arr2 = np.array([[-0.3180, -1.8249, 0.0499], [-0.4184, 0.6495, -0.4911]])
X = torch.Tensor(arr1)
Y = torch.Tensor(arr2)
I want to do torch.dot on every tensor 1D (2 vectors) inside my 2D tensor
torch.dot(X, Y)
I want to get the result like this tensor([dotResult1, dotResult2]).
But I got the error like this:
RuntimeError: 1D tensors expected, but got 2D and 2D tensors
My main purpose is to do "something" operation on every vector inside my matrix but I don't want to use looping here, does anyone know how to do that?

Assuming what you are looking for is the tensor : [torch.dot(X[0], Y[0]), torch.dot(X[1], Y[1])]
Then you can do:
(X*Y).sum(axis = 1)
Test:
(X*Y).sum(axis = 1) == torch.tensor([torch.dot(X[0], Y[0]),torch.dot(X[1], Y[1])])
outputs:
tensor([True, True])

How to do argmax in group in pytorch?

Is there any ways to implement maxpooling according to norm of sub vectors in a group in Pytorch? Specifically, this is what I want to implement:
Input:
x: a 2-D float tensor, shape #Nodes * dim
cluster: a 1-D long tensor, shape #Nodes
Output:
y, a 2-D float tensor, and:
y[i]=x[k] where k=argmax_{cluster[k]=i}(torch.norm(x[k],p=2)).
I tried torch.scatter with reduce="max", but this only works for dim=1 and x[i]>0.
Can someone help me to solve the problem?

I don't think there's any built-in function to do what you want. Basically this would be some form of scatter_reduce on the norm of x, but instead of selecting the max norm you want to select the row corresponding to the max norm.
A straightforward implementation may look something like this
"""
input
x: float tensor of size [NODES, DIMS]
cluster: long tensor of size [NODES]
output
float tensor of size [cluster.max()+1, DIMS]
"""
num_clusters = cluster.max().item() + 1
y = torch.zeros((num_clusters, DIMS), dtype=x.dtype, device=x.device)
for cluster_id in torch.unique(cluster):
x_cluster = x[cluster == cluster_id]
y[cluster_id] = x_cluster[torch.argmax(torch.norm(x_cluster, dim=1), dim=0)]
Which should work just fine if clusters.max() is relatively small. If there are many clusters though then this approach has to unnecessarily create masks over cluster for every unique cluster id. To avoid this you can make use of argsort. The best I could come up with in pure python was the following.
num_clusters = cluster.max().item() + 1
x_norm = torch.norm(x, dim=1)
cluster_sortidx = torch.argsort(cluster)
cluster_ids, cluster_counts = torch.unique_consecutive(cluster[cluster_sortidx], return_counts=True)
end_indices = torch.cumsum(cluster_counts, dim=0).cpu().tolist()
start_indices = [0] + end_indices[:-1]
y = torch.zeros((num_clusters, DIMS), dtype=x.dtype, device=x.device)
for cluster_id, a, b in zip(cluster_ids, start_indices, end_indices):
indices = cluster_sortidx[a:b]
y[cluster_id] = x[indices[torch.argmax(x_norm[indices], dim=0)]]
For example in random tests with NODES = 60000, DIMS = 512, cluster.max()=6000 the first version takes about 620ms whie the second version takes about 78ms.

Multiply a 3d tensor with a 2d matrix using torch.matmul

I have two tensors in PyTorch, z is a 3d tensor of shape (n_samples, n_features, n_views) in which n_samples is the number of samples in the dataset, n_features is the number of features for each sample, and n_views is the number of different views that describe the same (n_samples, n_features) feature matrix, but with other values.
I have another 2d tensor b, of shape (n_samples, n_views), which purpose is to rescale all the features of the samples across the different views. In other words, it encapsulates the importance of the features of each view for the same sample.
For example:
import torch
z = torch.Tensor(([[2,3], [1,1], [4,5]],
[[2,2], [1,2], [7,7]],
[[2,3], [1,1], [4,5]],
[[2,3], [1,1], [4,5]]))
b = torch.Tensor(([1, 0],
[0, 1],
[0.2, 0.8],
[0.5, 0.5]))
print(z.shape, b.shape)
>>>torch.Size([4, 3, 2]) torch.Size([4, 2])
I want to obtain a third tensor r of shape (n_samples, n_features) as a result of operations between z and b.
One possible solution is:
b = b.unsqueeze(1)
r = z * b
r = torch.sum(r, dim=-1)
print(r, r.shape)
>>>tensor([[2.0000, 1.0000, 4.0000],
[2.0000, 2.0000, 7.0000],
[2.8000, 1.0000, 4.8000],
[2.5000, 1.0000, 4.5000]]) torch.Size([4, 3])
Is it possible to achieve that same result using torch.matmul()?. I've tried many times to permute the dimensions of the two vector, but to no avail.

Yes that's possible. If you have mutiple batch dimensions in both operatns, you can use the broadcasting. In this case the last two dimensions of each operand are interpreted as a matrix size. (I recommend looking it up in the documentation.)
So you need an additional dimension for your vectors b, to make them a n x 1 "matrix" (column vector):
# original implementation
b1 = b.unsqueeze(1)
r1 = z * b1
r1 = torch.sum(r1, dim=-1)
print(r1.shape)
# using torch.matmul
r2 = torch.matmul(z, b.unsqueeze(2))[...,0]
print(r2.shape)
print((r1-r2).abs().sum()) # should be zero if we do the same operation
Alternatively, torch.einsum also makes this very straightforward.
# using torch.einsum
r3 = torch.einsum('ijk,ik->ij', z, b)
print((r1-r3).abs().sum()) # should be zero if we do the same operation
einsum is a very powerful operation that can do a lot of things: you can permute tensor dimensions, sum along them, or perform scalar products, all with or without broadcasting. It is derived from the Einstein summation convention mostly used in physics. The rough idea is that you give every dimension of your operans a name, and then, using these names define what the output should look like. I think it is best to read the documentation. In our case we have a 4 x 3 x 2 tensor as well as a 4 x 2 tensor. So the let's call the dimensions of the first tensor ijk. Here i and k should be considered the same as the dimensions of the second tensor, so this one can be described as ik. Finally the output should have clearly be ij (it mus be a 4 x 3 tensor). From this "signature" ijk, ik -> ij it is clear that the dimension i is preserved, and the dimensions k must be "summe/multiplied" away (scalar product).

Append a tensor vector to tensor matrix

I have a tensor matrix that i simply want to append a tensor vector as another column to it.
For example:
X = torch.randint(100, (100,5))
x1 = torch.from_numpy(np.array(range(0, 100)))
I've tried torch.cat([x1, X) with various numbers for both axis and dim but it always says that the dimensions don't match.

You can also use torch.hstack to combine and unsqueeze for reshape x1
torch.hstack([X, x1.unsqueeze(1)])

Shape of X is [100, 5], while the shape of X1 is 100. For concatenation torch requires similar shape on all the axis apart from the one in which we are trying to concatenate.
so, you will first need to
X1 = X1[:, None] # change the shape from 100 to [100, 1]
Xc = torch.cat([X, X1], axis=-1) /# tells the torch that we need to concatenate over the last dimension
Xc.shape should be [100, 6]

Combining the two answers into a pytorch 1.6 compatible version:
torch.cat((X, x1.unsqueeze(1)), dim = 1)

TensorFlow, how to index so that (batch_size x num_labels)+(batch_size) -> (batch_size)

Suppose I just have got a matrix (2D tensor) X, whose shape is (batch_size x num_labels). And the scores of labels for each sample are stored in the matrix. Now I want to extract the true labels' scores, while the true labels are stored in another 1D tensor y, whose shape is (batch_size).
What can I do ?
I know that in Theano or Numpy. It can be done with a single expression: X[y].
BUT in TensorFlow, what is the most convenient or cost-less way to achieve that ?
X = tf.get_variable("X",[batch_size,num_labels])
y = tf.placeholder(tf.int32,[batch_size])
Note 0 <= y[i] <= num_labels - 1. The output z should be 1D tensor where z[i]= X[i][y[i]]

I understand that X is a vector containing probabilities for each class and batch instance and that you want to get the probability of the true label. I propose on solution, though it may not be the optimal one:
# Create mask for values
increasing = tf.range(start=0, limit=tf.shape(X)[0], delta=1)
# Concatenate batch index and true label
# Note that in Tensorflow < 1.0.0 you must call tf.pack
mask = tf.stack([increasing, y], axis=1)
# Extract values
masked = tf.gather_nd(params=X, indices=mask)
Hope it helps.