I have a tensor matrix that i simply want to append a tensor vector as another column to it.
For example:
X = torch.randint(100, (100,5))
x1 = torch.from_numpy(np.array(range(0, 100)))
I've tried torch.cat([x1, X) with various numbers for both axis and dim but it always says that the dimensions don't match.
You can also use torch.hstack to combine and unsqueeze for reshape x1
torch.hstack([X, x1.unsqueeze(1)])
Shape of X is [100, 5], while the shape of X1 is 100. For concatenation torch requires similar shape on all the axis apart from the one in which we are trying to concatenate.
so, you will first need to
X1 = X1[:, None] # change the shape from 100 to [100, 1]
Xc = torch.cat([X, X1], axis=-1) /# tells the torch that we need to concatenate over the last dimension
Xc.shape should be [100, 6]
Combining the two answers into a pytorch 1.6 compatible version:
torch.cat((X, x1.unsqueeze(1)), dim = 1)
Related
I have two tensors in PyTorch, z is a 3d tensor of shape (n_samples, n_features, n_views) in which n_samples is the number of samples in the dataset, n_features is the number of features for each sample, and n_views is the number of different views that describe the same (n_samples, n_features) feature matrix, but with other values.
I have another 2d tensor b, of shape (n_samples, n_views), which purpose is to rescale all the features of the samples across the different views. In other words, it encapsulates the importance of the features of each view for the same sample.
For example:
import torch
z = torch.Tensor(([[2,3], [1,1], [4,5]],
[[2,2], [1,2], [7,7]],
[[2,3], [1,1], [4,5]],
[[2,3], [1,1], [4,5]]))
b = torch.Tensor(([1, 0],
[0, 1],
[0.2, 0.8],
[0.5, 0.5]))
print(z.shape, b.shape)
>>>torch.Size([4, 3, 2]) torch.Size([4, 2])
I want to obtain a third tensor r of shape (n_samples, n_features) as a result of operations between z and b.
One possible solution is:
b = b.unsqueeze(1)
r = z * b
r = torch.sum(r, dim=-1)
print(r, r.shape)
>>>tensor([[2.0000, 1.0000, 4.0000],
[2.0000, 2.0000, 7.0000],
[2.8000, 1.0000, 4.8000],
[2.5000, 1.0000, 4.5000]]) torch.Size([4, 3])
Is it possible to achieve that same result using torch.matmul()?. I've tried many times to permute the dimensions of the two vector, but to no avail.
Yes that's possible. If you have mutiple batch dimensions in both operatns, you can use the broadcasting. In this case the last two dimensions of each operand are interpreted as a matrix size. (I recommend looking it up in the documentation.)
So you need an additional dimension for your vectors b, to make them a n x 1 "matrix" (column vector):
# original implementation
b1 = b.unsqueeze(1)
r1 = z * b1
r1 = torch.sum(r1, dim=-1)
print(r1.shape)
# using torch.matmul
r2 = torch.matmul(z, b.unsqueeze(2))[...,0]
print(r2.shape)
print((r1-r2).abs().sum()) # should be zero if we do the same operation
Alternatively, torch.einsum also makes this very straightforward.
# using torch.einsum
r3 = torch.einsum('ijk,ik->ij', z, b)
print((r1-r3).abs().sum()) # should be zero if we do the same operation
einsum is a very powerful operation that can do a lot of things: you can permute tensor dimensions, sum along them, or perform scalar products, all with or without broadcasting. It is derived from the Einstein summation convention mostly used in physics. The rough idea is that you give every dimension of your operans a name, and then, using these names define what the output should look like. I think it is best to read the documentation. In our case we have a 4 x 3 x 2 tensor as well as a 4 x 2 tensor. So the let's call the dimensions of the first tensor ijk. Here i and k should be considered the same as the dimensions of the second tensor, so this one can be described as ik. Finally the output should have clearly be ij (it mus be a 4 x 3 tensor). From this "signature" ijk, ik -> ij it is clear that the dimension i is preserved, and the dimensions k must be "summe/multiplied" away (scalar product).
I have 2 tensors, namely X of shape (?, 32, 500) and indices of shape (?,). For both tensors, the 0th dimension is a batch dimension. Each element of indices specifies the index of X along the 1st dimension to select. In the end, I'd like to get a tensor of shape (?, 500). In numpy I would do it this way:
X[np.arange(len(X)), indices]
Does anyone know how to achieve the same in tensorflow (version 1)? I already looked at some examples of tf.gather and tf.gather_nd, but couldn't get my head around it. Thanks!
We can use tf.range, tf.stack and tf.gather_nd:
def fancy_index_arange(X, indices):
arange = tf.range(len(X))
fancy_index = tf.stack([arange, indices], axis=1)
result = tf.gather_nd(X, fancy_index)
return result
verify shape:
>>> X = tf.random.normal((10, 32, 500))
>>> indices = tf.random.uniform((10,), minval=0, maxval=32, dtype=tf.int32)
>>> fancy_index_arange(X, indices).shape
TensorShape([10, 500])
tested with tf.__version__ == "2.3.0"
A trick when you want to flatten a matrix X of shape (a,b,c,d) to a matrix X_flatten of shape (b ∗∗ c ∗∗ d, a) is to use:
X_flatten = X.reshape(X.shape[0], -1).T
I read this trick in coursera DL course, how does this work? Where did -1 come from and what does it mean?
X.shape[0] returns the first dimension of your original array:
X = np.random.rand(4, 4, 4, 4)
print(X.shape)
results in
(4, 4, 4, 4)
and therefore
X.shape[0]
returns
4
Using the reshape command, you can omit one of the target matrix dimensions by using -1 as a placeholder,
because one of the dimensions can be inferred by numpy.
I.e. by supplying the 4 from X.shape[0], numpy knows what the remaining first dimension must be for the array to contain all your values.
In the example
new_X = X.reshape(X.shape[0], -1).T
print(new_X.shape)
it is
(64, 4)
which would be equivalent to calling
new_X = X.reshape(X.shape[0], 64).T
print(new_X.shape)
The .T function just transposes the array resulting from the reshape command.
We can do the same in, Using the basic concepts of python
nested_list=[10,20,[30,40,[50]],[80,[10,[20]],90],60]
flat_list=[]
def unpack(list1):
for item in list1:
try:
len(item)
unpack(item)
except:
flat_list.append(item)
unpack(nested_list)
print (flat_list)
The problem I have using tensorflow is as follows:
For two tensors
[[x11,x12...],[x21,x22...],...[xn1,xn2...]]
and
[y1,y2,...yn],
I want to multiply them along axis 0 to get
[[x11*y1,x12*y1...],[x21*y2,x22*y2...]...]
For example, for
[[1,2],[3,4]] and [1,2], I want to get the result tensor [[1,2],[6,8]].
The real scenario is that I have two tensors A and B shaped (batches,height,width,n_channels) and (batches,1). Both are tensors defined in tensorflow. For each image of A in the batch I want to multiply it with the corresponding value in B.
Given a 2-dimensional tensor x and a vector y, you just need to do:
result = x * tf.expand_dims(y, axis=-1)
Or, if you like it more:
result = x * y[:, tf.newaxis]
Given indices with shape [batch_size, sequence_len], updates with shape [batch_size, sequence_len, sampled_size], to_shape with shape [batch_size, sequence_len, vocab_size], where vocab_size >> sampled_size, I'd like to use tf.scatter to map the updates to a huge tensor with to_shape, such that to_shape[bs, indices[bs, sz]] = updates[bs, sz]. That is, I'd like to map the updates to to_shape row by row. Please note that sequence_len and sampled_size are scalar tensors, while others are fixed. I tried to do the following:
new_tensor = tf.scatter_nd(tf.expand_dims(indices, axis=2), updates, to_shape)
But I got an error:
ValueError: The inner 2 dimension of output.shape=[?,?,?] must match the inner 1 dimension of updates.shape=[80,50,?]: Shapes must be equal rank, but are 2 and 1 for .... with input shapes: [80, 50, 1], [80, 50,?], [3]
Could you please tell me how to use scatter_nd properly? Thanks in advance!
So assuming you have:
A tensor updates with shape [batch_size, sequence_len, sampled_size].
A tensor indices with shape [batch_size, sequence_len, sampled_size].
Then you do:
import tensorflow as tf
# Create updates and indices...
# Create additional indices
i1, i2 = tf.meshgrid(tf.range(batch_size),
tf.range(sequence_len), indexing="ij")
i1 = tf.tile(i1[:, :, tf.newaxis], [1, 1, sampled_size])
i2 = tf.tile(i2[:, :, tf.newaxis], [1, 1, sampled_size])
# Create final indices
idx = tf.stack([i1, i2, indices], axis=-1)
# Output shape
to_shape = [batch_size, sequence_len, vocab_size]
# Get scattered tensor
output = tf.scatter_nd(idx, updates, to_shape)
tf.scatter_nd takes an indices tensor, an updates tensor and some shape. updates is the original tensor, and the shape is just the desired output shape, so [batch_size, sequence_len, vocab_size]. Now, indices is more complicated. Since your output has 3 dimensions (rank 3), for each of the elements in updates you need 3 indices to determine where in the output each element is going to be placed. So the shape of the indices parameter should be the same as updates with an additional dimension of size 3. In this case, we want the first to dimensions to be the same, but we still have to specify the 3 indices. So we use tf.meshgrid to generate the indices that we need and we tile them along the third dimension (the first and second index for each element vector in the last dimension of updates is the same). Finally, we stack these indices with the previously created mapping indices and we have our full 3-dimensional indices.
I think you might be looking for this.
def permute_batched_tensor(batched_x, batched_perm_ids):
indices = tf.tile(tf.expand_dims(batched_perm_ids, 2), [1,1,batched_x.shape[2]])
# Create additional indices
i1, i2 = tf.meshgrid(tf.range(batched_x.shape[0]),
tf.range(batched_x.shape[2]), indexing="ij")
i1 = tf.tile(i1[:, tf.newaxis, :], [1, batched_x.shape[1], 1])
i2 = tf.tile(i2[:, tf.newaxis, :], [1, batched_x.shape[1], 1])
# Create final indices
idx = tf.stack([i1, indices, i2], axis=-1)
temp = tf.scatter_nd(idx, batched_x, batched_x.shape)
return temp