Say I have a tensor with shape (?, 5, 1, 20)
For each occurence of the last dimension I do some computation (getting the k max values) on the last dimension that produces a smaller tensor b. What do I do if want to replace the last dimension of my original tensor with b?
What (preferably pure tensorflow) path should I take?
You're doing some computation on last dimension...That is you want to go from (?, 5, 1, 20) -> (?, 5, 1, b) if I understood correctly?
What kind of computation?
You could reshape your tensor, do the computation (such as matrix multiplication) and reshape back.
a = tf.reshape(X, [-1, 20])
a = tf.matmul(a, X)
a = tf.reshape(a, [-1, b])
Or you could use tf.einsum() to achieve similar feat. For non-linear computation depends what you want to do.
EDIT:
Also you could hack it with Conv2D and using filter of size [1,1, 20, b]. Does the same thing and more efficiently
Related
I am processing symmetric second order tensors (of stress) using numpy. In order to transform the tensors I have to generate a fully populated tensor, do the transformation and then recover the symmetric tensor in the rotated frame.
My input is a 2D numpy array of symmetric tensors (nx6). The code below works, but I'm pretty sure there must be a more efficient and/or elegant way to manipulate the arrays but I can't seem to figure it out.
I anyone can anyone suggest an improvement I'd be very grateful? The sample input is just 2 symmetric tensors but in use this could be millions of tensors, hence the concernr with efficiency
Thanks,
Doug
# Sample symmetric input (S11, S22, S33, S12, S23, S13)
sym_tens_in=np.array([[0,9], [1,10], [2,11], [3,12], [4,13], [5,14]])
# Expand to full tensor
tens_full=np.array([[sym_tens_in[0], sym_tens_in[3], sym_tens_in[4]],
[sym_tens_in[3], sym_tens_in[1], sym_tens_in[5]],
[sym_tens_in[4], sym_tens_in[5], sym_tens_in[2]]])
# Transpose and reshape to n x 3 x 3
tens_full=np.transpose(tens_full, axes=(2, 0, 1))
# This where the work on the full tensor will go....
# Reshape for extraction of the symmetric tensor
tens_full=np.reshape(tens_full, (2,9))
# Create an array for the test ouput symmetric tensor
sym_tens_out=np.empty((2,6), dtype=np.int32)
# Extract the symmetric components
sym_tens_out[:,0]=tens_full[:,0]
sym_tens_out[:,1]=tens_full[:,4]
sym_tens_out[:,2]=tens_full[:,8]
sym_tens_out[:,3]=tens_full[:,2]
sym_tens_out[:,4]=tens_full[:,3]
sym_tens_out[:,5]=tens_full[:,5]
# Transpose....
sym_tens_out=np.transpose(sym_tens_out)
This won't be any faster, but it's more compact:
In [166]: idx=np.array([0,3,4,3,1,5,4,5,2]).reshape(3,3)
In [167]: sym_tens_in[idx].transpose(2,0,1)
Out[167]:
array([[[ 0, 3, 4],
[ 3, 1, 5],
[ 4, 5, 2]],
[[ 9, 12, 13],
[12, 10, 14],
[13, 14, 11]]])
The transpose could be done first:
sym_tens_in.T[:,idx]
Similarly the reverse mapping can be done with:
In [168]: idx1 = [0,4,8,1,2,5]
In [171]: tens_full.reshape(2,-1)[:,idx1]
Out[171]:
array([[ 0, 1, 2, 3, 4, 5],
[ 9, 10, 11, 12, 13, 14]])
with the optional transpose.
OK - Based on the answers provided here I found a really cool solution. Now, I have to say that in my original question I omitted the actual reason that I was trying to get the full tensor into nx3x3 form. Basically, I'm implementing a function to rotate 2nd order stress tensors which requires solution of σ′=R⋅σ⋅RT.
I was planning to use numpy.matmul for the matrix multiplication but to transform multiple stress tensors, matmul requires the 3x3 tensors to be in the last two indices of the nx3x3 matrix - hence the effort to get the data into nx3x3 from from the original 3x3xn form....
However, after I let go of numpy.matmul as my target solution and embraced numpy.einsum instead....... everything became much easier....
# Sample symmetric input (S11, S22, S33, S12, S23, S13)
sym_tens_in=np.array([[0,9], [1,10], [2,11], [3,12], [4,13], [5,14]])
idx=np.array([0,3,5,3,1,4,5,4,2]).reshape(3,3)
full=sym_tens_in[idx]
full_transformed=np.einsum('ij, jkn, lk->nil', rot_mat, full, rot_mat)
Thanks for the inspiration!!!!
In the following,
x_6 = torch.cat((x_1, x_2_1, x_3_1, x_5_1), dim=-3)
Sizes of tensors x_1, x_2_1, x_3_1, x_5_1 are
torch.Size([1, 256, 7, 7])
torch.Size([1, 256, 7, 7])
torch.Size([1, 256, 7, 7])
torch.Size([1, 256, 7, 7]) respectively.
The size of x_6 turns out to be torch.Size([1, 1024, 7, 7])
I couldn't understand & visualise this concatenation along a negative dimension(-3 in this case).
What exactly is happening here?
How does the same go if dim = 3?
Is there any constraint on dim for a given set of tensors?
The answer by danin is not completely correct, actually wrong when looked from the perspective of tensor algebra, since the answer indicates that the problem has to do with accessing or indexing a Python list. It isn't.
The -3 means that we concatenate the tensors along the 2nd dimension. (you could've very well used 1 instead of the confusing -3).
From taking a closer look at the tensor shapes, it seems that they represent (b, c, h, w) where b stands for batch_size, c stands for number of channels, h stands for height and w stands for width.
This is usually the case, somewhere at the final stages of encoding (possibly) images in a deep neural network and we arrive at these feature maps.
The torch.cat() operation with dim=-3 is meant to say that we concatenate these 4 tensors along the dimension of channels c (see above).
4 * 256 => 1024
Hence, the resultant tensor ends up with a shape torch.Size([1, 1024, 7, 7]).
Notes: It is hard to visualize a 4 dimensional space since we humans live in an inherently 3D world. Nevertheless, here are some answers that I wrote a while ago which will help to get some mental picture.
How to understand the term tensor in TensorFlow?
Very Basic Numpy array dimension visualization
Python provides negative indexing, so you can access elements starting from the end of the list e.g, -1 is the last element of a list.
In this case the tensor has 4 dimensions, so -3 is actually the 2nd element.
I have a tensor T of the shape (8, 5, 300), where 8 is the batch size, 5 is the number of documents in each batch, and 300 is the encoding of each of the document. If I reshape the Tensor as follows, does the properties of my Tensor remain the same?
T = T.reshape(5, 300, 8)
T.shape
>> Size[5, 300, 8]
So, does this new Tensor indicate the same properties as the original one? By the properties, I mean, can I say that this is also a Tensor of batch size 8, with 5 documents for each batch, and a 300 dimensional encoding for each document?
Does this affect the training of the model? If reshaping of Tensor messes up the datapoints, then there is no point in training. For example, If reshaping like above gives output as a batch of 5 samples, with 300 documents of size 8 each. If it happens so, then it's useless, since I do not have 300 documents, neither do I have batch of 5 samples.
I need to reshape it like this because my model in between produces output of the shape [8, 5, 300], and the next layer accepts input as [5, 300, 8].
NO
You need to understand the difference between reshape/view and permute.
reshape and view only changes the "shape" of the tensor, without re-ordering the elements. Therefore
orig = torch.rand((8, 5, 300))
resh = orig.reshape(5, 300, 8)
orig[0, 0, :] != resh[0, :, 0]
If you want to change the order of the elements as well, you need to permute it:
perm = orig.permute(1, 2, 0)
orig[0, 0, :] == perm[0, :, 0]
NOOO!
I made a similar mistake.
Imagine you converting 2-d Tensor( Matrix) into 1-D Tensor(Array) and applying transform functionality on it. This would create serious issues in code as your new tensor has characteristic of an array.
Hope you got my point.
I have two torch tensors a and b. Tensor a has the shape of [batch_size, emb_size] and Tensor b has the shape of [num_of_words, emb_size]. I want to do the element-wise product on these two tensors instead of dot product.
I noticed that "*" can perform element-wise product but it doesn't fit my case.
For example, batch_size = 3, emb_size = 2, num_of_words = 5.
a = torch.rand((3,2))
b = torch.rand((5,2))
I want to get something like:
torch.cat([a[0]*b, a[1]*b, a[2]*b]).view(3, 5, 2)
but I want to do this in an efficient and elegant way.
You can use
a.unsqueeze(1) * b
PyTorch supports broadcasting semantics but you need to make sure the singleton dimensions are in the correct locations.
I'm trying to efficiently replicate numpy's ndarray.choose() method.
Here's a numpy example of what I'm looking for:
b = np.arange(15).reshape(3, 5)
c = np.array([1,0,4])
c.choose(b.T) # trying to replicate in tensorflow
-> array([ 1, 5, 14])
The best I've been able to do with this is generate a batch_size square matrix (which is huge if batch size is huge) and take the diagonal of it:
tf_b = tf.constant(b)
tf_c = tf.constant(c)
sess.run(tf.diag_part(tf.gather(tf.transpose(tf_b), tf_c)))
-> array([ 1, 5, 14])
Is there a way to do this that is just linear in the first dimension (instead of squared)?
Yeah, there's an easier way to do this. Flatten your b array to 1-d, so it's [0, 1, 2, ..., 13, 14]. Take an array of indices that are in the range of the number of 'choices' you are taking (3 in your case). That will be [0, 1, 2]. Multiply this range by the second dimension of your original shape, which is the number of options for each choice (5 in your case). That gives you [0, 5, 10]. Then add your indices to this to obtain [1, 5, 14]. Now you're good to call tf.gather().
Here is some code that I've taken from here that does a similar thing for RNN outputs. Yours will be slightly different, but the idea is the same.
index = tf.range(0, batch_size) * max_length + (length - 1)
flat = tf.reshape(output, [-1, out_size])
relevant = tf.gather(flat, index)
return relevant
In a big picture, the operation is pretty straightforward. You use the range operation to get the index of the beginning of each row, then add the index of where you are in each row. I think doing it in 1D is easiest, so that's why we flatten it.