I want to make a dynamic loss function in tensorflow. I want to calculate the energy of a signal's FFT, more specifically only a window of size 3 around the most dominant peak. I am unable to implement in TF, as it throws a lot of errors like Stride and InvalidArgumentError (see above for traceback): Expected begin, end, and strides to be 1D equal size tensors, but got shapes [1,64], [1,64], and [1] instead.
My code is this:
self.spec = tf.fft(self.signal)
self.spec_mag = tf.complex_abs(self.spec[:,1:33])
self.argm = tf.cast(tf.argmax(self.spec_mag, 1), dtype=tf.int32)
self.frac = tf.reduce_sum(self.spec_mag[self.argm-1:self.argm+2], 1)
Since I am computing batchwise of 64 and dimension of data as 64 too, the shape of self.signal is (64,64). I wish to calculate only the AC components of the FFT. As the signal is real valued, only half the spectrum would do the job. Hence, the shape of self.spec_mag is (64,32).
The max in this fft is located at self.argm which has a shape (64,1).
Now I want to calculate the energy of 3 elements around the max peak via: self.spec_mag[self.argm-1:self.argm+2].
However when I run the code and try to obtain the value of self.frac, I get thrown with multiple errors.
It seems like you were missing and index when accessing argm. Here is the fixed version of the 1, 64 version.
import tensorflow as tf
import numpy as np
x = np.random.rand(1, 64)
xt = tf.constant(value=x, dtype=tf.complex64)
signal = xt
print('signal', signal.shape)
print('signal', signal.eval())
spec = tf.fft(signal)
print('spec', spec.shape)
print('spec', spec.eval())
spec_mag = tf.abs(spec[:,1:33])
print('spec_mag', spec_mag.shape)
print('spec_mag', spec_mag.eval())
argm = tf.cast(tf.argmax(spec_mag, 1), dtype=tf.int32)
print('argm', argm.shape)
print('argm', argm.eval())
frac = tf.reduce_sum(spec_mag[0][(argm[0]-1):(argm[0]+2)], 0)
print('frac', frac.shape)
print('frac', frac.eval())
and here is the expanded version (batch, m, n)
import tensorflow as tf
import numpy as np
x = np.random.rand(1, 1, 64)
xt = tf.constant(value=x, dtype=tf.complex64)
signal = xt
print('signal', signal.shape)
print('signal', signal.eval())
spec = tf.fft(signal)
print('spec', spec.shape)
print('spec', spec.eval())
spec_mag = tf.abs(spec[:, :, 1:33])
print('spec_mag', spec_mag.shape)
print('spec_mag', spec_mag.eval())
argm = tf.cast(tf.argmax(spec_mag, 2), dtype=tf.int32)
print('argm', argm.shape)
print('argm', argm.eval())
frac = tf.reduce_sum(spec_mag[0][0][(argm[0][0]-1):(argm[0][0]+2)], 0)
print('frac', frac.shape)
print('frac', frac.eval())
you may want to fix function names since I edit this code at a newer version of tensorflow.
Tensorflow indexing uses tf.Tensor.getitem:
This operation extracts the specified region from the tensor. The notation is similar to NumPy with the restriction that currently only support basic indexing. That means that using a tensor as input is not currently allowed
So using tf.slice and tf.strided_slice is out of the question as well.
Whereas in tf.gather indices defines slices into the first dimension of Tensor, in tf.gather_nd, indices defines slices into the first N dimensions of the Tensor, where N = indices.shape[-1]
Since you wanted the 3 values around the max, I manually extract the first, second and third element using a list comprehension, followed be a tf.stack
import tensorflow as tf
signal = tf.placeholder(shape=(64, 64), dtype=tf.complex64)
spec = tf.fft(signal)
spec_mag = tf.abs(spec[:,1:33])
argm = tf.cast(tf.argmax(spec_mag, 1), dtype=tf.int32)
frac = tf.stack([tf.gather_nd(spec,tf.transpose(tf.stack(
[tf.range(64), argm+i]))) for i in [-1, 0, 1]])
frac = tf.reduce_sum(frac, 1)
This will fail for the corner case where argm is the first or last element in the row, but it should be easy to resolve.
Related
Is there any ways to implement maxpooling according to norm of sub vectors in a group in Pytorch? Specifically, this is what I want to implement:
Input:
x: a 2-D float tensor, shape #Nodes * dim
cluster: a 1-D long tensor, shape #Nodes
Output:
y, a 2-D float tensor, and:
y[i]=x[k] where k=argmax_{cluster[k]=i}(torch.norm(x[k],p=2)).
I tried torch.scatter with reduce="max", but this only works for dim=1 and x[i]>0.
Can someone help me to solve the problem?
I don't think there's any built-in function to do what you want. Basically this would be some form of scatter_reduce on the norm of x, but instead of selecting the max norm you want to select the row corresponding to the max norm.
A straightforward implementation may look something like this
"""
input
x: float tensor of size [NODES, DIMS]
cluster: long tensor of size [NODES]
output
float tensor of size [cluster.max()+1, DIMS]
"""
num_clusters = cluster.max().item() + 1
y = torch.zeros((num_clusters, DIMS), dtype=x.dtype, device=x.device)
for cluster_id in torch.unique(cluster):
x_cluster = x[cluster == cluster_id]
y[cluster_id] = x_cluster[torch.argmax(torch.norm(x_cluster, dim=1), dim=0)]
Which should work just fine if clusters.max() is relatively small. If there are many clusters though then this approach has to unnecessarily create masks over cluster for every unique cluster id. To avoid this you can make use of argsort. The best I could come up with in pure python was the following.
num_clusters = cluster.max().item() + 1
x_norm = torch.norm(x, dim=1)
cluster_sortidx = torch.argsort(cluster)
cluster_ids, cluster_counts = torch.unique_consecutive(cluster[cluster_sortidx], return_counts=True)
end_indices = torch.cumsum(cluster_counts, dim=0).cpu().tolist()
start_indices = [0] + end_indices[:-1]
y = torch.zeros((num_clusters, DIMS), dtype=x.dtype, device=x.device)
for cluster_id, a, b in zip(cluster_ids, start_indices, end_indices):
indices = cluster_sortidx[a:b]
y[cluster_id] = x[indices[torch.argmax(x_norm[indices], dim=0)]]
For example in random tests with NODES = 60000, DIMS = 512, cluster.max()=6000 the first version takes about 620ms whie the second version takes about 78ms.
Let's say I have the following tensors:
X = np.zeros((3,201, 340))
Y = np.zeros((340, 28))
Making a dot product of X, Y is successful with numpy, and yields a tensor of shape (3, 201, 28).
However with tensorflow I get the following error: Shape must be rank 2 but is rank 3 error ...
minimal code example:
X = np.zeros((3,201, 340))
Y = np.zeros((340, 28))
print(np.dot(X,Y).shape) # successful (3, 201, 28)
tf.matmul(X, Y) # errornous
Any idea how to achieve the same result with tensorflow?
Since, you are working with tensors, it would be better (for performance) to use tensordot there than np.dot. NumPy allows it (numpy.dot) to work on tensors through lowered performance and it seems tensorflow simply doesn't allow it.
So, for NumPy, we would use np.tensordot -
np.tensordot(X, Y, axes=((2,),(0,)))
For tensorflow, it would be with tf.tensordot -
tf.tensordot(X, Y, axes=((2,),(0,)))
Related post to understand tensordot.
Tensorflow doesn't allow for multiplication of matrices with different ranks as numpy does.
To cope with this, you can reshape the matrix. This essentially casts a matrix of,
say, rank 3 to one with rank 2 by "stacking the matrices" one on top of the other.
You can use this:
tf.reshape(tf.matmul(tf.reshape(Aijk,[i*j,k]),Bkl),[i,j,l])
where i, j and k are the dimensions of matrix one and k and l are the dimensions of matrix 2.
Taken from here.
What I am trying to do is have a weight matrix for my neural network which grows in size (i.e. a neuron is added to it each iteration). However, I do not want to use tf.Variable again as this will waste memory by copying the values in the previous matrix not expanding the matrix itself.
I have seen that people use tf.assign with validate_shape set to False, however, this does not change the shape of the variable correctly which I believed was a bug but the tensorflow GitHub did not seem to agree (I don't understand why from their reply).
Below is a simplified example of the problem. x is the matrix that I want to expand so that it can be added to z. If anyone knows a solution to what I am trying to achieve here I would be very grateful =)
import tensorflow as tf
import numpy as np
# Initialise some variables
sess = tf.Session()
x = tf.Variable(tf.truncated_normal([2, 4], stddev = 0.04))
z = tf.Variable(tf.truncated_normal([3, 4], stddev = 0.04))
sess.run(tf.variables_initializer([x, z]))
# Enlarge the matrix by assigning it a new set of values
sess.run(tf.assign(x, tf.concat((x, tf.cast(tf.truncated_normal([1, 4], stddev = 0.04), tf.float32)), 0), validate_shape=False))
# Print shapes of matrices, notice that x's actual shape is different for the
# shape tensorflow has recorded for it
print(x.get_shape())
print(x.eval(session=sess).shape)
print(z.get_shape())
print(z.eval(session=sess).shape)
# Add two matrices with equal shapes
print(tf.add(x, z).eval(session=sess))
Note: I realize that if I initialized z to the shape (2, 4) and then expanded it with tf.assign (as I do with x) the above example will work. But due to another constraint, I cannot control the original shape of z.
Tensors in tensorflow are immutable, so you can't re-scale them easily.
You can attempt to pad with 0's and then access parts of the matrix with tf.gather() as shown here How to select rows from a 3-D Tensor in TensorFlow?
to effect the "submatrix" within the larger padded matrix. This however does not seem to be an easy or elegant solution.
I have a set of differently-shaped matrices M = (M_1, M_2, ... M_K). For efficiency purposes, I can store all of M into a single tensor of size K x max(M_k.shape[0]) x max(M_k.shape[1]). This works fine for doing things like batch matrix multiplication and elementwise additions. But what if I want to do elementwise divisions where the zero elements are ignored?
The best version of this I've come up with is:
import numpy as np
import tensorflow as tf
M = tf.constant(np.array([[1.,2.,0],[3.,4.,5.],[6.,0,0]]), tf.float32)
Minv = tf.select(tf.equal(M, 0), tf.zeros_like(M), tf.inv(M))
Is this the fastest way? Does tf.select still get accelerated well via a GPU?
The 2D problem
For each datapoint, I have an index matrix which I want to use to gather vectors from a 2D lookup matrix.
For a single datapoint, Theano and Keras allow easy indexing.
import keras.backend as K
result = K.gather(reference, indices)
E.g.:
result = K.gather(reference, indices)
#let:
indices.shape = (100, 5)
reference.shape = (101, 68)
#where:
max(indices) < reference.shape[0]
#then:
result.shape = (100, 5, 68)
The 3D problem
However, I need to repeat this process for each datapoint in a batch. E.g. I want to parallelise the lookup.
I have a 3D matrix that I want to convert into a 4D matrix.
E.g.
#let:
indices.shape = (batch_n, 100, 5)
reference.shape = (batch_n, 101, 68)
#desired result
result.shape = (batch_n, 100, 5, 68)
More formally, I am looking for an operation such that:
result[i,j,k,:] = lookup[i, indices[i,j,k], :]
or
result[i,j,k,l] = lookup[i, indices[i,j,k], l]
I implemented a Theano solution using scan. It is actually quite straightforward:
import theano
import theano.tensor as T
def parallel_gather(references, indices):
result, _ = theano.scan(fn=lambda reference, indices:reference[indices], outputs_info=None, sequences=[references, indices])
return result
Rewriting this to the Keras backend seems troublesome given that keras.rnn is the Keras alternative. It does not seem to support iteration of a list of tensors, and has some weird requirements.
I also wonder if this is the fastest option, perhaps some clever reshaping could also solve the problem.