We Keep Coding

Concat tensor of[None, 192] with tensor of [1,128]

How to concatenate tensors of shapes [None, 128] with tensor of [1,128]. Here the first tensor will some data of unknown length and the second tensor is fixed tensor not dependant on data size. The final output should be of shape[None, 328]. This is a part of a neural network concatenation.
I tried
> c = Concatenate(axis = -1, name = 'DQN_Input')([ a, b])
Here a.shape = (None, 192) and b.shape = (1,128)
But this does not work.
The error is
ValueError: A Concatenate layer requires inputs with matching
shapes except for the concat axis. Got inputs shapes: [(None, 192),
(1, 128)]

What you can do is use tf.repeat on b based on the first dimension of a to generate the same shape tensor. Here is a simple working example:
import tensorflow as tf
a = tf.keras.layers.Input((192, ), name = 'a')
alpha = tf.keras.layers.Input((1,),name = 'Alpha')
b = tf.matmul(alpha, a, transpose_a=True)
b = tf.repeat(b, repeats=tf.shape(a)[0], axis=0)
c = tf.keras.layers.Concatenate(axis = -1, name = 'DQN_Input')([ a, b])
model = tf.keras.Model([a, alpha], c)
tf.print(model((tf.random.normal((5, 192)), tf.random.normal((5, 1)))).shape)
TensorShape([5, 384])

Tensorflow multiplying 2D tensor elementwise with 3D tensor [duplicate]

I'm trying to do this operation between two tensors:
green_mat = sio.loadmat('green.mat')
green = np.array(green_mat['G2'])
green = tf.convert_to_tensor(green)
green = tf.cast(green, dtype='complex64') # >>>green.shape = TensorShape([64, 40000])
tensor = tf.ones(128,1) # tensor.shape = TensorShape([128])
def mul_and_sum(tensor):
real = tensor[0:64]
imag = tensor[64:128]
complex_tensor = tf.complex(real, imag)
return tf.reduce_sum((tf.multiply(green, complex_tensor), 1))
res = mul_and_sum(tensor)
basically, at the end what I want to obtain is a tensor with 40000 elements to use as a layer for a neural network, but when I run this function as a test I have this error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64,40000] vs. [64] [Op:Mul]
It's the first time I'm working on tensors operation and maybe I'm a bit confused on how to treat the dimensions, any suggestions? Thanks :)
EDIT: ok, I have understood the point, indeed for the example that I provided everything works, but then I have another problem in my network:
def convolution(tensor):
tf.cast(tensor, dtype='float64')
real = tensor[0:64]
imag = tensor[64:128]
complex_tensor = tf.complex(real, imag)
a = tf.math.real(tf.reduce_sum((tf.multiply(green, complex_tensor)), 0))
return a
def get_model3(mask_kind):
epochs = 200
learning_rate = 0.1
decay_rate = learning_rate / epochs
inp_1 = keras.Input(shape=(64, 101, 129), name="RST_inputs")
x = layers.Conv2D(1, kernel_size=(1, 1), strides=(1, 1), padding="valid", trainable=False)(inp_1)
x = layers.Conv2D(256, kernel_size=(3, 3), kernel_regularizer=l2(1e-6), strides=(3, 3), padding="same")(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Conv2D(128, kernel_size=(3, 3), kernel_regularizer=l2(1e-6), strides=(3, 3), padding="same")(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Conv2D(64, kernel_size=(2, 2), kernel_regularizer=l2(1e-6), strides=(2, 2), padding="same")(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Conv2D(32, kernel_size=(2, 2), kernel_regularizer=l2(1e-6), strides=(2, 2), padding="same")(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Flatten()(x)
x = layers.Dense(512)(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Dense(256)(x)
x = layers.LeakyReLU(alpha=0.3)(x)
out1 = layers.Dense(128, name="ls_weights")(x)
if mask_kind == 1:
binary_mask = layers.Lambda(mask_layer1, name="lambda_layer", dtype='float64')(out1)
elif mask_kind == 2:
binary_mask = layers.Lambda(mask_layer2, name="lambda_layer", dtype='float64')(out1)
else:
binary_mask = out1
#here the binary mask shape is [?,128]
binary_mask = tf.expand_dims(binary_mask, axis=2) #here the shape is [?,128,1]
binary_mask = tf.squeeze(binary_mask, axis=0) #here the shape is [128,1]
print('binary shape:', binary_mask.shape)
lambda_layer = layers.Lambda(convolution, name="convolutional_layer")(binary_mask)
print(lambda_layer.shape)
model3 = keras.Model(inp_1, lambda_layer, name="2_out_model")
model3.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=decay_rate), # in caso
# rimettere 0.001
loss="mean_squared_error")
plot_model(model3, to_file='model.png', show_shapes=True, show_layer_names=True)
model3.summary()
return model3
and I get this error:
ValueError: Input 0 of layer sf_vec is incompatible with the layer: : expected min_ndim=2, found ndim=1. Full shape received: [40000]
I know that it is because there is a mismatch between the dimension, but the fact is that the shape of the output layer(tensor) should be [?,40000] and instead I get only a tensor of [40000], any suggestion?
EDIT 2.0 I didn't noticed that my output was already the lambda layer so in the way the model is written I get no errors, but from the summary in this way I get as lambda shape (1, 40000) while usually should be (None,40000).
Where is the error?

If you want to do a multiplication between 2 tensors, they need to have compatible shape, i.e, either the same shape, or a shape that is broadcastable. Quoting the numpy documentation (tensorflow follows the same broadcasting rules):
When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions and works its way forward. Two dimensions are compatible when
they are equal, or
one of them is 1
In your case, if you want to use tf.multiply, you need to add a dimension to your vector so that it has a the same number of dimensions. You can do that by using tf.expand_dims or using advanced indexing with tf.newaxis.
An example (using complexes, like in your question):
>>> a = tf.complex(tf.random.normal((64,128)),tf.random.normal((64,128)))
>>> a.shape
TensorShape([64, 128])
>>> b = tf.complex(tf.ones(64),tf.ones(64))
>>> b.shape
TensorShape([64])
To be able to use tf.multiply, you need to add a dimension to b :
>>> b_exp = tf.exand_dims(b, axis=1)
>>> b_exp.shape
TensorShape([64, 1])
>>> tf.multiply(a,b_exp).shape
TensorShape([64, 128])
Note: doing tf.reduce_sum on tf.multiply can be akin to just do a matrix multiplication
In your case, you can probably do something similar to
>>> tf.matmul(b[tf.newaxis,:], a).shape
TensorShape([1, 128])
If the extra dimension bothers you, you can get rid of it with tf.squeeze.

"InvalidArgumentError: Incompatible shapes: [64,40000] vs. [64] [Op:Mul]" while doing operations between tensors?

I'm trying to do this operation between two tensors:
green_mat = sio.loadmat('green.mat')
green = np.array(green_mat['G2'])
green = tf.convert_to_tensor(green)
green = tf.cast(green, dtype='complex64') # >>>green.shape = TensorShape([64, 40000])
tensor = tf.ones(128,1) # tensor.shape = TensorShape([128])
def mul_and_sum(tensor):
real = tensor[0:64]
imag = tensor[64:128]
complex_tensor = tf.complex(real, imag)
return tf.reduce_sum((tf.multiply(green, complex_tensor), 1))
res = mul_and_sum(tensor)
basically, at the end what I want to obtain is a tensor with 40000 elements to use as a layer for a neural network, but when I run this function as a test I have this error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64,40000] vs. [64] [Op:Mul]
It's the first time I'm working on tensors operation and maybe I'm a bit confused on how to treat the dimensions, any suggestions? Thanks :)
EDIT: ok, I have understood the point, indeed for the example that I provided everything works, but then I have another problem in my network:
def convolution(tensor):
tf.cast(tensor, dtype='float64')
real = tensor[0:64]
imag = tensor[64:128]
complex_tensor = tf.complex(real, imag)
a = tf.math.real(tf.reduce_sum((tf.multiply(green, complex_tensor)), 0))
return a
def get_model3(mask_kind):
epochs = 200
learning_rate = 0.1
decay_rate = learning_rate / epochs
inp_1 = keras.Input(shape=(64, 101, 129), name="RST_inputs")
x = layers.Conv2D(1, kernel_size=(1, 1), strides=(1, 1), padding="valid", trainable=False)(inp_1)
x = layers.Conv2D(256, kernel_size=(3, 3), kernel_regularizer=l2(1e-6), strides=(3, 3), padding="same")(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Conv2D(128, kernel_size=(3, 3), kernel_regularizer=l2(1e-6), strides=(3, 3), padding="same")(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Conv2D(64, kernel_size=(2, 2), kernel_regularizer=l2(1e-6), strides=(2, 2), padding="same")(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Conv2D(32, kernel_size=(2, 2), kernel_regularizer=l2(1e-6), strides=(2, 2), padding="same")(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Flatten()(x)
x = layers.Dense(512)(x)
x = layers.LeakyReLU(alpha=0.3)(x)
x = layers.Dense(256)(x)
x = layers.LeakyReLU(alpha=0.3)(x)
out1 = layers.Dense(128, name="ls_weights")(x)
if mask_kind == 1:
binary_mask = layers.Lambda(mask_layer1, name="lambda_layer", dtype='float64')(out1)
elif mask_kind == 2:
binary_mask = layers.Lambda(mask_layer2, name="lambda_layer", dtype='float64')(out1)
else:
binary_mask = out1
#here the binary mask shape is [?,128]
binary_mask = tf.expand_dims(binary_mask, axis=2) #here the shape is [?,128,1]
binary_mask = tf.squeeze(binary_mask, axis=0) #here the shape is [128,1]
print('binary shape:', binary_mask.shape)
lambda_layer = layers.Lambda(convolution, name="convolutional_layer")(binary_mask)
print(lambda_layer.shape)
model3 = keras.Model(inp_1, lambda_layer, name="2_out_model")
model3.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=decay_rate), # in caso
# rimettere 0.001
loss="mean_squared_error")
plot_model(model3, to_file='model.png', show_shapes=True, show_layer_names=True)
model3.summary()
return model3
and I get this error:
ValueError: Input 0 of layer sf_vec is incompatible with the layer: : expected min_ndim=2, found ndim=1. Full shape received: [40000]
I know that it is because there is a mismatch between the dimension, but the fact is that the shape of the output layer(tensor) should be [?,40000] and instead I get only a tensor of [40000], any suggestion?
EDIT 2.0 I didn't noticed that my output was already the lambda layer so in the way the model is written I get no errors, but from the summary in this way I get as lambda shape (1, 40000) while usually should be (None,40000).
Where is the error?

If you want to do a multiplication between 2 tensors, they need to have compatible shape, i.e, either the same shape, or a shape that is broadcastable. Quoting the numpy documentation (tensorflow follows the same broadcasting rules):
When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions and works its way forward. Two dimensions are compatible when
they are equal, or
one of them is 1
In your case, if you want to use tf.multiply, you need to add a dimension to your vector so that it has a the same number of dimensions. You can do that by using tf.expand_dims or using advanced indexing with tf.newaxis.
An example (using complexes, like in your question):
>>> a = tf.complex(tf.random.normal((64,128)),tf.random.normal((64,128)))
>>> a.shape
TensorShape([64, 128])
>>> b = tf.complex(tf.ones(64),tf.ones(64))
>>> b.shape
TensorShape([64])
To be able to use tf.multiply, you need to add a dimension to b :
>>> b_exp = tf.exand_dims(b, axis=1)
>>> b_exp.shape
TensorShape([64, 1])
>>> tf.multiply(a,b_exp).shape
TensorShape([64, 128])
Note: doing tf.reduce_sum on tf.multiply can be akin to just do a matrix multiplication
In your case, you can probably do something similar to
>>> tf.matmul(b[tf.newaxis,:], a).shape
TensorShape([1, 128])
If the extra dimension bothers you, you can get rid of it with tf.squeeze.

How to use Reshape keras layer with two None dimension?

I have a keras 3D/2D model. In this model a 3D layer has a shape of [None, None, 4, 32]. I want to reshape this into [None, None, 128]. However, if I simply do the following:
reshaped_layer = Reshape((-1, 128))(my_layer)
my_layer has a shape of [None, 128] and therefore I cannot apply afterwards any 2D convolution, like:
conv_x = Conv2D(16, (1,1))(reshaped_layer)
I've tried to use tf.shape(my_layer) and tf.reshape, but I have not been able to compile the model since tf.reshape is not a Keras layer.
Just to clarify, I'm using channels last; this is not tf.keras, this is just Keras. Here I send a debug of the reshape function: Reshape in keras
This is what I'm doing right now, following the advice of anna-krogager:
def reshape(x):
x_shape = K.shape(x)
new_x_shape = K.concatenate([x_shape[:-2], [x_shape[-2] * x_shape[-1]]])
return K.reshape(x, new_x_shape)
reshaped = Lambda(lambda x: reshape(x))(x)
reshaped.set_shape([None,None, None, 128])
conv_x = Conv2D(16, (1,1))(reshaped)
I get the following error: ValueError: The channel dimension of the inputs should be defined. Found None

You can use K.shape to get the shape of your input (as a tensor) and wrap the reshaping in a Lambda layer as follows:
def reshape(x):
x_shape = K.shape(x)
new_x_shape = K.concatenate([x_shape[:-2], [x_shape[-2] * x_shape[-1]]])
return K.reshape(x, new_x_shape)
reshaped = Lambda(lambda x: reshape(x))(x)
reshaped.set_shape([None, None, None, a * b]) # when x is of shape (None, None, a, b)
This will reshape a tensor with shape (None, None, a, b) to (None, None, a * b).

Digging into the base_layer.py, I have found that reshaped is:
tf.Tensor 'lambda_1/Reshape:0' shape=(?, ?, ?, 128) dtype=float32.
However its atribute "_keras_shape" is (None, None, None, None) even after the set_shape. Therefore, the solution is to set this attribute:
def reshape(x):
x_shape = K.shape(x)
new_x_shape = K.concatenate([x_shape[:-2], [x_shape[-2] * x_shape[-1]]])
return K.reshape(x, new_x_shape)
reshaped = Lambda(lambda x: reshape(x))(x)
reshaped.set_shape([None, None, None, 128])
reshaped.__setattr__("_keras_shape", (None, None, None, 128))
conv_x = Conv2D(16, (1,1))(reshaped)

Since you are reshaping the best you can obtain from (4,32), without losing dimensions, is either (128, 1) or (1, 128). Thus you can do the following:
# original has shape [None, None, None, 4, 32] (including batch)
reshaped_layer = Reshape((-1, 128))(original) # shape is [None, None, 128]
conv_layer = Conv2D(16, (1,1))(K.expand_dims(reshaped_layer, axis=-2)) # shape is [None, None, 1, 16]

Tensorflow: How to index one tensor using another tensor with unknown dimensions?

I have a tensor params with shape (?, 70, 64) and another tensor indices with shape (?, 1). I want to index into the first tensor's axis 1 using the second tensor, to get a result with shape (?, 64).
I can't figure how to go about it. Here's what I've tried:
tf.gather(params, indices) # returns a tensor of shape (?, 1, 70, 64)
tf.gather(params, indices, axis=1) # returns a tensor of shape (?, ?, 1, 64)
tf.gather_nd(params, indices) # returns a tensor of shape (?, 70, 64)
(I have an older version of TensorFlow, which doesn't have batch_gather. )
Any help would be appreciated.
Thanks!

You can use tf.stack to convert your indices to a tensor of shape (?, 2) with the first number in the second dimension being the batch number. Then using this new indices with tf.gather_nd should give you what you want if I understand your goal correctly.
Since your indices is a tensor of shape (?, 1), batch_gather would give you (?, 1, 64), meaning one reshape step from your expected result tensor of shape (?, 64). The following code shows two methods give you the same result:
import numpy as np
import tensorflow as tf
params = tf.constant(np.arange(3*70*64).reshape(3, 70, 64))
init_indices = tf.constant([[2], [1], [0]])
indices = tf.stack(
[tf.range(init_indices.shape[0]), tf.reshape(init_indices, [-1])],
axis=1
)
output = tf.gather_nd(params, indices)
batch_gather = tf.reshape(tf.batch_gather(params, init_indices),
[params.shape[0], -1])
with tf.Session() as sess:
print('tf.gather_nd')
print(output.shape)
print(sess.run(output))
print('batch_gather')
print(batch_gather.shape)
print(sess.run(batch_gather))
Edit on comment "first dimension unknown"
Overall, the optimal solution depends on the specific use case, and to use tf.gather_nd with tf.stack, the key is to get the batch size, i.e. the first dimension. One way, which again may not be optimal, is to use tf.shape:
import numpy as np
import tensorflow as tf
params = tf.placeholder(shape=(None, 70, 64), dtype=tf.int32)
init_indices = tf.placeholder(shape=(None, 1), dtype=tf.int32)
indices = tf.stack(
[tf.range(tf.shape(init_indices)[0]), tf.reshape(init_indices, [-1])],
axis=1
)
output = tf.gather_nd(params, indices)
batch_gather = tf.reshape(tf.batch_gather(params, init_indices),
[tf.shape(params)[0], -1])
with tf.Session() as sess:
print('tf.gather_nd')
print(output.shape)
print(sess.run(
output, feed_dict={params: np.arange(3*70*64).reshape(3, 70, 64),
init_indices: [[2], [1], [0]]}
))
print('batch_gather')
print(batch_gather.shape)
print(sess.run(
batch_gather, feed_dict={params: np.arange(3*70*64).reshape(3, 70, 64),
init_indices: [[2], [1], [0]]}
))
One thing to point out is because batch size is unknown, print(batch_gather.shape) gives (?, ?) rather than (?, 64).