We Keep Coding

Python (TensorFlow) - Concatenating tensor objects with different dimensions - python

I have a question regarding concatenating two Tensor object in Tensorflow. As you can see in the code below, I would like to concatenate a2 and b1. a2 has shape (None, 1, 512) and b1 has shape (None, 34, 512). I would like to concatenate them along the second argument, thus axis=1.
a_input = Input(shape=(20480,))
b_input = Input(shape=(34,))
a1 = Dense(embedding_dim)(image_input) #N: Activation specification needed here? # shape = (None, 512)
a2 = K.expand_dims(image_emb, axis=1) # shape = (None, 1, 512)
b1 = Embedding(num_words, embedding_dim, mask_zero=True)(caption_input) # shape = (None, 34, 512)
c = concatenate((a2, b1), axis=1)
However, if I execute the code above, I obtain the following error
ValueError: Dimension 0 in both shapes must be equal, but are 512 and 1. Shapes are [512] and [1]. for '{{node concatenate_28/concat_1}} = ConcatV2[N=2, T=DT_BOOL, Tidx=DT_INT32](concatenate_28/ones_like, concatenate_28/ExpandDims, concatenate_28/concat_1/axis)' with input shapes: [?,1,512], [?,34,1], [] and with computed input tensors: input[2] = <1>.
What am I doing wrong here? How can this be solved?
Looking forward to some suggestions!

Providing the solution here (Answer Section), even though it is present in the comment section for the benefit of the community.
ValueError: Dimension 0 in both shapes must be equal, but are 512 and 1. Shapes are [512] and [1]. for '{{node concatenate_28/concat_1}} = ConcatV2[N=2, T=DT_BOOL, Tidx=DT_INT32](concatenate_28/ones_like, concatenate_28/ExpandDims, concatenate_28/concat_1/axis)' with input shapes: [?,1,512], [?,34,1], [] and with computed input tensors: input[2] = <1>.
This error was resolved when modifying code from
b1 = Embedding(num_words, embedding_dim, mask_zero=True)(caption_input)
to
b1 = Embedding(num_words, embedding_dim, mask_zero=False)(caption_input)
Complete updated code in below
a_input = Input(shape=(20480,))
b_input = Input(shape=(34,))
a1 = Dense(embedding_dim)(image_input) #N: Activation specification needed here? # shape = (None, 512)
a2 = K.expand_dims(image_emb, axis=1) # shape = (None, 1, 512)
b1 = Embedding(num_words, embedding_dim, mask_zero=False)(caption_input) # shape = (None, 34, 512)
c = concatenate((a2, b1), axis=1)

Related

How to concatenate tensors of shapes [None, 128] with tensor of [1,128]. Here the first tensor will some data of unknown length and the second tensor is fixed tensor not dependant on data size. The final output should be of shape[None, 328]. This is a part of a neural network concatenation.
I tried
> c = Concatenate(axis = -1, name = 'DQN_Input')([ a, b])
Here a.shape = (None, 192) and b.shape = (1,128)
But this does not work.
The error is
ValueError: A Concatenate layer requires inputs with matching
shapes except for the concat axis. Got inputs shapes: [(None, 192),
(1, 128)]

What you can do is use tf.repeat on b based on the first dimension of a to generate the same shape tensor. Here is a simple working example:
import tensorflow as tf
a = tf.keras.layers.Input((192, ), name = 'a')
alpha = tf.keras.layers.Input((1,),name = 'Alpha')
b = tf.matmul(alpha, a, transpose_a=True)
b = tf.repeat(b, repeats=tf.shape(a)[0], axis=0)
c = tf.keras.layers.Concatenate(axis = -1, name = 'DQN_Input')([ a, b])
model = tf.keras.Model([a, alpha], c)
tf.print(model((tf.random.normal((5, 192)), tf.random.normal((5, 1)))).shape)
TensorShape([5, 384])

I'm working with a CNN-LSTM model on Tensorflow 2.0 + Keras to perform sequence classification. My model is defined as following:
inp = Input(input_shape)
rshp = Reshape((input_shape[0]*input_shape[1], 1), input_shape=input_shape)(inp)
cnn1 = Conv1D(100, 9, activation='relu')(rshp)
cnn2 = Conv1D(100, 9, activation='relu')(cnn1)
mp1 = MaxPooling1D((3,))(cnn2)
cnn3 = Conv1D(50, 3, activation='relu')(mp1)
cnn4 = Conv1D(50, 3, activation='relu')(cnn3)
gap1 = AveragePooling1D((3,))(cnn4)
dropout1 = Dropout(rate=dropout[0])(gap1)
flt1 = Flatten()(dropout1)
rshp2 = Reshape((input_shape[0], -1), input_shape=flt1.shape)(flt1)
bilstm1 = Bidirectional(LSTM(240,
return_sequences=True,
recurrent_dropout=dropout[1]),
merge_mode=merge)(rshp2)
dense1 = TimeDistributed(Dense(30, activation='relu'))(rshp2)
dropout2 = Dropout(rate=dropout[2])(dense1)
prediction = TimeDistributed(Dense(1, activation='sigmoid'))(dropout2)
model = Model(inp, prediction, name="CNN-bLSTM_per_segment")
print(model.summary(line_length=75))
Where input_shape = (60, 60). This definition, however, raises the following error:
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
At first, I thought it was because the rshp2 layer could not reshape the flt1 output to shape (60, X). So I added a printing block before the Bidirectional(LSTM)) layer:
print('reshape1: ', rshp.shape)
print('cnn1: ', cnn1.shape)
print('cnn2: ', cnn2.shape)
print('mp1: ', mp1.shape)
print('cnn3: ', cnn3.shape)
print('cnn4: ', cnn4.shape)
print('gap1: ', gap1.shape)
print('flatten 1: ', flt1.shape)
print('reshape 2: ', rshp2.shape)
And the shapes were:
reshape 1: (None, 3600, 1)
cnn1: (None, 3592, 100)
cnn2: (None, 3584, 100)
mp1: (None, 1194, 100)
cnn3: (None, 1192, 50)
cnn4: (None, 1190, 50)
gap1: (None, 396, 50)
flatten 1: (None, 19800)
reshape 2: (None, 60, None)
Looking at the flt1 layer, its output shape is (19800,), which can be reshaped as (60, 330), but for some reason the (60, -1) of the rshp2 layer is not working as intended, evidenced by the print reshape 2: (None, 60, None). When I try to reshape as (60, 330) it works just fine. Does anyone knows why the (-1) is not working?

-1 is working.
From Reshape documentation, https://www.tensorflow.org/api_docs/python/tf/keras/layers/Reshape
the layer returns a tensor with shape (batch_size,) + target_shape
So, the batch size stays the same, the other dimensions are calculated based on your target_shape.
From the doc, look at the last example,
# also supports shape inference using `-1` as dimension
model.add(tf.keras.layers.Reshape((-1, 2, 2)))
model.output_shape
(None, None, 2, 2)
If you pass -1 in your target shape, the Keras will store None, this is useful if you expect variable-length data in that axis, but if your data shape is always same, just put the dimension hard-coded that will place the dimension when you print the shape later.
N.B: Also no need to specify input_shape=input_shape for your intermediate layers in functional API. The model will infer that for you.

Hi I trained an auto encoder network with convolution layer of 96*96*32
Now I get the weights of my model named autoencoder by
layer=autoencoder.layers[1]
W=layer.get_weights()
As w is a list, please help me sort it's elements and visualize the trained kernels.
I am guessing it should be 32 kernels with 96×96 size
When I type
len(w)
It gives 2 so I have 2 arrays
The top array has 9 subarrays each with 32 numbers
The last array has 32 elements. So it must be bias
[array([[[[-6.56146603e-03, -1.51752336e-02, -3.76937017e-02,
-4.55160812e-03, 1.26366820e-02, -2.97747254e-02,
3.76312323e-02, -1.56892575e-02, 2.03932393e-02,
3.29606095e-03, 3.76580656e-02, 6.99581252e-03,
-4.97130565e-02, 3.63005586e-02, 3.70187908e-02,
2.63699284e-03, 4.42482866e-02, 8.26128479e-03,
3.44854854e-02, 1.94760375e-02, 3.91177870e-02,
-6.67006942e-03, 5.64308763e-02, -1.55166145e-02,
-3.46037326e-03, -3.14556211e-02, -2.31548538e-03,
5.77888393e-04, 2.17472352e-02, -8.16953406e-02,
1.54041937e-02, -3.55066173e-02]],
[[ 7.61649990e-03, -6.52475432e-02, 2.02584285e-02,
-4.36152853e-02, -7.94242844e-02, -6.29556971e-03,
-2.17294712e-02, 3.30206454e-02, 3.47386077e-02,
-2.77627818e-03, 4.49984707e-02, -3.03241126e-02,
-3.36903334e-02, 2.34354921e-02, 3.31020765e-02,
-7.81059638e-03, -9.54489596e-03, -1.07985372e-02,
4.10569459e-02, 5.06392084e-02, -1.64809041e-02,
8.42852518e-03, -6.24148361e-03, 1.38165271e-02,
4.47277874e-02, -1.68551356e-02, 2.87279133e-02,
-4.17906158e-02, -3.29194516e-02, 5.37550561e-02,
-3.10864598e-02, -4.53849025e-02]],
[[ 5.37880100e-02, 2.00091377e-02, -8.04780126e-02,
2.05146279e-02, -6.41385652e-03, 2.94176023e-02,
2.42049675e-02, 2.98423916e-02, 1.30865928e-02,
-9.23016574e-03, -2.63463743e-02, -1.58412699e-02,
-4.76215854e-02, -1.53328422e-02, -2.54222248e-02,
1.03113698e-02, 1.97005924e-02, -1.09527409e-02,
-4.29149866e-02, 1.15255425e-02, 3.65356952e-02,
2.26275604e-02, 8.76231957e-03, -1.82650369e-02,
4.30952013e-02, -1.58966344e-03, 1.01399068e-02,
7.15927547e-03, 2.70794444e-02, -1.93151142e-02,
2.06329934e-02, -3.24055366e-02]]],
[[[ 7.32885906e-04, -5.99233769e-02, 1.01583647e-02,
2.62707975e-02, -1.60765275e-02, 4.54364009e-02,
1.22182900e-02, 1.77695882e-02, 3.40870097e-02,
-3.20678158e-03, 1.94115974e-02, -5.89495376e-02,
5.51430099e-02, 1.08586736e-02, -2.14386974e-02,
-1.10124948e-03, -1.41514605e-02, -8.40184465e-03,
-4.09237854e-02, 2.27938611e-02, 2.82027805e-03,
3.99805643e-02, -5.23957238e-02, -6.65743649e-02,
-1.86213956e-03, 1.84283289e-03, 8.22036352e-04,
-2.04587094e-02, -4.95675243e-02, 5.40869832e-02,
4.00022417e-02, -4.74570543e-02]],
[[-3.73015292e-02, 9.84914601e-03, 9.94246900e-02,
3.19805741e-02, 8.14174674e-03, 2.72354241e-02,
-1.58177980e-03, -5.65455444e-02, -2.13499945e-02,
2.36055311e-02, 4.57456382e-03, 5.87781705e-02,
-4.50953143e-03, -3.05559561e-02, 8.65572542e-02,
-2.87776738e-02, 7.56273838e-03, -2.02421043e-02,
4.32164557e-02, 1.07650533e-02, 1.74834915e-02,
-2.26386450e-02, -4.51299828e-03, -7.19766971e-03,
-5.64673692e-02, -3.46505865e-02, -9.57003422e-03,
-4.17267382e-02, 2.74983943e-02, 7.50013590e-02,
-1.39447292e-02, -2.10063234e-02]],
[[-4.99953330e-03, -1.95915010e-02, 7.38414973e-02,
3.00457701e-02, 4.11909744e-02, -4.93509434e-02,
-3.72827090e-02, -4.84874584e-02, -1.73344277e-02,
2.13540550e-02, 2.63152272e-02, 5.11181913e-02,
5.94335012e-02, -8.46157200e-04, -3.79960015e-02,
-2.01609023e-02, 2.21411046e-02, -1.14003820e-02,
-1.78077854e-02, -6.17240835e-03, -9.96494666e-03,
-2.70768851e-02, 3.32489684e-02, -1.18451891e-02,
7.48611614e-02, 3.68427448e-02, -1.70680200e-04,
2.78645731e-03, 3.37152109e-02, -6.00774325e-02,
3.43431458e-02, 6.80516511e-02]]],
[[[ 4.51148823e-02, 4.12209071e-02, -1.92945134e-02,
-2.68811788e-02, 4.68725041e-02, -2.08357088e-02,
-3.62888947e-02, -1.60191804e-02, 3.19913588e-02,
1.54639455e-02, -7.92380888e-03, -4.85247411e-02,
-3.52074914e-02, -1.04825860e-02, -6.63231388e-02,
4.35819328e-02, 1.74060687e-02, -3.14022303e-02,
-2.88435258e-02, -2.56987382e-03, -4.61222306e-02,
9.01424140e-03, -3.54990773e-02, 3.61517034e-02,
-4.51472104e-02, -1.96188372e-02, 2.76502203e-02,
-3.39846462e-02, -5.75804268e-04, -4.55158725e-02,
2.47761561e-03, 5.08131757e-02]],
[[ 3.74217257e-02, 4.53428067e-02, -4.36269939e-02,
-1.65079869e-02, -2.69084796e-02, -2.38134293e-03,
2.26788968e-02, -3.10470518e-02, -4.33242172e-02,
1.89485904e-02, -5.52747138e-02, 6.01334386e-02,
-1.70235410e-02, -4.17503342e-02, -1.59652822e-03,
-3.10646854e-02, -1.94913559e-02, 5.42740058e-03,
5.47912866e-02, 2.19548331e-03, -2.94116754e-02,
2.24571414e-02, -1.57341175e-02, -5.24678500e-03,
4.41270098e-02, 1.79115515e-02, -3.40841003e-02,
-2.95497216e-02, 4.40835916e-02, 4.28234115e-02,
-4.25039157e-02, 5.90493456e-02]],
[[-2.71476209e-02, 6.84098527e-02, -2.91980486e-02,
-2.52507403e-02, -6.22444265e-02, 3.67519422e-03,
5.06899729e-02, 3.09969904e-03, 4.50362265e-02,
8.56801707e-05, 4.21552844e-02, -3.78406122e-02,
-1.73772611e-02, 4.68185954e-02, -6.93227863e-03,
-4.71074954e-02, 5.72011899e-03, -1.59831103e-02,
-1.66428182e-02, 1.12894354e-02, 5.62585844e-03,
1.36870472e-02, -2.89466791e-02, -2.87153292e-03,
-3.21626514e-02, -3.75866666e-02, -1.62240565e-02,
3.01954672e-02, -2.69964593e-03, -2.27513053e-02,
2.10835561e-02, -4.13369946e-02]]]], dtype=float32),
array([-1.1922461e-03, -2.0752363e-04, 1.1357996e-05, 1.6377015e-05,
-2.5950783e-04, 1.9307183e-05, -1.5572178e-06, -1.3648998e-03,
-8.6763187e-04, 4.4856939e-04, 2.7988455e-03, -7.7398616e-04,
-5.1178242e-04, -6.8265648e-04, 1.8571866e-04, -7.1992702e-04,
-5.5880222e-04, -3.6114815e-04, -9.7678707e-04, 2.6443407e-03,
1.1190268e-03, -1.0251488e-03, -1.1638318e-03, 7.1209669e-04,
4.9417594e-04, 2.3746442e-04, -4.8552561e-04, 1.4480414e-03,
-1.8445569e-05, 4.2989667e-04, 1.0579359e-04, -3.2821635e-04],
dtype=float32)]
The summary of model few starting layers
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) (None, 96, 96, 1) 0
__________________________________________________________________________________________________
conv2d_1 (Conv2D) (None, 96, 96, 32) 320 input_1[0][0]
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 96, 96, 32) 128 conv2d_1[0][0]
Now how do I order them and visualize
I am using keras
Thanks

Usually, if you are using a Dense layer then the first lenth 2 corresponds to the weight vector and bias vector.
As I don't know the type of your layer I'm adding an example explaining shapes for Dense, Conv2D layers.
The first length always corresponds to weight and biases, the second shapes for weight and biases are different, for biases it's always an array, for Dense the weight has a shape (input_dim, output_dim), for a Conv2D (channels, kernel_h, kernel_w, num_filters).
from tensorflow.keras.layers import *
from tensorflow.keras.models import *
import numpy as np
i1 = Input(shape=(32,32,3))
c1 = Conv2D(32, 3)(i1)
f1 = Flatten()(c1)
d1 = Dense(5)(f1)
m = Model(i1, d1)
m.summary()
y = m(np.zeros((1, 32, 32, 3)))
print(m.layers)
cw1 = np.array(m.layers[1].get_weights())
print(cw1.shape) # 2 weight, 1 weight, 1 bias
print(cw1[0].shape) # 3 channels, 3 by 3 kernels, 32 filters
print(cw1[1].shape) # 32 biases
cw1 = np.array(m.layers[2].get_weights())
print(cw1.shape) # this is just a flatten operations, so no weights
cw1 = np.array(m.layers[3].get_weights())
print(cw1.shape) # 2 -> 1 weight, 1 bias
print(cw1[0].shape) # 28800 inputs, 5 outputs, 28800 by 5 weight matrix
print(cw1[1].shape) # 5 biases
Model: "model_13"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_14 (InputLayer) [(None, 32, 32, 3)] 0
_________________________________________________________________
conv2d_13 (Conv2D) (None, 30, 30, 32) 896
_________________________________________________________________
flatten_13 (Flatten) (None, 28800) 0
_________________________________________________________________
dense_13 (Dense) (None, 5) 144005
=================================================================
Total params: 144,901
Trainable params: 144,901
Non-trainable params: 0
_________________________________________________________________
[<tensorflow.python.keras.engine.input_layer.InputLayer object at 0x7fb8ce3bb828>, <tensorflow.python.keras.layers.convolutional.Conv2D object at 0x7fb8ce5fd6d8>, <tensorflow.python.keras.layers.core.Flatten object at 0x7fb8ce3bb940>, <tensorflow.python.keras.layers.core.Dense object at 0x7fb8ce3bbb70>]
(2,)
(3, 3, 3, 32)
(32,)
(0,)
(2,)
(28800, 5)
(5,)
The visualization completely depends on the dimension.
If it's 1-D,
import matplotlib.pyplot as plt
plt.plot(weight)
plt.show()
If it's 2-D,
import matplotlib.pyplot as plt
plt.imshow(weight)
plt.show()
If it's 3-D,
you can choose a channel and plot that part only.
# plotting the 32 conv filter
import matplotlib.pyplot as plt
cw1 = np.array(m.layers[1].get_weights())
for i in range(32):
plt.imshow(cw1[0][:,:,:,i])
plt.show()

I'm not sure if this is possible or not in Keras, but I would like to know is there any way to concatenate specific values from LSTM layers.
I would like to encode whole sequence using LSTM but for prediction use only specific columns.
For example:
I'm using two input sequences with size of 5 (so my input shape is for one input is (None, 5) and (None, 5).
Then I embed sequences and my input shape is (None, 5, 300) and (None, 5, 300)
Then I encode sequence with LSTM layers with 200 LSTM cells and my final shape is (None, 5, 200) and (None, 5, 200).
Now I would like not to concatenate the whole sequence but the last 4 words encoded in lstm_1 and the first word in lstm_2.
> input_1 = Input(shape=(5, ))
> emb_1 = Embedding(..., 300, ...)(input_1)
> lstm_1 = CuDNNLSTM(200, ...)(emb_1)
>
> input_2 = Input(shape=(5, ))
> emb_2 = Embedding(..., 300, ...)(input_2)
> lstm_2 = CuDNNLSTM(200, ...)(emb_2)
>
> # here is the problem
> emb = concatenate([lstm_1[??], lstm_2[??])
>
> d1 = Dense(...)(emb)
> out = Dense(..., activation="softmax")(d1)
Not sure if I'm making any sense but I would like to know if it's possible using Keras functional API.
Best regards,
Daniel

So I didn't realize that lstm_1 and lstm_2 are actually Numpy arrays that I can concatenate. So the solution is simple, you just need to wrap it in Lambda layer.
lstm_1_lambda = Lambda(lambda x: x[:, -4:, :])(lstm_1)
lstm_2_lambda = Lambda(lambda x: x[:, :1, :])(lstm_2)
emb = concatenate([lstm_1_lambda, lstm_1_lambda)
Best regards,
Daniel

I have two separately designed CNNs for two different features(image and text) of the same data, and the output has two classes
In the very last layer:
for image (resnet), I would like to use "he_normal" as the initializer
flatten1 = Flatten()(image_maxpool)
dense = Dense(output_dim=2, kernel_initializer="he_normal")(flatten1)
but for the text CNNs, i would like to use the default "glorot_normal"
flatten2 = Flatten()(text_maxpool)
output = Dense(output_dim=2, kernel_initializer="glorot_normal")(flatten2)
the flatten1 and flatten2 have sizes:
flatten_1 (Flatten) (None, 512)
flatten_2 (Flatten) (None, 192)
is there anyway i can concate these two flatten layers and have a long dense layer with a size 192+512 = 704, where the first 192 and second 512 has two seperate kernel_initializer, and produce a 2-class outputs?
something like this:
merged_tensor = merge([flatten1, flatten2], mode='concat', concat_axis=1)
output = Dense(output_dim=2,
kernel_initializer for [:512]='he_normal',
kernel_initializer for [512:]='glorot_normal')(merged_tensor)
Edit: I think I have gotten this work by having the following codes(thanks to #Aechlys):
def my_init(shape, shape1, shape2):
x = initializers.he_normal()(shape1)
y = initializers.glorot_normal()(shape2)
return tf.concat([x,y], 0)
class_num = 2
flatten1 = Flatten()(image_maxpool)
flatten2 = Flatten()(text_maxpool)
merged_tensor = concatenate([flatten1, flatten2],axis=-1)
output = Dense(output_dim=class_num, kernel_initializer=lambda shape: my_init(shape,\
shape1=(512,class_num),\
shape2=(192,class_num)),\
activation='softmax')(merged_tensor)
I have to manually add the shape size 512 and 192, because I failed to get the size of flatten1 and flatten1 via the code
flatten1.get_shape().as_list()
,which gave me [none, none], althought it should be [None, 512], other than that it should be fine

Oh my, have I had fun with this one. You have to create your own kernel intializer:
def my_init(shape, dtype=None, *, shape1, shape2):
x = keras.initializers.he_normal()(shape1, dtype=dtype)
y = keras.initializers.glorot_normal()(shape2, dtype=dtype)
return tf.concat([x,y], 0)
Then you will call it via lambda function within the Dense function:
Unfortunately, as you can see, I have not been able to deduce the shape programatically, yet. I may update this answer when I do. But, if you know the shape beforehand you can pass them as constants:
DENSE_UNITS = 64
input_t = Input((1,25))
input_i = Input((1,35))
input_a = Concatenate(axis=-1)([input_t, input_i])
dense = Dense(DENSE_UNITS, kernel_initializer=lambda shape: my_init(shape,
shape1=(int(input_t.shape[-1]), DENSE_UNITS),
shape2=(int(input_i.shape[-1]), DENSE_UNITS)))(input_a)
tf.keras.Model(inputs=[input_t, input_i], outputs=dense)
Out: <tensorflow.python.keras._impl.keras.engine.training.Model at 0x19ff7baac88>