We Keep Coding

How to connect a LSTM layer to a Linear layer in Pytorch

I am doing a classification task of MFCC (time-series data) using LSTM.
I have input (16,60,40) (Batch,step,features)
class model(nn.Module):
def __init__(self,ninp,num_layers,class_num,nhid=128):
super().__init__()
self.lstm_nets = nn.LSTM(input_size=ninp,hidden_size=nhid,num_layers=num_layers,
batch_first=True,dropout=0.2,bidirectional=False)
self.FC = nn.Linear(nhid,class_num)
self.tanh = nn.Tanh()
self.softmax = nn.LogSoftmax(1)
def forward(self,X):
device = 'cuda:0'
out, (ht, ct) = self.lstm_nets(X)
# out = ht.contiguous().view(16,-1)
out = self.tanh(out)
out = self.FC(out)
Out = self.softmax(out)
return Out
model = model(ninp=X.shape[2],num_layers=1,class_num=32,nhid=128)
loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.5e-4)
If I have out = ht.contiguous().view(16,-1) that flattens the LSTM output, I got error
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-96-a7e2ba68dcd9> in <module>()
11
12 optimizer.zero_grad()
---> 13 y_pred = model(X)
14 # calculate loss function
15 loss = loss_function(y_pred, y)
3 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py in forward(self, input)
101
102 def forward(self, input: Tensor) -> Tensor:
--> 103 return F.linear(input, self.weight, self.bias)
104
105 def extra_repr(self) -> str:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (16x32 and 128x32)
If I have out = out.contiguous().view(16,-1) that flattens the LSTM output, I got error RuntimeError: mat1 and mat2 shapes cannot be multiplied (16x7680 and 128x32)
If I remove the Flatten Step, I got such an error. RuntimeError: Expected target size [16, 32], got [16]
In addition, I found examples online do not flatten the output of LSTM
Thanks for any help.

In each timestep of an LSTM the input goes through a simple neural network and the output gets passed to the next timestep.
The output out of function
out, (ht, ct) = self.lstm_nets(X)
contains a list of ALL outputs (i.e the output of the neural networks of every timestep). Yet, in classification, you mostly only really care about the LAST output. You can get it like this:
out = out[:, -1]
This output has now the shape (hidden-size, 1).
So in your case your forward function should look like this:
def forward(self,X):
device = 'cuda:0'
out, (ht, ct) = self.lstm_nets(X)
out = out[: ,-1]
out = self.tanh(out)
out = self.FC(out)
Out = self.softmax(out)
return Out

Layer count mismatch when loading weights from file. Model expected 106 layers, found 4 saved layers

I have this network (from SimClR paper):
base_model = tf.keras.applications.ResNet50(include_top=False, weights=None, input_shape=(224, 224, 3))
base_model.trainable = True
#inputs = Input((224, 224, 3))
inputs = Input((224, 224, 3))
h = base_model(inputs, training=True)
h = GlobalAveragePooling2D()(h)
projection_1 = Dense(256)(h)
projection_1 = Activation("relu")(projection_1)
projection_2 = Dense(128)(projection_1)
projection_2 = Activation("relu")(projection_2)
projection_3 = Dense(50)(projection_2)
resnet_simclr = Model(inputs, projection_3)
thane after training the model, I saved it:
resnet_simclr.save_weights(filename)
However, after trying to load the weight in ResNet50 like this:
inputs = tf.keras.layers.Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3), name="input_image")
Resmodel = tf.keras.applications.ResNet50(input_tensor=inputs, weights=weights_path, include_top=False)
I got this error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-92-cb642379c2ae> in <module>()
3 from keras.utils.vis_utils import plot_model
4 inputs = tf.keras.layers.Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3), name="input_image")
----> 5 Resmodel = tf.keras.applications.ResNet50(input_tensor=inputs, weights=weights_path, include_top=False) # weights="imagenet"
6 #Resmodel.load_weights(weights_path)
7 Resmodel.summary()
3 frames
/usr/local/lib/python3.7/dist-packages/keras/saving/hdf5_format.py in load_weights_from_hdf5_group(f, model)
718 if len(layer_names) != len(filtered_layers):
719 raise ValueError(
--> 720 f'Layer count mismatch when loading weights from file. '
721 f'Model expected {len(filtered_layers)} layers, found '
722 f'{len(layer_names)} saved layers.')
ValueError: Layer count mismatch when loading weights from file. Model expected 106 layers, found 4 saved layers.
EDIT
The issue I'm trying to do transfer learning (down stream) from RestNet50 with have also 3 projection layer to RestNet50 ( part of Unet backbone) but this Rasnet do not have project head like the trained model. I'm not sure how to fix that.

Try Saving the model again
resnet_simclr.layers[1].save_weights(filename)

InvalidArgumentError while Building a deep-CNN [duplicate]

This question already has an answer here:
InvalidArgumentError: Received a label value of 8825 which is outside the valid range of [0, 8825) SEQ2SEQ model
(1 answer)
Closed last year.
I’m new to TensorFlow and python...
I’m trying to build a deep CNN for cell image classification Hep-2 dataset. The data set consists of 13596 images and I’m using 8701 images as my training data for CNN. Also, I have.CSV file which consists of image ID and its cell-type. I extracted the content and using image_ID from .CSV file as my labels. Both training data and Image ID has been converted to .astype(‘float32’). But, somehow I’m getting InvalidArgumentError which I have no idea what’s going on in there.
I’ve posted my code and error, any tips or help would be highly appreciated. Thank you in advance :)
I'm new to Stack Overflow as well. sorry for my messy formatting.
My CODE:
from PIL import Image
import glob
import os
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Dropout, Flatten, MaxPooling2D
from keras.optimizers import SGD
def extract_labels(image_names, Original_Labels):
temp = np.array([image.split('.')[0] for image in image_names])
temp2 = np.array([j[0] for i in temp for j in Original_Labels if(int(i) == int(j[0]))])
return temp2
def get_Labels():
df=pd.read_csv('gt_training.csv', sep=',')
labels = np.asarray(df)
path = 'path..../training/'
image_names_train = [f for f in os.listdir(path) if os.path.splitext(f)[-1] == '.png']
return labels, image_names_train
Train_images = glob.glob('path.../training/*.png')
train_data = np.array([np.array(Image.open(fname)) for fname in Train_images])
train_data = train_data.astype('float32')
train_data /= 255
#getting labels from .csv file for training data
labels, image_names_train = get_Labels()
train_labels = extract_labels(image_names_train, labels)
train_labels = train_labels.astype('float32')
print(train_labels.shape)
train_data = train_data.reshape(train_data.shape[0],78,78,1) #reshaping into 4-Dim
input_shape = (78, 78, 1) #1 because the provided dataset is in grey scale
#Adding pooling, dense layers to an an non-optimized empty CNN
model = Sequential()
model.add(Conv2D(6, kernel_size=(7,7),activation = tf.nn.tanh, input_shape = input_shape))
model.add(MaxPooling2D(pool_size = (2, 2)))
model.add(Conv2D(16, kernel_size=(4,4),activation = tf.nn.tanh))
model.add(MaxPooling2D(pool_size = (3, 3)))
model.add(Conv2D(32, kernel_size=(3,3),activation = tf.nn.tanh))
model.add(MaxPooling2D(pool_size = (3, 3)))
model.add(Flatten())
model.add(Dense(150, activation = tf.nn.tanh, kernel_regularizer = keras.regularizers.l2(0.00005)))
model.add(Dropout(0.5))
model.add(Dense(6, activation = tf.nn.softmax))
#setting an optimizer with a given loss function
opt = SGD(lr = 0.01, momentum = 0.9)
model.compile(optimizer = opt, loss = 'sparse_categorical_crossentropy', metrics = ['accuracy'])
model.fit(x = train_data, y = train_labels, epochs = 10, batch_size = 77)
The error message I got:
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
InvalidArgumentError: Received a label value of 13269 which is outside the valid range of [0, 6). Label values: 8823 3208 9410 5223 8817 3799 6588 1779 1371 5017 9788 9886 3345 1815 5943 37 675 2396 4485 9528 11082 12457 13269 5488 3250 12896 13251 1854 10942 6287 6232 2944
[[node loss_24/dense_55_loss/sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at C:\Users\vardh\Anaconda3\envs\tf\lib\site-packages\keras\backend\tensorflow_backend.py:3009) ]] [Op:__inference_keras_scratch_graph_676176]
Function call stack:
keras_scratch_graph

Somehow I realized that my question is related to this Question InvalidArgumentError: Received a label value of 8825.....
Solution From that post:
#shaili posted,
In the last layer for eg you used model.add(Dense(1, activation='softmax')). Here 1 restricts its value from [0, 1) change its shape to the maximum output label. For eg your output is from label [0,7) then use model.add(Dense(7, activation='softmax'))
input_text = Input(shape=(max_len,), dtype=tf.string)
embedding = Lambda(ElmoEmbedding, output_shape=(max_len, 1024))(input_text)
x = Bidirectional(LSTM(units=512, return_sequences=True,
recurrent_dropout=0.2, dropout=0.2))(embedding)
x_rnn = Bidirectional(LSTM(units=512, return_sequences=True,
recurrent_dropout=0.2, dropout=0.2))(x)
x = add([x, x_rnn]) # residual connection to the first biLSTM
out = TimeDistributed(Dense(n_tags, activation="softmax"))(x)
Here in TimeDistributed layer n_tags is the length of tags from which I want to classify.
If I predict some other quantity such as q_tag whose length is different from n_tags i.e suppose 10 and length of n_tags is 7 and I received 8 as output label it will give the invalid argument error Received a label value of 8 which is outside the valid range of [0, 7).
From my experience,
Usually, this error generates due to the, no of classifications to be classified, inaccurately. Here in my code, model.add(Dense(6, activation = tf.nn.softmax) I gave 6 types of classification to be generated instead of 13596. However, this is not a fully working code, at least it gets my code running.

This model has not yet been built error on model.summary()

I've keras model defined as follow
class ConvLayer(Layer) :
def __init__(self, nf, ks=3, s=2, **kwargs):
self.nf = nf
self.grelu = GeneralReLU(leak=0.01)
self.conv = (Conv2D(filters = nf,
kernel_size = ks,
strides = s,
padding = "same",
use_bias = False,
activation = "linear"))
super(ConvLayer, self).__init__(**kwargs)
def rsub(self): return -self.grelu.sub
def set_sub(self, v): self.grelu.sub = -v
def conv_weights(self): return self.conv.weight[0]
def build(self, input_shape):
# No weight to train.
super(ConvLayer, self).build(input_shape) # Be sure to call this at the end
def compute_output_shape(self, input_shape):
output_shape = (input_shape[0],
input_shape[1]/2,
input_shape[2]/2,
self.nf)
return output_shape
def call(self, x):
return self.grelu(self.conv(x))
def __repr__(self):
return f'ConvLayer(nf={self.nf}, activation={self.grelu})'
class ConvModel(tf.keras.Model):
def __init__(self, nfs, input_shape, output_shape, use_bn=False, use_dp=False):
super(ConvModel, self).__init__(name='mlp')
self.use_bn = use_bn
self.use_dp = use_dp
self.num_classes = num_classes
# backbone layers
self.convs = [ConvLayer(nfs[0], s=1, input_shape=input_shape)]
self.convs += [ConvLayer(nf) for nf in nfs[1:]]
# classification layers
self.convs.append(AveragePooling2D())
self.convs.append(Dense(output_shape, activation='softmax'))
def call(self, inputs):
for layer in self.convs: inputs = layer(inputs)
return inputs
I'm able to compile this model without any issues
>>> model.compile(optimizer=tf.keras.optimizers.Adam(lr=lr),
loss='categorical_crossentropy',
metrics=['accuracy'])
But when I query the summary for this model, I see this error
>>> model = ConvModel(nfs, input_shape=(32, 32, 3), output_shape=num_classes)
>>> model.summary()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-220-5f15418b3570> in <module>()
----> 1 model.summary()
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py in summary(self, line_length, positions, print_fn)
1575 """
1576 if not self.built:
-> 1577 raise ValueError('This model has not yet been built. '
1578 'Build the model first by calling `build()` or calling '
1579 '`fit()` with some data, or specify '
ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.
I'm providing input_shape for the first layer of my model, why is throwing this error?

The error says what to do:
This model has not yet been built. Build the model first by calling build()
model.build(input_shape) # `input_shape` is the shape of the input data
# e.g. input_shape = (None, 32, 32, 3)
model.summary()

There is a very big difference between keras subclassed model and other keras models (Sequential and Functional).
Sequential models and Functional models are datastructures that represent a DAG of layers. In simple words, Functional or Sequential model are static graphs of layers built by stacking one on top of each other like LEGO. So when you provide input_shape to first layer, these (Functional and Sequential) models can infer shape of all other layers and build a model. Then you can print input/output shapes using model.summary().
On the other hand, subclassed model is defined via the body (a call method) of Python code. For subclassed model, there is no graph of layers here. We cannot know how layers are connected to each other (because that's defined in the body of call, not as an explicit data structure), so we cannot infer input / output shapes. So for a subclass model, the input/output shape is unknown to us until it is first tested with proper data. In the compile() method, we will do a deferred compile and wait for a proper data. In order for it to infer shape of intermediate layers, we need to run with a proper data and then use model.summary(). Without running the model with a data, it will throw an error as you noticed. Please check GitHub gist for complete code.
The following is an example from Tensorflow website.
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
class ThreeLayerMLP(keras.Model):
def __init__(self, name=None):
super(ThreeLayerMLP, self).__init__(name=name)
self.dense_1 = layers.Dense(64, activation='relu', name='dense_1')
self.dense_2 = layers.Dense(64, activation='relu', name='dense_2')
self.pred_layer = layers.Dense(10, name='predictions')
def call(self, inputs):
x = self.dense_1(inputs)
x = self.dense_2(x)
return self.pred_layer(x)
def get_model():
return ThreeLayerMLP(name='3_layer_mlp')
model = get_model()
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 784).astype('float32') / 255
x_test = x_test.reshape(10000, 784).astype('float32') / 255
model.compile(loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=keras.optimizers.RMSprop())
model.summary() # This will throw an error as follows
# ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.
# Need to run with real data to infer shape of different layers
history = model.fit(x_train, y_train,
batch_size=64,
epochs=1)
model.summary()
Thanks!

Another method is to add the attribute input_shape() like this:
model = Sequential()
model.add(Bidirectional(LSTM(n_hidden,return_sequences=False, dropout=0.25,
recurrent_dropout=0.1),input_shape=(n_steps,dim_input)))

# X is a train dataset with features excluding a target variable
input_shape = X.shape
model.build(input_shape)
model.summary()

Make sure you create your model properly. A small typo mistake like the following code may also cause a problem:
model = Model(some-input, some-output, "model-name")
while the correct code should be:
model = Model(some-input, some-output, name="model-name")

If your Tensorflow, Keras version is 2.5.0 then just add Tensorflow when you import Keras package
Not this:
from tensorflow import keras
from keras.models import Sequential
import tensorflow as tf
Like this:
from tensorflow import keras
from tensorflow.keras.models import Sequential
import tensorflow as tf

Version issues of your Tensorflow, Keras, can be the reason for this.
Same problem I encountered during training for the LSTM model for regression.
Error:
ValueError: This model has not yet been built. Build the model first
by calling build() or by calling the model on a batch of data.
Earlier:
from tensorflow.keras.models import Sequential
from tensorflow.python.keras.models import Sequential
Corrected:
from keras.models import Sequential

I was also facing same error, so I have removed model.summary(). Then issue is resolved. As it arises if model of summary is defined before the model is built.
Here is the LINK for description which states that
Raises:
ValueError: if `summary()` is called before the model is built.**

Use part of the layers of VGG19 in Keras with TimeDistributed layer

I want to use the first 9 layers of the trained VGG19 model combined with TimeDistributed layers. But I get an InvalidArgumentError.
def build_vgg(in_shape):
vgg = VGG19(weights="imagenet")
vgg.outputs = [vgg.layers[9].output]
img = keras.Input(in_shape)
img_features = vgg(img)
return keras.Model(img, img_features)
vggmodel = build_vgg((50,50,3))
input_layer = keras.Input(batch_shape=(10,10,50,50,3))
h2 = keras.layers.wrappers.TimeDistributed(vggmodel)(input_layer)
model = keras.Model(input_layer,h2)
model.summary()
I'm getting this error:
InvalidArgumentError Traceback (most recent call last)
~/.conda/envs/py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs)
1566 try:
-> 1567 c_op = c_api.TF_FinishOperation(op_desc)
1568 except errors.InvalidArgumentError as e:
InvalidArgumentError: Dimensions must be equal, but are 512 and 25088 for 'time_distributed_1/vgg19/fc1/MatMul' (op: 'MatMul') with input shapes: [10,512], [25088,4096].

First, your model should not use an additional input in build_vgg. You should only take the tensors you want.
Second, you should use a compatible input shape.
Third, you can't include top if you're changing the input shape AND loading imagenet weights:
def build_vgg(in_shape):
vgg = VGG19(weights="imagenet", input_shape= in_shape, include_top = False)
outputs = vgg.layers[9].output
return keras.Model(vgg.input, outputs)
Then the rest
vggmodel = build_vgg((50,50,3))
#vggmodel.summary()
input_layer = keras.Input(batch_shape=(10,10,50,50,3))
h2 = keras.layers.wrappers.TimeDistributed(vggmodel)(input_layer)
model = keras.Model(input_layer,h2)
model.summary()
model.predict(np.ones((10,10,50,50,3)))