We Keep Coding

Error Pop Out while Add Layers from Pre-Trained Model

I am trying to fine-tune a pre-trained time-series model with 12 dimension physiological signals as inputs, yet, my dataset is only one dimension. Therefore, I built the first conv1d layer and set the weight from the pre-trained model. After that, I add the rest parts from the pre-trained model. But, an error popout and says the dimension mismatch.
Here are the details of my code and error:
#Load Whole Model and Weight of the First Conv1d Layer
BaseModel = keras.models.load_model('model.hdf5')
input_w = BaseModel.layers[1].get_weights()[0]
input_w_1_lead = input_w[:,1,:].reshape(input_w.shape[0], 1, input_w.shape[2])
#Creat the Input and First Conv1d Layer
model = tf.keras.Sequential([
keras.Input(shape=(4096, 1), dtype=np.float32, name='signal'),
keras.layers.Conv1D(64, 16, padding='same',
use_bias=False, name='conv1d_1',),
])
#Set the Weight from Pre-Trained Model
model.layers[0].set_weights([input_w_1_lead])
#Add the Rest Parts from Pre-Trained Model
for layer in BaseModel.layers[2:]:
model.add(layer)
#And here is the error:
ValueError: Exception encountered when calling layer "add_1" (type Add).
A merge layer should be called on a list of inputs. Received: inputs=Tensor("Placeholder:0", shape=(None, 256, 128), dtype=float32) (not a list of tensors)
Call arguments received by layer "add_1" (type Add):
• inputs=tf.Tensor(shape=(None, 256, 128), dtype=float32)
Then, I looked into the details of my code and found out that I could only add layers until the 11th one.
test_model = tf.keras.Sequential()
for layer in BaseModel.layers[:11]:
#Not work if I set BaseModel.layers[:12]
test_model.add(layer)
With model.summary(), dimension information seems missing after the pooling layer. Here are the outputs for both BaseModel.summary() and test_model.summary():
BaseModel
test_model
However, I couldn't find the solution.

Dear All,
I figure out an alternative solution for my task. I modified the model's input and output layers from source code. Then set the weight from source model.
Here is the solution code:
'Load Pre-Trained Model to Get Weight'
BaseModel = keras.models.load_model('model.hdf5')
input_w = BaseModel.layers[1].get_weights()[0]
input_w_1_lead = input_w[:,1,:].reshape(input_w.shape[0], 1, input_w.shape[2])
BaseModel.summary()
'Build the Fine-Tuned Model'
model = get_model(1)
model.layers[1].set_weights([input_w_1_lead])
'Load Weight from Source Model'
for i in range(2, len(BaseModel.layers) -1):
model.layers[i].set_weights(BaseModel.layers[i].get_weights())
'Check the Weight is Equal to Source Code'
for i in range(2, len(BaseModel.layers) -1):
if np.allclose(
BaseModel.layers[i].get_weights(),
model.layers[i].get_weights()) is False:
print('this way is no correct')
Ta-da! It's work and nothing printed in the console.

Change intermediate activations/output and observe prediction in TF2/Keras

Let us say I have a simple NN model classifying MNIST like so:
model = models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])
And I extract the output tensor from the 2nd layer, i.e. Dense layer (after activation relu) like so:
mid_layer = model.get_layer("dense")
get_output = K.function([model.layers[0].input], [mid_layer.output])
For some particular data input x, I extract the output tensor activation values now:
mid_layer_outputs = get_output([test_images[0:1]])
And make some changes to it:
mid_layer_outputs = ...
And now, I would like the model to continue from this layer's modified output values and predict results accordingly. How would I do that?
I tried to construct another K.function([mid_layer_outputs], model.layers[-1].output) starting from this layer to the end, but I got the following error:
AttributeError: 'numpy.ndarray' object has no attribute 'op'
which is understandable as it cannot continue predictions with a NP array object instead of a model layer object. How do I do this?

I figured it out - I need to specify the next layer inputs in the second K.function and then pass in the modified outputs to it:
get_pred = K.function([model.layers[2].input], model.layers[-1].output)
pred = get_pred([mid_layer_outputs])
According to this answer, K.function runs the computation graph we created in the code, taking input from the first parameter and extracting the number of outputs as per the layers mentioned. So K.function([mid_layer_outputs], model.layers[-1].output) was not working earlier because instead of passing the layer for K.function to work with and connect to the output, I was passing in the values directly.

I cannot get the input shape right

The following code gives me an input error and i cannot figure it out.
import tensorflow as tf
import neural_structured_learning as nsl
.
.
.
b_size = 132
m = tf.keras.Sequential()
m.add(tf.keras.layers.Dense(980, activation = 'relu', input_shape = (2206,2,)))
m.add(tf.keras.layers.Dense(560, activation = 'relu'))
m.add(tf.keras.layers.Dense(10, activation = 'softmax'))
adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.5)
adv_model = nsl.keras.AdversarialRegularization(m, adv_config=adv_config)
adv_model.compile(optimizer = "adam",
loss = "sparse_categorical_crossentropy",
metrics = ['accuracy'])
adv_model.fit({"feature" : x_Train, "label" : y}, epochs = 50, batch_size=b_size)
My x_Train has shape (5002, 2206, 2) (5002 samples of size (2206,2)). I have tried to add a Flatten() layer at the beginning but it gives me a object of type 'NoneType' has no len() error, even though this works perfectly with tf.keras. I also have tried different shapes for the input but none of them work. So it throws me one of the following errors
KeyError: 'dense_115_input'
ValueError: Input 0 of layer sequential_40 is incompatible with the layer: expected axis -1 of input shape to have value 2206 but received input with shape [None, 2206, 2]
TypeError: object of type 'NoneType' has no len()

To train an NSL model with an input dictionary (like your {"feature" : x_Train, "label" : y}), the base model has to know which feature(s) in the dictionary to look at.
One way to specify the feature names is to add an Input layer:
m = tf.keras.Sequential()
m.add(tf.keras.Input(name="feature", shape=(2206, 2)))
Also as this answer pointed out, the input feature has to be flatten before passing to dense layers:
m.add(tf.keras.layers.Flatten())
m.add(tf.keras.layers.Dense(...))

If you want to use a dense layer the input should be (5002, 2206*2), i.e a matrix.
Maybe the simplest solution is to reshape your input x_train before the "fit".
Alternatively, you can use a TimeDistributed layer (see here), but the usage of this kind of layer depends on the physical meaning behind the input dimensions. Basically, TimeDistributed applies a certain operation many times, in your case twice.
Hoping this can help you.

Tensorflow Tf.tf.squared_difference is showing a value error with dense layer

Whenever I try to multiply two tensors and then feed them as an input to a dense layer, it is working perfectly. But, when I try to calculate the squared difference between them, it's showing me an error.
# working well
out= multiply([user, book])
result = Dense(1, activation='sigmoid', kernel_initializer=initializers.lecun_normal(),
name='prediction')(out)
# error
out= tf.reduce_sum(tf.squared_difference(user, book),1)
result = Dense(1, activation='sigmoid', kernel_initializer=initializers.lecun_normal(),
name='prediction')(out)
Here is the error I get:
Input 0 is incompatible with layer prediction: expected min_ndim=2, found ndim=1 error

You probably need to pass keepdims=True argument to reduce_sum function in order to keep the dimensions with length 1 (otherwise, the shape of out would be (batch_size), whereas the Dense layer expects (batch_size, N)):
out= tf.reduce_sum(tf.squared_difference(user, book), axis=1, keepdims=True)
Update: The input of Keras layers must be the output of other Keras layers. Therefore, if you want to use TensorFlow operations, you need to wrap them inside a Lambda layer in Keras. For example:
from keras.layers import Lambda
out = Lambda(lambda x: tf.reduce_sum(tf.squared_difference(x[0], x[1]), axis=1, keepdims=True))([user, book])

Keras - Embedding Layer and GRU Layer Shape Error

# input_shape = (137861, 21, 1)
# output_sequence_length = 21
# english_vocab_size = 199
# french_vocab_size = 344
def embed_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
'''
Build and train a RNN model using word embedding on x and y
:param input_shape: Tuple of input shape
:param output_sequence_length: Length of output sequence
:param english_vocab_size: Number of unique English words in the dataset
:param french_vocab_size: Number of unique French words in the dataset
:return: Keras model built, but not trained
'''
learning_rate = 1e-3
model = Sequential()
model.add(Embedding(english_vocab_size, 128, input_length=output_sequence_length, input_shape=input_shape[1:]))
model.add(GRU(units=128, return_sequences=True))
model.add(TimeDistributed(Dense(french_vocab_size)))
model.add(Activation('softmax'))
model.summary()
model.compile(loss=sparse_categorical_crossentropy,
optimizer=Adam(learning_rate),
metrics=['accuracy'])
return model
When invoking this method to train a model, it gets the error:
ValueError: Input 0 is incompatible with layer gru_1: expected ndim=3, found ndim=4
How to fix the shape error between Embedding Layer and GRU Layer?

The problem is that the Embedding layer takes a 2D array as the input. However, the shape of the input array is (137861, 21, 1) which makes it a 3D array. Simply remove the last axis using squeeze() method from numpy:
data = np.squeeze(data, axis=-1)
As a side, there is no need to use TimeDistributed layer here, since the Dense layer is applied on the last axis by defualt.