i have two cnn models both follow same architecture. I trained 'train set 1' on cnn1 and 'train set 2; on cnn2.Then i exracted features using following code.
# cnn1
model.pop() #removes softmax layer
model.pop() #removes dropoutlayer
model.pop() #removes activation layer
model.pop() #removes batch-norm layer
model.build() #here lies dense 512
features1 = model.predict(train set 1)
print(features1.shape) #600,512
# cnn2
model.pop() #removes softmax layer
model.pop() #removes dropoutlayer
model.pop() #removes activation layer
model.pop() #removes batch-norm layer
model.build() #here lies dense 512
features2 = model.predict(train set 2)
print(features2.shape) #600,512
How to combine these feature 1 and feature 2, so that output shape is 600,1024?
SIMPLEST SOLUTION:
you can simply concatenate the output of the two networks in this way:
features = np.concatenate([features1, features2], 1)
ALTERNATIVE:
given two trained models that have the same structure, whatever their structures are, you can combine them in this way
# generate dummy data
n_sample = 600
set1 = np.random.uniform(0,1, (n_sample,30))
set2 = np.random.uniform(0,1, (n_sample,30))
# model 1
inp1 = Input((30,))
x1 = Dense(512,)(inp1)
x1 = Dropout(0.3)(x1)
x1 = BatchNormalization()(x1)
out1 = Dense(3, activation='softmax')(x1)
m1 = Model(inp1, out1)
# m1.fit(...)
# model 2
inp2 = Input((30,))
x2 = Dense(512,)(inp2)
x2 = Dropout(0.3)(x2)
x2 = BatchNormalization()(x2)
out2 = Dense(3, activation='softmax')(x2)
m2 = Model(inp2, out2)
# m2.fit(...)
# concatenate the desired output
concat = Concatenate()([m1.layers[1].output, m2.layers[1].output]) # get the outputs of dense 512 layers
merge = Model([m1.input, m2.input], concat)
# make combined predictions
merge.predict([set1,set2]).shape # (n_sample, 1024)
Related
I have a custom ResNet model that I define through the Keras Functional API. Also my model has multiple outputs. The last element of the output array is the fully connected dense layer with num_class nodes. I want to be able to increment the number of nodes of this layer. This is the relevant code for the creation of my network:
from tensorflow.keras import layers, models, Input, regularizers
res = []
inputs = Input(shape=(height, width, channels), name='data')
x = MyLayer()(inputs)
# ... other layers
x = MyLayer()(x)
res.append(x)
# ... other layers
x = layers.Dense(num_class, name='fc1', use_bias=True)(x)
res.append(x)
model = models.Model(inputs=inputs, outputs=[res[-2], res[-3], res[-4], res[-1]])
At the question Adding new nodes to output layer in Keras I found an answer similar to what I'm searching for that I'm adding here below:
def add_outputs(self, n_new_outputs):
#Increment the number of outputs
self.n_outputs += n_new_outputs
weights = self.model.get_layer('fc8').get_weights()
#Adding new weights, weights will be 0 and the connections random
shape = weights[0].shape[0]
weights[1] = np.concatenate((weights[1], np.zeros(n_new_outputs)), axis=0)
weights[0] = np.concatenate((weights[0], -0.0001 * np.random.random_sample((shape, n_new_outputs)) + 0.0001), axis=1)
#Deleting the old output layer
self.model.layers.pop()
last_layer = self.model.get_layer('batchnormalization_1').output
#New output layer
out = Dense(self.n_outputs, activation='softmax', name='fc8')(last_layer)
self.model = Model(input=self.model.input, output=out)
#set weights to the layer
self.model.get_layer('fc8').set_weights(weights)
print(weights[0])
However in this question there was only one layer as output and I'm not sure how to replicate the same with my architecture.
This is the solution I've come up with. I assigned the layers that I wanted to keep as output to variables:
from tensorflow.keras import layers, models, Input, regularizers
inputs = Input(shape=(height, width, channels), name='data')
x = MyLayer()(inputs)
# ... other layers
a = MyLayer(name="a")(x)
b = MyLayer(name="b")(a)
c = MyLayer(name="c")(b)
x = layers.Dense(num_class, name='fc1', use_bias=True)(c)
model = models.Model(inputs=inputs, outputs=[c, b, a, x])
And then in order to increase the number of nodes in the last layer just call the function increment_classes. The total number of nodes will be the sum of old_num_class and num_class.
def increment_classes(model, old_num_class, num_class):
weights = model.get_layer("fc1").get_weights()
new_num_class = old_num_class + num_class
# Adding new weights, weights will be 0 and the connections random
shape = weights[0].shape[0]
weights[1] = np.concatenate((weights[1], np.zeros(num_class)), axis=0)
weights[0] = np.concatenate((weights[0], -0.0001 * np.random.random_sample((shape, num_class)) + 0.0001), axis=1)
# Deleting the old dense output layer
model.layers.pop()
# get the output layers
a = model.get_layer("a").output
b = model.get_layer("b").output
c = model.get_layer('c').output
# Replace dense output layer (x)
out = layers.Dense(new_num_class, name='fc1', use_bias=True)(c)
model = models.Model(inputs=model.input, outputs=[c, b, a, out])
# set weights to the layer
model.get_layer('fc1').set_weights(weights)
return model
I have a model like the one below. I want to add a matrix of learnable weights in the end, which is initialized to the variable matrix that I pass to the function create_model.
To get the intuitive idea of what I want to do, imagine the matrix is supposed to be the one I pass to the model, but I have the feeling that it can still be finetuned during training. Therefore, I want it to be initialized to the values I pass, and then refined during training.
The code below works, but as you see from the model.summary() output, the matrix multiplication contains no learnable weights, which makes me think that the weights of the matrix are not beeing finetuned.
What am I doing wrong?
def create_model(num_columns, matrix):
inp_layer = tfl.Input((num_columns,))
dense = tfl.Dense(512, activation = 'relu')(inp_layer)
dense = tfl.Dense(256, activation = 'relu')(dense)
dense = tfl.Dense(128, activation = 'relu')(dense)
va = tf.Variable(matrix, dtype = tf.float32)
dense = K.dot(dense, va )
model = tf.keras.Model(inputs = inp_layer, outputs = dense)
model.compile(optimizer='adam', loss=['binary_crossentropy'])
model.summary()
return model
matrix = np.random.randint(0,2,(128, 206)) # In reality, this is not random, but it has sensed values
num_columns = 750
model = create_model(num_columns,matrix)
you can simply use a dense layer with no bias to do this multiplication. After the model is built I change the weight of interest with the matrix you provided
def create_model(num_columns, matrix):
inp_layer = Input((num_columns,))
x = Dense(512, activation = 'relu')(inp_layer)
x = Dense(256, activation = 'relu')(x)
x = Dense(128, activation = 'relu')(x)
dense = Dense(206, use_bias=False)(x)
model = Model(inputs = inp_layer, outputs = dense)
model.compile(optimizer='adam', loss=['binary_crossentropy'])
model.set_weights(model.get_weights()[:-1] + [matrix])
model.summary()
return model
matrix = np.random.randint(0,2,(128, 206)) # In reality, this is not random, but it has sensed values
num_columns = 750
model = create_model(num_columns,matrix)
check
(model.get_weights()[-1] == matrix).all() # True
In this way, the weights can be fine-tuned
My NN has to learn image similarity with a custom triplet loss. The positive image is similar to the anchor, while the negative is not.
My task is to predict whether the second image or the third image of an unseen triplet is more similar to the anchor or not.
The triplets are given for both train and test sets in the task, so I did not have to mine them or randomly generate them: they are fixed in my task.
---> Idea: To improve my model, I try to use feature learning with Xception layers frozen and adding a Dense layer on top.
Problem:
When training the below model with Xception layers frozen, after 1-2 epochs it learns to just set all positive images to a very low distance to the anchor and all negative images to a very high distance. Hence, the 100% val accuracy.
I immediately thought of overfitting but I only have one fully connected layer that I train? How can I combat this? Or is my triplet loss somehow wrongly defined?
I dont use data augmentation so could that potentially help?
Somehow this happens only when using a pretrained model. When I use a simple model I get realistic accuracy...
What am I missing here?
My triplet loss:
def triplet_loss(y_true, y_pred, alpha = 0.4):
"""
Implementation of the triplet loss function
Arguments:
y_true -- true labels, required when you define a loss in Keras, you don't need it in this function.
y_pred -- python list containing three objects:
anchor -- the encodings for the anchor data
positive -- the encodings for the positive data (similar to anchor)
negative -- the encodings for the negative data (different from anchor)
Returns:
loss -- real number, value of the loss
"""
total_length = y_pred.shape.as_list()[-1]
anchor = y_pred[:,0:int(total_length*1/3)]
positive = y_pred[:,int(total_length*1/3):int(total_length*2/3)]
negative = y_pred[:,int(total_length*2/3):int(total_length*3/3)]
# distance between the anchor and the positive
pos_dist = K.sum(K.square(anchor-positive),axis=1)
# distance between the anchor and the negative
neg_dist = K.sum(K.square(anchor-negative),axis=1)
# compute loss
basic_loss = pos_dist-neg_dist+alpha
loss = K.maximum(basic_loss,0.0)
return loss
Then my model:
def baseline_model():
input_1 = Input(shape=(256, 256, 3))
input_2 = Input(shape=(256, 256, 3))
input_3 = Input(shape=(256, 256, 3))
pretrained_model = Xception(include_top=False, weights="imagenet")
for layer in pretrained_model.layers:
layer.trainable = False
x1 = pretrained_model(input_1)
x2 = pretrained_model(input_2)
x3 = pretrained_model(input_3)
x1 = Flatten(name='flatten1')(x1)
x2 = Flatten(name='flatten2')(x2)
x3 = Flatten(name='flatten3')(x3)
x1 = Dense(128, activation=None,kernel_regularizer=l2(0.01))(x1)
x2 = Dense(128, activation=None,kernel_regularizer=l2(0.01))(x2)
x3 = Dense(128, activation=None,kernel_regularizer=l2(0.01))(x3)
x1 = Lambda(lambda x: K.l2_normalize(x,axis=-1))(x1)
x2 = Lambda(lambda x: K.l2_normalize(x,axis=-1))(x2)
x3 = Lambda(lambda x: K.l2_normalize(x,axis=-1))(x3)
concat_vector = concatenate([x1, x2, x3], axis=-1, name='concat')
model = Model([input_1, input_2, input_3], concat_vector)
model.compile(loss=triplet_loss, optimizer=Adam(0.00001), metrics=[accuracy])
model.summary()
return model
Fitting my model:
model.fit(
gen(X_train,batch_size=batch_size),
steps_per_epoch=13281 // batch_size,
epochs=10,
validation_data=gen(X_val,batch_size=batch_size),
validation_steps=1666 // batch_size,
verbose=1,
callbacks=callbacks_list
)
model.save_weights('try_6.h5')
Please note that you use different Dense layers for each input (you define 3 different Dense layers. each time you create a new Dense object it generate a new layer, with new parameters, independent of the previous layers you created). If the input is consistent, meaning input 1 is always the anchor, input 2 is always the positive, and input 3 is always the negative - it will be super easy for the model to overfit. What you should probably do is use only a single Dense layer for all 3 inputs.
For example, based on your code you can define the model like this:
pretrained_model = Xception(include_top=False, weights="imagenet")
for layer in pretrained_model.layers:
layer.trainable = False
general_input = Input(shape=(256, 256, 3))
x = pretrained_model(general_input)
x = Flatten()(x)
x = Dense(128, activation=None,kernel_regularizer=l2(0.01))(x)
base_model = Model([general_input], [x])
input_1 = Input(shape=(256, 256, 3))
input_2 = Input(shape=(256, 256, 3))
input_3 = Input(shape=(256, 256, 3))
x1 = base_model(input_1)
x2 = base_model(input_2)
x3 = base_model(input_3)
# ... continue with your code - normalize, concat, etc.
I have a CNN with a concatenate-layer at the beginning of Dense network. I use the layer.concatenate() to merge the features retrieved by CNN and my hand-crafted features. Now, this 2 data arrays, have 2 different scale (because the first are values calculated by the CNN and the other are features calculated by me). So I would a way to normalize this 2 type of data in a common scale for simplify the job. This is my CNN:
emb = Embedding()(image_input)
conv = Conv1D(64, 2, activation = 'relu', strides = 1, padding = 'same')(emb)
conv = MaxPooling1D()(conv1)
first_part_output = Flatten()(conV)
merged_model = layers.concatenate([first_part_output, other_data_input])
Here i would do the normalization:
normal = *here i would do the normalization*
primoDense = Dense(256, activation = 'relu')(normal)
drop = Dropout(0.45)(primoDense)
predictions = Dense(1, activation = 'sigmoid')(drop)
[first_part_output, other_data_input] is my new merged-array.
first_part_output are CNN featues.
other_data_input are my hand-crafted features.
I want to try to implement the neural network architecture of the attached image: 1DCNN_model
Consider that I've got a dataset X which is (N_signals, 1500, 40) where 40 is the number of features where I want to do the 1d convolution on.
My Y is (N_signals, 1500, 2) and I'm working with keras.
Every 1d convolution needs to take one feature vector like in this picture:1DCNN_convolution
So it has to take one chunk of the 1500 timesamples, pass it through the 1d convolutional layer (sliding along time-axis) then feed all the output features to the LSTM layer.
I tried to implement the first convolutional part with this code but I'm not sure what it's doing, I can't understand how it can take in one chunk at a time (maybe I need to preprocess my input data before?):
input_shape = (None, 40)
model_input = Input(input_shape, name = 'input')
layer = model_input
convs = []
for i in range(n_chunks):
conv = Conv1D(filters = 40,
kernel_size = 10,
padding = 'valid',
activation = 'relu')(layer)
conv = BatchNormalization(axis = 2)(conv)
pool = MaxPooling1D(40)(conv)
pool = Dropout(0.3)(pool)
convs.append(pool)
out = Merge(mode = 'concat')(convs)
conv_model = Model(input = layer, output = out)
Any advice? Thank you very much
Thank you very much, I modified my code in this way:
input_shape = (1500,40)
model_input = Input(shape=input_shape, name='input')
layer = model_input
layer = Conv1D(filters=40,
kernel_size=10,
padding='valid',
activation='relu')(layer)
layer = BatchNormalization(axis=2)(layer)
layer = MaxPooling1D(pool_size=40,
padding='same')(layer)
layer = Dropout(self.params.drop_rate)(layer)
layer = LSTM(40, return_sequences=True,
activation=self.params.lstm_activation)(layer)
layer = Dropout(self.params.lstm_dropout)(layer)
layer = Dense(40, activation = 'relu')(layer)
layer = BatchNormalization(axis = 2)(layer)
model_output = TimeDistributed(Dense(2,
activation='sigmoid'))(layer)
I was actually thinking that maybe I have to permute my axes in order to make maxpooling layer work on my 40 mel feature axis...
If you want to perform an individual 1D convolution over the 40 feature channels you should add a dimension to your input:
(1500,40,1)
if you perform 1D convolution on a input with shape
(1500,40)
the filters are applied on the time dimension and the pictures you posted indicate that this is not what you want to do.