I have written a structure for Conv1D in keras. I want to merge the 6 different inputs of same shape. Previously, Merge([ model1, model2, model3, model4, model5, model6], mode = 'concat') worked just fine but after new updates, I cant use Merge anymore.
Concatenate can be used as follows,
from keras.layers import Concatenate
model = Concatenate([ model1, model2, model3, model4, model5, model6])
But I want to add Dense layers before the softmax layer to this merged model, which I cant add to Concatenate as it accepts only tensor inputs.
How do I merge the 6 inputs before passing it to 2 dense layers and softmax layer??
My current code is as follows,
input_shape = (64,250)
model1 = Sequential()
model1.add(Conv1D(64, 2, activation='relu', input_shape=input_shape))
model1.add(Conv1D(64, 2, activation='relu'))
model1.add(MaxPooling1D(2))
model1.add(Dropout(0.75))
model1.add(Flatten())
model2 = Sequential()
model2.add(Conv1D(128, 2, activation='relu', input_shape=input_shape))
model2.add(Conv1D(128, 2, activation='relu'))
model2.add(MaxPooling1D(2))
model2.add(Dropout(0.75))
model2.add(Flatten())
model3 = Sequential()
model3.add(Conv1D(128, 2, activation='relu', input_shape=input_shape))
model3.add(Conv1D(128, 2, activation='relu'))
model3.add(MaxPooling1D(2))
model3.add(Dropout(0.75))
model3.add(Flatten())
model4 = Sequential()
model4.add(Conv1D(128, 2, activation='relu', input_shape=input_shape))
model4.add(Conv1D(128, 2, activation='relu'))
model4.add(MaxPooling1D(2))
model4.add(Dropout(0.75))
model4.add(Flatten())
model5 = Sequential()
model5.add(Conv1D(128, 2, activation='relu', input_shape=input_shape))
model5.add(Conv1D(128, 2, activation='relu'))
model5.add(MaxPooling1D(2))
model5.add(Dropout(0.75))
model5.add(Flatten())
model6 = Sequential()
model6.add(Conv1D(128, 2, activation='relu', input_shape=input_shape))
model6.add(Conv1D(128, 2, activation='relu'))
model6.add(MaxPooling1D(2))
model6.add(Dropout(0.75))
model6.add(Flatten())
from keras.layers import Concatenate
model = Concatenate([ model1, model2, model3, model4, model5, model6])
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.75))
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.75))
model.add(Dense(40, activation='softmax'))
opt = keras.optimizers.adam(lr=0.001, decay=1e-6)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])
model.fit([d1, d2, d3, d4, d5, d6], label, validation_split=0.2, batch_size=25, epochs=30)
The way you are calling Concatenate Function is not correct. Concatenate expects one argument which specifies the axis of concatenation. What you are trying to achieve can be done using keras's functional API.
just change the following code
model = Concatenate([ model1, model2, model3, model4, model5, model6])
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.75))
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.75))
model.add(Dense(40, activation='softmax'))
to
merged = Concatenate()([ model1.output, model2.output, model3.output, model4.output, model5.output, model6.output])
merged = Dense(512, activation='relu')(merged)
merged = Dropout(0.75)(merged)
merged = Dense(1024, activation='relu')(merged)
merged = Dropout(0.75)(merged)
merged = Dense(40, activation='softmax')(merged)
model = Model(inputs=[model1.input, model2.input, model3.input, model4.input, model5.input, model6.input], outputs=merged)
N.B
Though it is not the question being asked, I've noticed that you have been using very large dropout rate( But this may depend on the problem you are trying to solve). 0.75 drop rate means you are dropping 75% of the neurons while training. Please consider using small drop rate because the model might not converge.
Related
For a chess engine I want to use two autoencoder models, which extract key-features out of a chess-position, concatenate them and build a model on top to compare two chess positions.
My code looks like this so far:
enc1 = keras.models.load_model("autoencoder.h5")
enc2 = keras.models.load_model("autoencoder.h5")
encoder1 = Model(
inputs=enc1.input,
outputs=[enc1.get_layer(index=2).output,
enc1.get_layer(index=4).output,
enc1.get_layer(index=6).output,
enc1.get_layer(index=7).output
]
)
encoder1.trainable = False
encoder2 = Model(
inputs=enc2.input,
outputs=[enc2.get_layer(index=2).output,
enc2.get_layer(index=4).output,
enc2.get_layer(index=6).output,
enc2.get_layer(index=7).output
]
)
encoder2.trainable = False
model = Sequential()
model.add(concatenate([encoder1, encoder2]))
model.add(Dense(400, activation="relu", input_shape=(2,769,)))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(200, activation='relu', kernel_regularizer=l2(), bias_regularizer=l2()))
model.add(Dropout(0.2))
model.add(Dense(100, activation='relu', kernel_regularizer=l2(), bias_regularizer=l2()))
model.add(Dropout(0.2))
model.add(Dense(2, activation='softmax'))
metric = tf.keras.metrics.CategoricalAccuracy()
model.compile(optimizer=Adam(learning_rate=0.001), loss="categorical_crossentropy", metrics=metric)
This is giving errors. How do I concatenate these two autoencoder layers?
Thanks so much!
I am working on the CNN model. In the given CNN model I can not handle how to divide the 4th layer into two streams and get output.
I also built a model in Keras.
def _build_model(self):
model = Sequential()
model.add(Conv2D(8, (3, 3), strides=4, padding='same', input_shape=self.state_size))
model.add(Activation('relu'))
model.add(Conv2D(2, (2, 2), strides=4, padding='same'))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(self.action_size, activation='relu'))
model.compile(loss='mse', optimizer=Adam())
return model
How to handle it? An example would be appreciated.
I'm trying to build a 2 layered neural network for MNIST dataset and I want to get weights from my model.
I found a similar question her on SO and I tried this,
model.get_weights()
But It returned 11 values when I check the len(model.get_weights()) Isn't it suppose to return 3 weights? I have even disabled bias.
model = Sequential()
model.add(Flatten(input_shape = (28, 28)))
model.add(Dense(512, activation='relu', kernel_initializer='he_normal', use_bias=False,))
model.add(BatchNormalization())
model.add(Dropout(0.3))
model.add(Dense(128, activation='relu', kernel_initializer='he_normal', use_bias=False,))
model.add(BatchNormalization())
model.add(Dropout(0.1))
model.add(Dense(10, activation='relu', kernel_initializer='he_normal', use_bias=False,))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
result = model.fit(x_train, y_train, validation_split=0.25, epochs=10,
batch_size=128, verbose=1)
To get the weights of a particular layer, you could retrieve this layer by using its name and call get_weights on it (as shubham-panchal said in its comment).
For example:
model.get_layer('dense').get_weights()
or
model.get_layer('dense_2').get_weights()
You could go though the layers of your model and retrieve its name and weights:
{layer.name: layer.get_weights() for layer in model.layers}
I am working in python and tensor flow but I miss 'units' argument and I do not know how to solve it, It looks like your post is mostly code; please add some more details.It looks like your post is mostly code; please add some more details.
here the code
def createModel():
model = Sequential()
# first set of CONV => RELU => MAX POOL layers
model.add(Conv2D(32, (3, 3), padding='same', activation='relu', input_shape=inputShape))
model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(output_dim=NUM_CLASSES, activation='softmax'))
# returns our fully constructed deep learning + Keras image classifier
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
# use binary_crossentropy if there are two classes
model.compile(loss="categorical_crossentropy", optimizer=opt, metrics=["accuracy"])
return model
print("Reshaping trainX at..."+ str(datetime.now()))
#print(trainX.sample())
print(type(trainX)) # <class 'pandas.core.series.Series'>
print(trainX.shape) # (750,)
from numpy import zeros
Xtrain = np.zeros([trainX.shape[0],HEIGHT, WIDTH, DEPTH])
for i in range(trainX.shape[0]): # 0 to traindf Size -1
Xtrain[i] = trainX[i]
print(Xtrain.shape) # (750,128,128,3)
print("Reshaped trainX at..."+ str(datetime.now()))
print("Reshaping valX at..."+ str(datetime.now()))
print(type(valX)) # <class 'pandas.core.series.Series'>
print(valX.shape) # (250,)
from numpy import zeros
Xval = np.zeros([valX.shape[0],HEIGHT, WIDTH, DEPTH])
for i in range(valX.shape[0]): # 0 to traindf Size -1
Xval[i] = valX[i]
print(Xval.shape) # (250,128,128,3)
print("Reshaped valX at..."+ str(datetime.now()))
# initialize the model
print("compiling model...")
sys.stdout.flush()
model = createModel()
# print the summary of model
from keras.utils import print_summary
print_summary(model, line_length=None, positions=None, print_fn=None)
# add some visualization
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot
SVG(model_to_dot(model).create(prog='dot', format='svg'))
Try changing this line:
model.add(Dense(output_dim=NUM_CLASSES, activation='softmax'))
to
model.add(Dense(NUM_CLASSES, activation='softmax'))
I'm not experience in keras but I could not find a parameter called output_dim on the documentation page for Dense. I think you meant to provide units but labelled it as output_dim
The Keras Dense layer documentation is as follows:
keras.layers.Dense(units, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)
Using the following :
classifier.add(Dense(6, activation='relu', kernel_initializer='glorot_uniform',input_dim=11))
Will work as here the units means the output_dim saying that we need 6 neurons in the hidden layer. The weights are initialized with the uniform function and the input layer has 11 independent variables of the dataset (input_dim) to feed the above-hidden layer.
I think it's a version issue. In updated version of keras for Dense there is no "output_dim" argument.
You can see this documentation link for Dense arguments.
https://keras.io/api/layers/core_layers/dense/
tf.keras.layers.Dense(
units,
activation=None,
use_bias=True,
kernel_initializer="glorot_uniform",
bias_initializer="zeros",
kernel_regularizer=None,
bias_regularizer=None,
activity_regularizer=None,
kernel_constraint=None,
bias_constraint=None,
**kwargs
)
So the first argument is "units", Which is mandatory.
instead of this line:
model.add(Dense(output_dim=NUM_CLASSES, activation='softmax'))
use this:
model.add(Dense(units=NUM_CLASSES, activation='softmax'))
or
model.add(Dense(NUM_CLASSES, activation='softmax'))
I created a CNN model using the cifar100 dataset from Keras. When adding the top_k_categorical_accuracy metric, I should be seeing the accuracy of when one of the top 5 predicted classes is the correct class. However, when training, top_k_categorical_accuracy stays very small, around 4-5%, as accuracy and validation accuracy increase all the way to 40-50%. Top 5 accuracy should be much higher than normal accuracy, instead its giving very odd results. I wrote my own metric using different k values but still the same issue. Even when I use k=1, which should then give the same value of accuracy, the same issue occurs.
Model code:
cnn = Sequential()
cnn.add(Conv2D(filters=200, kernel_size=2, padding='same', activation='relu', input_shape=(train_images.shape[1:])))
cnn.add(Conv2D(filters=200, kernel_size=2, padding='same', activation='relu'))
cnn.add(Conv2D(filters=200, kernel_size=2, padding='same', activation='relu'))
cnn.add(MaxPooling2D(pool_size=2, padding='same'))
cnn.add(Dropout(0.4))
cnn.add(Conv2D(filters=200, kernel_size=2, padding='same', activation='relu'))
cnn.add(Conv2D(filters=200, kernel_size=2, padding='same', activation='relu'))
cnn.add(Conv2D(filters=200, kernel_size=2, padding='same', activation='relu'))
cnn.add(Conv2D(filters=200, kernel_size=2, padding='same', activation='relu'))
cnn.add(Dropout(0.4))
cnn.add(MaxPooling2D(pool_size=2, padding='same'))
cnn.add(Dropout(0.5))
cnn.add(Flatten())
cnn.add(Dense(550, activation='relu'))
cnn.add(Dropout(0.4))
cnn.add(Dense(100, activation='softmax'))
Compile code:
cnn.compile(loss='sparse_categorical_crossentropy', optimizer=opt.Adam(lr=learn_rate), metrics=['accuracy', 'top_k_categorical_accuracy'])
Turns out, since I am using the sparse_categorical_crossentropy loss function I need to use the sparse_top_k_categorical_accuracy function. This metric also requires your labels be flattened to one dimension. After that, metric is correct and model is training!