I am using Keras with TensorFlow as backend.
Now i want to use the TensorBoard callback to visualize my conv layer kernels.
But i can only see the first conv layer kernel in TensorBoard and my Dense layers at the end.
For the other conv layers i can just the the bias values and not the kernels.
Here is my sample code for the Keras model.
tb = TensorBoard(
log_dir=log_dir,
histogram_freq=epochs,
write_images=True)
# Define the DNN
model = Sequential()
model.add(Conv2D(filters=16, kernel_size=3, input_shape=(width, height, depth), name="conv1"))
model.add(Activation("relu"))
model.add(Conv2D(filters=16, kernel_size=3, name="conv2"))
model.add(Activation("relu"))
model.add(MaxPool2D())
model.add(Conv2D(filters=32, kernel_size=3, name="conv3"))
model.add(Activation("relu"))
model.add(Conv2D(filters=32, kernel_size=3, name="conv4"))
model.add(Activation("relu"))
model.add(MaxPool2D())
model.add(Flatten())
model.add(Dense(128))
model.add(Activation("relu"))
model.add(Dense(num_classes, name="features"))
model.add(Activation("softmax"))
# Print the DNN layers
model.summary()
# Train the DNN
lr = 1e-3
optimizer = Adam(lr=lr)
model.compile(loss="categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
model.fit(x_train, y_train, verbose=1,
batch_size=batch_size, epochs=epochs,
validation_data=(x_test, y_test),
callbacks=[tb])
And this is what i see in TensorBoard.
(I minimized the Kernels of my first conv layer)
TB Screenshot
What am i missing to visulize all my kernels?
This is the expected (but not specified in the documentation) behaviour of the Tensorboard callback. See the answer on this related bug report of Tensorboard GitHub page:
The TensorBoard Keras callback calls tf.summary.image without
overriding the default for max_outputs, so there’s no way to visualize
more than the first 3 kernels via the callback at this time.
You need to visualize the kernels with your own call of the tf.summary.image.
Related
I am creating a LSTM model for human activity recognition and I keep always getting fluctuating but increasing Train and loss curves.
The following architecture gave these curves Train and loss curves :
model = Sequential()
model.add(LSTM(units = 128, return_sequences=True , input_shape=(3500, 11)))
model.add(Dropout(0.5))
model.add(Dense(units= 64, activation='relu'))
model.add(LSTM(units = 128, return_sequences=False , input_shape=(3500, 11)))
model.add(Dropout(0.5))
model.add(Dense(units= 64, activation='relu'))
model.add(Dense(4, activation='softmax'))
adam = tf.keras.optimizers.Adam(learning_rate=0.0020, beta_1=0.9, beta_2=0.999,
epsilon=None, decay=0.0, amsgrad=False, clipnorm=1.)
model.compile(optimizer=adam ,loss='categorical_crossentropy', metrics=['accuracy'])
history = model.fit(Gen, validation_data=val_Gen, epochs=30, callbacks=[tensorboard_callback],
verbose=1).history
I tried changing the models architecture with different hyperparameters but nothing improved.
I am using TimeSeriesGenerator from keras to generate batches.
Does anyone have a suggestion ?
In PyTorch the cross entropy loss function works something like
CrossEntropyLoss(x, y) = H(one_hot(y), softmax(x))
so you can have a linear output layer. Is there a way to do that with tf.keras.Sequential?
I have wirtten this little CNN for MNIST
model = tf.keras.Sequential()
model.add(tfkl.Input(shape=(28, 28, 1)))
model.add(tfkl.Conv2D(32, (5, 5), padding="valid", activation=tf.nn.relu))
model.add(tfkl.MaxPool2D((2, 2)))
model.add(tfkl.Conv2D(64, (5, 5), padding="valid", activation=tf.nn.relu))
model.add(tfkl.MaxPool2D((2, 2)))
model.add(tfkl.Flatten())
model.add(tfkl.Dense(1024, activation=tf.nn.relu))
model.add(tfkl.Dense(10, activation=tf.nn.softmax))
model.compile(optimizer="adam",
loss="sparse_categorical_crossentropy",
metrics=["accuracy"])
model.summary()
model.fit(x_train, y_train, epochs=1)
and I would like to have
model.add(tfkl.Dense(10))
as the last layer.
I am trying to implement the ADef algorithm but the entries of the gradient wrt. the input seem to be too small and I guess with a linear output they would be right.
I know there is tf.nn.softmax_cross_entropy_with_logits but I don't know how to use it in this context.
Edit:
Changing
model.compile(optimizer="adam",
loss="sparse_categorical_crossentropy",
metrics=["accuracy"])
to
model.compile(optimizer="adam",
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=["accuracy"])
has done the trick.
Thank you #Moe1234. For the benefit of community providing solution here
Issue was resolved after changing
model.compile(optimizer="adam",
loss="sparse_categorical_crossentropy",
metrics=["accuracy"])
to
model.compile(optimizer="adam",
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=["accuracy"])
I have an alexnet neural network that I wrote it from scratch using tensorflow and I used 6000 images as train_data.
but while training, the validation accuracy is not changing and it is greater than training accuracy, I guess it is overfitting. In addition, validation loss is increasing.
Is it possible to solve overfitting problem with 1000 data for saving time?
How can I prevent overfitting?
I attached my alexnet code below.
Thanks
def CreateModel():
model = Sequential()
# 1st Convolutional Layer
model.add(Conv2D(filters=96, input_shape=(227,227,3), kernel_size=(11,11), strides=(4,4), padding='valid'))
model.add(Activation('relu'))
# Max Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# 2nd Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(11,11), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Max Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# 3rd Convolutional Layer
model.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# 4th Convolutional Layer
model.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# 5th Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Max Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Passing it to a Fully Connected layer
model.add(Flatten())
# 1st Fully Connected Layer
model.add(Dense(4096, input_shape=(224*224*3,)))
model.add(Activation('relu'))
# Add Dropout to prevent overfitting
model.add(Dropout(0.5))
# 2nd Fully Connected Layer
model.add(Dense(4096))
model.add(Activation('relu'))
# Add Dropout
model.add(Dropout(0.5))
# 3rd Fully Connected Layer
model.add(Dense(1000))
model.add(Activation('relu'))
# Add Dropout
model.add(Dropout(0.5))
# Output Layer
model.add(Dense(2))
model.add(Activation('softmax'))
model.summary()
return model
alexNet_model = CreateModel()
alexNet_model.compile(loss='sparse_categorical_crossentropy' , optimizer='adam', metrics=["accuracy"])
batch_size = 4
epochs = 5
history = alexNet_model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1,
validation_data=(x_validation, y_validation))
To mitigate overfitting. You can try to implement below steps
1.Shuffle the Data, by using shuffle=True in alexNet_model.fit. Code is shown below:
history = alexNet_model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1,
validation_data=(x_validation, y_validation), shuffle = True)
2.Use Early Stopping. Code is shown below
callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=15)
3.Use Regularization. Code for Regularization is shown below (You can try l1 Regularization or l1_l2 Regularization as well):
from tensorflow.keras.regularizers import l2
Regularizer = l2(0.001)
alexNet_model.add(Conv2D(96,11, 11, input_shape = (227,227,3),strides=(4,4), padding='valid', activation='relu', data_format='channels_last',
activity_regularizer=Regularizer, kernel_regularizer=Regularizer))
alexNet_model.add(Dense(units = 2, activation = 'sigmoid',
activity_regularizer=Regularizer, kernel_regularizer=Regularizer))
4.You can try using BatchNormalization.
5.Perform Image Data Augmentation using ImageDataGenerator. Refer this link for more info about that.
6.If the Pixels are not Normalized, Dividing the Pixel Values with 255 also helps
I'm trying to build a 2 layered neural network for MNIST dataset and I want to get weights from my model.
I found a similar question her on SO and I tried this,
model.get_weights()
But It returned 11 values when I check the len(model.get_weights()) Isn't it suppose to return 3 weights? I have even disabled bias.
model = Sequential()
model.add(Flatten(input_shape = (28, 28)))
model.add(Dense(512, activation='relu', kernel_initializer='he_normal', use_bias=False,))
model.add(BatchNormalization())
model.add(Dropout(0.3))
model.add(Dense(128, activation='relu', kernel_initializer='he_normal', use_bias=False,))
model.add(BatchNormalization())
model.add(Dropout(0.1))
model.add(Dense(10, activation='relu', kernel_initializer='he_normal', use_bias=False,))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
result = model.fit(x_train, y_train, validation_split=0.25, epochs=10,
batch_size=128, verbose=1)
To get the weights of a particular layer, you could retrieve this layer by using its name and call get_weights on it (as shubham-panchal said in its comment).
For example:
model.get_layer('dense').get_weights()
or
model.get_layer('dense_2').get_weights()
You could go though the layers of your model and retrieve its name and weights:
{layer.name: layer.get_weights() for layer in model.layers}
I am experimenting with the Merku molecular activity challenge and I have created the train and test dataset.
The shape of the data is the following:
x_train.shape=(1452, 4306)
y_train.shape=(1452, 1)
x_test.shape=(363, 4306)
y_test.shape=(363, 1)
I have used the Dense layer for defining the model as follows:
model = Sequential()
model.add(Dense(100, activation="relu", input_shape=(4306,)))
model.add(Dense(50, activation="relu"))
model.add(Dense(25, activation="relu"))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(1))
# Compile the model
model.compile(
loss='categorical_crossentropy',
optimizer="adam",
)
model.summary()
# Train the model
model.fit(
x_train,
y_train,
batch_size=300,
epochs=900,
validation_data=(x_test, y_test),
shuffle=True
)
While trying the above code, the following error occurred:
ValueError: Input 0 is incompatible with layer flatten_23: expected min_ndim=3, found ndim=2
How can I resolve this error?
Just remove the flatten layer:
model = Sequential()
model.add(Dense(100, activation="relu", input_shape=(4306,)))
model.add(Dense(50, activation="relu"))
model.add(Dense(25, activation="relu"))
model.add(Dropout(0.25))
model.add(Dense(1))
The data sent to sequential layers is essentially 1-D (ignoring the batch column) so there's nothing to flatten. The data entering the flatten layer is already 1D.
EDIT -- for regression:
Categorical crossentropy is not an appropriate cost function for regression, you need to use mean-square error, which is commonly used for all regression tasks:
model.compile(
loss='mse',
optimizer="adam",
)