I'm relatively new to keras/CNNs. I've built a model that concatenates the output from 3 sequential CNNs with some other metadata into a final Dense network. The outputs from the 3 individual layers are sensible and perform ok, but the final output when concatenated with the meta data is showing worse performance and doesn't seem to learn - even though some of this meta data should be very useful for prediction. The labels are one-hot coded classification data (4 different labels). I'm a bit confused why the final concatenated model is performing so poorly compared to the individual pieces, would appreciate any insight into what I might be doing wrong here. Thanks!
# create first conv layers
first_input = Input(shape=input_shape, dtype='int32', name='first_input')
x = Embedding(input_dim=num_features,output_dim=embedding_dim,input_length=input_shape[0])(first_input)
#x = Dropout(rate = dropout_rate)(x)
x = Conv1D(filters=filters,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = Conv1D(filters=filters,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = MaxPooling1D(pool_size=pool_size)(x)
x = Conv1D(filters=filters * 2,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = Conv1D(filters=filters * 2,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = GlobalAveragePooling1D()(x)
aux_predictions = Dense(op_units, activation=op_activation)(x)
# now create a convolutional model for second
second_input = Input(shape=input_shape, dtype='int32', name='second_input')
x = Embedding(input_dim=num_features,output_dim=embedding_dim,input_length=input_shape[0])(second_input)
#x = Dropout(rate = dropout_rate)(x)
x = Conv1D(filters=filters,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = Conv1D(filters=filters,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = MaxPooling1D(pool_size=pool_size)(x)
x = Conv1D(filters=filters * 2,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = Conv1D(filters=filters * 2,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = GlobalAveragePooling1D()(x)
aux_predictions2 = Dense(op_units, activation=op_activation)(x)
# now create a convolutional model for second
third_input = Input(shape=input_shape, dtype='int32', name='third_input')
x = Embedding(input_dim=num_features,output_dim=embedding_dim,input_length=input_shape[0])(third_input)
#x = Dropout(rate = dropout_rate)(x)
x = Conv1D(filters=filters,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = Conv1D(filters=filters,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = MaxPooling1D(pool_size=pool_size)(x)
x = Conv1D(filters=filters * 2,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = Conv1D(filters=filters * 2,
kernel_size=kernel_size,
strides = 1,
activation='relu',
bias_initializer='random_uniform',
padding='same')(x)
x = GlobalAveragePooling1D()(x)
aux_predictions3 = Dense(op_units, activation=op_activation)(x)
# Now combine three CNN layers with metadata
auxiliary_input = Input(shape=metadata_dim, name='aux_input')
x = keras.layers.concatenate([aux_predictions, aux_predictions2, aux_predictions3, auxiliary_input ])
#x = Dropout(rate = dropout_rate)(x)
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)
model_output = Dense(op_units, activation=op_activation, name='model_output')(x)
opt = SGD(lr=0.01)
fullmetamodel = Model(inputs=[first_input, second_input, third_input, auxiliary_input], outputs=[aux_predictions, aux_predictions2, aux_predictions3, model_output])
fullmetamodel.compile(
metrics=['categorical_accuracy'], loss='categorical_crossentropy',
loss_weights=[0.2, 0.2, 0.2, 1.], optimizer = opt)
callbacks = [keras.callbacks.EarlyStopping(monitor='val_loss', patience=2), TQDMNotebookCallback(leave_inner=False, leave_outer = True)]
fullmetamodel.fit(x=[first_x_train, second_x_train, third_x_train, training_meta], y=[training_labels,training_labels,training_labels, training_labels],
batch_size=32, epochs=40, validation_data=([first_x_val, second_x_val, third_x_val, val_meta], [val_labels, val_labels, val_labels, val_labels])
, verbose = 0, callbacks = callbacks) # starts training
# Output, three conv layers working ok, concatenated model performing poorly
Training
50% 20/40 [2:49:34<2:49:23, 508.20s/it]
Epoch 20
[loss: 8.002, dense_118_loss: 0.749, dense_119_loss: 0.769, dense_120_loss: 0.876, model_output_loss: 7.523, dense_118_categorical_accuracy: 0.686
, dense_119_categorical_accuracy: 0.626, dense_120_categorical_accuracy: 0.620, model_output_categorical_accuracy: 0.532] : 66% 265184/400000 [05:13<02:40, 840.90it/s]
Related
I am trying to online-train a neural network. I want to use the Tensorflow Keras train_on_batch function on a convolutional neural network. Here it is:
look_back=1600
inputTensor = keras.layers.Input([look_back+3,2])
inputTensorReshaped = tf.reshape(inputTensor, [1, look_back + 3, 2, 1])
#split into 2 groups
inputgroup1 = keras.layers.Lambda(lambda x: x[:, :3], output_shape=((1, 3, 2, 1)))(inputTensorReshaped)
inputgroup2 = keras.layers.Lambda(lambda x: x[:, 3:look_back + 3], output_shape=((1, look_back,2, 1)))(inputTensorReshaped)
conv1 = keras.layers.Conv2D(filters=1024, kernel_size=(10, 2), activation='relu')(inputgroup2)#10
pool1 = keras.layers.MaxPooling2D(pool_size=(2, 1))(conv1)
dropout1 = keras.layers.Dropout(rate=0.1)(pool1)
norm1 = keras.layers.LayerNormalization()(dropout1)
conv2 = keras.layers.Conv2D(filters=512, kernel_size=(8, 1), activation='relu')(norm1)
pool2 = keras.layers.MaxPooling2D(pool_size=(2, 1))(conv2)
dropout2 = keras.layers.Dropout(rate=0.1)(pool2)
norm2 = keras.layers.LayerNormalization()(dropout2)
conv3 = keras.layers.Conv2D(filters=256, kernel_size=(6, 1), activation='relu')(norm2)
pool3 = keras.layers.MaxPooling2D(pool_size=(2, 1))(conv3)
dropout3 = keras.layers.Dropout(rate=0.1)(pool3)
norm3 = keras.layers.LayerNormalization()(dropout3)
conv4 = keras.layers.Conv2D(filters=128, kernel_size=(4, 1), activation='relu')(norm3)
pool4 = keras.layers.MaxPooling2D(pool_size=(2, 1))(conv4)
dropout4 = keras.layers.Dropout(rate=0.1)(pool4)
norm4 = keras.layers.LayerNormalization()(dropout4)
conv5 = keras.layers.Conv2D(filters=64, kernel_size=(2, 1), activation='relu')(norm4)
pool5 = keras.layers.MaxPooling2D(pool_size=(2, 1))(conv5)
dropout5 = keras.layers.Dropout(rate=0.1)(pool5)
norm5 = keras.layers.LayerNormalization()(dropout5)
flatten1 = keras.layers.Flatten()(norm5)
dense1 = keras.layers.Dense(32, activation='relu')(flatten1)
misclayer1 = keras.layers.Dense(32, activation='relu')(inputgroup1)
miscdropout1 = keras.layers.Dropout(rate=0.1)(misclayer1)
miscnorm1 = keras.layers.LayerNormalization()(miscdropout1)
misclayer2 = keras.layers.Dense(128, activation='relu')(miscnorm1)
miscdropout2 = keras.layers.Dropout(rate=0.1)(misclayer2)
miscnorm2 = keras.layers.LayerNormalization()(miscdropout2)
misclayer3 = keras.layers.Dense(32, activation='relu')(miscnorm2)
miscdropout3 = keras.layers.Dropout(rate=0.1)(misclayer3)
miscnorm3 = keras.layers.LayerNormalization()(miscdropout3)
miscflatten1 = keras.layers.Flatten()(miscnorm3)
misclayer4 = keras.layers.Dense(32, activation='relu')(miscflatten1)
rejoinlayer = keras.layers.Concatenate()([dense1, misclayer4])
processing1 = keras.layers.Dense(64, activation='relu')(rejoinlayer)
totalnorm1 = keras.layers.LayerNormalization()(processing1)
processing2 = keras.layers.Dense(32, activation='relu')(totalnorm1)
totaldropout1 = keras.layers.Dropout(rate=0.2)(processing2)
processing3 = keras.layers.Dense(16, activation='relu')(totaldropout1)
totalnorm2 = keras.layers.LayerNormalization()(processing3)
processing4 = keras.layers.Dense(8, activation='relu')(totalnorm2)
totaldropout2 = keras.layers.Dropout(rate=0.2)(processing4)
processing5 = keras.layers.Dense(4, activation='relu')(totaldropout2)
output = keras.layers.Dense(1, activation='linear')(processing5)
model = keras.Model(inputTensor,output)
model.compile(optimizer=keras.optimizers.SGD(learning_rate=0.00005, momentum=0.1, nesterov=True), loss="mean_squared_error")
#trains the model with the 1st state, action, and value
def train():
global qtable
x = []
y = []
for i in range(0, 8):
state = qtable.loc[qtable.index[i], "state"]
action = [qtable.loc[qtable.index[i], "action"], qtable.loc[qtable.index[0], "action"]]
x.append([action])
x[i].extend(state)
y.append([qtable.loc[qtable.index[i], "value"]])
print("training...loss:")
with tf.device('/gpu:0'):
print(model.train_on_batch(np.nan_to_num(np.array(x)), np.nan_to_num(np.array(y))))
In this case the variable "state" would be a 1202-by-2 list [[a,b],[c,d],[e,f],...] and the variable "action" would be a 1-by-2 list [a,b] before being appended/extended to x. In theory, the training I want is a batch size of 8 with a 1203-by-2 input shape. However, I get this error:
ValueError: Cannot reshape a tensor with 19248 elements to shape [1,1203,2,1] (2406 elements) for '{{node model/tf.reshape/Reshape}} = Reshape[T=DT_FLOAT, Tshape=DT_INT32](IteratorGetNext, model/tf.reshape/Reshape/shape)' with input shapes: [8,1203,2], [4] and with input tensors computed as partial shapes: input[1] = [1,1203,2,1].
This shows that all the inputs and outputs are being put into the CNN at once which is not what I want. Instead, I want the data to be in a batch of 8. How can I do this??? Am I even using "train_on_batch" correctly
batch_size:: Integer or None. Number of samples per batch of
computation. If unspecified, batch_size will default to 32. Do not
specify the batch_size if your data is in the form of a dataset,
generators, or keras.utils.Sequence instances (since they generate
batches).
Find the below example with batch_size
num_classes = 5
model = Sequential([
layers.experimental.preprocessing.Rescaling(1./255, input_shape=(img_height, img_width, 3)),
layers.Conv2D(16, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Conv2D(32, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Conv2D(64, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
layers.Flatten(),
layers.Dense(128, activation='relu'),
layers.Dense(num_classes)
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(
train_ds,
validation_data=val_ds,
batch_size = 32
epochs=10
)
Packages Version:
Tensorflow==2.5
Python==3.8
Keras==2.3
Here is the code:
# Pipe Line
(x_train, y_train), (x_test, y_test), (x_val, y_val) = (X_train, Y_train), (X_test, Y_test), (X_val, Y_val)
def model_seg():
# Convolution Layers (BatchNorm after non-linear activation)
img_input = Input(shape= (192, 256, 3))
x = Conv2D(16, (3, 3), padding='same', name='conv1')(img_input)
x = BatchNormalization(name='bn1')(x)
x = Activation('relu')(x)
x = Conv2D(32, (3, 3), padding='same', name='conv2')(x)
x = BatchNormalization(name='bn2')(x)
x = Activation('relu')(x)
x = MaxPooling2D()(x)
x = Conv2D(64, (3, 3), padding='same', name='conv3')(x)
x = BatchNormalization(name='bn3')(x)
x = Activation('relu')(x)
x = Conv2D(64, (3, 3), padding='same', name='conv4')(x)
x = BatchNormalization(name='bn4')(x)
x = Activation('relu')(x)
x = MaxPooling2D()(x)
x = Conv2D(128, (3, 3), padding='same', name='conv5')(x)
x = BatchNormalization(name='bn5')(x)
x = Activation('relu')(x)
x = Conv2D(128, (4, 4), padding='same', name='conv6')(x)
x = BatchNormalization(name='bn6')(x)
x = Activation('relu')(x)
x = MaxPooling2D()(x)
x = Conv2D(256, (3, 3), padding='same', name='conv7')(x)
x = BatchNormalization(name='bn7')(x)
x = Dropout(0.5)(x)
x = Activation('relu')(x)
x = Conv2D(256, (3, 3), padding='same', name='conv8')(x)
x = BatchNormalization(name='bn8')(x)
x = Activation('relu')(x)
x = MaxPooling2D()(x)
x = Conv2D(512, (3, 3), padding='same', name='conv9')(x)
x = BatchNormalization(name='bn9')(x)
x = Activation('relu')(x)
x = Dense(1024, activation = 'relu', name='fc1')(x)
x = Dense(1024, activation = 'relu', name='fc2')(x)
# Deconvolution Layers (BatchNorm after non-linear activation)
x = Conv2DTranspose(256, (3, 3), padding='same', name='deconv1')(x)
x = BatchNormalization(name='bn19')(x)
x = Activation('relu')(x)
x = UpSampling2D()(x)
x = Conv2DTranspose(256, (3, 3), padding='same', name='deconv2')(x)
x = BatchNormalization(name='bn12')(x)
x = Activation('relu')(x)
x = Conv2DTranspose(128, (3, 3), padding='same', name='deconv3')(x)
x = BatchNormalization(name='bn13')(x)
x = Activation('relu')(x)
x = UpSampling2D()(x)
x = Conv2DTranspose(128, (4, 4), padding='same', name='deconv4')(x)
x = BatchNormalization(name='bn14')(x)
x = Activation('relu')(x)
x = Conv2DTranspose(128, (3, 3), padding='same', name='deconv5')(x)
x = BatchNormalization(name='bn15')(x)
x = Activation('relu')(x)
x = UpSampling2D()(x)
x = Conv2DTranspose(64, (3, 3), padding='same', name='deconv6')(x)
x = BatchNormalization(name='bn16')(x)
x = Activation('relu')(x)
x = Conv2DTranspose(32, (3, 3), padding='same', name='deconv7')(x)
x = BatchNormalization(name='bn20')(x)
x = Activation('relu')(x)
x = UpSampling2D()(x)
x = Conv2DTranspose(16, (3, 3), padding='same', name='deconv8')(x)
x = BatchNormalization(name='bn17')(x)
x = Dropout(0.5)(x)
x = Activation('relu')(x)
x = Conv2DTranspose(1, (3, 3), padding='same', name='deconv9')(x)
x = BatchNormalization(name='bn18')(x)
x = Activation('sigmoid')(x)
pred = Reshape((192,256))(x)
model = Model(inputs=img_input, outputs=pred)
model.compile(optimizer= Adam(lr = 0.003), loss= [jaccard_distance], metrics=[iou])
hist = model.fit(x_train, y_train, epochs= 300, batch_size= 16,validation_data=(x_test, y_test), verbose=1)
model.save("model.h5")
accuracy = model.evaluate(x=x_test,y=y_test,batch_size=16)
print("Accuracy: ",accuracy[1])
Gives me this error in type conversion and I don't know how to fix it:
return gen_math_ops.mul(x, y, name)
D:\road-damage\road-damage-detection\rdd\lib\site-packages\tensorflow\python\ops\gen_math_ops.py:6248 mul
_, _, _op, _outputs = _op_def_library._apply_op_helper(
D:\road-damage\road-damage-detection\rdd\lib\site-packages\tensorflow\python\framework\op_def_library.py:555 _apply_op_helper
raise TypeError(
TypeError: Input 'y' of 'Mul' Op has type float32 that does not match type uint8 of argument 'x'.
Somewhere in your code, there's a tensor that is floats and a tensor that is integers, and it's not sure which one the result should be.
The architecture of your network does not tell us much; it is most likely to do with the way your data is being prepared.
If it is OK to treat your X and Y as floats, try explicitly converting them to floats like this before passing them to fit:
x_train = x_train.astype(np.float)
x_test = x_test.astype(np.float)
y_train = y_train.astype(np.float)
y_test = y_test.astype(np.float)
I'm trying to separate close objects as was shown in the U-Net paper (here). For this, one generates weight maps which can be used for pixel-wise losses. The following code describes the network I use from this blog post.
x_train_val = # list of images (imgs, 256, 256, 3)
y_train_val = # list of masks (imgs, 256, 256, 1)
y_weights = # list of weight maps (imgs, 256, 256, 1) according to the blog post
# visual inspection confirms the correct calculation of these maps
# Blog posts' loss function
def my_loss(target, output):
return - tf.reduce_sum(target * output,
len(output.get_shape()) - 1)
# Standard Unet model from blog post
_epsilon = tf.convert_to_tensor(K.epsilon(), np.float32)
def make_weighted_loss_unet(input_shape, n_classes):
ip = L.Input(shape=input_shape)
weight_ip = L.Input(shape=input_shape[:2] + (n_classes,))
conv1 = L.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(ip)
conv1 = L.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv1)
conv1 = L.Dropout(0.1)(conv1)
mpool1 = L.MaxPool2D()(conv1)
conv2 = L.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(mpool1)
conv2 = L.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv2)
conv2 = L.Dropout(0.2)(conv2)
mpool2 = L.MaxPool2D()(conv2)
conv3 = L.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(mpool2)
conv3 = L.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv3)
conv3 = L.Dropout(0.3)(conv3)
mpool3 = L.MaxPool2D()(conv3)
conv4 = L.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(mpool3)
conv4 = L.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv4)
conv4 = L.Dropout(0.4)(conv4)
mpool4 = L.MaxPool2D()(conv4)
conv5 = L.Conv2D(1024, 3, activation='relu', padding='same', kernel_initializer='he_normal')(mpool4)
conv5 = L.Conv2D(1024, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv5)
conv5 = L.Dropout(0.5)(conv5)
up6 = L.Conv2DTranspose(512, 2, strides=2, kernel_initializer='he_normal', padding='same')(conv5)
conv6 = L.Concatenate()([up6, conv4])
conv6 = L.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv6)
conv6 = L.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv6)
conv6 = L.Dropout(0.4)(conv6)
up7 = L.Conv2DTranspose(256, 2, strides=2, kernel_initializer='he_normal', padding='same')(conv6)
conv7 = L.Concatenate()([up7, conv3])
conv7 = L.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv7)
conv7 = L.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv7)
conv7 = L.Dropout(0.3)(conv7)
up8 = L.Conv2DTranspose(128, 2, strides=2, kernel_initializer='he_normal', padding='same')(conv7)
conv8 = L.Concatenate()([up8, conv2])
conv8 = L.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv8)
conv8 = L.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv8)
conv8 = L.Dropout(0.2)(conv8)
up9 = L.Conv2DTranspose(64, 2, strides=2, kernel_initializer='he_normal', padding='same')(conv8)
conv9 = L.Concatenate()([up9, conv1])
conv9 = L.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv9)
conv9 = L.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv9)
conv9 = L.Dropout(0.1)(conv9)
c10 = L.Conv2D(n_classes, 1, activation='softmax', kernel_initializer='he_normal')(conv9)
# Mimic crossentropy loss
c11 = L.Lambda(lambda x: x / tf.reduce_sum(x, len(x.get_shape()) - 1, True))(c10)
c11 = L.Lambda(lambda x: tf.clip_by_value(x, _epsilon, 1. - _epsilon))(c11)
c11 = L.Lambda(lambda x: K.log(x))(c11)
weighted_sm = L.multiply([c11, weight_ip])
model = Model(inputs=[ip, weight_ip], outputs=[weighted_sm])
return model
I then compile and fit the model as is shown below:
model = make_weighted_loss_unet((256, 256, 3), 1) # shape of input, number of classes
model.compile(optimizer='adam', loss=my_loss, metrics=['acc'])
model.fit([x_train_val, y_weights], y_train_val, validation_split=0.1, epochs=1)
The model can then train as usual. However, the loss doesn't seem to improve much. Furthermore, when I try to predict on new images, I obviously don't have the weight maps (because they are calculated on the labeled masks). I tried to use empty / zero arrays shaped like the weight map but that only yields in blank / zero predictions. I also tried different metrics and more standards losses without any success.
Did anyone face the same issue or have an alternative in implementing this weighted loss? Thanks in advance. BBQuercus
A simpler way to write custom loss with pixel weights
In your code, the loss is scattered around, between my_loss and make_weighted_loss_unet functions. You can add targets as an input and use model.add_loss to structure the code better :
def make_weighted_loss_unet(input_shape, n_classes):
ip = L.Input(shape=input_shape)
weight_ip = L.Input(shape=input_shape[:2] + (n_classes,))
targets = L.input(shape=input_shape[:2] + (n_classes,))
# .... rest of your model definition code ...
c10 = L.Conv2D(n_classes, 1, activation='softmax', kernel_initializer='he_normal')(conv9)
model.add_loss(pixel_weighted_cross_entropy(weights_ip, targets, c10))
# .... return Model .... NO NEED to specify loss in model.compile
def pixel_weighted_cross_entropy(weights, targets, predictions)
loss_val = keras.losses.categorical_crossentropy(targets, predictions)
weighted_loss_val = weights * loss_val
return K.mean(weighted_loss_val)
If you don't refactor your code to the above approach, next section shows how to still run inference without issues
How to run your model in inference
Option 1 : Use another Model object for inference
You can create a Model used for training and another used for inference. Both are largely the same except that the inference Model does not take weights_ip, and gives an early output c10.
Here's an example code that adds an argument is_training=True to decide which Model to return :
def make_weighted_loss_unet(input_shape, n_classes, is_training=True):
ip = L.Input(shape=input_shape)
conv1 = L.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(ip)
# .... rest of your model definition code ...
c10 = L.Conv2D(n_classes, 1, activation='softmax', kernel_initializer='he_normal')(conv9)
if is_training:
# Mimic crossentropy loss
c11 = L.Lambda(lambda x: x / tf.reduce_sum(x, len(x.get_shape()) - 1, True))(c10)
c11 = L.Lambda(lambda x: tf.clip_by_value(x, _epsilon, 1. - _epsilon))(c11)
c11 = L.Lambda(lambda x: K.log(x))(c11)
weight_ip = L.Input(shape=input_shape[:2] + (n_classes,))
weighted_sm = L.multiply([c11, weight_ip])
return Model(inputs=[ip, weight_ip], outputs=[weighted_sm])
else:
return Model(inputs=[ip], outputs=[c10])
return model
Option 2 : Use K.function
If you don't want to mess with your Model definition method (make_weighted_loss_unet) and want to achieve the same result outside, you can use a function that extracts the subgraph relevant for inference.
In your inference function:
from keras import backend as K
model = make_weighted_loss_unet(input_shape, n_classes)
inference_function = K.function([model.get_layer("input_layer").input],
[model.get_layer("output_softmax_layer").output])
predicted_heatmap = inference_function(new_image)
Note that you'll have to give name= to your ip layer and c10 layer to be able to retrieve them via model.get_layer(name) :
ip = L.Input(shape=input_shape, name="input_layer")
and
c10 = L.Conv2D(n_classes, 1, activation='softmax', kernel_initializer='he_normal', name="output_softmax_layer")(conv9)
I am training the following autoencoder on float numbers.
input_img = Input(shape=(2623,1), name='input')
x = ZeroPadding1D(1)(input_img)
x = Conv1D(32, 3, activation='relu', padding='same', use_bias=False)(input_img)
x = BatchNormalization(axis=-1)(x)
x = MaxPooling1D(2, padding='same')(x)
x = Conv1D(16, 3, activation='relu', padding='same', use_bias=False)(x)
x = BatchNormalization(axis=-1)(x)
x = MaxPooling1D(2, padding='same')(x)
x = Conv1D(16,3, activation='relu', padding='same', use_bias=False)(x)
x = BatchNormalization(axis=-1)(x)
encoded = MaxPooling1D(2, padding='same')(x)
x = Conv1D(16,3, activation='relu', padding='same', use_bias=False)(encoded)
x = BatchNormalization(axis=-1)(x)
x = UpSampling1D(2)(x)
x = Conv1D(16,3, activation='relu', padding='same', use_bias=False)(x)
x = BatchNormalization(axis=-1)(x)
x = UpSampling1D(2)(x)
x = Conv1D(32, 3, activation='relu', padding='same', use_bias=False)(x) #input_shape=(30, 1))
x = BatchNormalization(axis=-1)(x)
x = UpSampling1D(2)(x)
x = Cropping1D(cropping=(0, 1))(x) #Crop nothing from input but crop 1 element from the end
decoded = Conv1D(1, 3, activation='sigmoid', padding='same', use_bias=False)(x)
autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='rmsprop', loss='binary_crossentropy')
x = Input(shape=(16, 300), name="input")
h = x
h = Conv1D(filters=300, kernel_size=16,
activation="relu", padding='same', name='Conv1')(h)
h = MaxPooling1D(pool_size=16, name='Maxpool1')(h)
I had to convert the data to a numpy array in order to process it but when the model starts training i get:
ValueError: could not convert string to float:
This is happening because my training data looks like :
train[0][1] # one number of training data
array(['0.001758873'], dtype=object)
What can i do in order to avoid the "dtype=object" in my training data, or maybe do i have to convert it to something else?
Thank you!
Perhaps, as a preprocessing step, you can typecast your object type array to a float array using something like:
# if float32 is the desired & appropriate datatype
train = train.astype(numpy.float32)
I get the following binary classification Keras model, which trains not well, but trains:
def vgg_stack(self):
def func(x):
x = layers.Conv2D(64, (3, 3), activation='relu')(x)
x = layers.MaxPooling2D((3, 3), strides=(2, 2))(x)
x = layers.Conv2D(128, (3, 3), activation='relu')(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2))(x)
x = layers.Conv2D(128, (3, 3), activation='relu')(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2))(x)
x = layers.Conv2D(64, (3, 3), activation='relu')(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2))(x)
x = layers.Flatten()(x)
x = layers.Dense(512, activation='relu')(x)
x = layers.Dense(256, activation='relu')(x)
x = layers.Dense(1, activation='sigmoid')(x)
return x
return func
def implement(self):
self.inputs = layers.Input((self.input_width, self.input_height, self.input_depth))
self.outputs = self.vgg_stack()(self.inputs)
self.opt = optimizers.Adam(lr=self.learning_rate)
self.model = models.Model(inputs=self.inputs, outputs=self.outputs)
self.model.compile(loss='binary_crossentropy', optimizer=self.opt)
def fit_predict(self):
...
self.model.fit(data_train, actuals_train, batch_size=self.batch_size, epochs=10, verbose=1,
validation_data=[data_validation, actuals_validation], callbacks=[self])
it's predictions look like following
[[ 0.58952832]
[ 0.89163774]
[ 0.99083483]
...,
[ 0.52727282]
[ 0.72056866]
[ 0.99504411]]
I.e. it's something.
I tried to convert the model to pure tensroflow and got
def conv2drelu(self, x, filters, kernel_size, padding='VALID'):
input_depth = int(x.get_shape()[-1])
weights = tf.Variable(tf.truncated_normal([kernel_size[0], kernel_size[0], input_depth, filters],
dtype=tf.float32, stddev=self.init_stddev))
self.var_list.append(weights)
biases = tf.Variable(tf.constant(0.0, shape=[filters], dtype=tf.float32))
self.var_list.append(biases)
y = tf.nn.conv2d(x, weights, [1, 1, 1, 1], padding=padding)
y = tf.nn.bias_add(y, biases)
y = tf.nn.relu(y)
return y
def maxpooling(self, x, pool_size, strides, padding='VALID'):
y = tf.nn.max_pool(x, ksize=[1, pool_size[0], pool_size[1], 1], strides=[1, strides[0], strides[1], 1],
padding=padding)
return y
def flatten(self, x):
shape = int(np.prod(x.get_shape()[1:]))
y = tf.reshape(x, [-1, shape])
return y
def dense(self, x, units, activation):
shape = int(x.get_shape()[1])
weights = tf.Variable(tf.truncated_normal([shape, units], dtype=tf.float32, stddev=self.init_stddev))
self.var_list.append(weights)
biases = tf.Variable(tf.constant(0.0, shape=[units], dtype=tf.float32))
self.var_list.append(biases)
y = tf.matmul(x, weights)
y = tf.nn.bias_add(y, biases)
if activation == 'relu':
y = tf.nn.relu(y)
elif activation == 'sigmoid':
y = tf.nn.sigmoid(y)
return y
def vgg_stack(self, x):
x = self.conv2drelu(x, 64, (3, 3))
x = self.maxpooling(x, (3, 3), strides=(2, 2))
x = self.conv2drelu(x, 128, (3, 3))
x = self.maxpooling(x, (2, 2), strides=(2, 2))
x = self.conv2drelu(x, 128, (3, 3))
x = self.maxpooling(x, (2, 2), strides=(2, 2))
x = self.conv2drelu(x, 64, (3, 3))
x = self.maxpooling(x, (2, 2), strides=(2, 2))
x = self.flatten(x)
x = self.dense(x, 512, activation='relu')
x = self.dense(x, 256, activation='relu')
x = self.dense(x, 1, activation='sigmoid')
return x
def implement(self):
self.var_list = []
self.input_data = tf.placeholder(tf.float32, shape=(None, self.width, self.height, self.depth))
self.prediction = self.vgg_stack(self.input_data)
self.actual = tf.placeholder(tf.float32, shape=(None, 1))
self.log_loss = tf.losses.log_loss(self.actual, self.prediction)
opt = tf.train.AdamOptimizer(learning_rate=self.learning_rate)
# self.step = opt.minimize(self.mean_squared_error, var_list=self.var_list)
self.step = opt.minimize(self.log_loss, var_list=self.var_list)
i.e. I tries to write functions equivalent to each Keras layer and then combine them into the same structure.
I used all the same numbers. Unfortunately, network provides something degraded:
[[ 0.46732453]
[ 0.46732453]
[ 0.46732453]
...,
[ 0.46732453]
[ 0.46732453]
[ 0.46732453]]
I.e. the same values for all samples.
What can be the reason of this?
Conversion was correct. I wrote unittests for convolution layers from Keras and Tensorflow and found they produce numerically identical results.
Additionally, I replaced optimization goal from just log-loss to sigmoid_cross_entropy_with_logits but this didn't helped alone.
The problem was with too small stdev of initialization values.
I was thinking it is enough to have it very small to break symmetry, and was setting it to 1e-8 or 1e-5, but this was wrong: such small values were nearly identical to zeros and after several layers network was starting to produce identical results for all samples.
After I changed stdev to 1e-1, then netwrok started to perfor as in Keras.