I want to build a network that should be able to verificate images (e.g. human faces). As I understand, that the best solution for that is Siamese network with a triplet loss. I didn't found any ready-made implementations, so I decided to create my own.
But I have question about Keras. For example, here's the structure of the network:
And the code is something like that:
embedding = Sequential([
Flatten(),
Dense(1024, activation='relu'),
Dense(64),
Lambda(lambda x: K.l2_normalize(x, axis=-1))
])
input_a = Input(shape=shape, name='anchor')
input_p = Input(shape=shape, name='positive')
input_n = Input(shape=shape, name='negative')
emb_a = embedding(input_a)
emb_p = embedding(input_p)
emb_n = embedding(input_n)
out = Concatenate()([emb_a, emb_p, emp_n])
model = Model([input_a, input_p, input_n], out)
model.compile(optimizer='adam', loss=<triplet_loss>)
I defined only one embedding model. Does this mean that once the model starts training weights would be the same for each input?
If it is, how can I extract embedding weights from the model?
Yes, In triplet loss function weights should be shared across all three networks, i.e Anchor, Positive and Negetive.
In Tensorflow 1.x to achieve weight sharing you can use reuse=True in tf.layers.
But in Tensorflow 2.x since the tf.layers has been moved to tf.keras.layers and reuse functionality has been removed.
To achieve weight sharing you can write a custom layer that takes the parent layer and reuses its weights.
Below is the sample example to do the same.
class SharedConv(tf.keras.layers.Layer):
def __init__(
self,
filters,
kernel_size,
strides=None,
padding=None,
dilation_rates=None,
activation=None,
use_bias=True,
**kwargs
):
self.filters = filters
self.kernel_size = kernel_size
self.strides = strides
self.padding = padding
self.dilation_rates = dilation_rates
self.activation = activation
self.use_bias = use_bias
super().__init__(*args, **kwargs)
def build(self, input_shape):
self.conv = Conv2D(
self.filters,
self.kernel_size,
padding=self.padding,
dilation_rate=self.dilation_rates[0]
)
self.net1 = Activation(self.activation)
self.net2 = Activation(self.activation)
def call(self, inputs, **kwargs):
x1 = self.conv(inputs)
x1 = self.act1(x1)
x2 = tf.nn.conv2d(
inputs,
self.conv.weights[0],
padding=self.padding,
strides=self.strides,
dilations=self.dilation_rates[1]
)
if self.use_bias:
x2 = x2 + self.conv.weights[1]
x2 = self.act2(x2)
return x1, x2
I will answer on how to extract the embeddings (reference from my Github post):
My trained siamese model looked like this:
siamese_model.summary()
Note that my newly redefined model is basically the same as the one highlighted in yellow
I then redefined my model which I wanted to use for extracting embeddings (It should be the same model you defined except now it will not have those multiple inputs like siamese) which looked like this:
siamese_embeddings_model = build_siamese_model(input_shape)
siamese_embeddings_model .summary()
Then I just extracted the weights from my trained siamese model and set them into my new model
embeddings_weights = siamese_model.layers[-3].get_weights()
siamese_embeddings_model.set_weights(embeddings_weights )
Then you can supply the new Image to extract the embeddings from the new model
vector = siamese.predict(image)
len(vector[0]) it will print 150 because of my fine dense layer (which are the output vector)
Related
Background:
I am trying to use an HW classification engine that performs as follows:
Input is 12x12 image
On each 4x4 block (there are 3x3 blocks) it is applying a transform of the format Out4x4=Weight4x4 x In4x4 x Weight4x4' (a matrix representation of 4x4 transform) - by default, it is using a dct4x4 but this kernel is programable
As a result, there are 3 x 3 x 16 "dct coefficients", each 3x3 are weighted averaged to generate 16 "dct coefficients"
A complex classification process is deciding which of the 2 classes this image belongs to.
What I tried:
I wanted to find the "best" transform instead of using the default DCT.
I tried creating a CNN using Keras and Tensorflow that will simulate the feature extraction process. I tried reading examples and looking at the Keras code of DepthwiseConv2D, for example, and created the following customized Keras layer :
class ChildDenseTensor(keras.layers.Layer):
def __init__(self, units, activation=None):
super().__init__()
self.units = units
self.activation = activation
def get_config(self):
config = super().get_config()
config.update({
"units": self.units,
"activation": self.activation,
})
return config
def build(self, input_shape):
input_dim = input_shape[-1]
self.W = self.add_weight(shape=(1, 4,4, self.units), initializer='random_normal')
def call(self, inputs):
input_dim = inputs.shape[0]
if input_dim is not None:
input_dim=input_dim
else:
input_dim =1
out = tf.zeros((input_dim,12,12))
tmp_img_ts = tf.reshape(tf.convert_to_tensor(()), (0, 0, 0))
for img in range(0, input_dim): #quite a lot of work here to get the correct formating
for i in [0,4,8]:
for j in [0,4,8]:
y = tf.matmul(tf.matmul(tf.reshape(self.W[0],(4,4)), tf.reshape(inputs[img,i:(i+4),j:(j+4)],(4,4))), tf.reshape(self.W,(4,4)), transpose_b=True)
if j==0:
dim0 = tf.reshape(y,[1,16])
else:
dim0=tf.concat([dim0, tf.reshape(y,[1,16])],0)
if i==0:
dim1=tf.reshape(dim0,[1,3,16])
else:
dim1 = tf.concat([dim1, tf.reshape(dim0,[1,3,16])],0)
if img==0:
dim2= tf.reshape(dim1,[1,3,3,16])
else:
dim2 = tf.concat([dim2, tf.reshape(dim1,[1,3,3,16])],0)
return dim2
and built the following model
model = tf.keras.Sequential([
layers.Input((12,12,1)),
ChildDenseTensor(units=1, activation=tf.nn.relu), #, input_shape=(12,12,1) output_shape=(3,3,16)
tf.keras.layers.MaxPooling2D( pool_size=(3,3)),# -> (1,1,16) #averaging the "coefficients", I also tried 3x3 conv averages
tf.keras.layers.Conv2D(hparams[HP_FILTER_NUM], 1, activation='relu'),# classification layers - I also tried more layers and various number of filters but with no substential improvement
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(2)
])
I tried also more complex models but in all of them, the model training is very slow and does not converge to reasonable results.
My questions:
Am I required to produce the gradients somehow? I did not see that this was done in the examples or the Keras code but if I should do it then how and if I don't need then is it calculating it by some approximation?
Is there an easier solution to my problem? Is there something incorrect in my implementation?
The below code is just pseudo-code
I have this network architecture which is composed of multiple subclassed Keras.models. The basic layout of the entire network architecture is as follows:
class Main(tf.keras.Model):
def __init__(self, block):
super(Main, self).__init__(dynamic=True)
self.conv_1 = Conv2D(ncOut, kernel_size=(3,3), strides=(1,1),
padding="same", use_bias=False)
self.conv_2 = Conv2D(ncOut, kernel_size=(3,3), strides=(1,1),
padding="same", use_bias=False)
self.block1 = block(...)
self.block2 = block(next = block1)
self.block3 = block(next = block2)
def call(self, inputs):
x = inputs[0]
out = self.conv_1(x)
self.block3(out)
.
.
return out
class Block(tf.keras.Model):
def __init__(self, next):
super(Block, self).__init__(dynamic=True)
self.conv_1 = Conv2D(ncOut, kernel_size=(3,3), strides=(1,1),
padding="same", use_bias=False)
self.conv_2 = Conv2D(ncOut, kernel_size=(3,3), strides=(1,1),
padding="same", use_bias=False')
self.next = next
def call(self, inputs):
x = inputs[0]
out = self.conv_1(x)
out = self.next(out)
.
.
return out
This should get the gist of the architecture.
I.e, we have a subclassed model which creates entries other subclassed models that are then called.
After the network has trained, I save the weights. I then wish to make some changes to the network, i.e remove some layers or update the structure. Obviously, now it is no longer possible to load in the weights as there would be a mismatch.
What I then tried to do was give each layer a unique name (so during the creation of the blocks, I also pass a unique name). For the layers, like convolution and so forth, add a name argument with this unique name (plus an identifier for conv_1 and so on), in hopes that I would be able to load_weights(by_name=True).
Imagine that the structure is changed, I.e I removed conv_2 from the Block structure. If I then went to load the weights it would complain that layer # block expected ____ weights but received ___. Even if by_name=True is enabled. This tells me that it tries to load the weights for the entire block, and not specifically for each layer in this block, i.e the named convolution.
How would I load the weights, by the name, for each of the layers in a subsubclassed model after the structure of the network has been changed?
I ended up doing it by recursion.
def recursion(input, i = 0):
try:
if input.layers and not input.layers[i:]:
#we have exhausted the entire list
return
if input.layers:
recursion(input.layers[i], 0)
recursion(input, i+1)
except:
#print(input.name)
if input.name in layers_dict:
print(input.name)
#Take weights and set.
weights = input.get_weights()
layers_dict[input.name].set_weights(weights)
This seems to get the weights and set the weights for each layer as I wanted. For now, this works. If anyone has a better solution. Do not hesitate to suggest.
layers_dict is {"name_block_conv1": stripped_model.block.conv_1 ...}
Input: a model with weights
In the keras doc, it says that if we want to pick the intermediate layer's output of the model (sequential and functional), all we need to do as follows:
model = ... # create the original model
layer_name = 'my_layer'
intermediate_layer_model = keras.Model(inputs=model.input,
outputs=model.get_layer(layer_name).output)
intermediate_output = intermediate_layer_model(data)
So, here we get two models, the intermediate_layer_model is the sub-model of its parent model. And they're independent as well. Likewise, if we get the intermediate layer's output feature maps of the parent model (or base model), and do some operation with it and get some output feature maps from this operation, then we can also impute this output feature maps back to the parent model.
input = tf.keras.Input(shape=(size,size,3))
model = tf.keras.applications.DenseNet121(input_tensor = input)
layer_name = "conv1_block1" # for example
output_feat_maps = SomeOperationLayer()(model.get_layer(layer_name).output)
# assume, they're able to add up
base = Add()([model.output, output_feat_maps])
# bind all
imputed_model = tf.keras.Model(inputs=[model.input], outputs=base)
So, in this way we have one modified model. It's quite easy with functional API. All the keras imagenet models are written with functional API (mostly). In model subclassing API, we can use these models. My concern here is, what to do if we need the intermediate output feature maps of these functional API models' inside call function.
class Subclass(tf.keras.Model):
def __init__(self, dim):
super(Subclass, self).__init__()
self.dim = dim
self.base = DenseNet121(input_shape=self.dim)
# building new model with the desired output layer of base model
self.mid_layer_model = tf.keras.Model(self.base.inputs,
self.base.get_layer(layer_name).output)
def call(self, inputs):
# forward with base model
x = self.base(inputs)
# forward with mid_layer_model
mid_feat = self.mid_layer_model(inputs)
# do some op with it
mid_x = SomeOperationLayer()(mid_feat)
# assume, they're able to add up
out = tf.keras.layers.add([x, mid_x])
return out
The issue is, here we've technically two models in a joint fashion. But unlike building a model like this, here we simply want the intermediate output feature maps (from some inputs) of the base model forward manner and use it somewhere else and get some output. Like this
mid_x = SomeOperationLayer()(self.base.get_layer(layer_name).output)
But it gives ValueError: Graph disconnected. So, currently, we have to build a new model from the base model based on our desired intermediate layer. In the init method we define or create new self.mid_layer_model model that gives our desired output feature maps like this: mid_feat = self.mid_layer_model(inputs). Next, we take the mid_faet and do some operation and get some output and lastly add them with tf.keras.layers.add([x, mid_x]). So by creating a new model with desired intermediate out works but by the same time, we repeat the same operation twice i.e the base model and its subset model. Maybe I'm missing something obvious, please add up something. Is it how it is! or there some strategies we can adopt. I've asked in the forum here, no response yet.
Update
Here is a working example. Let's say we have a custom layer like this
import tensorflow as tf
from tensorflow.keras.applications import DenseNet121
from tensorflow.keras.layers import Add
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Flatten
class ConvBlock(tf.keras.layers.Layer):
def __init__(self, kernel_num=32, kernel_size=(3,3), strides=(1,1), padding='same'):
super(ConvBlock, self).__init__()
# conv layer
self.conv = tf.keras.layers.Conv2D(kernel_num,
kernel_size=kernel_size,
strides=strides, padding=padding)
# batch norm layer
self.bn = tf.keras.layers.BatchNormalization()
def call(self, input_tensor, training=False):
x = self.conv(input_tensor)
x = self.bn(x, training=training)
return tf.nn.relu(x)
And we want to impute this layer into an ImageNet model and construct a model like this
input = tf.keras.Input(shape=(32, 32, 3))
base = DenseNet121(weights=None, input_tensor = input)
# get output feature maps of at certain layer, ie. conv2_block1_0_relu
cb = ConvBlock()(base.get_layer("conv2_block1_0_relu").output)
flat = Flatten()(cb)
dense = Dense(1000)(flat)
# adding up
adding = Add()([base.output, dense])
model = tf.keras.Model(inputs=[base.input], outputs=adding)
from tensorflow.keras.utils import plot_model
plot_model(model,
show_shapes=True, show_dtype=True,
show_layer_names=True,expand_nested=False)
Here the computation from input to layer conv2_block1_0_relu is computed one time. Next, if we want to translate this functional API to subclassing API, we had to build a model from the base model's input to layer conv2_block1_0_relu first. Like
class ModelWithMidLayer(tf.keras.Model):
def __init__(self, dim=(32, 32, 3)):
super().__init__()
self.dim = dim
self.base = DenseNet121(input_shape=self.dim, weights=None)
# building sub-model from self.base which gives
# desired output feature maps: ie. conv2_block1_0_relu
self.mid_layer = tf.keras.Model(self.base.inputs,
self.base.get_layer("conv2_block1_0_relu").output)
self.flat = Flatten()
self.dense = Dense(1000)
self.add = Add()
self.cb = ConvBlock()
def call(self, x):
# forward with base model
bx = self.base(x)
# forward with mid layer
mx = self.mid_layer(x)
# make same shape or do whatever
mx = self.dense(self.flat(mx))
# combine
out = self.add([bx, mx])
return out
def build_graph(self):
x = tf.keras.layers.Input(shape=(self.dim))
return tf.keras.Model(inputs=[x], outputs=self.call(x))
mwml = ModelWithMidLayer()
plot_model(mwml.build_graph(),
show_shapes=True, show_dtype=True,
show_layer_names=True,expand_nested=False)
Here model_1 is actually a sub-model from DenseNet, which probably leads the whole model (ModelWithMidLayer) to compute the same operation twice. If this observation is correct, then this gives us concern.
I thought it might be much complex but it's actually rather very simple. We just need to build a model with desired output layers at the __init__ method and use it normally in the call method.
import tensorflow as tf
from tensorflow.keras.applications import DenseNet121
from tensorflow.keras.layers import Add
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Flatten
class ConvBlock(tf.keras.layers.Layer):
def __init__(self, kernel_num=32, kernel_size=(3,3), strides=(1,1), padding='same'):
super(ConvBlock, self).__init__()
# conv layer
self.conv = tf.keras.layers.Conv2D(kernel_num,
kernel_size=kernel_size,
strides=strides, padding=padding)
# batch norm layer
self.bn = tf.keras.layers.BatchNormalization()
def call(self, input_tensor, training=False):
x = self.conv(input_tensor)
x = self.bn(x, training=training)
return tf.nn.relu(x)
class ModelWithMidLayer(tf.keras.Model):
def __init__(self, dim=(32, 32, 3)):
super().__init__()
self.dim = dim
self.base = DenseNet121(input_shape=self.dim, weights=None)
# building sub-model from self.base which gives
# desired output feature maps: ie. conv2_block1_0_relu
self.mid_layer = tf.keras.Model(
inputs=[self.base.inputs],
outputs=[
self.base.get_layer("conv2_block1_0_relu").output,
self.base.output])
self.flat = Flatten()
self.dense = Dense(1000)
self.add = Add()
self.cb = ConvBlock()
def call(self, x):
# forward with base model
bx = self.mid_layer(x)[1] # output self.base.output
# forward with mid layer
mx = self.mid_layer(x)[0] # output base.get_layer("conv2_block1_0_relu").output
# make same shape or do whatever
mx = self.dense(self.flat(mx))
# combine
out = self.add([bx, mx])
return out
def build_graph(self):
x = tf.keras.layers.Input(shape=(self.dim))
return tf.keras.Model(inputs=[x], outputs=self.call(x))
mwml = ModelWithMidLayer()
tf.keras.utils.plot_model(mwml.build_graph(),
show_shapes=True, show_dtype=True,
show_layer_names=True,expand_nested=False)
We're trying to add a custom layer inside a pre-trained imagenet model. For a sequential or non-sequential model, we can easily do that. But here are some requirements.
First of all, we don't wanna disclose the whole imagenet model and deal with the desired inside layer. Let's say for DenseNet we need the following layers and further get the output shape of theirs to connect with some custom layers.
vision_model = tf.keras.applications.DenseNet121(
input_shape=(224,224,3),
include_top = False,
weights='imagenet')
for i, layer in enumerate(vision_model.layers):
if layer.name in ['conv3_block12_concat', 'conv4_block24_concat']:
print(i,'\t',layer.trainable,'\t :',layer.name)
if layer.name == 'conv3_block12_concat':
print(layer.get_output_shape_at(0)[1:]) # (28, 28, 512)
if layer.name == 'conv4_block24_concat':
print(layer.get_output_shape_at(0)[1:]) # (14, 14, 1024)
The whole requirement can be demonstrated as follows
The green indicator is basically the transition layer of the dense net.
In the above diagram, the dense net model has (let's say) 5 blocks and among them, we want to pick block 3 and block 4 and add some custom layers followed by merging them to lead the final output.
Also, the blocks of DenseNet (block 1 to 5), should be as disclose as possible with their pre-trained imagenet weights. We like to have control to freeze and unfreeze pre-trained layers when we need them.
How can we efficiently achieve with tf.keras? or, If you think there some better approach to do the same thing, please suggest.
Let's say, a custom block is something like this
class MLPBlock(tf.keras.layers.Layer):
def __init__(self, kernel_num=32, kernel_size=(3,3), strides=(1,1), padding='same'):
super(ConvModule, self).__init__()
# conv layer
self.conv = tf.keras.layers.Conv2D(kernel_num,
kernel_size=kernel_size,
strides=strides, padding=padding)
# batch norm layer
self.bn = tf.keras.layers.BatchNormalization()
def call(self, input_tensor, training=False):
x = self.conv(input_tensor)
x = self.bn(x, training=training)
return tf.nn.relu(x)
Motivation
I'm trying to implement this paper-work where they did something like this. Initially, the paper was free to get but now it's not. But below is the main block diagram of their approach.
I don't have access to the paper so I just build an example like the one your draw:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, models
class ConvBlock(layers.Layer):
def __init__(self, kernel_num=32, kernel_size=(3,3), strides=(1,1), padding='same'):
super(ConvBlock, self).__init__()
# conv layer
self.conv = layers.Conv2D(kernel_num,
kernel_size=kernel_size,
strides=strides, padding=padding)
# batch norm layer
self.bn = layers.BatchNormalization()
def call(self, input_tensor, training=False):
x = self.conv(input_tensor)
x = self.bn(x, training=training)
return tf.nn.relu(x)
vision_model = keras.applications.DenseNet121(
input_shape=(224,224,3),
include_top = False,
weights='imagenet')
# Control freeze and unfreeze over blocks
def set_freeze(block, unfreeze):
for layer in block:
layer.trainable = unfreeze
block_1 = vision_model.layers[:7]
block_2 = vision_model.layers[7:53]
block_3 = vision_model.layers[53:141]
block_4 = vision_model.layers[141:313]
block_5 = vision_model.layers[313:]
set_freeze(block_1, unfreeze=False)
set_freeze(block_2, unfreeze=False)
for i, layer in enumerate(vision_model.layers):
print(i,'\t',layer.trainable,'\t :',layer.name)
layer_names = ['conv3_block12_concat', 'conv4_block24_concat', 'conv5_block16_concat']
vision_model_outputs = [vision_model.get_layer(name).output for name in layer_names]
custom_0 = ConvBlock()(vision_model_outputs[0])
custom_1 = ConvBlock()(layers.UpSampling2D(2)(vision_model_outputs[1]))
cat_layer = layers.concatenate([custom_0, custom_1])
last_conv_num = 2
custom_2 = layers.UpSampling2D(4)(vision_model_outputs[2])
outputs = layers.concatenate([ConvBlock()(cat_layer) for i in range(last_conv_num)] + [custom_2])
model = models.Model(vision_model.input, outputs)
keras.utils.plot_model(model, "./Model_structure.png", show_shapes=True)
Run the code and you will see the block1 and block2 are frozen,
Because the plot of full model is long so I just post few snippet of it:
I'm trying to train a neural network in keras but I'm getting as error that there are no gradients for any variable, which may imply that the graph is disconnected.
I'm copying here a stripped down version of the code with only the bit related to the model definition.
The model accepts two inputs that will be fed, one at time, to the same shared model: the encoder.
The two outputs of the encoder are then concatenated and sent to a dense layer to compute the final output.
I don't get what's wrong, it looks like that when instantiating the encoder I'm creating additional trainable variables that are not used anywhere.
For the network layout I was getting inspiration from the official keras docs:
https://keras.io/guides/functional_api/#all-models-are-callable-just-like-layers
def _get_encoder(self, model_input_shape):
encoder_input = Input(shape=model_input_shape)
x = encoder_input
x = Conv2D(32, (3, 3), strides=1, padding="same")(x)
x = BatchNormalization(axis=-1)(x)
x = LeakyReLU(alpha=0.1)(x)
latent_z = Flatten()(x)
latent_z = Dense(self.latent_dim)(latent_z)
encoder = Model(
encoder_input,
latent_z,
name='encoder'
)
return encoder
def build_model(self):
model_input_shape = (self.height, self.width, self.depth)
model_input_1 = Input(shape=model_input_shape)
model_input_2 = Input(shape=model_input_shape)
self.encoder = self._get_encoder(model_input_shape)
z_1 = self.encoder(model_input_1)
z_2 = self.encoder(model_input_2)
x = concatenate([z_1, z_2])
prediction = Dense(1, activation='sigmoid')(x)
self.network = Model(
inputs=[model_input_1, model_input_2],
outputs=[prediction],
name = 'network'
)
network.network.compile(
optimizer='rmsprop',
loss='mse',
metrics=['mae'])
H = network.network.fit(
x=train_gen,
validation_data=test_gen,
epochs=EPOCHS,
steps_per_epoch=STEPS,
validation_steps=STEPS)
I found the problem. My custom data generator was returning a list [x,y] instead of a tuple (x,y). Where x is the input and y the target. A simple mistake that was causing totally unrelated errors.