We Keep Coding

TF2, Tensorflow Probability random seed generator and VAE

Playing around with Variational Autoencoders for some days. I am trying to fit a small toy function with a small model.
I first implemented the model using the Keras Functional API, with the following code:
def define_tfp_encoder(latent_dim, n_inputs=2, kl_weight=1):
prior = tfd.MultivariateNormalDiag(loc=tf.zeros(latent_dim))
input_x = Input((n_inputs,))
input_c = Input((1,))
dense = Dense(25, activation='relu', name='tfpenc/dense_1')(input_x)
dense = Dense(32, activation='relu', name='tfpenc/dense_2')(dense)
dense_z_params = Dense(tfpl.MultivariateNormalTriL.params_size(latent_dim), name='tfpenc/z_params')(dense)
dense_z = tfpl.MultivariateNormalTriL(latent_dim, name='tfpenc/z')(dense_z_params)
#activity_regularizer=tfpl.KLDivergenceRegularizer(prior) # weight=kl_weight
kld = tfpl.KLDivergenceAddLoss(prior, name='tfpenc/kld_add')(dense_z)
model = Model(inputs=input_x, outputs=kld)
return model
def define_tfp_decoder(latent_dim, n_inputs=2):
input_c = Input((1,), name='tfpdec/cond_input')
input_n = Input((latent_dim,))
dense = Dense(15, activation='relu', name='tfpdec/dense_1')(input_n)
dense = Dense(32, activation='relu', name='tfpdec/dense_2')(dense)
dense = Dense(tfpl.IndependentNormal.params_size(n_inputs), name='tfpdec/output')(dense)
output = tfpl.IndependentNormal((n_inputs,))(dense)
model = Model(input_n, output)
return model
def get_custom_unconditional_vae():
latent_size = 5
encoder = define_tfp_encoder(latent_dim=latent_size)
decoder = define_tfp_decoder(latent_dim=latent_size)
encoder.trainable = True
decoder.trainable = True
x = encoder.input
z = encoder.output
out = decoder(z)
vae = Model(inputs=x, outputs=out)
vae.compile(loss=lambda x, pred: -pred.log_prob(x), optimizer='adam')
return encoder, decoder, vae
The vae-model was then fitted and trained on 3000 epochs.
However, it only produced garbage for a very simple quadratic function to fit.
Now it comes:
When creating the exact same model using the sequential API it works as expected and the desired function gets approximated nicely:
And it becomes even stranger for me:
After running tf.random.set_seed(None) the model created using the Functional API also works as expected - What am I missing or not understanding correctly so far? - I assume that there are some differences regarding tf.random.set_seed when using the Sequential vs. the Functional API but... ?
Thanks in advance,
codax
EDIT: I forgot to mention that setting a seed (e.g. tf.random.set_seed(123) leads to identical results for both models not fitting the desired function.

Tensorflow: Use model inside another model as layer

I want to use a classification model inside another model as layer, since I thought that keras models can be used as layers also. This is the code of the first model:
cencoder_inputs = keras.layers.Input(shape=[pad_len], dtype=np.int32)
ccondi_input = keras.layers.Input(shape=[1], dtype=np.int32)
ccondi_layer = tf.keras.layers.concatenate([cencoder_inputs, ccondi_input], axis=1)
cembeddings = keras.layers.Embedding(vocab_size, 4)
cencoder_embeddings = cembeddings(ccondi_layer)
clstm = keras.layers.LSTM(128)(cencoder_embeddings)
cout_layer = keras.layers.Dense(16, activation="softmax")(clstm)
classification_model = keras.Model(inputs=[cencoder_inputs, ccondi_input], outputs=[cout_layer])
classification_model.compile(optimizer="Nadam", loss="sparse_categorical_crossentropy", metrics=["accuracy"], experimental_run_tf_function=False)
I train this model, save and reload it as class_model and set trainable=False
This is the code of my model, which should use the model above as layer:
encoder_inputs = keras.layers.Input(shape=[pad_len], dtype=np.int32)
decoder_inputs = keras.layers.Input(shape=[pad_len], dtype=np.int32)
condi_input = keras.layers.Input(shape=[1], dtype=np.int32)
class_layer = class_model((encoder_inputs, condi_input))
#Thats how I use the class model. Compilation goes fine so far
class_pred_layer = keras.layers.Lambda(lambda x: tf.reshape(tf.cast(tf.keras.backend.argmax(x, axis=1), dtype=tf.int32),shape=(tf.shape(encoder_inputs)[0],1)))(class_layer)
# Lambda and reshape layer, so I get 1 prediction per batch as integer
condi_layer = tf.keras.layers.concatenate([encoder_inputs, condi_input, class_pred_layer], axis=1)
embeddings = keras.layers.Embedding(vocab_size, 2)
encoder_embeddings = embeddings(condi_layer)
decoder_embeddings = embeddings(decoder_inputs)
encoder_1 = keras.layers.LSTM(64, return_sequences=True, return_state=True)
encoder_lstm_bidirectional_1 = keras.layers.Bidirectional(encoder_1)
encoder_output, state_h1, state_c1, state_h2, state_c2 = encoder_lstm_bidirectional_1(encoder_embeddings)
encoder_state = [Concatenate()([state_h1, state_h2]), Concatenate()([state_c1, state_c2])]
decoder_lstm = keras.layers.LSTM(64*2, return_sequences=True, return_state=True, name="decoder_lstm")
print(encoder_output.shape)
decoder_outputs,decoder_fwd_state, decoder_back_state = decoder_lstm(decoder_embeddings,initial_state=encoder_state)
print(decoder_outputs.shape)
attn_layer = AttentionLayer(name="attention_layer")
attn_out, attn_states = attn_layer([encoder_output, decoder_outputs])
decoder_concat_input = Concatenate(axis=-1, name="decoder_concat_layer")([decoder_outputs, attn_out])
decoder_dense_out = keras.layers.TimeDistributed(keras.layers.Dense(vocab_size, activation="softmax"))
decoder_outputs = decoder_dense_out(decoder_concat_input)
model = keras.Model(inputs=[encoder_inputs, decoder_inputs, condi_input], outputs=[decoder_outputs])
When I execute model.fit(), I receive the following error:
Inputs to eager execution function cannot be Keras symbolic tensors, but found [<tf.Tensor 'input_21:0' shape=(None, 35) dtype=int32>]
I thought trained models could be used easily as layers, what am I doing wrong?
I also already looked into this Post but it didnt help me either.
Thanks for your help!

Ok, i will do 2 things: (1) I will give you an example that works where i had to do call a model inside an other model, and (2) try to give you a hint on what could be your problem here ( i cant really undertand the code but i had in the past the same error )
1.
This is an example of a model that use an other model as an hidden layer:
def model_test(input_shape, sub_model):
inputs = Input(input_shape)
eblock_1_1 = dense_convolve(inputs, n_filters=growth_rate)
eblock_1_2 = dense_convolve(eblock_1_1, n_filters=growth_rate);
dblock_1_1 = dense_convolve(eblock_1_2, n_filters=growth_rate);
dblock_1_2 = dense_convolve(dblock_1_1, n_filters=growth_rate);
final_convolution = Conv3D(2, (1, 1, 1), padding='same', activation='relu')(dblock_1_2)
intermedio = sub_model(final_convolution)
layer = LeakyReLU(alpha=0.3)(intermedio)
model = Model(inputs=inputs, outputs=layer)
return model
I call it like this:
with strategy.scope():
sub_model = tf.keras.models.load_model('link_to_the_model')
sub_model.trainable = False
model = model_test(INPUT_SIZE, sub_model)
model.compile(optimizer=Adam(lr=0.1),
loss=tf.keras.losses.MeanSquaredError(),
metrics=None)
I just tested this on google colab with keras.
I had the same error some time ago when i tried to call a function with eager execution inside a model, the problem here is that the training is executed in graph mode ( you can find online some info about https://towardsdatascience.com/eager-execution-vs-graph-execution-which-is-better-38162ea4dbf6).
If the problem is the call of the model maybe try to do what i did, pass the model as a parameter and call it inside with a layer as argument and use it as a simple layer

Gradcam with guided backprop for transfer learning in Tensorflow 2.0

I get an error using gradient visualization with transfer learning in TF 2.0. The gradient visualization works on a model that does not use transfer learning.
When I run my code I get the error:
assert str(id(x)) in tensor_dict, 'Could not compute output ' + str(x)
AssertionError: Could not compute output Tensor("block5_conv3/Identity:0", shape=(None, 14, 14, 512), dtype=float32)
When I run the code below it errors. I think there's an issue with the naming conventions or connecting inputs and outputs from the base model, vgg16, to the layers I'm adding. Really appreciate your help!
"""
Broken example when grad_model is created.
"""
!pip uninstall tensorflow
!pip install tensorflow==2.0.0
import cv2
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers
import matplotlib.pyplot as plt
IMAGE_PATH = '/content/cat.3.jpg'
LAYER_NAME = 'block5_conv3'
model_layer = 'vgg16'
CAT_CLASS_INDEX = 281
imsize = (224,224,3)
img = tf.keras.preprocessing.image.load_img(IMAGE_PATH, target_size=(224, 224))
plt.figure()
plt.imshow(img)
img = tf.io.read_file(IMAGE_PATH)
img = tf.image.decode_jpeg(img)
img = tf.cast(img, dtype=tf.float32)
# img = tf.keras.preprocessing.image.img_to_array(img)
img = tf.image.resize(img, (224,224))
img = tf.reshape(img, (1, 224,224,3))
input = layers.Input(shape=(imsize[0], imsize[1], imsize[2]))
base_model = tf.keras.applications.VGG16(include_top=False, weights='imagenet',
input_shape=(imsize[0], imsize[1], imsize[2]))
# base_model.trainable = False
flat = layers.Flatten()
dropped = layers.Dropout(0.5)
global_average_layer = tf.keras.layers.GlobalAveragePooling2D()
fc1 = layers.Dense(16, activation='relu', name='dense_1')
fc2 = layers.Dense(16, activation='relu', name='dense_2')
fc3 = layers.Dense(128, activation='relu', name='dense_3')
prediction = layers.Dense(2, activation='softmax', name='output')
for layr in base_model.layers:
if ('block5' in layr.name):
layr.trainable = True
else:
layr.trainable = False
x = base_model(input)
x = global_average_layer(x)
x = fc1(x)
x = fc2(x)
x = prediction(x)
model = tf.keras.models.Model(inputs = input, outputs = x)
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
loss='binary_crossentropy',
metrics=['accuracy'])
This portion of the code is where the error lies. I'm not sure what is the correct way to label inputs and outputs.
# Create a graph that outputs target convolution and output
grad_model = tf.keras.models.Model(inputs = [model.input, model.get_layer(model_layer).input],
outputs=[model.get_layer(model_layer).get_layer(LAYER_NAME).output,
model.output])
print(model.get_layer(model_layer).get_layer(LAYER_NAME).output)
# Get the score for target class
# Get the score for target class
with tf.GradientTape() as tape:
conv_outputs, predictions = grad_model(img)
loss = predictions[:, 1]
The section below is for plotting a heatmap of gradcam.
print('Prediction shape:', predictions.get_shape())
# Extract filters and gradients
output = conv_outputs[0]
grads = tape.gradient(loss, conv_outputs)[0]
# Apply guided backpropagation
gate_f = tf.cast(output > 0, 'float32')
gate_r = tf.cast(grads > 0, 'float32')
guided_grads = gate_f * gate_r * grads
# Average gradients spatially
weights = tf.reduce_mean(guided_grads, axis=(0, 1))
# Build a ponderated map of filters according to gradients importance
cam = np.ones(output.shape[0:2], dtype=np.float32)
for index, w in enumerate(weights):
cam += w * output[:, :, index]
# Heatmap visualization
cam = cv2.resize(cam.numpy(), (224, 224))
cam = np.maximum(cam, 0)
heatmap = (cam - cam.min()) / (cam.max() - cam.min())
cam = cv2.applyColorMap(np.uint8(255 * heatmap), cv2.COLORMAP_JET)
output_image = cv2.addWeighted(cv2.cvtColor(img.astype('uint8'), cv2.COLOR_RGB2BGR), 0.5, cam, 1, 0)
plt.figure()
plt.imshow(output_image)
plt.show()
I also asked this to the tensorflow team on github at https://github.com/tensorflow/tensorflow/issues/37680.

I figured it out. If you set up the model extending the vgg16 base model with your own layers, rather than inserting the base model into a new model like a layer, then it works.
First set up the model and be sure to declare the input_tensor.
inp = layers.Input(shape=(imsize[0], imsize[1], imsize[2]))
base_model = tf.keras.applications.VGG16(include_top=False, weights='imagenet', input_tensor=inp,
input_shape=(imsize[0], imsize[1], imsize[2]))
This way we don't have to include a line like x=base_model(inp) to show what input we want to put in. That's already included in tf.keras.applications.VGG16(...).
Instead of putting this vgg16 base model inside another model, it's easier to do gradcam by adding layers to the base model itself. I grab the output of the last layer of VGG16 (with the top removed), which is the pooling layer.
block5_pool = base_model.get_layer('block5_pool')
x = global_average_layer(block5_pool.output)
x = fc1(x)
x = prediction(x)
model = tf.keras.models.Model(inputs = inp, outputs = x)
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
loss='binary_crossentropy',
metrics=['accuracy'])
Now, I grab the layer for visualization, LAYER_NAME='block5_conv3'.
# Create a graph that outputs target convolution and output
grad_model = tf.keras.models.Model(inputs = [model.input],
outputs=[model.output, model.get_layer(LAYER_NAME).output])
print(model.get_layer(LAYER_NAME).output)
# Get the score for target class
# Get the score for target class
with tf.GradientTape() as tape:
predictions, conv_outputs = grad_model(img)
loss = predictions[:, 1]
print('Prediction shape:', predictions.get_shape())
# Extract filters and gradients
output = conv_outputs[0]
grads = tape.gradient(loss, conv_outputs)[0]

We (I plus a number of team members developing a project) found a similar problem with a code implementing Grad-CAM that we found in a tutorial.
That code didn't work with a model consisting of the base model of VGG19 plus a few extra layers added on top of it. The problem was that the VGG19 base model was inserted as a "layer" inside our model, and apparently the GradCAM code didn't know how to deal with it - we were getting a "Graph disconnected..." error. Then after some debugging (carried out by another team member, not me) we managed to modify the original code to make it work for this kind of model that contains another model inside it. The idea is to add the inner model as an extra argument of the class GradCAM. Since this may be helpful to others I am including the modified code below (we also renamed the GradCAM class as My_GradCAM).
class My_GradCAM:
def __init__(self, model, classIdx, inner_model=None, layerName=None):
self.model = model
self.classIdx = classIdx
self.inner_model = inner_model
if self.inner_model == None:
self.inner_model = model
self.layerName = layerName
[...]
gradModel = tensorflow.keras.models.Model(inputs=[self.inner_model.inputs],
outputs=[self.inner_model.get_layer(self.layerName).output,
self.inner_model.output])
Then the class can be instantiated by adding the inner model as the extra argument, e.g.:
cam = My_GradCAM(model, None, inner_model=model.get_layer("vgg19"), layerName="block5_pool")
I hope this helps.
Edit: Credit to Mirtha Lucas for doing the debugging and finding the solution.

After a lot of struggle, I condense the way to draw the heat map when you are using transfer learning. Here is the keras official tutorial
The issue I encounter is that when I'm trying to draw the heat map
from my model, the densenet can be only seen as functional layer in my
model. So the make_gradcam_heatmap can not figure out the layer that
inside functional layer. As the 5th layer shows.
Therefore, to simulate the Keras official document, I need to only use the densenet as the model for visualization. Here is the step
Only Take out the model from your model
dense_model = dense_model.get_layer('densenet121')
Copy the weight from dense model to your new initiated model
inputs = tf.keras.Input(shape=(224, 224, 3))
model = model_builder(weights="imagenet", include_top=True, input_tensor=inputs)
for layer, dense_layer in zip(model.layers[1:], dense_model.layers[1:]):
layer.set_weights(dense_layer.get_weights())
relu = model.get_layer('relu')
x = tf.keras.layers.GlobalAveragePooling2D()(relu.output)
outputs = tf.keras.layers.Dense(5)(x)
model = tf.keras.models.Model(inputs = inputs, outputs = outputs)
Draw the heat map
preprocess_input = keras.applications.densenet.preprocess_input
img_array = preprocess_input(get_img_array(img_path, size=(224, 224)))
heatmap = make_gradcam_heatmap(img_array, model, 'bn')
plt.matshow(heatmap)
plt.show()
get_img_array, make_gradcam_heatmap and save_and_display_gradcam are kept in still. Follow the keras tutorial then you are good to go.

Including advanced computation (scikit-like) in a keras custom layer

Normally I would preprocess the data before I feed it into my model for classification.
This is however not possible and thus am stuck either to enhance the performance of the model further (somehow) or include useful preprocessing steps directly inside the model.
How can I do that? The best solution I found thus far, included re-implementing the functionality I want using Keras backend. This is far from a good solution and thus I am hoping someone has an idea, how to salavage the situation.
Below are links I found useful + my current code.
Useful links:
Keras Custom Layer with advanced calculations
How to Switch from Keras Tensortype to numpy array for a custom layer?
How to create a Keras Custom Layer using functions not included in the Backend, to perform tensor sampling?
My code thus far:
def freezeBaseModelLayers(baseModel):
for layer in baseModel.layers:
layer.trainable = False
def preprocess_input(x):
# TODO: Not working, but intention should be clear
numpy_array = tf.unstack(tf.unstack(tf.unstack(x, 224, 0), 224, 0), 1, 0)
from skimage.feature import hog
from skimage import data, exposure
img_adapteq = exposure.equalize_adapthist(numpy_array, orientations=8, pixels_per_cell=(3, 3),
cells_per_block=(1, 1), visualize=True, multichannel=False)
[x1, x2, x3] = tf.constant(img_adapteq), tf.constant(img_adapteq), tf.constant(img_adapteq)
img_conc = Concatenate([x1, x2, x3])
return img_conc
def create(x):
is_training = tf.get_variable('is_training', (), dtype=tf.bool, trainable=False)
with tf.name_scope('pretrained'):
# Add preprocess step here...
input_layer = Lambda(preprocess_input(x), input_shape=(224, 224, 1), output_shape=(224, 224, 3))
baseModel = vgg16.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
freezeBaseModelLayers(baseModel)
layer = baseModel(input_layer)
layer = GlobalMaxPooling2D()(layer)
layer = Dense(1024, activation='relu')(layer)
layer = Dense(2, activation=None)(layer)
model = Model(input=input_layer.input, output=layer)
output = model(x)
return output
I would like to include prepocessing steps inside my model
The models I am working with are receiving noisy data. In order to enhance the performance of the models, I would like to do some preprocessing steps e.g. equalize_adapthist.

A better way to do this is via a custom keras layer. Here is an example:
import tensorflow as tf
from keras.layers import Layer, Input, Conv2D
from keras.models import Model
from keras import backend as K
from skimage.feature import hog
from skimage import data, exposure
def equalize(img):
img_adapteq = exposure.equalize_adapthist(img)
return img_adapteq
def preprocess_input(img):
return tf.py_func(equalize,
[img],
'float32',
stateful=False,
name='custom_image_op')
class CustomLayer(Layer):
def __init__(self, output_dim, **kwargs):
self.output_dim = output_dim
self.trainable = False
super(CustomLayer, self).__init__(**kwargs)
def call(self, x):
res = tf.map_fn(preprocess_input, x)
res.set_shape([x.shape[0],
self.output_dim[1],
self.output_dim[0],
x.shape[-1]])
return res
output_dim = (224,224)
inputs = Input(shape=(224,224,3))
x = CustomLayer(output_dim)(inputs)
x = Conv2D(32, (3,3))(x)
x = Flatten()(x)
x = Dense(1)(x)
model = Model(inputs, x)
model.summary()
# test
sample = np.random.rand(4, 224,224,3).astype(np.float32)
y = np.random.randint(2, size=(4,))
model.compile("sgd", "mse")
model.fit(sample, y)

To do this with a Lambda layer, you would need to write histogram equalization in pure tensorflow.
Indeed, when building the graph, the function (preprocess_input) will be called on tensorflow placeholders which, in the case of this skimage function which expects numpy arrays, will not work.
This question shows how to write it in pure tensorflow. Copy-pasting here for the sake of redundancy/ease-of-reading (I have not tested it myself but a test is available in the question):
def tf_equalize_histogram(image):
values_range = tf.constant([0., 255.], dtype = tf.float32)
histogram = tf.histogram_fixed_width(tf.to_float(image), values_range, 256)
cdf = tf.cumsum(histogram)
cdf_min = cdf[tf.reduce_min(tf.where(tf.greater(cdf, 0)))]
img_shape = tf.shape(image)
pix_cnt = img_shape[-3] * img_shape[-2]
px_map = tf.round(tf.to_float(cdf - cdf_min) * 255. / tf.to_float(pix_cnt - 1))
px_map = tf.cast(px_map, tf.uint8)
eq_hist = tf.expand_dims(tf.gather_nd(px_map, tf.cast(image, tf.int32)), 2)
return eq_hist
Btw you should write the pre-processing step as (once you made the preprocessing step pure tensorflow):
input_layer = Lambda(preprocess_input, input_shape=(224, 224, 1), output_shape=(224, 224, 3))(x)
Another way to do this is to write a custom layer as pointed out by mlRocks.

load_model and Lamda layer in Keras

How to load model that have lambda layer?
Here is the code to reproduce behaviour:
MEAN_LANDMARKS = np.load('data/mean_shape_68.npy')
def add_mean_landmarks(x):
mean_landmarks = np.array(MEAN_LANDMARKS, np.float32)
mean_landmarks = mean_landmarks.flatten()
mean_landmarks_tf = tf.convert_to_tensor(mean_landmarks)
x = x + mean_landmarks_tf
return x
def get_model():
inputs = Input(shape=(8, 128, 128, 3))
cnn = VGG16(include_top=False, weights='imagenet', input_shape=(128, 128, 3))
x = TimeDistributed(cnn)(inputs)
x = TimeDistributed(Flatten())(x)
x = LSTM(256)(x)
x = Dense(68 * 2, activation='linear')(x)
x = Lambda(add_mean_landmarks)(x)
model = Model(inputs=inputs, outputs=x)
optimizer = Adadelta()
model.compile(optimizer=optimizer, loss='mae')
return model
Model compiles and I can save it, but when I tried to load it with load_model function I get an error:
in add_mean_landmarks
mean_landmarks = np.array(MEAN_LANDMARKS, np.float32)
NameError: name 'MEAN_LANDMARKS' is not defined
Аs I understand MEAN_LANDMARKS is not incorporated in graph as constant tensor. Also it's related to this question: How to add constant tensor in Keras?

You need to pass custom_objects argument to load_model function:
model = load_model('model_file_name.h5', custom_objects={'MEAN_LANDMARKS': MEAN_LANDMARKS})
Look for more info in Keras docs: Handling custom layers (or other custom objects) in saved models
.