Normally I would preprocess the data before I feed it into my model for classification.
This is however not possible and thus am stuck either to enhance the performance of the model further (somehow) or include useful preprocessing steps directly inside the model.
How can I do that? The best solution I found thus far, included re-implementing the functionality I want using Keras backend. This is far from a good solution and thus I am hoping someone has an idea, how to salavage the situation.
Below are links I found useful + my current code.
Useful links:
Keras Custom Layer with advanced calculations
How to Switch from Keras Tensortype to numpy array for a custom layer?
How to create a Keras Custom Layer using functions not included in the Backend, to perform tensor sampling?
My code thus far:
def freezeBaseModelLayers(baseModel):
for layer in baseModel.layers:
layer.trainable = False
def preprocess_input(x):
# TODO: Not working, but intention should be clear
numpy_array = tf.unstack(tf.unstack(tf.unstack(x, 224, 0), 224, 0), 1, 0)
from skimage.feature import hog
from skimage import data, exposure
img_adapteq = exposure.equalize_adapthist(numpy_array, orientations=8, pixels_per_cell=(3, 3),
cells_per_block=(1, 1), visualize=True, multichannel=False)
[x1, x2, x3] = tf.constant(img_adapteq), tf.constant(img_adapteq), tf.constant(img_adapteq)
img_conc = Concatenate([x1, x2, x3])
return img_conc
def create(x):
is_training = tf.get_variable('is_training', (), dtype=tf.bool, trainable=False)
with tf.name_scope('pretrained'):
# Add preprocess step here...
input_layer = Lambda(preprocess_input(x), input_shape=(224, 224, 1), output_shape=(224, 224, 3))
baseModel = vgg16.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
freezeBaseModelLayers(baseModel)
layer = baseModel(input_layer)
layer = GlobalMaxPooling2D()(layer)
layer = Dense(1024, activation='relu')(layer)
layer = Dense(2, activation=None)(layer)
model = Model(input=input_layer.input, output=layer)
output = model(x)
return output
I would like to include prepocessing steps inside my model
The models I am working with are receiving noisy data. In order to enhance the performance of the models, I would like to do some preprocessing steps e.g. equalize_adapthist.
A better way to do this is via a custom keras layer. Here is an example:
import tensorflow as tf
from keras.layers import Layer, Input, Conv2D
from keras.models import Model
from keras import backend as K
from skimage.feature import hog
from skimage import data, exposure
def equalize(img):
img_adapteq = exposure.equalize_adapthist(img)
return img_adapteq
def preprocess_input(img):
return tf.py_func(equalize,
[img],
'float32',
stateful=False,
name='custom_image_op')
class CustomLayer(Layer):
def __init__(self, output_dim, **kwargs):
self.output_dim = output_dim
self.trainable = False
super(CustomLayer, self).__init__(**kwargs)
def call(self, x):
res = tf.map_fn(preprocess_input, x)
res.set_shape([x.shape[0],
self.output_dim[1],
self.output_dim[0],
x.shape[-1]])
return res
output_dim = (224,224)
inputs = Input(shape=(224,224,3))
x = CustomLayer(output_dim)(inputs)
x = Conv2D(32, (3,3))(x)
x = Flatten()(x)
x = Dense(1)(x)
model = Model(inputs, x)
model.summary()
# test
sample = np.random.rand(4, 224,224,3).astype(np.float32)
y = np.random.randint(2, size=(4,))
model.compile("sgd", "mse")
model.fit(sample, y)
To do this with a Lambda layer, you would need to write histogram equalization in pure tensorflow.
Indeed, when building the graph, the function (preprocess_input) will be called on tensorflow placeholders which, in the case of this skimage function which expects numpy arrays, will not work.
This question shows how to write it in pure tensorflow. Copy-pasting here for the sake of redundancy/ease-of-reading (I have not tested it myself but a test is available in the question):
def tf_equalize_histogram(image):
values_range = tf.constant([0., 255.], dtype = tf.float32)
histogram = tf.histogram_fixed_width(tf.to_float(image), values_range, 256)
cdf = tf.cumsum(histogram)
cdf_min = cdf[tf.reduce_min(tf.where(tf.greater(cdf, 0)))]
img_shape = tf.shape(image)
pix_cnt = img_shape[-3] * img_shape[-2]
px_map = tf.round(tf.to_float(cdf - cdf_min) * 255. / tf.to_float(pix_cnt - 1))
px_map = tf.cast(px_map, tf.uint8)
eq_hist = tf.expand_dims(tf.gather_nd(px_map, tf.cast(image, tf.int32)), 2)
return eq_hist
Btw you should write the pre-processing step as (once you made the preprocessing step pure tensorflow):
input_layer = Lambda(preprocess_input, input_shape=(224, 224, 1), output_shape=(224, 224, 3))(x)
Another way to do this is to write a custom layer as pointed out by mlRocks.
Related
I have a network written in tensorflow keras functional API.
I'd like to use the gradient of one layer w.r.t to the previous layer as input for another layer.
I tried gradient tape and tf.gradients and none of them worked. I get the following error:
ValueError: tf.function-decorated function tried to create variables on non-first call.
There is no input at this point and I have input layer.
Is it possible to do this in tenserflow?
My code:
def Geo_branch(self, geo_inp):
Fully_Connected1 = layers.TimeDistributed(layers.Dense(128, activation='tanh'))(geo_inp)
Fully_Connected2 = layers.TimeDistributed(layers.Dense(64, activation='tanh'))(Fully_Connected1)
return Fully_Connected2
#tf.function
def geo_extension(self, geo_branch):
Fully_Connected = layers.TimeDistributed(layers.Dense(100, activation='tanh'))(geo_branch)
geo_ext = layers.LSTM(6,
activation="tanh",
recurrent_activation="sigmoid",
unroll=False,
use_bias=True,
name='Translation'
)(Fully_Connected)
grads = tf.gradients(geo_ext, geo_branch)
return geo_ext, grads
inp_geo = layers.Input(shape=(self.time_size, 6), name='geo_input')
Geo_branch = Geo_branch(inp_geo)
geo_ext, grads = geo_extension(Geo_branch)
Any solution is appreciated. It doesn't have to be GradientTape, if there is any other way to compute these gradients.
I would just inherit from tensorflow's Layer class and creating your own custom Layer. Also, it would probably be beneficial to put everything under one call so as to minimize the likelihood that there are disconnections in the graph.
Example:
import tensorflow as tf
from typing import List
from typing import Optional
from typing import Tuple
from tensorflow.keras import Model
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Layer
from tensorflow.keras.layers import LSTM
from tensorflow.keras.layers import TimeDistributed
class CustomGeoLayer(Layer):
"""``CustomGeoLayer``."""
def __init__(self, num_units: List[int], name: Optional[str] = None):
super().__init__(name=name)
self.num_units = num_units
self.dense_0 = TimeDistributed(Dense(num_units[0], activation="tanh"))
self.dense_1 = TimeDistributed(Dense(num_units[1], activation="tanh"))
self.dense_2 = TimeDistributed(Dense(num_units[2], activation="tanh"))
self.rnn = LSTM(units=num_units[3], activation="tanh",
recurrent_activation="sigmoid",
unroll=False, use_bias=True,
name="Translation")
#tf.function
def call(self,
input_tensor: tf.Tensor,
training: bool = True) -> Tuple[tf.Tensor, tf.Tensor]:
x = self.dense_0(input_tensor)
x = self.dense_1(x)
r = self.dense_2(x)
x = self.rnn(r, training=training)
return x, tf.gradients(x, r)[0]
# create model
x_in = Input(shape=(10, 6))
x_out = CustomGeoLayer([128, 64, 100, 6])(x_in)
model = Model(x_in, x_out)
# fake input data
arr = tf.random.normal((3, 10, 6))
# forward pass
out, g = model(arr)
print(out.shape)
# (3, 6)
print(g.shape)
# (3, 10, 100)
I have a model that I am using transfer learning for MobileNetV2 and I'd like to quantize it and compare the accuracy difference against a non-quantized model with transfer learning. However, they do not entirely support recursive quantization, but according to this, this method should quantize my model: https://github.com/tensorflow/model-optimization/issues/377#issuecomment-820948555
What I tried doing was:
import tensorflow as tf
import tensorflow_model_optimization as tfmot
pretrained_model = tf.keras.applications.MobileNetV2(include_top=False)
pretrained_model.trainable = True
for layer in pretrained_model.layers[:-1]:
layer.trainable = False
quantize_model_pretrained = tfmot.quantization.keras.quantize_model
q_pretrained_model = quantize_model_pretrained(pretrained_model)
original_inputs = tf.keras.layers.Input(shape=(224, 224, 3))
y = tf.keras.layers.experimental.preprocessing.Rescaling(1./255)(original_inputs)
y = base_model(original_inputs)
y = tf.keras.layers.GlobalAveragePooling2D()(y)
original_outputs = tf.keras.layers.Dense(5, activation="softmax")(y)
model_1 = tf.keras.Model(original_inputs, original_outputs)
quantize_model = tfmot.quantization.keras.quantize_model
q_aware_model = quantize_model(model_1)
It is still giving me the following error:
ValueError: Quantizing a tf.keras Model inside another tf.keras Model is not supported.
I'd like to understand what is the correct way to perform quantization-aware-training in this case?
According to the issue you mentioned, you should quantize each model separately and then bring them together afterwards. Something like this:
import tensorflow as tf
import tensorflow_model_optimization as tfmot
pretrained_model = tf.keras.applications.MobileNetV2(input_shape=(224, 224, 3), include_top=False)
pretrained_model.trainable = True
for layer in pretrained_model.layers[:-1]:
layer.trainable = False
q_pretrained_model = tfmot.quantization.keras.quantize_model(pretrained_model)
q_base_model = tfmot.quantization.keras.quantize_model(tf.keras.Sequential([tf.keras.layers.GlobalAveragePooling2D(input_shape=(7, 7, 1280)), tf.keras.layers.Dense(5, activation="softmax")]))
original_inputs = tf.keras.layers.Input(shape=(224, 224, 3))
y = tf.keras.layers.experimental.preprocessing.Rescaling(1./255)(original_inputs)
y = q_pretrained_model(original_inputs)
original_outputs = q_base_model(y)
model = tf.keras.Model(original_inputs, original_outputs)
It does not look like it is already supported out of the box, even though this is claimed.
I'm using Tensorflow 2 and I need to build a multi-input multi-output model, and my data is timeseries data, which does not have a consistent shape for its time dimension. I've tried many ways but none worked due to the inconsistent shape.
There are three data and one of them is used twice. They have the format of (number of files, None, 5), with the None dimension to be the inconsistent dimension.
Here's some testing codes that reproduce my issues, and I'm using a generator in this case, but feel free to change to whatever method. Could someone help me with this input pipeline?
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
dummy_1 = [[[1.1,2,3,4,5],[2,3,4,5,6],[3,4,5,6,7]],
[[1.2,2,3,4,5],[2,3,4,5,6.8]],
[[1.3,2,3,4,5],[2,3,4,5,6],[3,4,5,6,7],[4,5,6,7,8.9]]]
dummy_2 = [[[1.1,2,3,4,5],[2,3,4,5,6]],
[[1.1,2,3,4,5],[2,3,4,5,6]],[3,4,5,6,7],
[[1.3,2,3,4,5],[2,3,4,5,6]]]
dummy_3 = [[[1.5,2,3,4,5],[2,3,4,5,6]],
[[1.6,2,3,4,5],[2,3,4,5,6]],[3,4,5,6,7],
[[1.7,2,3,4,5],[2,3,4,5,6]]]
def gen():
for i in range(len(dummy_1)):
yield(dummy_1[i],dummy_2[i],dummy_2[i],dummy_3[i])
def custom_loss(y_true, y_pred):
return tf.reduce_mean(tf.abs(y_pred - y_true))
class network():
def __init__(self):
input_1 = keras.Input(shape=(None,5))
input_2 = keras.Input(shape=(None,5))
output_1 = layers.Conv1DTranspose(16, 3, padding='same', activation='relu')(input_1)
output_2 = layers.Conv1DTranspose(16, 3, padding='same', activation='relu')(input_2)
self.model = keras.Model(inputs=[input_1, input_2],
outputs=[output_1, output_2])
# compile model
self.model.compile(optimizer=keras.optimizers.SGD(learning_rate=0.001),
loss={"mel_loss":custom_loss, "mag_loss":custom_loss})
def train(self):
self.dataset = tf.data.Dataset.from_generator(gen,
(tf.float32, tf.float32, tf.float32, tf.float32))
self.dataset.batch(32).repeat()
self.model.fit(self.dataset,epochs=3)
#self.model.fit([dummy_1, dummy_2],
# [dummy_2, dummy_3],
# epochs=3)
net = network()
net.train()
This is not possible currently for TF2, referring to https://github.com/tensorflow/tensorflow/issues/45112
I get an error using gradient visualization with transfer learning in TF 2.0. The gradient visualization works on a model that does not use transfer learning.
When I run my code I get the error:
assert str(id(x)) in tensor_dict, 'Could not compute output ' + str(x)
AssertionError: Could not compute output Tensor("block5_conv3/Identity:0", shape=(None, 14, 14, 512), dtype=float32)
When I run the code below it errors. I think there's an issue with the naming conventions or connecting inputs and outputs from the base model, vgg16, to the layers I'm adding. Really appreciate your help!
"""
Broken example when grad_model is created.
"""
!pip uninstall tensorflow
!pip install tensorflow==2.0.0
import cv2
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers
import matplotlib.pyplot as plt
IMAGE_PATH = '/content/cat.3.jpg'
LAYER_NAME = 'block5_conv3'
model_layer = 'vgg16'
CAT_CLASS_INDEX = 281
imsize = (224,224,3)
img = tf.keras.preprocessing.image.load_img(IMAGE_PATH, target_size=(224, 224))
plt.figure()
plt.imshow(img)
img = tf.io.read_file(IMAGE_PATH)
img = tf.image.decode_jpeg(img)
img = tf.cast(img, dtype=tf.float32)
# img = tf.keras.preprocessing.image.img_to_array(img)
img = tf.image.resize(img, (224,224))
img = tf.reshape(img, (1, 224,224,3))
input = layers.Input(shape=(imsize[0], imsize[1], imsize[2]))
base_model = tf.keras.applications.VGG16(include_top=False, weights='imagenet',
input_shape=(imsize[0], imsize[1], imsize[2]))
# base_model.trainable = False
flat = layers.Flatten()
dropped = layers.Dropout(0.5)
global_average_layer = tf.keras.layers.GlobalAveragePooling2D()
fc1 = layers.Dense(16, activation='relu', name='dense_1')
fc2 = layers.Dense(16, activation='relu', name='dense_2')
fc3 = layers.Dense(128, activation='relu', name='dense_3')
prediction = layers.Dense(2, activation='softmax', name='output')
for layr in base_model.layers:
if ('block5' in layr.name):
layr.trainable = True
else:
layr.trainable = False
x = base_model(input)
x = global_average_layer(x)
x = fc1(x)
x = fc2(x)
x = prediction(x)
model = tf.keras.models.Model(inputs = input, outputs = x)
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
loss='binary_crossentropy',
metrics=['accuracy'])
This portion of the code is where the error lies. I'm not sure what is the correct way to label inputs and outputs.
# Create a graph that outputs target convolution and output
grad_model = tf.keras.models.Model(inputs = [model.input, model.get_layer(model_layer).input],
outputs=[model.get_layer(model_layer).get_layer(LAYER_NAME).output,
model.output])
print(model.get_layer(model_layer).get_layer(LAYER_NAME).output)
# Get the score for target class
# Get the score for target class
with tf.GradientTape() as tape:
conv_outputs, predictions = grad_model(img)
loss = predictions[:, 1]
The section below is for plotting a heatmap of gradcam.
print('Prediction shape:', predictions.get_shape())
# Extract filters and gradients
output = conv_outputs[0]
grads = tape.gradient(loss, conv_outputs)[0]
# Apply guided backpropagation
gate_f = tf.cast(output > 0, 'float32')
gate_r = tf.cast(grads > 0, 'float32')
guided_grads = gate_f * gate_r * grads
# Average gradients spatially
weights = tf.reduce_mean(guided_grads, axis=(0, 1))
# Build a ponderated map of filters according to gradients importance
cam = np.ones(output.shape[0:2], dtype=np.float32)
for index, w in enumerate(weights):
cam += w * output[:, :, index]
# Heatmap visualization
cam = cv2.resize(cam.numpy(), (224, 224))
cam = np.maximum(cam, 0)
heatmap = (cam - cam.min()) / (cam.max() - cam.min())
cam = cv2.applyColorMap(np.uint8(255 * heatmap), cv2.COLORMAP_JET)
output_image = cv2.addWeighted(cv2.cvtColor(img.astype('uint8'), cv2.COLOR_RGB2BGR), 0.5, cam, 1, 0)
plt.figure()
plt.imshow(output_image)
plt.show()
I also asked this to the tensorflow team on github at https://github.com/tensorflow/tensorflow/issues/37680.
I figured it out. If you set up the model extending the vgg16 base model with your own layers, rather than inserting the base model into a new model like a layer, then it works.
First set up the model and be sure to declare the input_tensor.
inp = layers.Input(shape=(imsize[0], imsize[1], imsize[2]))
base_model = tf.keras.applications.VGG16(include_top=False, weights='imagenet', input_tensor=inp,
input_shape=(imsize[0], imsize[1], imsize[2]))
This way we don't have to include a line like x=base_model(inp) to show what input we want to put in. That's already included in tf.keras.applications.VGG16(...).
Instead of putting this vgg16 base model inside another model, it's easier to do gradcam by adding layers to the base model itself. I grab the output of the last layer of VGG16 (with the top removed), which is the pooling layer.
block5_pool = base_model.get_layer('block5_pool')
x = global_average_layer(block5_pool.output)
x = fc1(x)
x = prediction(x)
model = tf.keras.models.Model(inputs = inp, outputs = x)
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
loss='binary_crossentropy',
metrics=['accuracy'])
Now, I grab the layer for visualization, LAYER_NAME='block5_conv3'.
# Create a graph that outputs target convolution and output
grad_model = tf.keras.models.Model(inputs = [model.input],
outputs=[model.output, model.get_layer(LAYER_NAME).output])
print(model.get_layer(LAYER_NAME).output)
# Get the score for target class
# Get the score for target class
with tf.GradientTape() as tape:
predictions, conv_outputs = grad_model(img)
loss = predictions[:, 1]
print('Prediction shape:', predictions.get_shape())
# Extract filters and gradients
output = conv_outputs[0]
grads = tape.gradient(loss, conv_outputs)[0]
We (I plus a number of team members developing a project) found a similar problem with a code implementing Grad-CAM that we found in a tutorial.
That code didn't work with a model consisting of the base model of VGG19 plus a few extra layers added on top of it. The problem was that the VGG19 base model was inserted as a "layer" inside our model, and apparently the GradCAM code didn't know how to deal with it - we were getting a "Graph disconnected..." error. Then after some debugging (carried out by another team member, not me) we managed to modify the original code to make it work for this kind of model that contains another model inside it. The idea is to add the inner model as an extra argument of the class GradCAM. Since this may be helpful to others I am including the modified code below (we also renamed the GradCAM class as My_GradCAM).
class My_GradCAM:
def __init__(self, model, classIdx, inner_model=None, layerName=None):
self.model = model
self.classIdx = classIdx
self.inner_model = inner_model
if self.inner_model == None:
self.inner_model = model
self.layerName = layerName
[...]
gradModel = tensorflow.keras.models.Model(inputs=[self.inner_model.inputs],
outputs=[self.inner_model.get_layer(self.layerName).output,
self.inner_model.output])
Then the class can be instantiated by adding the inner model as the extra argument, e.g.:
cam = My_GradCAM(model, None, inner_model=model.get_layer("vgg19"), layerName="block5_pool")
I hope this helps.
Edit: Credit to Mirtha Lucas for doing the debugging and finding the solution.
After a lot of struggle, I condense the way to draw the heat map when you are using transfer learning. Here is the keras official tutorial
The issue I encounter is that when I'm trying to draw the heat map
from my model, the densenet can be only seen as functional layer in my
model. So the make_gradcam_heatmap can not figure out the layer that
inside functional layer. As the 5th layer shows.
Therefore, to simulate the Keras official document, I need to only use the densenet as the model for visualization. Here is the step
Only Take out the model from your model
dense_model = dense_model.get_layer('densenet121')
Copy the weight from dense model to your new initiated model
inputs = tf.keras.Input(shape=(224, 224, 3))
model = model_builder(weights="imagenet", include_top=True, input_tensor=inputs)
for layer, dense_layer in zip(model.layers[1:], dense_model.layers[1:]):
layer.set_weights(dense_layer.get_weights())
relu = model.get_layer('relu')
x = tf.keras.layers.GlobalAveragePooling2D()(relu.output)
outputs = tf.keras.layers.Dense(5)(x)
model = tf.keras.models.Model(inputs = inputs, outputs = outputs)
Draw the heat map
preprocess_input = keras.applications.densenet.preprocess_input
img_array = preprocess_input(get_img_array(img_path, size=(224, 224)))
heatmap = make_gradcam_heatmap(img_array, model, 'bn')
plt.matshow(heatmap)
plt.show()
get_img_array, make_gradcam_heatmap and save_and_display_gradcam are kept in still. Follow the keras tutorial then you are good to go.
How to load model that have lambda layer?
Here is the code to reproduce behaviour:
MEAN_LANDMARKS = np.load('data/mean_shape_68.npy')
def add_mean_landmarks(x):
mean_landmarks = np.array(MEAN_LANDMARKS, np.float32)
mean_landmarks = mean_landmarks.flatten()
mean_landmarks_tf = tf.convert_to_tensor(mean_landmarks)
x = x + mean_landmarks_tf
return x
def get_model():
inputs = Input(shape=(8, 128, 128, 3))
cnn = VGG16(include_top=False, weights='imagenet', input_shape=(128, 128, 3))
x = TimeDistributed(cnn)(inputs)
x = TimeDistributed(Flatten())(x)
x = LSTM(256)(x)
x = Dense(68 * 2, activation='linear')(x)
x = Lambda(add_mean_landmarks)(x)
model = Model(inputs=inputs, outputs=x)
optimizer = Adadelta()
model.compile(optimizer=optimizer, loss='mae')
return model
Model compiles and I can save it, but when I tried to load it with load_model function I get an error:
in add_mean_landmarks
mean_landmarks = np.array(MEAN_LANDMARKS, np.float32)
NameError: name 'MEAN_LANDMARKS' is not defined
Аs I understand MEAN_LANDMARKS is not incorporated in graph as constant tensor. Also it's related to this question: How to add constant tensor in Keras?
You need to pass custom_objects argument to load_model function:
model = load_model('model_file_name.h5', custom_objects={'MEAN_LANDMARKS': MEAN_LANDMARKS})
Look for more info in Keras docs: Handling custom layers (or other custom objects) in saved models
.