I would like to add a gaussian layer model to a pretrained vgg16 network (after convolutional layer 1) and have the noisy layer active when I pass the images (I would like to obtain activation of layers after passing through a noise layer during inference phase).
I have managed to compile a 'new' model which has the gaussian layer added to it. I also have code for passing images and obtaining activations to them (using model.predict(img)). My problem is that now when I am passing the images through the gaussian noise layer the activation does not change from passing through the layer prior. I have read that gaussian noise layers are usually used in training (not testing - which is relevant for me), but I've also read that you can 'trick' a layer into believing you are in a training state and still have the gaussian noise active. I don't know how to practically implement this.
Here I am creating my new model with gaussian noise layer:
# load the model
model = VGG16(weights='imagenet')
# summarize the model
model.summary()
# loop over layers and add noise - create a new model
for i in range(1,len(model.layers)): # loop over all layers
#print ('Working on '+layer_names[i])
layer1 = model.layers[i]
if i==1: # adding the noise
noise = GaussianNoise(0.85)
x = noise(layer1.output)
if i<len(model.layers)-1: # reconnecting the layers
layer2 = model.layers[i+1]
x = layer2(x)
if i==len(model.layers)-1:
predictors = x # this is the last layer
# Create a new model
model2 = Model(input=model.input, output=predictors)
model2.summary()
Here is an example of how I would like to obtain activation of all layers (including Gaussian layer).
layer_names = ['block1_conv1',model2.layers[2].name,'block1_conv2','block2_conv1','block2_conv2','block3_conv1','block3_conv2','block3_conv3',\
'block4_conv1','block4_conv2','block4_conv3','block5_conv1','block5_conv2','block5_conv3','fc1','fc2','predictions'];
for i in range(1,len(layer_names)): # loop over all layers
print ('Working on '+layer_names[i])
layer_name = layer_names[i]
intermediate_layer_model = Model(inputs=model2.input,outputs=model2.get_layer(layer_name).output)
# load the image with the required shape
img = image.load_img('test.BMP', target_size=(224,224)) # plot with img
# convert the image to an array
img = image.img_to_array(img)
# expand dimensions so that it represents a single 'sample'
img = np.expand_dims(img, axis=0)
# prepare the image (e.g. scale pixel values for the vgg)
img = preprocess_input(img)
# Now get the activations to this image
intermediate_output = intermediate_layer_model.predict(img)
The output (intermediate_output) should be different after receiving output from block1_cov1 layer vs. gaussian_noise layer, but it isn't. I guess that's because the Gaussian noise layer is not active, but I don't know how to change that. I know my code is suboptimal (just starting to work with Keras), but any help on how to get a noise to a pretrained network and let it 'propagate' through a network would be highly appreciated.
Related
I am working on CNN model using Tensorflow frames in google collab. I am unable to extract the latent vectors from the convolutional layers. I want to extract the output of the convolutional layers, the layers before fully connected layer.
I have tried with the following code
a = dropout()(classifier_model.output)
print(a)
I am unable to understand the solution suggested on the link Stackoverflow solution to print the value of tensorflow object after applying a-conv-pool-layer
Anyone with any suggestion?
You can use get_layer method of the Model class to get a layer by its name, find bellow an example with a dummy 1D CNN and a binary classifier :
timesteps = 100
nfeatures = 2
# build the model using the functional API
# example of a 1D CNN inspired by the your stack overflow link, but using a model instead of successive *raw* layers
# the values of the Conv1D filters and kernels are different
input = Input((timesteps, nfeatures))
p = Conv1D(filters=16, kernel_size=10)(input)
p = ReLU()(p)
p = MaxPool1D(pool_size=2)(p)
p = Conv1D(filters=32, kernel_size=10)(p)
p = ReLU()(p)
p = MaxPool1D(pool_size=2)(p)
p = Conv1D(filters=64, kernel_size=10)(p)
p = ReLU()(p)
p = MaxPool1D(pool_size=2, name='conv1Dfeat')(p) # give a name to the CNN output
# fully connected part
p = Flatten()(p)
p = Dense(10)(p)
# could add a dropout layer to ease optimization
finaloutput = Dense(1, activation='sigmoid')(p)
# full model
model = Model(inputs=input, outputs=finaloutput)
# compile network, i.e. define optimizer, loss and metrics
model.compile(optimizer='Adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()
You need to train the model using the fit method with some data. Then you can get the output of the layer which name is conv1Dfeat (the last layer of the convolutive part) by defining the model:
modelCNN = Model(inputs=input, outputs=model.get_layer('conv1Dfeat').output)
modelCNN.summary()
If you want to get the output of the convolutive part, let's say based on a single numpy input array of shape (timesteps, nfeatures), you can use the predict of the Model class on batched data:
data = np.random.normal(size=(timesteps, nfeatures)) # dummy data
data_tf = tf.expand_dims(data, axis=0) # convert to TF tensor and add batch dimension at the same time
cnn_out_np = modelCNN.predict(data_tf)
cnn_out_np = np.squeeze(cnn_out_np, axis=0) # remove batch dimension
print(cnn_out_np.shape)
(4, 64)
I am trying to develop an auto-encoder for compressing images using Keras. I was able to train it and to compress images, but I am struggling with the decoder part of it. Specifically, given a compressed image, I don't know how to use the model to de-compress it.
This is what I have:
input_layer = keras.layers.Input(shape=(64, 64, 3))
code_layer = build_encoder(input_layer, size_of_code) # add some convolution layers and max-pooling
output_layer = build_decoder(code_layer) # add some convolution layers and up-sampling
autoencoder_model = keras.models.Model(input_layer, output_layer)
encoder_model = keras.models.Model(input_layer, code_layer)
decoder_model = ??
autoencoder_model.compile(optimizer='adam', loss='binary_crossentropy')
using the code above I can train the autoencoder_model and compress the images using the encoder_model, but I don't know how to construct the decoder_model, mainly because I don't know how to insert a new input to the middle of the model.
Like this. Instead of the code_layer, need to define an input layer and build the decoder model with that input.
latent_inputs = keras.layers.Input(shape=(size_of_code))
output_layer = build_decoder(latent_inputs) # add some convolution layers and up-sampling
decoder_model = keras.models.Model(latent_inputs, output_layer)
You can refer this complete VAE example:
https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py
I want to train a specific conditional GAN with some deterministic constraints at the end of my generator with Keras and to do so I need first to compute the embeddings of my Generator outputs with VGG-16 pre-trained model.
I'm using python 3.6.
In my computational Graph, I want to feed my Generator outputs img to a pre-trained VGG-16 model in order to get the embeddings.
My img is then a tensor of shape (None,224,224,3) since I am in the computational Graph. Thing is if i compile the following i get the error
When feeding symbolic tensors to a model, we expect the tensors to
have a static batch size. Got tensor with shape: (None, 224, 224, 3)
self.vgg = self.build_vgg()
def build_vgg(self):
vgg16_model = keras.applications.vgg16.VGG16()
return Model(inputs=vgg16_model.input,outputs=vgg16_model.get_layer('fc2').output)
#-------------------------------
# Construct Computational Graph
# for Generator
#-------------------------------
# For the generator we freeze the critic's layers
self.critic.trainable = False
self.generator.trainable = True
self.vgg.trainable = False
# Sampled noise for input to generator
noise = Input(shape=(self.latent_dim,))
# Input Embedding:
embedding = Input(shape=(self.embedding,))
# Generate images based of noise
img = self.generator([noise,embedding])
# Discriminator determines validity
valid = self.critic(img)
# Get the embeddings from vgg-16:
X = self.vgg.predict(img)
Obviously, I can't loop along the first axis since it's None index. I tried to apply a function to this 'img' tensor using the tensorflow function 'tf.map_fn' like the following :
def Embedding(self,img):
fn = lambda x: self.vgg.predict(preprocess_input(np.expand_dims(x, axis=0))).flatten()
embedding = tf.map_fn(fn,img,dtype=tf.float32)
return embedding
#-------------------------------
# Construct Computational Graph
# for Generator
#-------------------------------
# For the generator we freeze the critic's layers
self.critic.trainable = False
self.generator.trainable = True
self.vgg.trainable = False
# Sampled noise for input to generator
noise = Input(shape=(self.latent_dim,))
# Input Embedding:
embedding = Input(shape=(self.embedding,))
# Generate images based of noise
img = self.generator([noise,embedding])
# Discriminator determines validity
valid = self.critic(img)
# Get the embeddings from VGG16
X = self.Embedding(img)
But i get the following error:
ValueError: setting an array element with a sequence.
To recap, I want to apply a pre-trained VGG-16 model on a tensor with shape (None,224,224,3) along the Batch_Size Axis (0) in the computational graph in Keras. What i explained to you before is what I already tried...
Does anyone have any suggestion to this ?
(Apologies for the long post)
All,
I want to use the bottleneck features from a pretrained Inceptionv3 model to predict classification for my input images. Before training a model and predicting classification, I tried 3 different approaches for extracting the bottleneck features.
My 3 approaches yielded different bottleneck features (not just in values but even the size was different).
Size of my bottleneck features from Approach 1 and 2: (number of input images) x 3 x 3 x 2048
Size of my bottleneck features from Approach 3: (number of input images) x 2048
Why are the sizes different between the Keras based Inceptionv3 model and the native Tensorflow model? My guess is that when I say include_top=False in Keras, I'm not extracting the 'pool_3/_reshape:0' layer. Is this correct? If yes, how do I extract the 'pool_3/_reshape:0' layer in Keras? If my guess is incorrect, what 'am I missing?
I compared the bottleneck feature values from Approach 1 and 2 and they were significantly different. I think I'm feeding it the same input images because I resize and rescale my images before I even read it as input for my script. I have no options for my ImageDataGenerator in Approach 1 and according to the documentation for that function all the default values do not change my input image. I have set shuffle to false so I assumed that predict_generator and predict are reading images in the same order. What 'am I missing?
Please note:
My inputs images are in RGB format (so number of channels = 3) and I resized all of them to 150x150. I used the preprocess_input function in inceptionv3.py to preprocess all my images.
def preprocess_input(image):
image /= 255.
image -= 0.5
image *= 2.
return image
Approach 1: Used Keras with tensorflow as backend, an ImageDataGenerator to read my data and model.predict_generator to compute bottleneck features
I followed the example (Section Using the bottleneck features of a pre-trained network: 90% accuracy in a minute) from Keras' blog. Instead of VGG model listed there I used Inceptionv3. Below is the snippet of code I used
(code not shown here but what i did before the code below) : read all input images, resize to 150x150x3, rescale according to the preprocessing_input function mentioned above, save the resized and rescaled images
train_datagen = ImageDataGenerator()
train_generator = train_datagen.flow_from_directory(my_input_dir, target_size=(150,150),shuffle=False, batch_size=16)
# get bottleneck features
# use pre-trained model and exclude top layer - which is used for classification
pretrained_model = InceptionV3(include_top=False, weights='imagenet', input_shape=(150,150,3))
bottleneck_features_train_v1 = pretrained_model.predict_generator(train_generator,len(train_generator.filenames)//16)
Approach 2: Used Keras with tensorflow as backend, my own reader and model.predict to compute bottleneck features
Only difference between this approach and earlier one is that I used my own reader to read the input images.
(code not shown here but what i did before the code below) : read all input images, resize to 150x150x3, rescale according to the preprocessing_input function mentioned above, save the resized and rescaled images
# inputImages is a numpy array of size <number of input images x 150 x 150 x 3>
inputImages = readAllJPEGsInFolderAndMergeAsRGB(my_input_dir)
# get bottleneck features
# use pre-trained model and exclude top layer - which is used for classification
pretrained_model = InceptionV3(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))
bottleneck_features_train_v2 = pretrained_model.predict(trainData.images,batch_size=16)
Approach 3: Used tensorflow (NO KERAS) compute bottleneck features
I followed retrain.py to extract bottleneck features for my input images. Please note that that the weights from that script can be obtained from (http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz)
As mentioned in that example, I used the bottleneck_tensor_name = 'pool_3/_reshape:0' as the layer to extract and compute bottleneck features. Similar to the first 2 approaches, I used resized and rescaled images as input to the script and I called this feature list bottleneck_features_train_v3
Thank you so much
Different results between 1 and 2
Since you haven't shown your code, I (maybe wrongly) suggest that the problem is that you might not have used preprocess_input when declaring ImageDataGenerator ?
from keras.applications.inception_v3 import preprocess_input
train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
Make sure, though, that your saved image files range from 0 to 255. (Bit depth 24).
Different shapes between 1 and 3
There are three possible types of model in this case:
include_top = True -> this will return classes
include_top = False (only) -> this implies in pooling = None (no final pooling layer)
include_top = False, pooling='avg' or ='max' -> has a pooling layer
So, your declared model without an explicit pooling=something doesn't have the final pooling layer in keras. Then the outputs will still have the spatial dimensions.
Solve that simply by adding a pooling at the end. One of these:
pretrained_model = InceptionV3(include_top=False, pooling = 'avg', weights='imagenet', input_shape=(img_width, img_height, 3))
pretrained_model = InceptionV3(include_top=False, pooling = 'max', weights='imagenet', input_shape=(img_width, img_height, 3))
Not sure which one the model in the tgz file is using.
As an alternative, you can also get another layer from the Tensorflow model, the one coming immediately before 'pool_3'.
You can look into the Keras implementation of inceptionv3 here:
https://github.com/keras-team/keras/blob/master/keras/applications/inception_v3.py
so, the default parameter is:
def InceptionV3(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000):
Notice that default for pooling=None, then when building the model, the code is:
if include_top:
# Classification block
x = GlobalAveragePooling2D(name='avg_pool')(x)
x = Dense(classes, activation='softmax', name='predictions')(x)
else:
if pooling == 'avg':
x = GlobalAveragePooling2D()(x)
elif pooling == 'max':
x = GlobalMaxPooling2D()(x)
# Ensure that the model takes into account
# any potential predecessors of `input_tensor`.
if input_tensor is not None:
inputs = get_source_inputs(input_tensor)
else:
inputs = img_input
# Create model.
model = Model(inputs, x, name='inception_v3')
So if you do not specify the pooling the bottleneck feature is extracted without any pooling, you need to specify if you want to get an average pooling or max pooling on top of these feature.
I built a keras model that takes an image as input and performs several convolutions and a pooling operation, then performs a specialized convolution layer with pre-initialized weights. When run on an image, this model outputs an array of the correct shape, but with all the elements as NaN.
The first part of the model is the first "block" of the pretrained VGG16 model for keras. The specialized layer (keras.layers.Conv2D) takes its weights as a set of filters corresponding to certain features I want to extract from the image. It does not matter if i flip the filters (to do cross-correlation), or if i change the image, always NaN. Any ideas?
EDIT: here is code. Takes a numpy image array as input.
def make_model(features, layer_name="block2_conv1"):
vgg = VGG16(include_top=False)
layer = vgg.get_layer(layer_name)
x = layer.output
num_chars, char_w, char_h, char_filters = features.shape
filters = features.transpose((1, 2, 3, 0)).astype(int)
filters = filters / np.sqrt(np.sum(np.square(filters), axis=(0, 1), keepdims=True))
x = BatchNormalization()(x)
specialized_layer = Conv2D(num_chars, (char_w, char_h))
x = specialized_layer(x)
biases = np.zeros((num_chars, ))
specialized_layer.set_weights([filters, biases])
model = Model(inputs=vgg.input, outputs=x)
return model