In keras I would like to use the model with the initial layers of the structure for a given trained neuralnet with the weights I got for the training process.
Going to the case: Lets imagine we have a dataset df, after spliting into train, dev and test we train a neural network, for this example an autoencoder.
A real piece of code illustrating this concept, without providing data(i didn't consider it necessary):
from keras.models import Model
from keras.layers import Activation, Dense, Dropout, Input
# Define input layer
input_data = Input(shape=(train.shape[1],), name='Input')
# Define encoding layer
encoded = Dense(encoding_dim, activation='relu')(input_data)
# Define decoding layer
decoded = Dense(train.shape[1], activation='sigmoid')(encoded)
# Create the autoencoder model
autoencoder = Model(input_data, decoded, name='Simple AutoEncoder')
#Compile the autoencoder model
autoencoder.compile(optimizer='rmsprop',
loss='binary_crossentropy')
autoencoder.fit(train, train,
epochs=50,
batch_size=256,
shuffle=True,
validation_data=(dev_x, dev_x), verbose=0)
After compile and fit the model we have a neural network with their weights that we got from fitting process.
How could I use only the encoder part of this net by preserving the weight I got?
I believe something along this line should do the trick:
#...all the code from above, including training...
# Define the encoder model
encoder = Model(input_data, encoded, name='Encoder')
The encoder model can be treated as a fully-fledged Keras model (you can save/load/fit/evaluate/predict).
By training an Autoencoder, the encoder neuralnet part would be created with the encoded object that contains the trained weights of the autoencoder.
# Getting the trained weights of the first layer(dense layer of encoder)
weights_ae = autoencoder.layers[1].get_weights()[0]
# The previous code of the example...
# Creating the encoder model
encoder = Model(input_data, encoded, name='Encoder')
# Getting the weights of the encoder model
weights_e = encoder.layers[1].get_weights()[0]
So, finally it would be confirmed that by creating the model encoder would have the weights ("trainied experience") from the autoencoder.
Related
Sample code from https://blog.keras.io/building-autoencoders-in-keras.html
import keras
from keras import layers
# This is our input image
input_img = keras.Input(shape=(784,))
# "encoded" is the encoded representation of the input
encoded = layers.Dense(encoding_dim, activation='relu')(input_img)
# "decoded" is the lossy reconstruction of the input
decoded = layers.Dense(784, activation='sigmoid')(encoded)
# This model maps an input to its reconstruction
autoencoder = keras.Model(input_img, decoded)
# This model maps an input to its encoded representation
encoder = keras.Model(input_img, encoded)
# This is our encoded (32-dimensional) input
encoded_input = keras.Input(shape=(encoding_dim,))
# Retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]
# Create the decoder model
decoder = keras.Model(encoded_input, decoder_layer(encoded_input))
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
autoencoder.fit(x_train, x_train,
epochs=50,
batch_size=256,
shuffle=True,
validation_data=(x_test, x_test))
...
# Encode and decode some digits
# Note that we take them from the *test* set
encoded_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)
In the example, only model autoencoder had been compiled and fitted, encoder is not.
I am so confused, why encoder can predict new data directly without any compiling and fitting?
encoder is a sub-model inside autoencoder (specifically, the part between input_img and encoded), it's not a separate model: you're just referring to a part of autoencoder with an explicit name.
When you train autoencoder, you're training both the encoder and decoder parts simultaneously. After training, encoder refers to the trained sub-model, that you can then use for inference.
I am currently building an auto-encoder for the MNIST dataset with Kears, here is my code:
import all the dependencies
from keras.layers import Dense,Conv2D,MaxPooling2D,UpSampling2D
from keras import Input, Model
from keras.datasets import mnist
import numpy as np
import matplotlib.pyplot as plt
encoding_dim = 15
input_img = Input(shape=(784,))
# encoded representation of input
encoded = Dense(encoding_dim, activation='relu')(input_img)
# decoded representation of code
decoded = Dense(784, activation='sigmoid')(encoded)
# Model which take input image and shows decoded images
autoencoder = Model(input_img, decoded)
# This model shows encoded images
encoder = Model(input_img, encoded)
# Creating a decoder model
encoded_input = Input(shape=(encoding_dim,))
# last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]
# decoder model
decoder = Model(encoded_input, decoder_layer(encoded_input))
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
the last step is the compiling step, but I need to use a L1-norm reconstruction loss function. From Keras losses description, it seems they don't have this function. How can I apply a L1-norm reconstruction loss function to the autoencoder.compile() function? Thank you!
In loss function, we refer to the expected error values. Hence, for L1-norm you can use MAE (mean absolute error) with the name of mean_absolute_error. So, you can rewrite the last line of your code as follow:
autoencoder.compile(optimizer='adam', loss='mean_absolute_error')
I am trying to use the transfer learning example on keras.applications using VGG19. I am trying to train on the cifar10 dataset, so 10 classes. My model is (conceptually) simple as it's just VGG 19 minus the top three layers and then some extra layers that are trainable.
import tensorflow as tf
from keras.utils import to_categorical
from keras.applications import VGG19
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D, Input
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
#%%
# Specify input and number of classes
input_tensor = Input(shape=(32, 32, 3))
num_classes=10
#Load the data (cifar100), if label mode is fine then 100 classes
(X_train,y_train),(X_test,y_test)=tf.keras.datasets.cifar10.load_data()
#One_Hot_encode y data
y_test=to_categorical(y_test,num_classes=num_classes,dtype='int32')
y_train=to_categorical(y_train,num_classes=num_classes,dtype='int32')
#%%
# create the base pre-trained model
base_model = VGG19(weights='imagenet', include_top=False,
input_tensor=input_tensor)
# Add a fully connected layer and then a logistic layer
x = base_model.output
# # let's add a fully-connected layer
x = Dense(1024, activation='relu',name='Fully_Connected')(x)
# and a logistic layer -- let's say we have 200 classes
predictions = Dense(num_classes, activation='softmax',name='Logistic')(x)
# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
layer.trainable = False
# compile the model (should be done *after* setting layers to non-trainable)
model.compile(optimizer='adam', loss='categorical_crossentropy',metrics=['accuracy'])
# train the model on the new data for a few epochs
model.fit(X_train,y_train,epochs=10)
#%%
model.evaluate(X_test,y_test)
Now when I try to train using X_train [dimensions of (50000, 32, 32, 3)] and y_test (dimensions of (50000,10),
I get an error:
ValueError: Error when checking target: expected Logistic to have 4
dimensions, but got array with shape (50000, 10)
So for some reason the model isn't realizing that its output shape should be a 1x10 vector with one-hot encoding for the 10 classes.
How can I make it so that the dimensions agree? I don't fully understand the dimensions of output that keras is expecting here. When I do model.summary() the Logistic layer yields that the output shape should be (None, 1, 1, 10), which when flattened should just give a
VGG19 without the top layers does not return a fully-connected layer and instead returns a 2D feature space (output of a Conv2D/max pooling2d I believe). You'll probably want to place a flatten after the VGG, that'd be the best practical choice, as it will make your output shape (None,10).
Otherwise, you could do
y_train = np.reshape(y_train, (50000,1,1,10))
I tried to build a simple Autoencoder using Keras for this I started with a single fully-connected neural layer as an encoder and as a decoder.
> input_img = Input(shape=(784,))
>encoded = Dense(encoding_dim,activation='relu')(input_img)
>decoded = Dense(784, activation='sigmoid')(encoded)
>autoencoder =Model(input_img, decoded)
I also created a separate encoder module with the help of
encoder = Model(input_img, encoded)
As well as the decoder model:
encoded_input = Input(shape=(32,))
# retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]
# create the decoder model
decoder = Model(encoded_input, decoder_layer(encoded_input))
Then I trained the model
autoencoder.fit(x_train, x_train,
epochs=50,
batch_size=256,
shuffle=True,
validation_data=(x_test, x_test))
but even if i didn't train my encoder and decoder, Those are sharing the weights of autoencoder even if I passed the layers before training. I trained only the encoder but both encoder and decoder are getting trained.
encoded_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)
I should have been more careful while reading the text.
If two Keras models are sharing some layers, when you train the first model, the weights from the shared layers will be updated automatically in the other model.
https://keras.io/getting-started/functional-api-guide/
This blog illustrates the use of shared layers nicely.
https://blog.keras.io/building-autoencoders-in-keras.html/
A keras sequential model with embedding needs to be retrained starting from the currently known weights.
A Keras sequential model is trained on the provided (text) training data. The training data is tokenized by a (custom made) tokenizer. The input dimension for the first layer in the model, a embedding layer, is the number of words known by the tokenizer.
After a few days additional training data becomes available. The tokenizer needs to be refitted on this new data as it may contain additional words. That means that the input dimension of the embedding layer changes, so the previously trained model is not usable anymore.
self.model = Sequential()
self.model.add(Embedding(tokenizer.totalDistinctWords + 1,
hiddenSize + 1, batch_size=1,
input_length=int(self.config['numWords'])))
self.model.add(LSTM(hiddenSize, return_sequences=True,
stateful=True, activation='tanh', dropout = dropout))
self.model.add(LSTM(hiddenSize, return_sequences=True,
stateful=True, activation='tanh', dropout = dropout))
self.model.add(TimeDistributed(Dense(
len(self.controlSupervisionConfig.predictableOptionsAsList))))
self.model.add(Activation('softmax'))
I want to use the previously trained model as initializer for the new training session. For the new words in the tokenizer, the embedding layer should just use a random initialization. For the words already known by the tokenizer, it should use the previously trained embedding.
You can access (get and set) the layer's weights directly as numpy arrays with code like weights = model.layers[0].get_weights() and model.layers[0].set_weighs(weights, where model.layers[0] is your Embedding layer. This way you can store embeddings separately and set known embeddings by copying them from stored data.