I want to use keras.layers.Embedding in a customized sub-model. But output shape is 'multiple'.Then I try to write a demo and test it
The results of the two methods are different,and I'm not sure if it's my problem.
model1 return me a clear output shape
but model2 give me a 'multiple'
here is the full demo code:
import tensorflow as tf
import tensorflow.keras as keras
import numpy as np
myPath = "./data/"
embedding_matrix = np.load(myPath + "embeddings_matrix.npy",allow_pickle=True)
model1=keras.Sequential([
keras.layers.InputLayer(5),
keras.layers.Embedding(len(embedding_matrix), 100,weights=[embedding_matrix],trainable=False,mask_zero=True,name="embedding_layer"),
keras.layers.Dense(200)
],name="model1")
model1.build((None,5))
model1.summary()
class myModel(keras.Model):
def __init__(self,
embedding_matrix,
embedding_out=100,
**kwargs):
super(myModel, self).__init__(name='model2')
self.input_x = keras.layers.InputLayer(input_shape=5)
self.embedding_layer = keras.layers.Embedding(len(embedding_matrix), embedding_out,weights=[embedding_matrix],trainable=False,mask_zero=True,name="embedding",input_length=5)
self.dense = keras.layers.Dense(200)
def call(self,x):
x = self.input_x(x)
y = self.embedding_layer(x)
y = self.dense(y)
return y
model2 = myModel(embedding_matrix)
model2.build((None,5))
model2.summary()
Here is the model.summary() info:
Model: "model1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding_layer (Embedding) (None, 5, 100) 5000400
_________________________________________________________________
dense (Dense) (None, 5, 200) 20200
=================================================================
Total params: 5,020,600
Trainable params: 20,200
Non-trainable params: 5,000,400
_________________________________________________________________
Model: "model2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, 5)] 0
_________________________________________________________________
embedding (Embedding) multiple 5000400
_________________________________________________________________
dense_1 (Dense) multiple 20200
=================================================================
Total params: 5,020,600
Trainable params: 20,200
Non-trainable params: 5,000,400
_________________________________________________________________
I want to know the right way to use a keras.layers.Embedding in a customized sub-model
Related
What I want to do is to calculate correlations within the model and use the correlation results as input for a next layer. I could compute the correlations beforehand, however I have a lot of input features and calculating all feature inter-correlations is not feasible. My idea is to reduce the features down to a manageable size and then compute their correlations. Here is a minimal example where I stumbled upon a problem:
from tensorflow import keras
import tensorflow_probability as tfp
def the_corr(x):
return tfp.stats.correlation(x, sample_axis = 1)
input = keras.Input(shape=(100,3000,))
x = keras.layers.Conv1D(filters=64, kernel_size=1,activation='relu') (input)
x = keras.layers.Lambda(the_corr, output_shape=(64,64,)) (x)
#x = keras.layers.Dense(3) (x)
model = keras.Model(input, x)
model.summary()
However, this is the summary result:
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_9 (InputLayer) [(None, 100, 3000)] 0
conv1d_3 (Conv1D) (None, 100, 64) 192064
lambda_8 (Lambda) (None, None, None) 0
=================================================================
Total params: 192,064
Trainable params: 192,064
Non-trainable params: 0
_________________________________________________________________
The lambda layer produces not the correct output shape, and completely ignores the option output_shape=(64,64,). So obviously, if the commented line is brought back in, the following dense layer will throw an error:
ValueError: The last dimension of the inputs to a Dense layer should be defined. Found None. Full input shape received: (None, None, None)
I can also remove the sample_axis=1 option in tfp.stats.correlation(), however, then the batch axis (None) is thrown away:
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_12 (InputLayer) [(None, 100, 3000)] 0
conv1d_6 (Conv1D) (None, 100, 64) 192064
lambda_11 (Lambda) (100, 64, 64) 0
dense_5 (Dense) (100, 64, 3) 195
=================================================================
Total params: 192,259
Trainable params: 192,259
Non-trainable params: 0
_________________________________________________________________
This is also not what I want, as the batch samples are independent and should not be brought together.:
What am I doing wrong? Is this even possible?
You can try to set keepdims=True in tfp.stats.corr:
def the_corr(x):
x = tfp.stats.correlation(x,
sample_axis=1,
keepdims=True)
# Keepdims will give an extra dim.
x = tf.squeeze(x, axis = 1)
return x
input = keras.Input(shape=(100,3000,))
x = keras.layers.Conv1D(filters=64, kernel_size=1,activation='relu') (input)
x = keras.layers.Lambda(the_corr)(x)
x = keras.layers.Dense(3)(x)
model = keras.Model(input, x)
model.summary()
Summary:
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 100, 3000)] 0
conv1d (Conv1D) (None, 100, 64) 192064
lambda (Lambda) (None, 64, 64) 0
dense (Dense) (None, 64, 3) 195
=================================================================
Total params: 192,259
Trainable params: 192,259
Non-trainable params: 0
I am trying to train an auto encoder in tensorflow using the Keras Layer API. This API is quite nice and easy to use to setup the deep learning layers.
Just to review a quickly an autoencoder (in my mind) is a function $f(x) = z$ and its pseudo inverse \hat{x} = f^{-1}(z) such that f(f^{-1}(x)) \approx x. In a neural network model, you would setup a neural network with a bottleneck layer that tries to predict itself x using f^{-1}(f(x)). When the training error minimizes, you then have two components, z = f(x) is the prediction up until and including the bottleneck layer. f^{-1}(z) is the bottleneck layer to the end.
So I setup the encoder:
SZ = 6
model = tf.keras.Sequential()
model.add(layers.InputLayer(SZ))
model.add(layers.Dense(SZ))
model.add(layers.Dense(1))
model.add(layers.Dense(SZ))
model.summary()
model.compile('sgd','mse',metrics = ['accuracy'])
history= model.fit(returns.values,returns.values,epochs=100)
My difficulty here is that the weights and components (f being input+dense(SZ)+dense(1),f^{-1} being dense(1)+dense(SZ)) are trained but I do not know how to disentangle them. Is there some way to break off the two layers in the neural network and have them treated as their own separate models?
import tensorflow as tf
SZ=6
encoder_input = tf.keras.layers.Input(shape=(SZ,))
x = tf.keras.layers.Dense(SZ)(encoder_input)
x = tf.keras.layers.Dense(1)(x)
encoder_model = tf.keras.Model(inputs=encoder_input, outputs=x, name='encoder')
decoder_input = tf.keras.layers.Input(shape=(1,))
x2 = tf.keras.layers.Dense(SZ)(decoder_input)
decoder_model = tf.keras.Model(inputs=decoder_input, outputs=x2, name='decoder')
encoder_output = encoder_model(encoder_input)
decoder_output = decoder_model(encoder_output)
encoder_decoder_model = tf.keras.Model(inputs=encoder_input , outputs=decoder_output, name='encoder-decoder')
encoder_decoder_model.summary()
Here is the summary:
Model: "encoder-decoder"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_8 (InputLayer) [(None, 6)] 0
_________________________________________________________________
encoder (Model) (None, 1) 49
_________________________________________________________________
decoder (Model) (None, 6) 12
=================================================================
Total params: 61
Trainable params: 61
Non-trainable params: 0
you could train the encoder-decoder model and you separate encoder_model and decoder_model will be trained automatically. You could also retrieve them from your encoder_decoder model as follows:
retrieved_encoder = encoder_decoder_model.get_layer('encoder')
retrieved_encoder.summary()
it prints:
Model: "encoder"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_8 (InputLayer) [(None, 6)] 0
_________________________________________________________________
dense_11 (Dense) (None, 6) 42
_________________________________________________________________
dense_12 (Dense) (None, 1) 7
=================================================================
Total params: 49
Trainable params: 49
Non-trainable params: 0
and the decoder:
retrieved_decoder = encoder_decoder_model.get_layer('decoder')
retrieved_decoder.summary()
which prints:
Model: "decoder"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_9 (InputLayer) [(None, 1)] 0
_________________________________________________________________
dense_13 (Dense) (None, 6) 12
=================================================================
Total params: 12
Trainable params: 12
Non-trainable params: 0
I'm trying to create a function that splits a Keras model on a user specified layer. I have the following code:
def return_split_models(model, layer):
model_f, model_h = Sequential(), Sequential()
for current_layer in range(0, layer+1):
model_f.add(model.layers[current_layer])
for current_layer in range(layer+1, len(model.layers)):
model_h.add(model.layers[current_layer])
return model_f, model_h
However, when we return model_h and call a summary, we will see a ValueError that the model has never been called. From looking at other posts it seems like this has to do with the inputs for model_h, however I cannot find examples which generalize to any specified layer. Does anyone have any guidance?
You need to add InputLayer to model_h.
from keras.layers import InputLayer
def return_split_models(model, layer):
model_f, model_h = Sequential(), Sequential()
for current_layer in range(0, layer+1):
model_f.add(model.layers[current_layer])
# add input layer
model_h.add(InputLayer(input_shape=model.layers[layer+1].input_shape[1:]))
for current_layer in range(layer+1, len(model.layers)):
model_h.add(model.layers[current_layer])
return model_f, model_h
An example:
model = Sequential()
model.add(Dense(50,input_shape=(100,)))
model.add(Dense(40))
model.add(Dense(30))
model.add(Dense(20))
model.add(Dense(10))
model_f, model_h = return_split_models(model, 2)
print(model_f.summary())
print(model_h.summary())
# print
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_1 (Dense) (None, 50) 5050
_________________________________________________________________
dense_2 (Dense) (None, 40) 2040
_________________________________________________________________
dense_3 (Dense) (None, 30) 1230
=================================================================
Total params: 8,320
Trainable params: 8,320
Non-trainable params: 0
_________________________________________________________________
None
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_4 (Dense) (None, 20) 620
_________________________________________________________________
dense_5 (Dense) (None, 10) 210
=================================================================
Total params: 830
Trainable params: 830
Non-trainable params: 0
_________________________________________________________________
None
inputs_bedding = Input(shape=(it.shape))
embedding = Embedding(9488, 512, trainable=False)(inputs_bedding)
There is no name parameter in keras Embedding layer. How to set the name to the layer?
You can set the name of the embedding layer just like any other layer.
from keras.layers import Embedding, Input
from keras import Model
inputs_bedding = Input(shape=(32,))
embedding = Embedding(9488, 512, trainable=False, name="test")(inputs_bedding)
model = Model(inputs=inputs_bedding, outputs=embedding)
model.summary() gives you:
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) (None, 32) 0
_________________________________________________________________
test (Embedding) (None, 32, 512) 4857856
=================================================================
Total params: 4,857,856
Trainable params: 0
Non-trainable params: 4,857,856
_________________________________________________________________
Here I have a GoogleNet model for Keras. Is there any possibile way to block the changing of the individual layers of network? I want to block the first two layers of pretrained model from changes.
By 'block the changing of the individual layers' I am assuming you don't want to train those layers, that is you don't want to modify the loaded weights(possibly learnt in previous training).
if so you can pass trainable=False to the layer and the parameters wont be used for the training update rule.
Example:
from keras.models import Sequential
from keras.layers import Dense, Activation
model = Sequential([
Dense(32, input_dim=100),
Dense(output_dim=10),
Activation('sigmoid'),
])
model.summary()
model2 = Sequential([
Dense(32, input_dim=100,trainable=False),
Dense(output_dim=10),
Activation('sigmoid'),
])
model2.summary()
You can see in the model summary for the 2nd model the parameters are counted as Non-trainable ones.
____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
dense_1 (Dense) (None, 32) 3232 dense_input_1[0][0]
____________________________________________________________________________________________________
dense_2 (Dense) (None, 10) 330 dense_1[0][0]
____________________________________________________________________________________________________
activation_1 (Activation) (None, 10) 0 dense_2[0][0]
====================================================================================================
Total params: 3,562
Trainable params: 3,562
Non-trainable params: 0
____________________________________________________________________________________________________
____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
dense_3 (Dense) (None, 32) 3232 dense_input_2[0][0]
____________________________________________________________________________________________________
dense_4 (Dense) (None, 10) 330 dense_3[0][0]
____________________________________________________________________________________________________
activation_2 (Activation) (None, 10) 0 dense_4[0][0]
====================================================================================================
Total params: 3,562
Trainable params: 330
Non-trainable params: 3,232