from keras.models import Sequential
from keras.layers.embeddings import Embedding
from theano import function
model = Sequential()
model.add(Embedding(max_features, 128, input_length = maxlen))
I want to get the outputs from the embedding layers. I read through the source in keras but didnt find any suitable function or attribute. Anyone can help me with this?
You can get the output of any layer, not just an embedding layer, as described here:
from keras import backend as K
get_3rd_layer_output = K.function([model.layers[0].input],
[model.layers[3].output])
layer_output = get_3rd_layer_output([X])[0]
In your case, you would want model.layers[0].output instead of model.layers[3].output.
Related
I want to apply this method to implement Bi-LSTM with attention. The method is discussed here: Bi-LSTM Attention model in Keras
I get the following error:
'module' object is not callable
It can not apply multiply in this line:
sent_representation = merge([lstm, attention], mode='mul')
from keras.layers import merge
import tensorflow as tf
from tensorflow.keras.layers import Concatenate, Dense, Input, LSTM, Embedding, Dropout, Activation, Flatten, Permute, RepeatVector
from tensorflow.keras.layers import Bidirectional
inp =Input(shape=(maxlen,), dtype='float32')
x = Embedding(max_features, embed_size, weights=[emb_matrix])(inp)
lstm = Bidirectional(LSTM(50, return_sequences=True), name="bi_lstm_0")(x)
attention = Dense(1, activation='tanh')(lstm)
attention = Flatten()(attention)
attention = Activation('softmax')(attention)
attention = RepeatVector(max_features*2)(attention)
attention = Permute([2,1])(attention)
sent_representation = merge([lstm, attention], mode='mul')
sent_representation = Lambda(lambda xin: K.sum(xin, axis=1))(sent_representation)
output = Dense(3, activation="softmax")(sent_representation)
In Keras, merge is a module that contains layers that implement various ways of merging outputs of other layers. You need to select a method that you want to use to merge the states.
In this particular case, you want to concatenate the outputs.
How do I set the initial state of the recurrent neural network rnn constructed below?
from tensorflow.keras.layers import Dense, SimpleRNN
from tensorflow.keras.models import Sequential
rnn = Sequential([SimpleRNN(3), Dense(1)])
I'd like to specify the initial state of the first layer before fitting the model with model.fit.
According to the tf.keras.layers.RNN documentation, you can specify the initial states symbolically using the argument initial_state or numerically by calling the function reset_states.
Symbolic specification means you need to add the initial states as a input to your model. Here is an example I adapted from the Keras tests:
from tensorflow.keras.layers import Dense, SimpleRNN, Input
from tensorflow.keras.models import Model
import numpy as np
import tensorflow as tf
timesteps = 3
embedding_dim = 4
units = 3
inputs = Input((timesteps, embedding_dim))
# initial state as Keras Input
initial_state = Input((units,))
rnn = SimpleRNN(units)
hidden = rnn(inputs, initial_state=initial_state)
output = Dense(1)(hidden)
model = Model([inputs] + [initial_state], output)
model.compile(loss='categorical_crossentropy',
optimizer=tf.compat.v1.train.AdamOptimizer())
And once your model is defined, you can perform training as follows:
num_samples = 2
inputs = np.random.random((num_samples, timesteps, embedding_dim))
# random initial state as additional input
some_initial_state = np.random.random((num_samples, units))
targets = np.random.random((num_samples, units))
model.train_on_batch([inputs] + [some_initial_state], targets)
Note that this approach requires you to use the Functional API. For Sequential models, you will need to use a stateful RNN, specify a batch_input_shape, and call the reset_states method:
input_shape = (num_samples, timesteps, embedding_dim)
model = Sequential([
SimpleRNN(3, stateful=True, batch_input_shape=input_shape),
Dense(1)])
some_initial_state = np.random.random((num_samples, units))
rnn = model.layers[0]
rnn.reset_states(states=some_initial_state)
I trained my model and saved it:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.preprocessing.image import img_to_array
new_model=tf.keras.models.load_model('the_model.h5')
new_model.summary()
img = load_img('e.jpg',target_size=(227,227))
img=img_to_array(img)
img = np.expand_dims(img,axis=0)
img=img/255.
print(img.shape)
#prints out (1,227,227,3) the expected shapes
so the architecture of my model is the following one, i'm using pre-trained resnet50
backbone = ResNet50(input_shape=(227,227,3),weights='imagenet', include_top=False)
model = Sequential()
model.add(backbone)
model.add(GlobalAveragePooling2D())
model.add(Dropout(0.5))
model.add(Dense(64,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1,activation='sigmoid'))
i tried visualize outputs of hidden layers, however with keras or keract i can't get the outputs
with keras :
layer_outputs=[]
for layer in new_model.layers:
if layer.name=='resnet50':
temp = [l.output for l in layer.layers]
layer_outputs=temp
else:
layer_outputs.append(layer.output)
activation_model = Model(inputs=new_model.input, outputs=layer_outputs)
the error caused by the last line :
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(None, 227, 227, 3), dtype=float32) at layer "input_1". The following previous layers were accessed without issue: []
i feel like my model inputs are matching with layer_outputs so i don't really understand the error, indeed when i'm checking :
print(new_model.layers[0].input)
#prints out :Tensor("input_1:0", shape=(None, 227, 227, 3), dtype=float32)
print(layer_outputs[0])
#prints out : Tensor("input_1:0", shape=(None, 227, 227, 3), dtype=float32)
when using keract :
a = keract.get_activations(new_model, img) # with just one sample.
keract.display_activations(a, directory='f', save=True)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'input_1' with dtype float and shape [?,227,227,3]
Any idea of how i can fix it, or another viable solution to get outputs from hidden layers using pre-trained model?
Thanks,
Okey, i found a convenient solution to my problem.
Indeed i think this problem occurs because my sequential model is itself made up of another model (resnet).
Since i didn't add many layers on top of the pre-trained resnet model, i just decided to visualize the feature maps from the resnet model
img = load_img('e.jpg',target_size=(227,227))
img=img_to_array(img)
img = np.expand_dims(img,axis=0)
img=img/255.
print(img.shape)
loaded=tf.keras.models.load_model('age_gender_train.h5')
layer_outputs=[ layer.output for layer in loaded.layers[0].layers]
res = loaded.layers[0]
activation_model = Model(inputs=res.input, outputs=layer_outputs)
activations=activation_model.predict(img)
img = np.squeeze(img,axis=0)
Then you can easily display features maps using activations variable.
Note that since you have the outputs of the resnet model, it might be possible to get the feature maps from the layers on top by repeating the process. Using the output of resnet as input and removing the resnet model from the layer_outputs.(i did not try this, could not work)
Hope it could help someone
I am trying to design a Bi-Directional LSTM model and I want to concatenate features after Max pooling and Average pooling layers.
I have this for my model:
from keras.layers import Dense, Embedding
from keras.layers.recurrent import LSTM
from keras.layers import Bidirectional
from keras.models import Sequential
from keras.layers.core import Dropout
from features import train,embedding_matrix,words
from keras.layers import concatenate,AveragePooling1D,GlobalMaxPooling1D
model=Sequential()
model.add(Embedding(words,300,input_length=train.shape[1],weights=[embedding_matrix]))
model.add(Bidirectional(LSTM(20,activation='tanh',kernel_initializer='glorot_uniform',recurrent_dropout = 0.2, dropout = 0.2,return_sequences=True)))
model.add(concatenate([GlobalMaxPooling1D(),AveragePooling1D()]))
model.add(Dropout(0.2))
model.add(Dense(2, activation='softmax'))
print model.summary()
But I am having:
ValueError: Layer concatenate_1 was called with an input that isn't a symbolic tensor which is because I believe the concatenating layer. As I am not adding the pooling in the model.
Can I add two layers in the same model? or Should I define two separate models and then add pooling layers in each of them?
The trick here is to use a graph model instead of a sequential model.
Before we get started, I assume
your network expects a 2D input tensor of shape (B=batch_size, N=num_of_words), where N is the longest sample length of your training data. (In case you have unequal length samples, you should use keras.preprocessing.sequence.pad_sequences to achieve equal length samples)
your vocabulary size is V (probably is 300 if I understand correctly)
your embedding layer encodes each word to a feature of F dimension, i.e. your embedding layer's weight matrix is VxF.
from keras.layers import Dense, Embedding, Input, Concatenate, Lambda
from keras.layers.recurrent import LSTM
from keras.layers import Bidirectional
from keras.models import Model
from keras.layers.core import Dropout
from keras import backend as BKN
from keras.layers import concatenate,AveragePooling1D,GlobalMaxPooling1D
words = Input( shape=(N,))
f = Embedding(input_dim=V,output_dim=F)( words )
f = Bidirectional(LSTM(20,activation='tanh',
kernel_initializer='glorot_uniform',
recurrent_dropout = 0.2,
dropout = 0.2,return_sequences=True))(f)
gpf = GlobalMaxPooling1D()(f)
gpf = Lambda( lambda t : BKN.expand_dims(t, axis=1) )(gpf)
apf = AveragePooling1D( pool_size=2 )(f)
pf = Concatenate(axis=1)([gpf, apf])
pf = Dropout(0.2)( pf )
pred = Dense(2, activation='softmax')(pf) # <-- make sure this is correct
model = Model( input=words, output=pred )
Finally, I fail to find that keras Embedding layer supports syntax like weights=[embedding_matrix].
I've been trying to implement an LSTM in Keras for several hours (using a sequential model with an embedding layer, two LSTM layers, and a dense layer), but I wind up getting different error messages.
From what I can tell, the problem is that the output of the embedding layer has two dimensions instead of three, because I get this value error (ValueError: Input 0 is incompatible with layer lstm_2: expected ndim=3, found ndim=2) when adding the second LSTM layer, and I get the error assert len(input_shape) >= 3 AssertionError when I delete the line for adding the second LSTM layer (which means the dense layer has the same issue).
These error occurs before I call the model's "train" method.
My code is here.
from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.layers import LSTM
from keras.layers import Embedding
from keras.layers import TimeDistributed
from keras.preprocessing import text
from keras.preprocessing.sequence import pad_sequences
# The data in X was preprocessed using Keras' built in pad_sequences.
# Before preprocessing, it consisted of plain lists of integers (which
# were just integers wit a one-to-one map to plain words as strings)
X = pad_sequences(X)
model = Sequential()
model.add(Embedding(batch_size=32, input_dim=len(filtered_vocabulary)+1, output_dim=256, input_length=38))
model.add(LSTM(128))
model.add(LSTM(128)) # error occurs in this line
model.add(TimeDistributed(Dense(len(filtered_vocabulary)+1, activation="softmax")))
model.compile(optimizer = "rmsprop", loss="categorical_crossentropy", metrics=["accuracy"])
model.fit(X, X, epochs=60, batch_size=32)
I'd be glad if any of you could help me out with this.
The error occurs because the first LSTM layer only returns the last output because you haven't specified return_sequences=True in the LSTM layers. This looks like a multi-output setup so you would need to return the LSTM output at every time step using that argument for both layers.
Just to be clear, without return_sequences=True the shape is (None, 128); with the shape is (None, 38, 128).