print("Building model...")
ques1_enc = Sequential()
ques1_enc.add(Embedding(output_dim=64, input_dim=vocab_size, weights=[embedding_weights], mask_zero=True))
ques1_enc.add(LSTM(100, input_shape=(64, seq_maxlen), return_sequences=False))
ques1_enc.add(Dropout(0.3))
ques2_enc = Sequential()
ques2_enc.add(Embedding(output_dim=64, input_dim=vocab_size, weights=[embedding_weights], mask_zero=True))
ques2_enc.add(LSTM(100, input_shape=(64, seq_maxlen), return_sequences=False))
ques2_enc.add(Dropout(0.3))
model = Sequential()
model.add(Merge([ques1_enc, ques2_enc], mode="sum"))
model.add(Dense(2, activation="softmax"))
model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
print("Building model costs:", time.time() - start)
print("Training...")
checkpoint = ModelCheckpoint(filepath=os.path.join("C:/Users/", "quora_dul_best_lstm.hdf5"), verbose=1, save_best_only=True)
model.fit([x_ques1train, x_ques2train], ytrain, batch_size=32, epochs=1, validation_split=0.1, verbose=2, callbacks=[checkpoint])
print("Training neural network costs:", time.time() - start)
I want to convert the above code into functional API in keras as in sequential API Merge() function is not supported. I have been trying it for long time but getting few errors. About the details of the attrributes:
ques_pairs contains the preprocessed data,
word2index contains the word count,
seq_maxlen contains the maximum length of question one or two.
iam trying to implement this model on Quora Question Pair Dataset https://www.kaggle.com/c/quora-question-pairs
I will give you a small example, that you can apply to your own model:
from keras.layers import Input, Dense, Add
input1 = Input(shape=(16,))
output1 = Dense(8, activation='relu')(input1)
output1 = Dense(4, activation='relu')(output1) # Add as many layers as you like like this
input2 = Input(shape=(16,))
output2 = Dense(8, activation='relu')(input2)
output2 = Dense(4, activation='relu')(output2) # Add as many layers as you like like this
output_full = Add()([output1, output2])
output_full = Dense(1, activation='sigmoid')(output_full) # Add as many layers as you like like this
model_full = Model(inputs=[input1, input2], outputs=output_full)
You need to define an Input for each of your model parts first, then add layers (as shown in the code) to both models. Then you can add them using the Add layer. Finally you call Model with a list of the input layers and the output layer.
model_full can then be compiled and trained like any other model.
Are you trying to achieve something like the following ?
from tensorflow.python import keras
from keras.layers import *
from keras.models import Sequential, Model
vocab_size = 1000
seq_maxlen = 32
embedding_weights = np.zeros((vocab_size, 64))
print("Building model...")
ques1_enc = Sequential()
ques1_enc.add(Embedding(output_dim=64, input_dim=vocab_size, weights=[embedding_weights], mask_zero=True))
ques1_enc.add(LSTM(100, input_shape=(64, seq_maxlen), return_sequences=False))
ques1_enc.add(Dropout(0.3))
ques2_enc = Sequential()
ques2_enc.add(Embedding(output_dim=64, input_dim=vocab_size, weights=[embedding_weights], mask_zero=True))
ques2_enc.add(LSTM(100, input_shape=(64, seq_maxlen), return_sequences=False))
ques2_enc.add(Dropout(0.3))
merge = Concatenate(axis=1)([ques1_enc.output, ques2_enc.output])
output = Dense(2, activation="softmax")(merge)
model = Model([ques1_enc.input, ques2_enc.input], output)
model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
model.summary()
Related
I had 5 LSTM layers and 2 MLP's which must be concatenate together into another MLP which produce the final output. Here is the code I wrote using the API approach, which works fine:
lstm_input = Input(shape=(X_dynamic_LSTM.shape[1], X_dynamic_LSTM.shape[2]))
x = LSTM(70, activation='tanh', return_sequences=True)(lstm_input )
x = Dropout(0.3)(x)
x = Dense(1, activation='tanh')(x)
mlp_input=Input(shape=(X_static_MLP.shape[1]))
mlp = Dense(30, activation='relu')(mlp_input)
mlp = Dense(10, activation='relu')(mlp)
merge = Concatenate()([x, mlp])
hidden1 = Dense(5, activation='relu')(merge)
mlp_out = Dense(1, activation='relu')(hidden1)
model = Model(inputs=[lstm_input, mlp_input], outputs=mlp_out)
model.compile(loss='mae', optimizer='Adam')
history = model.fit([X_dynamic_LSTM, X_static_MLP], y_train, batch_size=20,
epochs=10, validation_split=0.2)
If I want to convert this format to one similar to below:
x = Sequential()
x.add(LSTM(70, return_sequences=True))
x.add(Dropout(0.3))
x.add(Dense(1, activation='tanh'))
Can any one help me how should I define the MLP, the Concatenate and the part regarding the "model = Model(inputs=[lstm_input, mlp_input], outputs=mlp_out)" ??
My main problem is I want to add an Embedding layer to the LSTM. when I add the dollowing code to non-API approach the model works perfect.
x.add(Embedding(X_dynamic_LSTM.shape[0], 1,mask_zero=True))
But when instead I used
lstm_input = Embedding(X_dynamic_LSTM.shape[0], 1,mask_zero=True)
It gave me the error : TypeError: Inputs to a layer should be tensors, So I got to stick with non-API approach.
This is the error and data I entered into my model. I just can't figure out why it won't work since the dimensions are okay and it literally prints a list of arrays.
My Model + Code before:
import numpy as np
training = np.array(training)
training_inputs = list(training[:,0])
training_outputs = list(training[:,1])
print("train inputs ", training_inputs)
print("train outputs ", training_outputs)
# Now lets create our tensorflow model
# In[10]:
from tensorflow.python.keras import Sequential
from tensorflow.python.keras.layers import LSTM, Dense
model = Sequential()
model.add(Dense(training_inputs[0], activation='linear'))
model.add(Dense(15, activation='linear'))
model.add(Dense(15, activation='linear'))
model.add(Dense(15, activation='linear'))
model.add(Dense(len(training_outputs[0]), activation='softmax'))
model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy', 'loss']
)
model.fit(x=training_inputs, y=training_outputs,
epochs=10000,
batch_size=20,
verbose=True,
shuffle=True)
model.save('models/basic_chat.json')
You need an input layer to your model:
...
model = Sequential()
model.add(Dense(15, activation='linear', input_shape=( len(training_inputs[0]),)))
model.add(Dense(15, activation='linear'))
...
training_inputs = np.array(training[:,0])
training_outputs = np.array(training[:,1])
I have an existing LSTM model that looks as follows:
model_glove1 = Sequential()
model_glove1.add(Embedding(vocabulary_size, 25, input_length=50, weights=[embedding_matrix25],trainable=False))
model_glove1.add(LSTM(32))
model_glove1.add(Dense(128, activation='relu'))
model_glove1.add(Dense(64, activation='relu'))
model_glove1.add(Dense(1, activation='softmax'))
model_glove1.compile(loss='binary_crossentropy',optimizer='adam',metrics['accuracy',auc_roc])
model_glove1.fit(data, np.array(train_y), batch_size=32,
epochs=4,
verbose=1,
validation_split=0.1,
shuffle=True)
I want to add an additional auxiliary input layer which is present in a dataframe of 27 columns . I want that layer to be concatenated with the output of the LSTM layer. Is it possible ? If so how can I achieve it?
Before using the code, please check the secondary input has the same dimension like output of LSTM layer.
Moreover, in model1_glove.fit() function, you need to provide two inputs
def NNStructure():
initial_input= Embedding(vocabulary_size, 25, input_length=50, weights=
[embedding_matrix25],trainable=False)
lstm = LSTM(32)(initial_input)
secondary_input = Input(shape=(Number_of_row,27))
merge = concatenate([lstm, secondary_input])
first_dense = Dense(128, activation='relu')(merge)
second_dense=Dense(64, activation='relu')(first_dense)
output=Dense(1, activation='softmax')(second_dense)
model_glove1 = Model(inputs=[initial_input, secondary_input], outputs=output)
return model_glove1
model_glove1=NNStructure()
model_glove1.compile(loss='binary_crossentropy',optimizer='adam',metrics['accuracy',auc_roc])
model_glove1.fit(x=[data1,data2], y=np.array(train_y), batch_size=32,
epochs=4,
verbose=1,
validation_split=0.1,
shuffle=True)
I am trying to learn how vgg16 works. Below is my code, using vgg16 for another classification.
# Generate a model with all layers (with top)
model_vgg16_conv = VGG16(weights='imagenet', include_top=False)
model_vgg16_conv.summary()
# create your own input format
input = Input(shape=(128,128,3),name = 'image_input')
# Use the generated model
output_vgg16_conv = model_vgg16_conv(input)
# Add the fully-connected layers
x = Flatten(name='flatten')(output_vgg16_conv)
x = Dense(4096, activation='relu', name='fc1')(x)
x = Dense(4096, activation='relu', name='fc2')(x)
x = Dense(5, activation='softmax', name='predictions')(x)
#Create your own model
model = Model(input=input, output=x)
#In the summary, weights and layers from VGG part will be hidden, but they will be fit during the training
model.summary()
# Specify an optimizer to use
adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
# Choose loss function, optimization method, and metrics (which results to display)
model.compile(
optimizer = adam,
loss='categorical_crossentropy',
metrics=['accuracy']
)
model.fit(X_train,y_train,epochs=10,batch_size=10,verbose=2)
# model.fit(X_train,y_train,epochs=30,batch_size=100,verbose=2)
result = model.predict(y_test) # same result
For some reason, using different epoch size and batch size generate exactly the same result. Am I doing something wrong?
I am trying to create my first ensemble models in keras. I have 3 input values and a single output value in my dataset.
from keras.optimizers import SGD,Adam
from keras.layers import Dense,Merge
from keras.models import Sequential
model1 = Sequential()
model1.add(Dense(3, input_dim=3, activation='relu'))
model1.add(Dense(2, activation='relu'))
model1.add(Dense(2, activation='tanh'))
model1.compile(loss='mse', optimizer='Adam', metrics=['accuracy'])
model2 = Sequential()
model2.add(Dense(3, input_dim=3, activation='linear'))
model2.add(Dense(4, activation='tanh'))
model2.add(Dense(3, activation='tanh'))
model2.compile(loss='mse', optimizer='SGD', metrics=['accuracy'])
model3 = Sequential()
model3.add(Merge([model1, model2], mode = 'concat'))
model3.add(Dense(1, activation='sigmoid'))
model3.compile(loss='binary_crossentropy', optimizer='Adam', metrics=['accuracy'])
model3.input_shape
The ensemble model(model3) compiles without any error but while fitting the model I have to pass the same input two times model3.fit([X,X],y). Which I think is an unnecessary step and instead of passing input twice I want to have a common input nodes for my ensemble model. How can I do it?
Keras functional API seems to be a better fit for your use case, as it allows more flexibility in the computation graph. e.g.:
from keras.layers import concatenate
from keras.models import Model
from keras.layers import Input, Merge
from keras.layers.core import Dense
from keras.layers.merge import concatenate
# a single input layer
inputs = Input(shape=(3,))
# model 1
x1 = Dense(3, activation='relu')(inputs)
x1 = Dense(2, activation='relu')(x1)
x1 = Dense(2, activation='tanh')(x1)
# model 2
x2 = Dense(3, activation='linear')(inputs)
x2 = Dense(4, activation='tanh')(x2)
x2 = Dense(3, activation='tanh')(x2)
# merging models
x3 = concatenate([x1, x2])
# output layer
predictions = Dense(1, activation='sigmoid')(x3)
# generate a model from the layers above
model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
# Always a good idea to verify it looks as you expect it to
# model.summary()
data = [[1,2,3], [1,1,3], [7,8,9], [5,8,10]]
labels = [0,0,1,1]
# The resulting model can be fit with a single input:
model.fit(data, labels, epochs=50)
Notes:
There might be slight differences in the API between Keras versions (pre- and post- version 2)
The example above specifies different optimizer and loss function for each of the models. However, since fit() is being called only once (on model3), the same settings - those of model3 - will apply to the entire model. In order to have different settings when training the sub-models, they will have to be fit() separately -
see comment by #Daniel.
EDIT: updated notes based on comments
etov's answer is a great option.
But suppose you already have model1 and model2 ready and you don't want to change them, you can create the third model like this:
singleInput = Input((3,))
out1 = model1(singleInput)
out2 = model2(singleInput)
#....
#outN = modelN(singleInput)
out = Concatenate()([out1,out2]) #[out1,out2,...,outN]
out = Dense(1, activation='sigmoid')(out)
model3 = Model(singleInput,out)
And if you already have all the models ready and don't want to change them, you can have something like this (not tested):
singleInput = Input((3,))
output = model3([singleInput,singleInput])
singleModel = Model(singleInput,output)
Define new input layer and use model outputs directly (works in functional api):
assert model1.input_shape == model2.input_shape # make sure they got same shape
inp = tf.keras.layers.Input(shape=model1.input_shape[1:])
model = tf.keras.models.Model(inputs=[inp], outputs=[model1(inp), model2(inp)])