I was trying to make a model learned from difference between two model output. So I made code like below. But it occurred error read:
TypeError: Output tensors to a Model must be Keras tensors. Found:
Tensor("sub:0", shape=(?, 10), dtype=float32)
I have found related answer including lambda, but I couldn't solve this issue.
Does anyone know this issue?
It might be seen converting tensor to keras's tensor.
Thx in advance.
from keras.layers import Dense
from keras.models import Model
from keras.models import Sequential
left_branch = Sequential()
left_branch.add(Dense(10, input_dim=784))
right_branch = Sequential()
right_branch.add(Dense(10, input_dim=784))
diff = left_branch.output - right_branch.output
model = Model(inputs=[left_branch.input, right_branch.input], outputs=[diff])
model.compile(optimizer='rmsprop', loss='binary_crossentropy', loss_weights=[1.])
model.summary(line_length=150)
It's better to keep all operations done by a layer, do not subtract outputs like that (I wouldn't risk hidden errors for doing things differently from what the documentation expects):
from keras.layers import *
def negativeActivation(x):
return -x
left_branch = Sequential()
left_branch.add(Dense(10, input_dim=784))
right_branch = Sequential()
right_branch.add(Dense(10, input_dim=784))
negativeRight = Activation(negativeActivation)(right_branch.output)
diff = Add()([left_branch.output,negativeRight])
model = Model(inputs=[left_branch.input, right_branch.input], outputs=diff)
model.compile(optimizer='rmsprop', loss='binary_crossentropy', loss_weights=[1.])
When joining models like that, I do prefer using the Model way of doing it, with layers, instead of using Sequential:
def negativeActivation(x):
return -x
leftInput = Input((784,))
rightInput = Input((784,))
left_branch = Dense(10)(leftInput) #Dense(10) creates a layer
right_branch = Dense(10)(rightInput) #passing the input creates the output
negativeRight = Activation(negativeActivation)(right_branch)
diff = Add()([left_branch,negativeRight])
model = Model(inputs=[leftInput, rightInput], outputs=diff)
model.compile(optimizer='rmsprop', loss='binary_crossentropy', loss_weights=[1.])
With this, you can create other models with the same layers, they will share the same weights:
leftModel = Model(leftInput,left_branch)
rightModel = Model(rightInput,right_branch)
fullModel = Model([leftInput,rightInput],diff)
Training one of them will affect the others if they share the same layer.
You can train just the right part in the full model by making left_branch.trainable = False before compiling (or compile again for training), for instance.
I think I solve this, but it might be exact solution.
I added some code like below :
diff = left_branch.output - right_branch.output
setattr(diff, '_keras_history', getattr(right_branch.output, '_keras_history'))
setattr(diff, '_keras_shape', getattr(right_branch.output, '_keras_shape'))
setattr(diff, '_uses_learning_phase', getattr(right_branch.output, '_uses_learning_phase'))
The reason why the error occur is diff tensor doesn't have attr named _keras_history so on. So adding intentionally them to diff tensor could prevent above error. I checked original code ran and being possible to learn.
Related
I'm sorry if my knowledge in the concatenation is less I'm new to this.
this is NLP-based training. I have used onehot encoding to preprocess the data.
I need to create two LSTM and use concatenation.
this is my concatenation part.
from tensorflow.keras.layers import Dense, merge, Activation
first_lstm = Sequential()
first_lstm.add(Embedding(voc_size,embedding_vector_features,input_length=sent_length))
first_lstm.compile('adam','mse')
first_lstm.add(LSTM(50))
sec_lstm = Sequential()
sec_lstm.add(Embedding(voc_size,embedding_vector_features,input_length=sent_length))
sec_lstm.compile('adam','mse')
sec_lstm.add(LSTM(50))
model = Sequential()
model.add(merge([first_lstm,sec_lstm], mode='concat'))
model.add(Dense(1,activation='sigmoid'))
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
model.fit(x_train,y_train,validation_data=(x_test,y_test),epochs=2,batch_size=64)
model.summary()
I'm getting this error for this
cannot import name 'merge' from 'tensorflow.keras.layers' (C:\Users\asus\anaconda3\lib\site-packages\keras\api\_v2\keras\layers\__init__.py)
as many threads say I tried "Merge" with capital M but the same results.
I don't know whether It is my concatenate error or not.
also, is the way I have to concatenate correct?
any solutions!
Is there a way to add nodes to a layer in an existing Keras model? if so, what is the most efficient way to do so?
Also, is it possible to do the same but with layers? i.e. add a new layer to an existing Keras model (for example, right after the input layer).
One way I know of is to use Keras functional API by iterating and cloning each layer of the model in order to create a "copy" of the original model with the desired changes, but is it the most efficient way to accomplish this task?
You can take the output of a layer in a model and build another model starting from it:
import tensorflow as tf
# One simple model
inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation='relu')(inputs)
outputs = tf.keras.layers.Dense(5, activation='softmax')(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
# Make a second model starting from layer in previous model
x2 = tf.keras.layers.Dense(8, activation='relu')(model.layers[1].output)
outputs2 = tf.keras.layers.Dense(7, activation='softmax')(x2)
model2 = tf.keras.Model(inputs=model.input, outputs=outputs2)
Note that in this case model and model2 share the same input layer and first dense layer objects (model.layers[0] is model2.layers[0] and model.layers[1] is model2.layers[1]).
I am not sure how to get output of an intermediate layer in Keras. I have read the other questions on stackoverflow but they seem to be functions with a single sample as input. I want to get output features(at intermediate layer) in batches as well. Here is my model:
model = Sequential()
model.add(ResNet50(include_top = False, pooling = RESNET50_POOLING_AVERAGE, weights = resnet_weights_path)) #None
model.add(Dense(784, activation = 'relu'))
model.add(Dense(NUM_CLASSES, activation = DENSE_LAYER_ACTIVATION))
model.layers[0].trainable = True
After training the model, in my code I want to get the output after the first dense layer (784 dimensional). Is this the right way to do it?
pred = model.layers[1].predict_generator(data_generator, steps = len(data_generator), verbose = 1)
I am new to Keras so I am a little unsure. Do I need to compile the model again after training?
No, you don't need to compile again after training.
Based on your Sequential model.
Layer 0 :: model.add(ResNet50(include_top = False, pooling = RESNET50_POOLING_AVERAGE, weights = resnet_weights_path)) #None
Layer 1 :: model.add(Dense(784, activation = 'relu'))
Layer 2 :: model.add(Dense(NUM_CLASSES, activation = DENSE_LAYER_ACTIVATION))
Accessing the layers, may differ if used Functional API approach.
Using Tensorflow 2.1.0, you could try this approach when you want to access intermediate outputs.
model_dense_784 = Model(inputs=model.input, outputs = model.layers[1].output)
pred_dense_784 = model_dense_784.predict(train_data_gen, steps = 1) # predict_generator is deprecated
print(pred_dense_784.shape) # Use this to check Output Shape
It is highly advisable to use the model.predict() method, rather than model.predict_generator() as it is already deprecated.
You could also use shape() method to check whether the output generated is the same as indicated on the model.summary().
In building a model that uses TensorFlow 2.0 Attention I followed the example given in the TF docs. https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention
The last line in the example is
input_layer = tf.keras.layers.Concatenate()(
[query_encoding, query_value_attention])
Then the example has the comment
# Add DNN layers, and create Model.
# ...
So it seemed logical to do this
model = tf.keras.Sequential()
model.add(input_layer)
This produces the error
TypeError: The added layer must be an instance of class Layer.
Found: Tensor("concatenate/Identity:0", shape=(None, 200), dtype=float32)
UPDATE (after #thushv89 response)
What I am trying to do in the end is add an attention layer in the following model which works well (or convert it to an attention model).
model = tf.keras.Sequential()
model.add(layers.Embedding(vocab_size, embedding_nodes, input_length=max_length))
model.add(layers.LSTM(20))
#add attention here?
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(loss='mean_squared_error', metrics=['accuracy'])
My data looks like this
4912,5059,5079,0
4663,5145,5146,0
4663,5145,5146,0
4840,5117,5040,0
Where the first three columns are the inputs and the last column is binary and the goal is classification. The data was prepared similarly to this example with a similar purpose, binary classification. https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/
So, first thing is Keras has three APIs when it comes to creating models.
Sequential - (Which is what you're doing here)
Functional - (Which is what I'm using in the solution)
Subclassing - Creating Python classes to represent custom models/layers
The way the model created in the tutorial is not to be used with sequential models but a model from the Functional API. So you got to do the following. Note that, I've taken the liberty of defining the dense layers with arbitrary parameters (e.g. number of output classes, which you can change as needed).
import tensorflow as tf
# Variable-length int sequences.
query_input = tf.keras.Input(shape=(None,), dtype='int32')
value_input = tf.keras.Input(shape=(None,), dtype='int32')
# ... the code in the middle
# Concatenate query and document encodings to produce a DNN input layer.
input_layer = tf.keras.layers.Concatenate()(
[query_encoding, query_value_attention])
# Add DNN layers, and create Model.
# ...
dense_out = tf.keras.layers.Dense(50, activation='relu')(input_layer)
pred = tf.keras.layers.Dense(10, activation='softmax')(dense_out)
model = tf.keras.models.Model(inputs=[query_input, value_input], outputs=pred)
model.summary()
In my TensorFlow model I have some data that I feed into a stack of CNNs before it goes into a few fully connected layers. I have implemented that with Keras' Sequential model. However, I now have some data that should not go into the CNN and instead be fed directly into the first fully connected layer because that data contains some values and labels that are part of the input data but that data should not undergo convolutions as it is not image data.
Is such a thing possible with tensorflow.keras or should I do that with tensorflow.nn instead? As far as I understand Keras' sequential models is that the input goes in one end and comes out the other with no special wiring in the middle.
Am I correct that to do this I have to use tensorflow.concat on the data from the last CNN layer and the data that bypasses the CNNs before feeding it into the first fully connected layer?
Here is an simple example in which the operation is to sum the activations from different subnets:
import keras
import numpy as np
import tensorflow as tf
from keras.layers import Input, Dense, Activation
tf.reset_default_graph()
# this represents your cnn model
def nn_model(input_x):
feature_maker = Dense(10, activation='relu')(input_x)
feature_maker = Dense(20, activation='relu')(feature_maker)
feature_maker = Dense(1, activation='linear')(feature_maker)
return feature_maker
# a list of input layers, of course the input shapes can be different
input_layers = [Input(shape=(3, )) for _ in range(2)]
coupled_feature = [nn_model(input_x) for input_x in input_layers]
# assume you take the sum of the outputs
coupled_feature = keras.layers.Add()(coupled_feature)
prediction = Dense(1, activation='relu')(coupled_feature)
model = keras.models.Model(inputs=input_layers, outputs=prediction)
model.compile(loss='mse', optimizer='adam')
# example training set
x_1 = np.linspace(1, 90, 270).reshape(90, 3)
x_2 = np.linspace(1, 90, 270).reshape(90, 3)
y = np.random.rand(90)
inputs_x = [x_1, x_2]
model.fit(inputs_x, y, batch_size=32, epochs=10)
You can actually plot the model to gain more intuition
from keras.utils.vis_utils import plot_model
plot_model(model, show_shapes=True)
The model of the above code looks like this
With a little remodeling and the functional API you can:
#create the CNN - it can also be a sequential
cnn_input = Input(image_shape)
cnn_output = Conv2D(...)(cnn_input)
cnn_output = Conv2D(...)(cnn_output)
cnn_output = MaxPooling2D()(cnn_output)
....
cnn_model = Model(cnn_input, cnn_output)
#create the FC model - can also be a sequential
fc_input = Input(fc_input_shape)
fc_output = Dense(...)(fc_input)
fc_output = Dense(...)(fc_output)
fc_model = Model(fc_input, fc_output)
There is a lot of space for creativity, this is just one of the ways.
#create the full model
full_input = Input(image_shape)
full_output = cnn_model(full_input)
full_output = fc_model(full_output)
full_model = Model(full_input, full_output)
You can use any of the three models in any way you want. They share the layers and the weights, so internally they are the same.
Saving and loading the full model might be quirky. I'd probably save the other two separately and when loading create the full model again.
Notice also that if you save two models that share the same layers, after loading they will probably not share these layers anymore. (Another reason for saving/loading only fc_model and cnn_model, while creating full_model again from code)