I have two different types of data (image volumes and coordinates) and I would like to use a convolutional neural network on the image volume data and then after this I would like to append some additional information (ie. the coordinates of the volume).
Independently this should create a pretty solid predictor for my function. How can I implement this using Keras.
The only answers I have found online are either ambiguous or are using the deprecated methods which I have got to work. But I would really like to implement this using the current API that way I can more easily save the model for later use.
model = Sequential()
model.add(Conv3D(32, kernel_size=(3, 3, 3),
activation='relu',
input_shape=input_shape))
model.add(Conv3D(64, (3, 3, 3), activation='relu'))
model.add(MaxPooling3D(pool_size=(2, 2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
print(model.output_shape)
# The additional data (the coordinates x,y,z)
extra = Sequential()
extra.add(Activation('sigmoid', input_shape=(3,)))
print(extra.output_shape)
merged = Concatenate([model, extra])
# New model should encompass the outputs of the convolutional network and the coordinates that have been merged.
# But how?
new_model = Sequential()
new_model.add(Dense(128, activation='relu'))
new_model.add(Dropout(0.8))
new_model.add(Dense(32, activation='sigmoid'))
new_model.add(Dense(num_classes, activation='softmax'))
new_model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
Sequential models are not suited for creating models with branches.
You can have the two independent models as Sequential models, as you did, but from the Concatenate on, you should start using the functional Model API.
The idea is to get the output tensors of the two models and feed them in other layers to get new output tensors.
So, considering you have model and extra:
mergedOutput = Concatenate()([model.output, extra.output])
This mergetOutput is a tensor. You can either create the last part of the model using this tensor, or create the last part independently, and call it on this tensor. The second approach may be good if you want to train each model separately (doesn't seem to be your case).
Now, creating the new model as a functional API model:
out = Dense(128, activation='relu')(mergetOutput)
out = Dropout(0.8)(out)
out = Dense(32, activation='sigmoid')(out)
out = Dense(num_classes, activation='softmax')(out)
new_model = Model(
[model.input, extra.input], #model with two input tensors
out #and one output tensor
)
An easier approach is to take all three models you have already created and use them to create a combined model:
model = Sequential() #your first model
extra = Sequential() #your second model
new_model = Sequential() #all these three exactly as you did
#in this case, you just need to add an input shape to new_model, compatible with the concatenated output of the previous models.
new_model.add(FirstNewModelLayer(...,input_shape=(someValue,)))
Join them like this:
mergedOutput = Concatenate()([model.output, extra.output])
finalOutput = new_model(mergedOutput)
fullModel = Model([model.input,extra.input],finalOutput)
Use the functional API of Keras (https://keras.io/models/model/). You can just apply layers to your merged layer in Keras. The functional API works like this. You have a tensor and you apply a function to this Tensor. Then this is recursively evaluated. Because pretty much everything is a tensor in Keras this works quite nicely.
An example for this is:
activation = Dense(128, activation='relu')(merged)
Related
I need to see how I would initialize all layers of a Sequential model with data from a same-sized sequential model.
E.G. How would I initialize the weights for every layer of the following Sequential model?
model = tf.keras.Sequential([Dense(2000, activation='relu', input_shape=(11,)),
Dense(1, activation='relu'),
Dropout(0.5),
Dense(400, activation='relu'),
Dropout(0.5),
Dense(150, activation='relu'),
BatchNormalization(),
Dense(y_max+1, activation='softmax')
])
I am fairly new to CNN training and have managed to make the above code work through trial and error and extensive research.
Datatype is list and np.array() of dtype np.float64
The idea is that I grab the weights from one model (same as above) and return it to another model (also same as above). I just need to be able to visualize how I can initialize the weights and biases of all layers using the following:
weights = model.get_weights()[0]
biases = model.get_weights()[1]
return weights, biases
I have attempted the model.set_weights() method, but I keep getting the following error message, given the code before the TypeError:
if iteration == 1:
for layer in model.layers:
layer.set_weights(None, None)
TypeError: set_weights() takes 2 positional arguments but 3 were given
I'd be very appreciative of any help, thank you.
In the Sequential example above, each layer parameters can be accessed and assigned new weights as shown below,
#example of first layer
model.layers[0]
#weights of the first layer,
model.layers[0].weights #gives the weights of kernel and bias of dense in this case
#assign new_weights by
model.layers[0].kernel.assign(tf.Variable(new_kernel_weights))
model.layers[0].bias.assign(tf.Variable(new_bias_weights))
I am working with a CNN and my professor wants me to try and include some information that is relevant, but isn't available in the images itself. As of right now, the data is a 1-D array. He thinks that adding it after the flattening layer and before the dense layers should be possible but neither of us are quite knowledgeable enough for it yet.
model = Sequential()
for i, feat in enumerate(args.conv_f):
if i==0:
model.add(Conv2D(feat, input_shape=x[0].shape, kernel_size=3, padding = 'same',use_bias=False))
else:
model.add(Conv2D(feat, kernel_size=3, padding = 'same',use_bias=False))
model.add(BatchNormalization())
model.add(LeakyReLU(alpha=args.conv_act))
model.add(Conv2D(feat, kernel_size=3, padding = 'same',use_bias=False))
model.add(BatchNormalization())
model.add(LeakyReLU(alpha=args.conv_act))
model.add(Dropout(args.conv_do[i]))
model.add(Flatten())
#Input code here
denseArgs = {'use_bias':False}
for i,feat in enumerate(args.dense_f):
model.add(Dense(feat,**denseArgs))
model.add(BatchNormalization())
model.add(LeakyReLU(alpha=args.dense_act))
model.add(Dropout(args.dense_do[i]))
model.add(Dense(1))
We could be wrong, obviously, so any help is appreciated! Thanks!
One approach I know of requires the use of the functional API of keras.
This means you would have to drop the Sequential approach you are currently using.
Using a toy model as an example, let the bloc:
img_input = Input((64, 64, 1))
model = Conv2D(20, (5, 5))(img_input)
model = MaxPooling2D((2, 2))(model)
model = Flatten()(model)
be the convolutional layers of a CNN with final flattening.
It is possible to add information by concatenating the last model layer with the new information. The new information can be packaged by creating a short model (here af_input) with just an input layer.
As an example:
af_input = Input(shape=(2,))
model = Concatenate()([model, af_input])
model = Dense(120, activation='relu')(model)
model = Dropout(0.1)(model)
model = Dense(100, activation='relu')(model)
predictions = Dense(2)(model)
fullmodel = Model(inputs=[img_input,af_input], outputs=predictions)
So now the results of the flatten layer of the CNN will be concatenated with a vector of extra information (here 2 features).
You can then keep adding layers to the networks as usual.
I suggest you check the stackoverflow link:
How to concatenate two layers in keras?
for another example and a good explanation.
I've created a code to train my CNN which will determine if a image be one or another class ("pdr or nonPdr")
This is my keras model:
model = Sequential()
model.add(Conv2D(input_shape=(605,700,3), filters=64, kernel_size=(3,3), padding="same",activation="tanh"))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(32, activation='tanh', input_dim=100))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
data, labels = ReadImages(TRAIN_DIR)
model.fit(np.array(data), np.array(labels), epochs=10, batch_size=16)
model.save('model.h5')
And this is my test_predict file:
model = load_model('model.h5')
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
test_image = cv2.imread("test_data/im0203.ppm")
test_image = test_image.reshape(1,605,700,3).astype('float')
test_image = np.array(test_image)
test_image2 = cv2.imread("test_data/im0001.ppm")
test_image2 = test_image2.reshape(1,605,700,3).astype('float')
#predict the result
print(model.predict(test_image))
print(model.predict(test_image2))
After run the code below I get the same value for my 2 images (that are different, one is pdr and the other is nonPdr)
[[0.033681]]
[[0.033681]]
How can I fix it and improve my CNN. I appreciate your help.
Update
I tried to remove one dense layer and change the last one in a (2, activation='sigmoid') but it not works too.. I really don't have any ideia what to do**
This could be because of multiple reasons. Most plausible ones are:
The input fed to network and the image fed during prediction may not have the same preprocessing steps.
The model is over-fitted and when it finds a little different prediction image, it tends to bias its decision based on its training.
Model is not good
To avoid such complications, possible future solutions could be:
1. Try feeding a normalized or scaled image only during training and testing.
Do not use too many Dense layers (Especially for image classification) because dense layers are good at learning spatial information and for an image to classify correctly, the model needs to look the image in 2d matrix rather than 1d spatial matrix (since dense layers flatten the 2d matrix to 1d matrix).
Even if you are trying to use a dense layer, I would recommend using a dense layer only at the last layer, it, for the final number_of_classes layers.
A better architecture would be
[ Conv followed by few transition blocks and bottleneck layers ] x 2
flatten
Dense(number_of_classes)
I am trying to follow online tutorials and put together code for a wide and deep model in Keras. However, I'm having issues merging both models together
wide = Sequential()
wide.add(Dense(1, input_dim=X_train.shape[1], kernel_initializer ='uniform', activation='relu'))
deep = Sequential()
deep.add(Dense(1, input_dim=X_train.shape[1], kernel_initializer ='uniform', activation='relu'))
deep.add(Dense(100, activation='relu'))
deep.add(Dense(50, activation='relu'))
deep.add(Dense(1, activation='linear'))
model = Sequential()
model.add(Merge([wide, deep], mode='concat', concat_axis=1))
model.add(Dense(1, activation='linear'))
The following warning occurs:
model = Sequential()
model.add(Merge([wide, deep], mode='concat', concat_axis=1))
__main__:2: UserWarning: The `Merge` layer is deprecated and will be removed after 08/2017. Use instead layers from `keras.layers.merge`, e.g. `add`, `concatenate`, etc
The warning message tells me what to do, but I've yet to figure out how to combine the two models.
I've tried different things like the following, but keep getting errors.
from keras.layers import add
model = Sequential()
model.add([wide, deep])
Traceback (most recent call last):
File "<ipython-input-428-3e81d6d35c6f>", line 1, in <module>
model.add([wide, deep])
File "/Users/abrahammathew/anaconda3/lib/python3.6/site-packages/keras/models.py", line 430, in add
'Found: ' + str(layer))
TypeError: The added layer must be an instance of class Layer. Found: [<keras.models.Sequential object at 0x1a32876cc0>, <keras.models.Sequential object at 0x1a328761d0>]
Can anyone tell me how to concat the wide and deep models with keras.
It's because you are trying to merge models inside a Sequential model. In this case you will need to expand to the functional API because you need calculate the the outputs of both models before you can merge them. Something along the lines of:
in = Input(shape=(X_train.shape[1],)
wide_out = wide(in)
deep_out = deep(in)
wide_deep = concatenate([wide_out, deep_out]) # or any merge layer
out = Dense(1, activation='linear')(wide_deep)
model = Model(in, out) # Your final model
In a nutshell, in the functional API layers are like function that apply the set operations to the given layers creating the corresponding computation graph.
I a trying to merge 2 sequential models in keras. Here is the code:
model1 = Sequential(layers=[
# input layers and convolutional layers
Conv1D(128, kernel_size=12, strides=4, padding='valid', activation='relu', input_shape=input_shape),
MaxPooling1D(pool_size=6),
Conv1D(256, kernel_size=12, strides=4, padding='valid', activation='relu'),
MaxPooling1D(pool_size=6),
Dropout(.5),
])
model2 = Sequential(layers=[
# input layers and convolutional layers
Conv1D(128, kernel_size=20, strides=5, padding='valid', activation='relu', input_shape=input_shape),
MaxPooling1D(pool_size=5),
Conv1D(256, kernel_size=20, strides=5, padding='valid', activation='relu'),
MaxPooling1D(pool_size=5),
Dropout(.5),
])
model = merge([model1, model2], mode = 'sum')
Flatten(),
Dense(256, activation='relu'),
Dropout(.5),
Dense(128, activation='relu'),
Dropout(.35),
# output layer
Dense(5, activation='softmax')
return model
Here is the error log:
File
"/nics/d/home/dsawant/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py",
line 392, in is_keras_tensor
raise ValueError('Unexpectedly found an instance of type ' + str(type(x)) + '. ' ValueError: Unexpectedly found an instance of
type <class 'keras.models.Sequential'>. Expected a symbolic tensor
instance.
Some more log:
ValueError: Layer merge_1 was called with an input that isn't a
symbolic tensor. Received type: class 'keras.models.Sequential'.
Full input: [keras.models.Sequential object at 0x2b32d518a780,
keras.models.Sequential object at 0x2b32d521ee80]. All inputs to the
layer should be tensors.
How can I merge these 2 Sequential models that use different window sizes and apply functions like 'max', 'sum' etc to them?
Using the functional API brings you all possibilities.
When using the functional API, you need to keep track of inputs and outputs, instead of just defining layers.
You define a layer, then you call the layer with an input tensor to get the output tensor. Models and layers can be called exactly the same way.
For the merge layer, I prefer using other merge layers that are more intuitive, such as Add(), Multiply() and Concatenate() for instance.
from keras.layers import *
mergedOut = Add()([model1.output,model2.output])
#Add() -> creates a merge layer that sums the inputs
#The second parentheses "calls" the layer with the output tensors of the two models
#it will demand that both model1 and model2 have the same output shape
This same idea apply to all the following layers. We keep updating the output tensor giving it to each layer and getting a new output (if we were interested in creating branches, we would use a different var for each output of interest to keep track of them):
mergedOut = Flatten()(mergedOut)
mergedOut = Dense(256, activation='relu')(mergedOut)
mergedOut = Dropout(.5)(mergedOut)
mergedOut = Dense(128, activation='relu')(mergedOut)
mergedOut = Dropout(.35)(mergedOut)
# output layer
mergedOut = Dense(5, activation='softmax')(mergedOut)
Now that we created the "path", it's time to create the Model. Creating the model is just like telling at which input tensors it starts and where it ends:
from keras.models import Model
newModel = Model([model1.input,model2.input], mergedOut)
#use lists if you want more than one input or output
Notice that since this model has two inputs, you have to train it with two different X_training vars in a list:
newModel.fit([X_train_1, X_train_2], Y_train, ....)
Now, suppose you wanted only one input, and both model1 and model2 would take the same input.
The functional API allows that quite easily by creating an input tensor and feeding it to the models (we call the models as if they were layers):
commonInput = Input(input_shape)
out1 = model1(commonInput)
out2 = model2(commonInput)
mergedOut = Add()([out1,out2])
In this case, the Model would consider this input:
oneInputModel = Model(commonInput,mergedOut)