We Keep Coding

Can I add data after flattening in a CNN?

I am working with a CNN and my professor wants me to try and include some information that is relevant, but isn't available in the images itself. As of right now, the data is a 1-D array. He thinks that adding it after the flattening layer and before the dense layers should be possible but neither of us are quite knowledgeable enough for it yet.
model = Sequential()
for i, feat in enumerate(args.conv_f):
if i==0:
model.add(Conv2D(feat, input_shape=x[0].shape, kernel_size=3, padding = 'same',use_bias=False))
else:
model.add(Conv2D(feat, kernel_size=3, padding = 'same',use_bias=False))
model.add(BatchNormalization())
model.add(LeakyReLU(alpha=args.conv_act))
model.add(Conv2D(feat, kernel_size=3, padding = 'same',use_bias=False))
model.add(BatchNormalization())
model.add(LeakyReLU(alpha=args.conv_act))
model.add(Dropout(args.conv_do[i]))
model.add(Flatten())
#Input code here
denseArgs = {'use_bias':False}
for i,feat in enumerate(args.dense_f):
model.add(Dense(feat,**denseArgs))
model.add(BatchNormalization())
model.add(LeakyReLU(alpha=args.dense_act))
model.add(Dropout(args.dense_do[i]))
model.add(Dense(1))
We could be wrong, obviously, so any help is appreciated! Thanks!

One approach I know of requires the use of the functional API of keras.
This means you would have to drop the Sequential approach you are currently using.
Using a toy model as an example, let the bloc:
img_input = Input((64, 64, 1))
model = Conv2D(20, (5, 5))(img_input)
model = MaxPooling2D((2, 2))(model)
model = Flatten()(model)
be the convolutional layers of a CNN with final flattening.
It is possible to add information by concatenating the last model layer with the new information. The new information can be packaged by creating a short model (here af_input) with just an input layer.
As an example:
af_input = Input(shape=(2,))
model = Concatenate()([model, af_input])
model = Dense(120, activation='relu')(model)
model = Dropout(0.1)(model)
model = Dense(100, activation='relu')(model)
predictions = Dense(2)(model)
fullmodel = Model(inputs=[img_input,af_input], outputs=predictions)
So now the results of the flatten layer of the CNN will be concatenated with a vector of extra information (here 2 features).
You can then keep adding layers to the networks as usual.
I suggest you check the stackoverflow link:
How to concatenate two layers in keras?
for another example and a good explanation.

Keras weight file load exception: loading 2 layers into a model with 0 layers

Exception happened when I add dropout to the input layer.
The exception was mentioned in other threads as well related to another issues and most common suggested solution is to downgrade the Keras version. Is there a workaround for this exception?
def baseline_model() :
model = Sequential()
model.add(Dropout(0.35)) #THIS LINE CAUSES THE EXCEPTION
model.add(Dense(200, input_dim=1200, kernel_initializer='normal', activation='relu'))
model.add(Dropout(0.8))
rms = RMSprop(lr = 0.00050)
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer=rms, metrics=['accuracy'])
return model
Model throws the following exception during weight file loading:
ValueError: You are trying to load a weight file containing 2 layers into a model with 0 layers.

The problem is that you have not assigned the input shape for the first layer:
model.add(Dropout(0.35, input_shape=(1200,)))
And then remove the input_dim argument of second layer as it is redundant.

Merge two different deep learning models in Keras

I have two different types of data (image volumes and coordinates) and I would like to use a convolutional neural network on the image volume data and then after this I would like to append some additional information (ie. the coordinates of the volume).
Independently this should create a pretty solid predictor for my function. How can I implement this using Keras.
The only answers I have found online are either ambiguous or are using the deprecated methods which I have got to work. But I would really like to implement this using the current API that way I can more easily save the model for later use.
model = Sequential()
model.add(Conv3D(32, kernel_size=(3, 3, 3),
activation='relu',
input_shape=input_shape))
model.add(Conv3D(64, (3, 3, 3), activation='relu'))
model.add(MaxPooling3D(pool_size=(2, 2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
print(model.output_shape)
# The additional data (the coordinates x,y,z)
extra = Sequential()
extra.add(Activation('sigmoid', input_shape=(3,)))
print(extra.output_shape)
merged = Concatenate([model, extra])
# New model should encompass the outputs of the convolutional network and the coordinates that have been merged.
# But how?
new_model = Sequential()
new_model.add(Dense(128, activation='relu'))
new_model.add(Dropout(0.8))
new_model.add(Dense(32, activation='sigmoid'))
new_model.add(Dense(num_classes, activation='softmax'))
new_model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])

Sequential models are not suited for creating models with branches.
You can have the two independent models as Sequential models, as you did, but from the Concatenate on, you should start using the functional Model API.
The idea is to get the output tensors of the two models and feed them in other layers to get new output tensors.
So, considering you have model and extra:
mergedOutput = Concatenate()([model.output, extra.output])
This mergetOutput is a tensor. You can either create the last part of the model using this tensor, or create the last part independently, and call it on this tensor. The second approach may be good if you want to train each model separately (doesn't seem to be your case).
Now, creating the new model as a functional API model:
out = Dense(128, activation='relu')(mergetOutput)
out = Dropout(0.8)(out)
out = Dense(32, activation='sigmoid')(out)
out = Dense(num_classes, activation='softmax')(out)
new_model = Model(
[model.input, extra.input], #model with two input tensors
out #and one output tensor
)
An easier approach is to take all three models you have already created and use them to create a combined model:
model = Sequential() #your first model
extra = Sequential() #your second model
new_model = Sequential() #all these three exactly as you did
#in this case, you just need to add an input shape to new_model, compatible with the concatenated output of the previous models.
new_model.add(FirstNewModelLayer(...,input_shape=(someValue,)))
Join them like this:
mergedOutput = Concatenate()([model.output, extra.output])
finalOutput = new_model(mergedOutput)
fullModel = Model([model.input,extra.input],finalOutput)

Use the functional API of Keras (https://keras.io/models/model/). You can just apply layers to your merged layer in Keras. The functional API works like this. You have a tensor and you apply a function to this Tensor. Then this is recursively evaluated. Because pretty much everything is a tensor in Keras this works quite nicely.
An example for this is:
activation = Dense(128, activation='relu')(merged)

keras - model wont work after pop()

I have taken a standard ResNet50 model:
model = keras.applications.resnet50.ResNet50(include_top=False,
weights='imagenet',
classes=10,
input_shape=(224, 224, 3))
And added several dense layers of my own:
top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(2, activation='softmax'))
model = Model(input=model.input, output=top_model(model.output))
This way it works great, however, when I want to delete last Dense and Dropout layers with model.pop() keras wont work well:
model.layers[-1].layers
[<keras.layers.core.Flatten at 0x16b5c00b8>,
<keras.layers.core.Dense at 0x16b5c0320>,
<keras.layers.core.Dropout at 0x16b5c02e8>,
<keras.layers.core.Dense at 0x16b5c0d68>]
model.layers[-1].pop()
model.layers[-1].pop()
model.layers[-1].layers
[<keras.layers.core.Flatten at 0x1ae6e5940>,
<keras.layers.core.Dense at 0x1ae6e9e10>]
model.layers[-1].outputs = [model.layers[-1].layers[-1].output]
model.outputs = model.layers[-1].outputs
model.layers[-1].layers[-1].outbound_nodes = []
Then I just compile the model and when trying to predict, I get an error:
You must feed a value for placeholder tensor 'flatten_7_input_12' with dtype float and shape [?,1,1,2048]

model.pop() takes care of all the underlying settings, including setting the model.output to the output of the new last layer. Therefore, you don't need to handle anything regarding the outputs.
Please also note that you are assigning to the model variable; thus, model.outputs is already referring to the extended model's output.
Here is a sample code that works fine on keras 2.0.6 with TensorFlow backend (1.4.0):
import keras
from keras.models import Sequential, Model
from keras.layers import *
import numpy as np
model = keras.applications.resnet50.ResNet50(include_top=False,
weights='imagenet',
classes=10,
input_shape=(224, 224, 3))
top_model = Sequential()
top_model.add(keras.layers.Flatten(input_shape=model.output_shape[1:]))
top_model.add(keras.layers.Dense(256, activation='relu'))
top_model.add(keras.layers.Dropout(0.5))
top_model.add(keras.layers.Dense(2, activation='softmax'))
model_extended = Model(input=model.input, output=top_model(model.output))
model_extended.layers[-1].pop()
model_extended.layers[-1].pop()
model_extended.compile(optimizer='rmsprop',
loss='categorical_crossentropy',
metrics=['accuracy'])
model_extended.predict(np.zeros((1, 224, 224, 3)))

Merge 2 sequential models in Keras

I a trying to merge 2 sequential models in keras. Here is the code:
model1 = Sequential(layers=[
# input layers and convolutional layers
Conv1D(128, kernel_size=12, strides=4, padding='valid', activation='relu', input_shape=input_shape),
MaxPooling1D(pool_size=6),
Conv1D(256, kernel_size=12, strides=4, padding='valid', activation='relu'),
MaxPooling1D(pool_size=6),
Dropout(.5),
])
model2 = Sequential(layers=[
# input layers and convolutional layers
Conv1D(128, kernel_size=20, strides=5, padding='valid', activation='relu', input_shape=input_shape),
MaxPooling1D(pool_size=5),
Conv1D(256, kernel_size=20, strides=5, padding='valid', activation='relu'),
MaxPooling1D(pool_size=5),
Dropout(.5),
])
model = merge([model1, model2], mode = 'sum')
Flatten(),
Dense(256, activation='relu'),
Dropout(.5),
Dense(128, activation='relu'),
Dropout(.35),
# output layer
Dense(5, activation='softmax')
return model
Here is the error log:
File
"/nics/d/home/dsawant/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py",
line 392, in is_keras_tensor
raise ValueError('Unexpectedly found an instance of type ' + str(type(x)) + '. ' ValueError: Unexpectedly found an instance of
type <class 'keras.models.Sequential'>. Expected a symbolic tensor
instance.
Some more log:
ValueError: Layer merge_1 was called with an input that isn't a
symbolic tensor. Received type: class 'keras.models.Sequential'.
Full input: [keras.models.Sequential object at 0x2b32d518a780,
keras.models.Sequential object at 0x2b32d521ee80]. All inputs to the
layer should be tensors.
How can I merge these 2 Sequential models that use different window sizes and apply functions like 'max', 'sum' etc to them?

Using the functional API brings you all possibilities.
When using the functional API, you need to keep track of inputs and outputs, instead of just defining layers.
You define a layer, then you call the layer with an input tensor to get the output tensor. Models and layers can be called exactly the same way.
For the merge layer, I prefer using other merge layers that are more intuitive, such as Add(), Multiply() and Concatenate() for instance.
from keras.layers import *
mergedOut = Add()([model1.output,model2.output])
#Add() -> creates a merge layer that sums the inputs
#The second parentheses "calls" the layer with the output tensors of the two models
#it will demand that both model1 and model2 have the same output shape
This same idea apply to all the following layers. We keep updating the output tensor giving it to each layer and getting a new output (if we were interested in creating branches, we would use a different var for each output of interest to keep track of them):
mergedOut = Flatten()(mergedOut)
mergedOut = Dense(256, activation='relu')(mergedOut)
mergedOut = Dropout(.5)(mergedOut)
mergedOut = Dense(128, activation='relu')(mergedOut)
mergedOut = Dropout(.35)(mergedOut)
# output layer
mergedOut = Dense(5, activation='softmax')(mergedOut)
Now that we created the "path", it's time to create the Model. Creating the model is just like telling at which input tensors it starts and where it ends:
from keras.models import Model
newModel = Model([model1.input,model2.input], mergedOut)
#use lists if you want more than one input or output
Notice that since this model has two inputs, you have to train it with two different X_training vars in a list:
newModel.fit([X_train_1, X_train_2], Y_train, ....)
Now, suppose you wanted only one input, and both model1 and model2 would take the same input.
The functional API allows that quite easily by creating an input tensor and feeding it to the models (we call the models as if they were layers):
commonInput = Input(input_shape)
out1 = model1(commonInput)
out2 = model2(commonInput)
mergedOut = Add()([out1,out2])
In this case, the Model would consider this input:
oneInputModel = Model(commonInput,mergedOut)