I want to try to implement the neural network architecture of the attached image: 1DCNN_model
Consider that I've got a dataset X which is (N_signals, 1500, 40) where 40 is the number of features where I want to do the 1d convolution on.
My Y is (N_signals, 1500, 2) and I'm working with keras.
Every 1d convolution needs to take one feature vector like in this picture:1DCNN_convolution
So it has to take one chunk of the 1500 timesamples, pass it through the 1d convolutional layer (sliding along time-axis) then feed all the output features to the LSTM layer.
I tried to implement the first convolutional part with this code but I'm not sure what it's doing, I can't understand how it can take in one chunk at a time (maybe I need to preprocess my input data before?):
input_shape = (None, 40)
model_input = Input(input_shape, name = 'input')
layer = model_input
convs = []
for i in range(n_chunks):
conv = Conv1D(filters = 40,
kernel_size = 10,
padding = 'valid',
activation = 'relu')(layer)
conv = BatchNormalization(axis = 2)(conv)
pool = MaxPooling1D(40)(conv)
pool = Dropout(0.3)(pool)
convs.append(pool)
out = Merge(mode = 'concat')(convs)
conv_model = Model(input = layer, output = out)
Any advice? Thank you very much
Thank you very much, I modified my code in this way:
input_shape = (1500,40)
model_input = Input(shape=input_shape, name='input')
layer = model_input
layer = Conv1D(filters=40,
kernel_size=10,
padding='valid',
activation='relu')(layer)
layer = BatchNormalization(axis=2)(layer)
layer = MaxPooling1D(pool_size=40,
padding='same')(layer)
layer = Dropout(self.params.drop_rate)(layer)
layer = LSTM(40, return_sequences=True,
activation=self.params.lstm_activation)(layer)
layer = Dropout(self.params.lstm_dropout)(layer)
layer = Dense(40, activation = 'relu')(layer)
layer = BatchNormalization(axis = 2)(layer)
model_output = TimeDistributed(Dense(2,
activation='sigmoid'))(layer)
I was actually thinking that maybe I have to permute my axes in order to make maxpooling layer work on my 40 mel feature axis...
If you want to perform an individual 1D convolution over the 40 feature channels you should add a dimension to your input:
(1500,40,1)
if you perform 1D convolution on a input with shape
(1500,40)
the filters are applied on the time dimension and the pictures you posted indicate that this is not what you want to do.
Related
I am doing a binary regression problem using keras.
The input shape is: (None, 2, 94, 3) (channels is the last dimension)
I have the following architecture:
input1 = Input(shape=(time, n_rows, n_channels))
masking = Masking(mask_value=-999)(input1)
convlstm = ConvLSTM1D(filters=16, kernel_size=15,
data_format='channels_last',
activation="tanh")(masking)
dropout = Dropout(0.2)(convlstm)
flatten1 = Flatten()(dropout)
outputs = Dense(n_outputs, activation='sigmoid')(flatten1)
model = Model(inputs=input1, outputs=outputs)
model.compile(loss=keras.losses.BinaryCrossentropy(),
optimizer=tf.keras.optimizers.Adam(learning_rate=0.01))
However when training I get this error: Dimensions must be equal, but are 94 and 80 for '{{node conv_lstm1d/while/SelectV2}} = SelectV2[T=DT_FLOAT](conv_lstm1d/while/Tile, conv_lstm1d/while/mul_5, conv_lstm1d/while/Placeholder_2)' with input shapes: [?,94,16], [?,80,16], [?,80,16].
If I remove the masking layer this error disappears, what is the masking doing that triggers this error? Also the only way I was able to run the above architecture was with a kernel_size of 1.
Seems like the ConvLSTM1D layer needs a mask with the shape (samples, timesteps) according to the docs. The mask you are calculating has the shape (samples, time, rows). Here is one solution to fix your problem but I am not sure if it is the 'correct' way to go:
import tensorflow as tf
input1 = tf.keras.layers.Input(shape=(2, 94, 3))
masking = tf.keras.layers.Masking(mask_value=-999)(input1)
convlstm = tf.keras.layers.ConvLSTM1D(filters=16, kernel_size=15,
data_format='channels_last',
activation="tanh")(inputs = masking, mask = tf.reduce_all(masking._keras_mask, axis=-1))
dropout = tf.keras.layers.Dropout(0.2)(convlstm)
flatten1 = tf.keras.layers.Flatten()(dropout)
outputs = tf.keras.layers.Dense(1, activation='sigmoid')(flatten1)
model = tf.keras.Model(inputs=input1, outputs=outputs)
model.compile(loss=tf.keras.losses.BinaryCrossentropy(),
optimizer=tf.keras.optimizers.Adam(learning_rate=0.01))
This line mask = tf.reduce_all(masking._keras_mask, axis=-1) essentially reduces your mask to (samples, timesteps) by applying an AND operation to the last dimension of the mask. Alternatively, you could just create your own custom mask layer:
import tensorflow as tf
class Reduce(tf.keras.layers.Layer):
def __init__(self):
super(Reduce, self).__init__()
def call(self, inputs):
return tf.reduce_all(tf.reduce_any(tf.not_equal(inputs, -999), axis=-1, keepdims=False), axis=1)
input1 = tf.keras.layers.Input(shape=(2, 94, 3))
reduce_layer = Reduce()
boolean_mask = reduce_layer(input1)
convlstm = tf.keras.layers.ConvLSTM1D(filters=16, kernel_size=15,
data_format='channels_last',
activation="tanh")(inputs = input1, mask = boolean_mask)
dropout = tf.keras.layers.Dropout(0.2)(convlstm)
flatten1 = tf.keras.layers.Flatten()(dropout)
outputs = tf.keras.layers.Dense(1, activation='sigmoid')(flatten1)
model = tf.keras.Model(inputs=input1, outputs=outputs)
model.compile(loss=tf.keras.losses.BinaryCrossentropy(),
optimizer=tf.keras.optimizers.Adam(learning_rate=0.01))
print(model.summary(expand_nested=True))
x = tf.random.normal((50, 2, 94, 3))
y = tf.random.uniform((50, ), maxval=3, dtype=tf.int32)
model.fit(x, y)
We're trying to add a custom layer inside a pre-trained imagenet model. For a sequential or non-sequential model, we can easily do that. But here are some requirements.
First of all, we don't wanna disclose the whole imagenet model and deal with the desired inside layer. Let's say for DenseNet we need the following layers and further get the output shape of theirs to connect with some custom layers.
vision_model = tf.keras.applications.DenseNet121(
input_shape=(224,224,3),
include_top = False,
weights='imagenet')
for i, layer in enumerate(vision_model.layers):
if layer.name in ['conv3_block12_concat', 'conv4_block24_concat']:
print(i,'\t',layer.trainable,'\t :',layer.name)
if layer.name == 'conv3_block12_concat':
print(layer.get_output_shape_at(0)[1:]) # (28, 28, 512)
if layer.name == 'conv4_block24_concat':
print(layer.get_output_shape_at(0)[1:]) # (14, 14, 1024)
The whole requirement can be demonstrated as follows
The green indicator is basically the transition layer of the dense net.
In the above diagram, the dense net model has (let's say) 5 blocks and among them, we want to pick block 3 and block 4 and add some custom layers followed by merging them to lead the final output.
Also, the blocks of DenseNet (block 1 to 5), should be as disclose as possible with their pre-trained imagenet weights. We like to have control to freeze and unfreeze pre-trained layers when we need them.
How can we efficiently achieve with tf.keras? or, If you think there some better approach to do the same thing, please suggest.
Let's say, a custom block is something like this
class MLPBlock(tf.keras.layers.Layer):
def __init__(self, kernel_num=32, kernel_size=(3,3), strides=(1,1), padding='same'):
super(ConvModule, self).__init__()
# conv layer
self.conv = tf.keras.layers.Conv2D(kernel_num,
kernel_size=kernel_size,
strides=strides, padding=padding)
# batch norm layer
self.bn = tf.keras.layers.BatchNormalization()
def call(self, input_tensor, training=False):
x = self.conv(input_tensor)
x = self.bn(x, training=training)
return tf.nn.relu(x)
Motivation
I'm trying to implement this paper-work where they did something like this. Initially, the paper was free to get but now it's not. But below is the main block diagram of their approach.
I don't have access to the paper so I just build an example like the one your draw:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, models
class ConvBlock(layers.Layer):
def __init__(self, kernel_num=32, kernel_size=(3,3), strides=(1,1), padding='same'):
super(ConvBlock, self).__init__()
# conv layer
self.conv = layers.Conv2D(kernel_num,
kernel_size=kernel_size,
strides=strides, padding=padding)
# batch norm layer
self.bn = layers.BatchNormalization()
def call(self, input_tensor, training=False):
x = self.conv(input_tensor)
x = self.bn(x, training=training)
return tf.nn.relu(x)
vision_model = keras.applications.DenseNet121(
input_shape=(224,224,3),
include_top = False,
weights='imagenet')
# Control freeze and unfreeze over blocks
def set_freeze(block, unfreeze):
for layer in block:
layer.trainable = unfreeze
block_1 = vision_model.layers[:7]
block_2 = vision_model.layers[7:53]
block_3 = vision_model.layers[53:141]
block_4 = vision_model.layers[141:313]
block_5 = vision_model.layers[313:]
set_freeze(block_1, unfreeze=False)
set_freeze(block_2, unfreeze=False)
for i, layer in enumerate(vision_model.layers):
print(i,'\t',layer.trainable,'\t :',layer.name)
layer_names = ['conv3_block12_concat', 'conv4_block24_concat', 'conv5_block16_concat']
vision_model_outputs = [vision_model.get_layer(name).output for name in layer_names]
custom_0 = ConvBlock()(vision_model_outputs[0])
custom_1 = ConvBlock()(layers.UpSampling2D(2)(vision_model_outputs[1]))
cat_layer = layers.concatenate([custom_0, custom_1])
last_conv_num = 2
custom_2 = layers.UpSampling2D(4)(vision_model_outputs[2])
outputs = layers.concatenate([ConvBlock()(cat_layer) for i in range(last_conv_num)] + [custom_2])
model = models.Model(vision_model.input, outputs)
keras.utils.plot_model(model, "./Model_structure.png", show_shapes=True)
Run the code and you will see the block1 and block2 are frozen,
Because the plot of full model is long so I just post few snippet of it:
I have a question.
Working with CNN, there is a way to get output of an hidden layer during classification?
Example:
common_input = layers.Input(shape=(224, 224, 3))
x = model0(common_input) #model0 is a pretrain model on imagenet
x = layers.Flatten()(x)
p = layers.Dense(768, activation="relu")(x)
p = layers.Dropout(0.3)(p)
p = layers.Dense(8, activation="softmax", name="fc_out")(p)
model = Model(inputs=common_input, outputs=p)
My task is classification; so if I have 2000 images I will get a matrix 2000x8. If I need the output of layer dense, there is a way to get a matrix 2000x768 (both in the same computation)?
Thanks
I have a CNN with a concatenate-layer at the beginning of Dense network. I use the layer.concatenate() to merge the features retrieved by CNN and my hand-crafted features. Now, this 2 data arrays, have 2 different scale (because the first are values calculated by the CNN and the other are features calculated by me). So I would a way to normalize this 2 type of data in a common scale for simplify the job. This is my CNN:
emb = Embedding()(image_input)
conv = Conv1D(64, 2, activation = 'relu', strides = 1, padding = 'same')(emb)
conv = MaxPooling1D()(conv1)
first_part_output = Flatten()(conV)
merged_model = layers.concatenate([first_part_output, other_data_input])
Here i would do the normalization:
normal = *here i would do the normalization*
primoDense = Dense(256, activation = 'relu')(normal)
drop = Dropout(0.45)(primoDense)
predictions = Dense(1, activation = 'sigmoid')(drop)
[first_part_output, other_data_input] is my new merged-array.
first_part_output are CNN featues.
other_data_input are my hand-crafted features.
In my case I am using a set of sequential features and also non sequential features to train the model. Following is the architecture of my model
Sequential features -> LSTM -> Dense(1) --->>
\
\
-- Dense -> Dense -> Dense(1) ->output
/
Non-sequential features---/
I am using data generator to generate batches for sequential data. Here the batch size is varying for each batch. For one batch I am keeping the non-sequential feature fixed. Following is my data generator.
def training_data_generator(raw_data):
while True:
for index, row in raw_data.iterrows():
x_train, y_train = list(), list()
feature1 = row['xxx']
x_current_batch = []
y_current_batch = []
for j in range(yyy):
x_current_batch.append(row['zz1'])
y_current_batch.append(row['zz2'])
x_train.append(x_current_batch)
y_train.append(y_current_batch)
x_train = array(x_train)
y_train = array(y_train)
yield [x_train, np.reshape(feature1,1)], y_train
Note: x_train y_train sizes are varying.
Following is my model implementation.
seq_input = Input(shape=(None, 3))
lstm_layer = LSTM(50)(seq_input)
dense_layer1 = Dense(1)(lstm_layer)
non_seq_input = Input(shape=(1,))
hybrid_model = concatenate([dense_layer1, non_seq_input])
hidden1 = Dense(10, activation = 'relu')(hybrid_model)
hidden2 = Dense(10, activation='relu')(hidden1)
final_output = Dense(1, activation='sigmoid')(hidden2)
model = Model(inputs = [seq_input, non_seq_input], outputs = final_output)
model.compile(loss='mse',optimizer='adam')
model.fit_generator(training_data_generator(flatten), steps_per_epoch= 5017,
epochs = const.NUMBER_OF_EPOCHS, verbose=1)
I am getting error at the output dense layer
ValueError: Error when checking target:
expected dense_4 to have shape (1,) but got array with shape (4,)
I think the last layer is getting whole output of the generator but not as one by one.
What is the reason for this issue. Appreciate your insights on this issue.
The output gives a Dense layer with a size of 4. Since you've declared your output as a Dense layer with a size of 1, it crashes.
What you can do is change your output dense Layer to 4. And then manually convert this to one value.
Hopefully this answers your question.