We Keep Coding

How can I reduce outliers in the output of a regression model?

I am using a Keras model for regression which inputs are sensor measurements, and the output is the attitude of the sensor. This model consists of CuDNNLSTM and CNN. I need to reduce the number or range of outliers in the output.
The mean error is reasonable and low, but there are so many outliers in the output. The mean error is around 1, but as you can see in the boxplot, sometimes I get 180 errors (the maximum possible error).
The training data has no outlier and has been preprocessed before.
How can I reduce the outliers in the output?
Are there any specific network topologies or layers that could handle this?
I tried normalizing the input or adding gaussian noise, but none of them had any impact on the number of outliers in the outputs. Also, I tried all possible loss functions (more than 38), and this is the best result.
The model is:
Acc = Input((window_size, 3), name='acc')
Gyro = Input((window_size, 3), name='gyro')
AGconcat = concatenate([Acc, Gyro], axis=2, name='AGconcat')
fs = Input((1,), name='fs')
ACNN = Conv1D(filters=133,
kernel_size = 11,
padding = 'same',
activation = tfa.activations.mish,
name= 'ACNN')(Acc)
ACNN = Conv1D(filters=109,
kernel_size = 11,
padding = 'same',
activation = tfa.activations.mish,
name= 'ACNN1')(ACNN)
ACNN = MaxPooling1D(pool_size=3,
name = 'MaxPooling1D')(ACNN)
ACNN = Flatten(name='ACNNF')(ACNN)
GCNN = Conv1D(filters=142,
kernel_size = 11,
padding = 'same',
activation = tfa.activations.mish,
name= 'GCNN')(Gyro)
GCNN = Conv1D(filters=116,
kernel_size = 11,
padding = 'same',
activation = tfa.activations.mish,
name= 'GCNN1')(GCNN)
GCNN = MaxPooling1D(pool_size=3,
name = 'GyroMaxPool1D')(GCNN)
GCNN = Flatten(name='GCNNF')(GCNN)
AGconLSTM =Bidirectional(CuDNNGRU(128, return_sequences=True,
#return_state=True,
go_backwards=True,
name='BiLSTM1'))(AGconcat)
FlattenAG = Flatten(name='FlattenAG')(AGconLSTM)
AG = concatenate([ACNN, GCNN,FlattenAG])
AG = Dense(units=256,
activation= tfa.activations.mish)(AG)
Fdense = Dense(units=256,
activation= tfa.activations.mish,
name= 'Fdense')(fs)
AG = Flatten(name='AGF')(AG)
x = concatenate([AG, Fdense])
x = Dense(units=256,
activation= tfa.activations.mish)(x)
x = Flatten(name='output')(x)
output = Dense(4, activation='linear', name='quat')(x)

You can try weight decay and regularization. For the last line, you can add this:
quat = Lambda(lambda x: k.l2_normalize(x, axis=1), name='QuatNorm')(quat)

How to combine features extracted from two cnn models?

i have two cnn models both follow same architecture. I trained 'train set 1' on cnn1 and 'train set 2; on cnn2.Then i exracted features using following code.
# cnn1
model.pop() #removes softmax layer
model.pop() #removes dropoutlayer
model.pop() #removes activation layer
model.pop() #removes batch-norm layer
model.build() #here lies dense 512
features1 = model.predict(train set 1)
print(features1.shape) #600,512
# cnn2
model.pop() #removes softmax layer
model.pop() #removes dropoutlayer
model.pop() #removes activation layer
model.pop() #removes batch-norm layer
model.build() #here lies dense 512
features2 = model.predict(train set 2)
print(features2.shape) #600,512
How to combine these feature 1 and feature 2, so that output shape is 600,1024?

SIMPLEST SOLUTION:
you can simply concatenate the output of the two networks in this way:
features = np.concatenate([features1, features2], 1)
ALTERNATIVE:
given two trained models that have the same structure, whatever their structures are, you can combine them in this way
# generate dummy data
n_sample = 600
set1 = np.random.uniform(0,1, (n_sample,30))
set2 = np.random.uniform(0,1, (n_sample,30))
# model 1
inp1 = Input((30,))
x1 = Dense(512,)(inp1)
x1 = Dropout(0.3)(x1)
x1 = BatchNormalization()(x1)
out1 = Dense(3, activation='softmax')(x1)
m1 = Model(inp1, out1)
# m1.fit(...)
# model 2
inp2 = Input((30,))
x2 = Dense(512,)(inp2)
x2 = Dropout(0.3)(x2)
x2 = BatchNormalization()(x2)
out2 = Dense(3, activation='softmax')(x2)
m2 = Model(inp2, out2)
# m2.fit(...)
# concatenate the desired output
concat = Concatenate()([m1.layers[1].output, m2.layers[1].output]) # get the outputs of dense 512 layers
merge = Model([m1.input, m2.input], concat)
# make combined predictions
merge.predict([set1,set2]).shape # (n_sample, 1024)

How to do matrix multiplication with learnable weights in keras?

I have a model like the one below. I want to add a matrix of learnable weights in the end, which is initialized to the variable matrix that I pass to the function create_model.
To get the intuitive idea of what I want to do, imagine the matrix is supposed to be the one I pass to the model, but I have the feeling that it can still be finetuned during training. Therefore, I want it to be initialized to the values I pass, and then refined during training.
The code below works, but as you see from the model.summary() output, the matrix multiplication contains no learnable weights, which makes me think that the weights of the matrix are not beeing finetuned.
What am I doing wrong?
def create_model(num_columns, matrix):
inp_layer = tfl.Input((num_columns,))
dense = tfl.Dense(512, activation = 'relu')(inp_layer)
dense = tfl.Dense(256, activation = 'relu')(dense)
dense = tfl.Dense(128, activation = 'relu')(dense)
va = tf.Variable(matrix, dtype = tf.float32)
dense = K.dot(dense, va )
model = tf.keras.Model(inputs = inp_layer, outputs = dense)
model.compile(optimizer='adam', loss=['binary_crossentropy'])
model.summary()
return model
matrix = np.random.randint(0,2,(128, 206)) # In reality, this is not random, but it has sensed values
num_columns = 750
model = create_model(num_columns,matrix)

you can simply use a dense layer with no bias to do this multiplication. After the model is built I change the weight of interest with the matrix you provided
def create_model(num_columns, matrix):
inp_layer = Input((num_columns,))
x = Dense(512, activation = 'relu')(inp_layer)
x = Dense(256, activation = 'relu')(x)
x = Dense(128, activation = 'relu')(x)
dense = Dense(206, use_bias=False)(x)
model = Model(inputs = inp_layer, outputs = dense)
model.compile(optimizer='adam', loss=['binary_crossentropy'])
model.set_weights(model.get_weights()[:-1] + [matrix])
model.summary()
return model
matrix = np.random.randint(0,2,(128, 206)) # In reality, this is not random, but it has sensed values
num_columns = 750
model = create_model(num_columns,matrix)
check
(model.get_weights()[-1] == matrix).all() # True
In this way, the weights can be fine-tuned

Get output of hidden layer during classification task

I have a question.
Working with CNN, there is a way to get output of an hidden layer during classification?
Example:
common_input = layers.Input(shape=(224, 224, 3))
x = model0(common_input) #model0 is a pretrain model on imagenet
x = layers.Flatten()(x)
p = layers.Dense(768, activation="relu")(x)
p = layers.Dropout(0.3)(p)
p = layers.Dense(8, activation="softmax", name="fc_out")(p)
model = Model(inputs=common_input, outputs=p)
My task is classification; so if I have 2000 images I will get a matrix 2000x8. If I need the output of layer dense, there is a way to get a matrix 2000x768 (both in the same computation)?
Thanks

1d CNN audio in keras

I want to try to implement the neural network architecture of the attached image: 1DCNN_model
Consider that I've got a dataset X which is (N_signals, 1500, 40) where 40 is the number of features where I want to do the 1d convolution on.
My Y is (N_signals, 1500, 2) and I'm working with keras.
Every 1d convolution needs to take one feature vector like in this picture:1DCNN_convolution
So it has to take one chunk of the 1500 timesamples, pass it through the 1d convolutional layer (sliding along time-axis) then feed all the output features to the LSTM layer.
I tried to implement the first convolutional part with this code but I'm not sure what it's doing, I can't understand how it can take in one chunk at a time (maybe I need to preprocess my input data before?):
input_shape = (None, 40)
model_input = Input(input_shape, name = 'input')
layer = model_input
convs = []
for i in range(n_chunks):
conv = Conv1D(filters = 40,
kernel_size = 10,
padding = 'valid',
activation = 'relu')(layer)
conv = BatchNormalization(axis = 2)(conv)
pool = MaxPooling1D(40)(conv)
pool = Dropout(0.3)(pool)
convs.append(pool)
out = Merge(mode = 'concat')(convs)
conv_model = Model(input = layer, output = out)
Any advice? Thank you very much

Thank you very much, I modified my code in this way:
input_shape = (1500,40)
model_input = Input(shape=input_shape, name='input')
layer = model_input
layer = Conv1D(filters=40,
kernel_size=10,
padding='valid',
activation='relu')(layer)
layer = BatchNormalization(axis=2)(layer)
layer = MaxPooling1D(pool_size=40,
padding='same')(layer)
layer = Dropout(self.params.drop_rate)(layer)
layer = LSTM(40, return_sequences=True,
activation=self.params.lstm_activation)(layer)
layer = Dropout(self.params.lstm_dropout)(layer)
layer = Dense(40, activation = 'relu')(layer)
layer = BatchNormalization(axis = 2)(layer)
model_output = TimeDistributed(Dense(2,
activation='sigmoid'))(layer)
I was actually thinking that maybe I have to permute my axes in order to make maxpooling layer work on my 40 mel feature axis...

If you want to perform an individual 1D convolution over the 40 feature channels you should add a dimension to your input:
(1500,40,1)
if you perform 1D convolution on a input with shape
(1500,40)
the filters are applied on the time dimension and the pictures you posted indicate that this is not what you want to do.