We Keep Coding

Parallel Convolutions using Keras

What i am trying to do is create a text classification model which combines CNNS and word embeddings.The basic idea is that we have an Embedding layer at the start of the network and then 2 parallel convolutional networks for find 2,3 - grams.
Each of these convolution layers takes the output of the embedding layer as input.
After the outputs of the two cnn layers are concatenated,flattend and feeded to a Dense.
My input is tokenized,numerical sentences of length 27(shape = (None,27)) and i have 1244 of these sentences.
I've managed to create a sequential model wit ha single cnn layer but struggle wit hthe above
My code so far :
input_shape = Embedding(voc, 100,weights=[embedding_matrix], input_length=X.shape[1])
tower_1 = Conv1D(filters=100, kernel_size=2, activation='relu')(input_shape)
tower_1 = MaxPooling1D(pool_size=2)(tower_1)
tower_2 = Conv1D(filters=100, kernel_size=3, activation='relu')(input_shape)
tower_2 = MaxPooling1D(pool_size=2)(tower_2)
merged = keras.layers.concatenate([tower_1, tower_2,], axis=1)
merged = Flatten()(merged)
out = Dense(3, activation='softmax')(merged)
model = Model(input_shape, out)
This produces this error:
TypeError: Inputs to a layer should be tensors. Got: <keras.layers.embeddings.Embedding object at 0x7fadca016dd0>
i have also trid replacing
input_shape = Embedding(voc, 100,weights=[embedding_matrix], input_length=X.shape[1])
with:
input_tensor = Input(shape=(1244,27))
input_shape = Embedding(voc, 100,weights=[embedding_matrix], input_length=X.shape[1])(input_tensor)
which gives me this error:
ValueError: Input 0 of layer "max_pooling1d_23" is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: (None, 1244, 26, 100)

You should define your Input layer without the number of samples. Just the sentence length:
import tensorflow as tf
inputs = tf.keras.layers.Input((27,))
embedded = tf.keras.layers.Embedding(50, 100, input_length=27)(inputs)
tower_1 = tf.keras.layers.Conv1D(filters=100, kernel_size=2, activation='relu')(embedded)
tower_1 = tf.keras.layers.MaxPooling1D(pool_size=2)(tower_1)
tower_2 = tf.keras.layers.Conv1D(filters=100, kernel_size=3, activation='relu')(embedded)
tower_2 = tf.keras.layers.MaxPooling1D(pool_size=2)(tower_2)
merged = tf.keras.layers.concatenate([tower_1, tower_2,], axis=1)
merged = tf.keras.layers.Flatten()(merged)
out = tf.keras.layers.Dense(3, activation='softmax')(merged)
model = tf.keras.Model(inputs, out)
print(model.summary())
Usage:
samples = 5
random_input = tf.random.uniform((samples, 27), maxval=50, dtype=tf.int32)
print(model(random_input))
tf.Tensor(
[[0.31525075 0.33163014 0.3531191 ]
[0.3266019 0.3295619 0.34383622]
[0.32351935 0.32669052 0.34979013]
[0.32954428 0.33178467 0.33867106]
[0.32966062 0.3283257 0.34201372]], shape=(5, 3), dtype=float32)

How to change input dimensions for LSTM layer

I need to make a model that has 2 dropout layers and two LSTM layers. Unfortunately I have a problem with input shape that goes to my second LSTM layer. After searching for the problem I found out I need to change the input dimensions but I don't know how to do that. I found an option that requires using Lambda layer but I can't import it to my environmet (it's a coursera environment). Have you got any suggestions how to deal with my error?
model = Sequential()
Layer1 = model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))
Layer2 = model.add(Bidirectional(LSTM(20)))
Layer3 = model.add(Dropout(.03))
Layer4 = model.add(LSTM(20))
Layer5 = model.add(Dense(total_words,
kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4),
bias_regularizer=regularizers.l2(1e-4),
activity_regularizer=regularizers.l2(1e-5)))
# A Dense Layer including regularizers
Layer6 = model.add(Dense(total_words, activation = 'softmax'))
# Pick an optimizer
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.summary()
Error:
ValueError: Input 0 of layer lstm_20 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 40]

Thank you #Marco Cerliani and #Joanna Kastelik for the update.
For the benefit of community providing solution here using sample data as shown below
import tensorflow as tf
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Bidirectional, LSTM, Dropout, Dense
from tensorflow.keras.regularizers import l1_l2, l2
total_words = 478
max_sequence_len = 90
model = Sequential()
Layer1 = model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))
Layer2 = model.add(Bidirectional(LSTM(20)))
Layer3 = model.add(Dropout(.03))
Layer4 = model.add(LSTM(20))
Layer5 = model.add(Dense(total_words,
kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4),
bias_regularizer=l2(1e-4),
activity_regularizer=l2(1e-5)))
# A Dense Layer including regularizers
Layer6 = model.add(Dense(total_words, activation = 'softmax'))
# Pick an optimizer
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.summary()
Output:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-1-8ce04225c92d> in <module>()
12 Layer2 = model.add(Bidirectional(LSTM(20)))
13 Layer3 = model.add(Dropout(.03))
---> 14 Layer4 = model.add(LSTM(20))
15 Layer5 = model.add(Dense(total_words,
16 kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4),
8 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/input_spec.py in assert_input_compatibility(input_spec, inputs, layer_name)
221 'expected ndim=' + str(spec.ndim) + ', found ndim=' +
222 str(ndim) + '. Full shape received: ' +
--> 223 str(tuple(shape)))
224 if spec.max_ndim is not None:
225 ndim = x.shape.rank
ValueError: Input 0 of layer lstm_1 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: (None, 40)
Fixed code:
Your issue can be resolved once you add return_sequences=True to the LSTM (i.e, Layer2) layer.
model = Sequential()
Layer1 = model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))
Layer2 = model.add(Bidirectional(LSTM(20, return_sequences=True)))
Layer3 = model.add(Dropout(.03))
Layer4 = model.add(LSTM(20))
Layer5 = model.add(Dense(total_words,
kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4),
bias_regularizer=l2(1e-4),
activity_regularizer=l2(1e-5)))
# A Dense Layer including regularizers
Layer6 = model.add(Dense(total_words, activation = 'softmax'))
# Pick an optimizer
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.summary()
Output:
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding_1 (Embedding) (None, 89, 64) 30592
_________________________________________________________________
bidirectional_1 (Bidirection (None, 89, 40) 13600
_________________________________________________________________
dropout_1 (Dropout) (None, 89, 40) 0
_________________________________________________________________
lstm_3 (LSTM) (None, 20) 4880
_________________________________________________________________
dense (Dense) (None, 478) 10038
_________________________________________________________________
dense_1 (Dense) (None, 478) 228962
=================================================================
Total params: 288,072
Trainable params: 288,072
Non-trainable params: 0
_________________________________________________________________

Matrix Size Incompatibility with ragged tensor of variable size sequence and LSTM

I'm working on ECG classification using LSTM. I have about thousands of ECG sequences, but the length of each ECG sequence seems to be different.
The model I have built is inspired of :
LSTM Model
Actually I have a ragged tensor of 8528 sequences (Z) which has a shape of (8528, None, None), and corresponding labels (U with 4 possibilities) are a one hot encoded tensor of shape (8528, 4).
The model is supposed to be a sequence of LTSM with respectively 64,256 and 100 units followed by a Dense layer of 4.
I have understand that because of the variable sequence length, the batch_size has to be set to 1.
Here is the code I'm using :
max_seq = Z.bounding_shape()[-1]
print(f'Maximum length sequence : {max_seq}')
model = keras.Sequential([
tf.keras.layers.Input(shape=[None, max_seq], batch_size=1, dtype=tf.float32, ragged=True),
keras.layers.LSTM(units=64, activation='tanh', dropout=0.2, name = 'LSTM_1', return_sequences=True),
keras.layers.LSTM(units=256, activation='tanh', dropout=0.2, name = 'LSTM_2',return_sequences=True),
keras.layers.LSTM(units=100, activation='tanh', dropout=0.2, name = 'LSTM_3', return_sequences=False),
# tf.keras.layers.LSTM(4),
#tf.keras.layers.Flatten(name = 'Flatten'),
tf.keras.layers.Dense(units=4, activation='sigmoid', name = 'Dense_1')
])
model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001),loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True), metrics=['accuracy'])
model.summary()
model.fit(Z, W, batch_size=1, epochs=20)
That returns :
Maximum length sequence : 18286
Model: "sequential_23"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
LSTM_1 (LSTM) (1, None, 64) 4697856
_________________________________________________________________
LSTM_2 (LSTM) (1, None, 256) 328704
_________________________________________________________________
LSTM_3 (LSTM) (None, 100) 142800
_________________________________________________________________
Dense_1 (Dense) (None, 4) 404
=================================================================
Total params: 5,169,764
Trainable params: 5,169,764
Non-trainable params: 0
_________________________________________________________________
Size of first sequence : 9000
But I got this error about Matrix incompatibility :
Matrix size-incompatible: In[0]: [1,16866], In[1]: [18286,256]
[[{{node sequential_23/LSTM_1/while/body/_1/sequential_23/LSTM_1/while/lstm_cell_67/MatMul}}]] [Op:__inference_train_function_100065]
Function call stack:
train_function
Can someone explain me why I got such type of error ?
Thanks in advance
EDIT : Here's a minimal reproductible example :
xx = tf.ragged.constant([
[[0.1, 0.2]],
[[0.4, 0.7 , 0.5, 0.6]],
[[0.4, 0.7 , 0.5]]
])
"""
Labels represented as OneHotEncoding so you
should use CategoricalCrossentropy instade of SparseCategoricalCrossentropy
"""
yy = np.array([[0, 0, 1,0], [1,0,0,0], [1,0,0,0]])
model = keras.Sequential([
tf.keras.layers.Input(shape=[None, max_seq], batch_size=1, dtype=tf.float32, ragged=True),
keras.layers.LSTM(units=64, activation='tanh', dropout=0.2, name = 'LSTM_1', return_sequences=True),
keras.layers.LSTM(units=256, activation='tanh', dropout=0.2, name = 'LSTM_2',return_sequences=True),
keras.layers.LSTM(units=100, activation='tanh', dropout=0.2, name = 'LSTM_3', return_sequences=False),
tf.keras.layers.Dense(units=4, activation='sigmoid', name = 'Dense_1')
])
model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001),loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True), metrics=['accuracy'])
model.summary()
model.fit(xx, yy, batch_size=1, epochs=20)

How to replace dense layer with convolutional one?

I wanted to replace the Dense_out layer with a convolution one, can anybody tell me how to do it?
code:
model = Sequential()
conv_1 = Conv2D(filters = 32,kernel_size=(3,3),activation='relu')
model.add(conv_1)
conv_2 = Conv2D(filters=64,kernel_size=(3,3),activation='relu')
model.add(conv_2)
pool = MaxPool2D(pool_size = (2,2),strides = (2,2), padding = 'same')
model.add(pool)
drop = Dropout(0.5)
model.add(drop)
model.add(Flatten())
Dense_1 = Dense(128,activation = 'relu')
model.add(Dense_1)
Dense_out = Dense(57,activation = 'softmax')
model.add(Dense_out)
model.compile(optimizer='Adam',loss='categorical_crossentropy',metric=['accuracy'])
model.fit(train_image,train_label,epochs=10,verbose = 1,validation_data=(test_image,test_label))
print(model.summary())
when I'm trying this code :
model = Sequential()
conv_01 = Conv2D(filters = 32,kernel_size=(3,3),activation='relu')
model.add(conv_01)
conv_02 = Conv2D(filters=64,kernel_size=(3,3),activation='relu')
model.add(conv_02)
pool = MaxPool2D(pool_size = (2,2),strides = (2,2), padding = 'same')
model.add(pool)
conv_11 = Conv2D(filters=64,kernel_size=(3,3),activation='relu')
model.add(conv_11)
pool_2 = MaxPool2D(pool_size=(2,2),strides=(2,2),padding='same')
model.add(pool_2)
drop = Dropout(0.3)
model.add(drop)
model.add(Flatten())
Dense_1 = Dense(128,activation = 'relu')
model.add(Dense_1)
Dense_2 = Dense(64,activation = 'relu')
model.add(Dense_2)
conv_out = Conv2D(filters= 64,kernel_size=(3,3),activation='relu')
model.add(Dense_out)
model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(train_image,train_label,epochs=10,verbose = 1,validation_data=(test_image,test_label))
I get the following error
ValueError: Input 0 of layer conv2d_3 is incompatible with the layer:
expected ndim=4, found ndim=2. Full shape received: [None, 64]
I am new at this so an explanation would greatly help

You will need to reshape to be able to use a 2x2 filter as needed in a conv2D layer.
You can use:
out = keras.layers.Reshape(target_shape)
model.add(out)
and then do the convolution:
conv_out = Conv2D(filters=3,kernel_size=(3,3),activation='softmax')
model.add(conv_out)
with filters being the number of channels you want in you output layer (3 for RGB).
More info about the layers and parameters in Keras Documentation

Keras Sequential model, more inputs

I am quite new to machine learning and I am currently working on a "car value predictor" application. I stuck where I have to feed my data to my model. I have 4 inputs:
date: the car's first registration date (int)
km: the car's mileage meter (int)
consume: the car's consume type (one-hot encoded vector with 10 element e.g. for petrol: [1 0 0 0 0 0 0 0 0 0])
type: the car's type (for example: "BMW-320", stored in one-hot encoded vector with 440 element )
and one output:
the price of the car.
I would like to do something similar to this: https://imgur.com/wlvffn7
I have tried the following code which compiles but the output is not what I wanted:
model = Sequential([
Dense(128, input_shape=(1,), activation='relu', name='date'),
Dense(128, input_shape=(1,), activation='relu', name='km'),
Dense(128, input_shape=(10,), activation='relu', name='consume'),
Dense(128, input_shape=(440,), activation='relu', name='type'),
Dropout(0.5),
Dense(128, activation='relu'),
Dropout(0.5),
Dense(1, activation='linear')
])
model.compile(loss='mse', optimizer='adam')
model.fit( x = {'date' : samples_train['input'][:,0],
'km' : samples_train['input'][:,1],
'consume':samples_train['input'][:,2],
'type':samples_train['input'][:,3]},
y = samples_train['output'],
epochs=1000,
batch_size=16,
verbose=1,
validation_data = ({'date' : samples_valid['input'][:,0],
'km' : samples_valid['input'][:,1],
'consume':samples_valid['input'][:,2],
'type':samples_valid['input'][:,3]}, samples_valid['output']),
callbacks=callbacks)
Can anyone point out what am I doing wrong or how can I implement a model "structure" like on the picture?
EDIT:
I think this is what I was looking for. Can anyone confirm this? :)
input_1 = Input(shape=(1,), name='date') # input layers
input_2 = Input(shape=(1,), name='km')
input_3 = Input(shape=(10,), name='consume')
input_4 = Input(shape=(440,), name='type')
dense_1 = Dense(256, activation='relu')(input_1) # hidden layers
dense_1 = Dense(256, activation='relu')(input_2)
dense_1 = Dense(256, activation='relu')(input_3)
dense_1 = Dense(256, activation='relu')(input_4)
dropout_1 = Dropout(0.5)(dense_1)
dense_2 = Dense(256, activation='relu')(dropout_1)
dropout_2 = Dropout(0.5)(dense_2)
outputs = Dense(1, activation='linear')(dropout_2) # output layer
model = Model([input_1,input_2,input_3,input_4], outputs)
Thank you in advance.

I think your second Implementation is wrong.
By implementeing it like that, dense_1 will only have the value you gave it in the last line : input_4 = Input(shape=(440,), name='type') thus not taking into accunt the rest of the imputs for the rest of the network.
What you should do is concatenate your inputs into a single Line before feeding it to the first dense layer, like that :
from keras.layers import Concatenate
input_1 = Input(shape=(1,), name='date') # input layers
input_2 = Input(shape=(1,), name='km')
input_3 = Input(shape=(10,), name='consume')
input_4 = Input(shape=(440,), name='type')
x = Concatenate()([input_1 , input_2 , input_3 , input_4]) # Concatenation of the inputs.
dense_1 = Dense(256, activation='relu')(x) # hidden layers
dropout_1 = Dropout(0.5)(dense_1)
dense_2 = Dense(256, activation='relu')(dropout_1)
dropout_2 = Dropout(0.5)(dense_2)
outputs = Dense(1, activation='linear')(dropout_2) # output layer
model = Model([input_1,input_2,input_3,input_4], outputs)