We Keep Coding

Mismatch between expected batch size and model output batch size

Does the keras functional API impose that the number of batch elements in models output equals to the number of elements in their input? For instance, the code bellow raises an exception: ValueError: Mismatch between expected batch size and model output batch size. Output shape = (1, 1), expected output shape = shape (2, 1):
d = 2
input_ = Input(shape=(d, ))
output_ = keras.layers.Lambda(lambda x: tf.reduce_sum(x, keepdims=True))(input_)
model = keras.Model(name='model', inputs=input_, outputs=output_)
If I set keepdims to false, another exception is raised: ValueError: zero-dimensional arrays cannot be concatenated, which makes sense as the model expects a batch of elements, each of which being a 2d array.

Tensorflow Keras functional API doesn't necessarily impose that the number of batch elements in model output equals to the number of elements in their input.
Tensorflow handles this by leaving the index 0 of the Output shape to None, for it to be batch_size agnostic.
But in this case, when model.summary() is printed, you can observe that the last layer output shape is fixed to (1,1) rather than (None,1).
Which will work normally if you will only feed it data with batch_size = 1, but will raise an error when the data fed have batch_size != 1.
More specifically: Mismatch between expected batch size and model output batch size. Output shape = (1, 1), expected output shape = shape (BATCH_SIZE, 1).
Expected output shape (BATCH_SIZE,1) is the output shape of your data.
Output shape (1, 1) is the output shape of the model.

Bi-directional LSTM for entity recognition

Following a paper, I'm using word embeddings as a feature vector for entity recognition.
I've attempted to architect the network using Keras but have run into a dimensionality problem I cannot seem to resolve.
Take the following example sentence:
["I went to the shop"]
The sentence has 5 words, and after computing the feature matrix, I am left with a matrix of dimension: (1, 120, 1000) == (#examples, sequence_length, embedding).
Note that sequence_length appends 0. padding when not complete. In this example, the actual sequence_length would be 5.
My network architecture is as follows:
enc = encode()
claims_input = Input(shape=(120, 1000), dtype='float32', name='claims')
x = Masking(mask_value=0., input_shape=(120, 1000))(claims_input)
x = Bidirectional(LSTM(units=512, return_sequences=True, recurrent_dropout=0.2, dropout=0.2))(x)
x = Bidirectional(LSTM(units=512, return_sequences=True, recurrent_dropout=0.2, dropout=0.2))(x)
out = TimeDistributed(Dense(8, activation="softmax"))(x)
model = Model(inputs=claims_input, output=out)
model.compile(loss="sparse_categorical_crossentropy", optimizer='adam', metrics=["accuracy"])
model.fit(enc, y)
The architecture is straight forward, I mask specific time steps, run two bidirectional LSTMs, followed by a softmax output. My y variable in this case, is a (9,8) one-hot-encoded matrix corresponding to the gold label of each word.
When trying to fit() this model, I am running into a dimensionality problem relating to the TimeDistributed() layer and I'm unsure how to resolve, or even begin to debug this.
Error: ValueError: Error when checking target: expected time_distributed_1 to have 3 dimensions, but got array with shape (9, 8)
Any help would be appreciated.

You are doing entity recognition. So each element in your input sequence will be assigned an entity (probably some of them as null). If your model takes an input sample of shape (120, n_features), then the output must also be a sequence of length of 120, i.e. one entity for each element. Therefore, the labels, i.e. y, you provide to the model must have a shape of (n_samples, 120, n_entities) (or (n_samples, 120, 1) if you are using sparse labeling).
Side note: There is no difference between TimeDistributed(Dense(...)) and Dense(...), as the Dense layer is applied on the last axis.

Do I understand batch_size correctly in Keras?

I'm using Keras' built-in inception_resnet_v2 to train a CNN to recognize images. When training the model, I have a numpy array of data as inputs, with input shape (1000, 299, 299, 3),
model.fit(x=X, y=Y, batch_size=16, ...) # Output shape `Y` is (1000, 6), for 6 classes
At first, When trying to predict, I passed in a single image of shape (299, 299, 3), but got the error
ValueError: Error when checking input: expected input_1 to have 4 dimensions, but got array with shape (299, 299, 3)
I reshaped my input with:
x = np.reshape(x, ((1, 299, 299, 3)))
Now, when I predict,
y = model.predict(x, batch_size=1, verbose=0)
I don't get an error.
I want to make sure I understand batch_size correctly in both training and predicting. My assumptions are:
1) With model.fit, Keras takes batch_size elements from the input array (in this case, it works through my 1000 examples 16 samples at a time)
2) With model.predict, I should reshape my input to be a single 3D array, and I should explicitly set batch_size to 1.
Are these correct assumptions?
Also, would it be better (possible even) to provide training data to the model so that this sort of reshape before prediction was not necessary? Thank you for helping me learn this.

No, you got the idea wrong. batch_size specifies how many data examples are "forwarded" through the network at once (using GPU usually).
By default, this value is set to 32 inside model.predict method, but you may specify otherwise (as you did with batch_size=1). Because of this default value you got an error:
ValueError: Error when checking input: expected input_1 to have 4
dimensions, but got array with shape (299, 299, 3)
You should not reshape your input this way, rather you would provide it with the correct batch size.
Say, for the default case you would pass an array of shape (32, 299, 299, 3), analogous for different batch_size, e.g. with batch_size=64 this function requires you to pass an input of shape (64, 299, 299, 3.
EDIT:
It seems you need to reshape your single sample into a batch. I would advise you to use np.expand_dims for improved readability and portability of your code, like this:
y = model.predict(np.expand_dims(x, axis=0), batch_size=1)

Python, Keras - ValueError: Cannot feed value of shape (10, 70, 1025) for Tensor u'dense_2_target:0', which has shape '(?, ?)'

I am trying to train a RNN by batches.
The input input size
(10, 70, 3075),
where 10 is the batch size, 70 the time dimension, 3075 are the frequency dimension.
There are three outputs whose size is
(10, 70, 1025)
each, basically 10 spectrograms with size (70,1025).
I would like to train this RNN by regression, whose structure is
input_img = Input(shape=(70,3075 ) )
x = Bidirectional(LSTM(n_hid,return_sequences=True, dropout=0.5, recurrent_dropout=0.2))(input_img)
x = Dropout(0.2)(x)
x = Bidirectional(LSTM(n_hid, dropout=0.5, recurrent_dropout=0.2))(x)
x = Dropout(0.2)(x)
o0 = ( Dense(1025, activation='sigmoid'))(x)
o1 = ( Dense(1025, activation='sigmoid'))(x)
o2 = ( Dense(1025, activation='sigmoid'))(x)
The problem is that output dense layers cannot take into account three dimensions, they want something like (None, 1025), which I don't know how to provide, unless I concatenate along the time dimension.
The following error occurs:
ValueError: Cannot feed value of shape (10, 70, 1025) for Tensor u'dense_2_target:0', which has shape '(?, ?)'
Would be the batch_shape option useful in the input layer? I have actually tried it, but I've got the same error.

In this instance the second RNN is collapsing the sequence to a single vector because by default return_sequences=False. To make the model return sequences and run the Dense layer over each timestep separately just add return_sequences=True to the second RNN as well:
x = Bidirectional(LSTM(n_hid, return_sequences=True, dropout=0.5, recurrent_dropout=0.2))(x)
The Dense layers automatically apply to the last dimension so no need to reshape afterwards.

To get the right output shape, you can use the Reshape layer:
o0 = Dense(70 * 1025, activation='sigmoid')(x)
o0 = Reshape((70, 1025)))(o0)
This will output (batch_dim, 70, 1025). You can do exactly the same for the other two outputs.

Error when feeding numpy sequence to bidirectional LSTM in Keras

I am trying to feed the features extracted from 2 fine-tuned VGG16 (each on a different stream), then for each sequence of 9 data pairs, concatenate their numpy arrays and feed the sequence of 9 outputs (concatenated) to a bi-directional LSTM in Keras.
The problem is that I am running into an error when trying to build the LSTM part. The following shows the generator I wrote to read both RGB and Optical flow streams, extract features and concatenate each pair :
def generate_generator_multiple(generator,dir1, dir2, batch_rgb, batch_flow, img_height,img_width):
print("Processing inside generate multiple")
genX1 = generator.flow_from_directory(dir1,
target_size = (img_height,img_width),
class_mode = 'categorical',
batch_size = batch_rgb,
shuffle=False
)
genX2 = generator.flow_from_directory(dir2,
target_size = (img_height,img_width),
class_mode = 'categorical',
batch_size = batch_flow,
shuffle=False
)
while True:
imgs, labels = next(genX1)
X1i = RGB_model.predict(imgs, verbose=0)
imgs2, labels2 = next(genX2)
X2i = FLOW_model.predict(imgs2,verbose=0)
Xi = []
for i in range(9):
Xi.append(np.concatenate([X1i[i+1],X2i[i]]))
Xi = np.asarray(Xi)
if np.array_equal(labels[1:],labels2)==False:
print("ERROR !! problem of labels matching: RGB and FLOW have different labels")
yield Xi, labels2[2]
I am expecting the generator to yield a sequence of 9 arrays, so the shape of Xi when I force the loop to run twice is: (9, 14, 7, 512)
When I use while True (like in the code above) and try to call the method to check what it returs, after 3 iterations I get the error:
ValueError: too many values to unpack (expected 2)
Now, assuming that there is no problem with the generator, I try to feed the data returned by the generator to the bidirectional LSTM like the following:
n_frames = 9
seq = 100
Bi_LSTM = Sequential()
Bi_LSTM.add(Bidirectional(LSTM(seq, return_sequences=True, dropout=0.25, recurrent_dropout=0.1),input_shape=(n_frames,14,7,512)))
Bi_LSTM.add(GlobalMaxPool1D())
Bi_LSTM.add(TimeDistributed(Dense(100, activation="relu")))
Bi_LSTM.add(layers.Dropout(0.25))
Bi_LSTM.add(Dense(4, activation="relu"))
model.compile(Adam(lr=.00001), loss='categorical_crossentropy', metrics=['accuracy'])
But I keep getting the following error: (the error log is a bit long)
InvalidArgumentError: Shape must be rank 4 but is rank 2 for 'bidirectional_2/Tile_1' (op: 'Tile') with input shapes: [?,7,512,1], [2].
It seems to be caused by this line:
Bi_LSTM.add(Bidirectional(LSTM(seq, return_sequences=True, dropout=0.25, recurrent_dropout=0.1),input_shape=(n_frames,14,7,512)))
I am not sure anymore if the problem is the way I try to build the LSTM, the way I return the data from the generator, or the way I define the input of LSTM.
Thanks a lot for any help you can provide.

It seems like this error specifically is cause by the following line:
input_shape=(n_frames,14,7,512)
I was confused about the input for LSTM. Instead to explicitly giving the shape of the input, we just need to specify the dimensions of the input. In my case, this is 3 since the input is a 3D np array. I still have other problems with my code, but for this specific error, the solution is changing that part with:
input_shape=(n_frames,3)
Edit:
When predicting, We need to get the mean of the prediction since LSTM expects a 1D input.
Another issue in my code was the shape of Xi. It needs to be reshaped before yielding it so that it matches the input expected by LSTM.