We Keep Coding

Convolutional LSTM Model Dimension Incompatibility when making predictions & prediction dimension issues

I structured a Convolutional LSTM model to predict the forthcoming Bitcoin price data, using the analyzed past data of the Bitcoin close price and other features.
Let me jump straight to the code:
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import tensorflow as tf
import tensorflow.keras as keras
import keras_tuner as kt
from keras_tuner import HyperParameters as hp
from keras.models import Sequential
from keras.layers import InputLayer, ConvLSTM1D, LSTM, Flatten, RepeatVector, Dense, TimeDistributed
from keras.callbacks import EarlyStopping
from tensorflow.keras.metrics import RootMeanSquaredError
from tensorflow.keras.optimizers import Adam
import keras.backend as K
from keras.losses import Huber
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
DIR = '../input/btc-features-targets'
SEG_DIR = '../input/segmented'
segmentized_features = os.listdir(SEG_DIR)
btc_train_features = []
for seg in segmentized_features:
train_features = pd.read_csv(f'{SEG_DIR}/{seg}')
train_features.set_index('date', inplace=True)
btc_train_features.append(scaler.fit_transform(train_features.values))
btc_train_targets = pd.read_csv(f'{DIR}/btc_train_targets.csv')
btc_train_targets.set_index('date', inplace=True)
btc_test_features = pd.read_csv(f'{DIR}/btc_test_features.csv')
btc_tef1 = btc_test_features.iloc[:111]
btc_tef2 = btc_test_features.iloc[25:]
btc_tef1.set_index('date', inplace=True)
btc_tef2.set_index('date', inplace=True)
btc_test_targets = pd.read_csv(f'{DIR}/btc_test_targets.csv')
btc_test_targets.set_index('date', inplace=True)
btc_trt_log = np.log(btc_train_targets)
btc_tefs1 = scaler.fit_transform(btc_tef1.values)
btc_tefs2 = scaler.fit_transform(btc_tef2.values)
btc_tet_log = np.log(btc_test_targets)
scaled_train_features = []
for features in btc_train_features:
shape = features.shape
scaled_train_features.append(np.expand_dims(features, [0,3]))
shape_2 = btc_tefs1.shape
btc_tefs1 = np.expand_dims(btc_tefs1, [0,3])
shape_3 = btc_tefs2.shape
btc_tefs2 = np.expand_dims(btc_tefs2, [0,3])
btc_trt_log = btc_trt_log.values[0]
btc_tet_log = btc_tet_log.values[0]
def build(hp):
model = keras.Sequential()
# Input Layer
model.add(InputLayer(input_shape=(111,32,1)))
# ConvLSTM1D
convLSTM_hp_filters = hp.Int(name='convLSTM_filters', min_value=32, max_value=512, step=32)
convLSTM_hp_kernel_size = hp.Choice(name='convLSTM_kernel_size', values=[3,5,7])
convLSTM_activation = hp.Choice(name='convLSTM_activation', values=['selu', 'relu'])
model.add(ConvLSTM1D(filters=convLSTM_hp_filters,
kernel_size=convLSTM_hp_kernel_size,
padding='same',
activation=convLSTM_activation,
use_bias=True,
bias_initializer='zeros'))
# Flatten
model.add(Flatten())
# RepeatVector
model.add(RepeatVector(5))
# LSTM
LSTM_hp_units = hp.Int(name='LSTM_units', min_value=32, max_value=512, step=32)
LSTM_activation = hp.Choice(name='LSTM_activation', values=['selu', 'relu'])
model.add(LSTM(units=LSTM_hp_units, activation=LSTM_activation, return_sequences=True))
# TimeDistributed Dense
dense_units = hp.Int(name='dense_units', min_value=32, max_value=512, step=32)
dense_activation = hp.Choice(name='dense_activation', values=['selu', 'relu'])
model.add(TimeDistributed(Dense(units=dense_units, activation=dense_activation)))
# TimeDistributed Dense_Output
model.add(Dense(1))
# Set Learning Rate
hp_learning_rate = hp.Choice(name='learning_rate', values=[1e-2, 1e-3, 1e-4])
# Compile Model
model.compile(optimizer=Adam(learning_rate=hp_learning_rate),
loss=Huber(),
metrics=[RootMeanSquaredError()])
return model
tuner = kt.Hyperband(build,
objective=kt.Objective('root_mean_squared_error', direction='min'),
max_epochs=10,
factor=3)
early_stop = EarlyStopping(monitor='root_mean_squared_error', patience=5)
opt_hps = []
for train_features in scaled_train_features:
tuner.search(train_features, btc_trt_log, epochs=50, callbacks=[early_stop])
opt_hps.append(tuner.get_best_hyperparameters(num_trials=1)[0])
models, epochs = ([] for _ in range(2))
for hps in opt_hps:
model = tuner.hypermodel.build(hps)
models.append(model)
history = model.fit(train_features, btc_trt_log, epochs=70, verbose=0)
rmse = history.history['root_mean_squared_error']
best_epoch = rmse.index(min(rmse)) + 1
epochs.append(best_epoch)
hypermodel = tuner.hypermodel.build(opt_hps[0])
for train_features, epoch in zip(scaled_train_features, epochs): hypermodel.fit(train_features, btc_trt_log, epochs=epoch)
tp1 = hypermodel.predict(btc_tefs1).flatten()
tp2 = hypermodel.predict(btc_tefs2).flatten()
test_predictions = np.concatenate((tp1, tp2[86:]), axis=None)
The hyperparameters of the model are configured using keras_tuner; as there were ResourceExhaustError issues output by the notebook when training is done with the full features dataset, sequentially segmented datasets are used instead (and apparently, referring to the study done utilizing the similar model architecture, training is able to be efficiently done through this training approach).
The input dimension of each segmented dataset is (111,32,1).
There aren't any issues reported until before the last code block. The models work fine. Yet, when the .predict() function is executed, the notebook prints out an error, which states that the dimension of the input features for making predictions is incompatible with the dimension of the input features used while training. I did not understand the reason behind its occurrence, since as far as I know, the input dimensions of a train dataset for a DNN model cannot be identical as the input dimensions of a test dataset.
Even though all the price data from 2018 to early 2021 are used as training datasets, predictions are only needed for the mid 2021 timeframe.
The dataset used for prediction has a dimension of (136,32,1).
I tried matching the dimension of this dataset to (111,32,1), through index slicing.
Now this showed issues in the output dimension. While predictions should be made for 136 data points, the result only returned 10.
Are there any issues relevant to the model configuration? Cannot interpret the current situation.

Is there a way to ensemble two keras (h5) models trained for same classes

I have trained two keras models with different datasets for same class labels. How could I ensemble the models keras_model.h5 and keras_model2.h5 together and make another keras model say keras_ensemble.h5. I have tried referring various internet sources but not luck. Can someone help me with the code for ensembling it?
Here are the models I've trained
Please assist me through this.Thank you.
Edit:
This was my code which i was able to get through with the help of the one who responded to my question Frightera
import tensorflow.keras
import tensorflow as tf
from PIL import Image, ImageOps
import numpy as np
# Disable scientific notation for clarity
np.set_printoptions(suppress=True)
# Load the model
keras_model = tensorflow.keras.models.load_model('keras_model.h5', compile=False)
keras_model._name = 'model1'
keras_model2 = tensorflow.keras.models.load_model('keras_model2.h5', compile=False)
keras_model2._name = 'model2'
models = [keras_model, keras_model2]
#model_input = tf.keras.Input(shape=(125, 125, 3))
model_input = tf.keras.Input(shape=(224, 224, 3))
model_outputs = [model(model_input) for model in models]
ensemble_output = tf.keras.layers.Average()(model_outputs)
ensemble_model = tf.keras.Model(inputs=model_input, outputs=ensemble_output)
EDIT
How do i get the keras ensemble model in h5 format??

You can average-ensemble them like this:
models = [keras_model, keras_model2]
model_input = tf.keras.Input(shape=(125, 125, 3))
model_outputs = [model(model_input) for model in models]
ensemble_output = tf.keras.layers.Average()(model_outputs)
ensemble_model = tf.keras.Model(inputs=model_input, outputs=ensemble_output)
Edit: If you want to create weighted ensemble you can create a custom layer like this:
class WeightedAverageLayer(tf.keras.layers.Layer):
def __init__(self, w1, w2, **kwargs):
super(WeightedAverageLayer, self).__init__(**kwargs)
self.w1 = w1
self.w2 = w2
def call(self, inputs):
return self.w1 * inputs[0] + self.w2 * inputs[1]
So the output layer should look like this:
ensemble_output = WeightedAverageLayer(0.6, 0.4)(model_outputs)
Here, first model's output is scaled with a factor of 0.6. Same thing applies for the second model, with a factor of 0.4.
Note: WeightedAverageLayer(0.5, 0.5) will be identical to tf.keras.layers.Average().

import tensorflow.keras
import tensorflow as tf
from PIL import Image, ImageOps
import numpy as np
# Disable scientific notation for clarity
np.set_printoptions(suppress=True)
# Load the model
keras_model = tensorflow.keras.models.load_model('APPLE/apple_model1.h5', compile=False)
keras_model._name = 'model1'
keras_model2 = tensorflow.keras.models.load_model('APPLE/apple_model2.h5', compile=False)
keras_model2._name = 'model2'
models = [keras_model, keras_model2]
#model_input = tf.keras.Input(shape=(125, 125, 3))
model_input = tf.keras.Input(shape=(224, 224, 3))
model_outputs = [model(model_input) for model in models]
ensemble_output = tf.keras.layers.Average()(model_outputs)
ensemble_model = tf.keras.Model(inputs=model_input, outputs=ensemble_output)
ensemble_model.save('APPLE/ensemble_apple.h5')

Multi Class and Multi Label Simple Model

I have been working for a few weeks to complete a project. The project seems simple , but there's a lot of moving parts. So
- Create a simple CNN model that has 4 input images. The images are of trees. Images are the trunk of a tree, branches , leafs and the roots of the tree.
- There are 8 different trees.
- The goal feed the model the four images and have the model detect which tree it is.
- I must Create the raw data from google images (search)
As I work through this problem for a school project , I am struggling on how to pre-process the images for the hot-label and multi class. Then insure my model architecture is accurate.
# import the necessary packages
from keras.models import Sequential
from keras.layers.normalization import BatchNormalization
from keras.layers.convolutional import Conv2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers.core import Activation
from keras.layers.core import Dropout
from keras.layers.core import Dense
from keras.layers import Flatten
from keras.layers import Input
from keras.models import Model
from keras.layers import Dense , LSTM, concatenate, Input, Flatten
# define four inputs ( Different photos of trees)
inputA = Input(shape=(308, 308, 1)) # Trunk
inputB = Input(shape=(308, 308, 1)) # Branch
inputC = Input(shape=(308, 308, 1)) # Leafs
inputD = Input(shape=(308, 308, 1)) # Roots
# the first branch operates on the first input
a = Conv2D(16, (2, 2), activation='relu')(inputA)
a = Conv2D(16, (2, 2), activation='relu')(a)
a = Model(inputs=inputA, outputs=a)
# the second branch opreates on the second input
b = Conv2D(16, (2, 2), activation='relu')(inputB)
b = Conv2D(16, (2, 2), activation='relu')(b)
b = Model(inputs=inputB, outputs=b)
# the third branch opreates on the third input
c = Conv2D(16, (2, 2), activation='relu')(inputC)
c = Conv2D(16, (2, 2), activation='relu')(c)
c = Model(inputs=inputC, outputs=c)
# the fourth branch opreates on the fourth input
d = Conv2D(16, (2, 2), activation='relu')(inputD)
d = Conv2D(16, (2, 2), activation='relu')(d)
d = Model(inputs=inputD, outputs=d)
# combine the output of the four branches
combined = concatenate([a.output, b.output, c.output, d.output])
# apply a FC layer and then a regression prediction on the
# combined outputs
z = Dense(128, activation="relu")(combined)
z = Dense(4, activation="softmax")(z)
# our model will accept the inputs of the four branches and then output a single value
model = Model(inputs=[a.input, b.input, c.input, d.input], outputs=z)
model.summary()
model.compile(loss='categorical_crossentropy', optimizer="adam", metrics=['accuracy'])
This is how I have been trying to hot-label for pre-processing data.
from tqdm import tqdm
import tensorflow as tf
import matplotlib.pyplot as plt
%matplotlib inline
train_data = ""
test_data = ""
def one_hot_label(img):
label = img.split(".")[0]
if label == "Trunk";
ohl = np.array([1,0,0,0])
elif label == "Branch":
ohl = np.array([0,1,0,0])
elif label == "Leaf":
ohl = np.array([0,0,1,0])
elif label == "Root":
ohl = np.array([0,0,0,1])
return ohl
def train_data_with_label();
train_images = []
for i in tqdm(os.listdir(train_data)):
path = os.path.join(train_data, i)
img = cv2.imread(path, cv2.IMREAD_COLOR)
img = cv2.resize(img, (308, 308))
train_images.append([np.array(img), one_hot_label(i)])
shuffle(train_images)
return train_images
def test_data_with_label();
test_images = []
for i in tqdm(os.listdir(test_data)):
path = os.path.join(test_data, i)
img = cv2.imread(path, cv2.IMREAD_COLOR)
img = cv2.resize(img, (308, 308))
test_images.append([np.array(img), one_hot_label(i)])
shuffle(test_images)
return test_images
This is a huge post. I know I am a mess and really struggling. When you run the data its accuracy is so low , I know something is wrong. (around 18%)
If anyone can help !! pleaseeee . It doesnt have to be the exact answer even a direction is helpful.
I am grateful for this site and the people on it.
Thanks
Noob Coder Dyl

How to generate CNN heatmaps using built-in Keras in TF2.0 (tf.keras)

I used to generate heatmaps for my Convolutional Neural Networks, based on the stand-alone Keras library on top of TensorFlow 1. That worked fine, however, after my switch to TF2.0 and built-in tf.keras implementation (with eager execution) I cannot use my old heatmap generation code any longer.
So I re-wrote parts of my code for TF2.0 and ended up with the following:
from tensorflow.keras.applications.vgg16 import preprocess_input
from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.models import load_model
from tensorflow.keras import preprocessing
from tensorflow.keras import backend as K
from tensorflow.keras import models
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
image_size = 150
image_path = "/tmp/images/test-image.jpg"
model_path = "/tmp/models/prototype/basic_vgg16.h5"
# Load pre-trained Keras model and the image to classify
model = load_model(model_path) # VGG16 CNN with custom classifier head
image = load_img(image_path, target_size=(image_size, image_size))
img_tensor = preprocessing.image.img_to_array(image)
img_tensor = np.expand_dims(img_tensor, axis=0)
img_tensor = preprocess_input(img_tensor)
input_layer = model.get_layer("model_input")
conv_layer = model.get_layer("block5_conv3")
heatmap_model = models.Model([model.inputs], [conv_layer.output, model.output])
# Get gradient of the winner class w.r.t. the output of the (last) conv. layer
with tf.GradientTape() as gtape:
conv_output, predictions = heatmap_model(img_tensor)
loss = predictions[:, np.argmax(predictions[0])]
grads = gtape.gradient(loss, conv_output)
pooled_grads = K.mean(grads, axis=(0, 1, 2))
# Get values of pooled grads and model conv. layer output as Numpy arrays
iterate = K.function([model.inputs], [pooled_grads, conv_layer.output[0]])
pooled_grads_value, conv_layer_output_value = iterate([img_tensor])
# Multiply each channel in the feature-map array by "how important it is"
for i in range(pooled_grads_value.shape[0]):
conv_layer_output_value[:, :, i] *= pooled_grads_value[i]
# Channel-wise mean of resulting feature-map is the heatmap of class activation
heatmap = np.mean(conv_layer_output_value, axis=-1)
heatmap = np.maximum(heatmap, 0)
max_heat = np.max(heatmap)
if max_heat == 0:
max_heat = 1e-10
heatmap /= max_heat
# Render heatmap via pyplot
plt.matshow(heatmap)
plt.show()
But now the following line:
iterate = K.function([model.inputs], [pooled_grads, conv_layer.output[0]])
leads to this error message:
AttributeError: Tensor.op is meaningless when eager execution is enabled.
I always used Keras and did not work with TF directly, so I am bit lost here.
Any ideas what could be the problem here?
PS: If you want to c&p this code, you can create the VGG16-based model like so:
# Create Keras model from pre-trained VGG16 and custom classifier
input_layer = layers.Input(shape=(image_size, image_size, 3), name="model_input")
vgg16_model = VGG16(weights="imagenet", include_top=False, input_tensor=input_layer)
model_head = vgg16_model.output
model_head = layers.Flatten(name="model_head_flatten")(model_head)
model_head = layers.Dense(256, activation="relu")(model_head)
model_head = layers.Dense(3, activation="softmax")(model_head)
model = models.Model(inputs=input_layer, outputs=model_head)
model.compile(loss="categorical_crossentropy", optimizer=optimizers.Adam(), metrics=["accuracy"])

At the end of the GradientTape loop, conv_output and grads already holds the value. The iterate function is no longer need to compute the values.
Working example below:
from tensorflow.keras.applications.vgg16 import preprocess_input
from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.models import load_model
from tensorflow.keras import preprocessing
from tensorflow.keras import backend as K
from tensorflow.keras import models
import tensorflow as tf
import numpy as np
image_size = 224
# Load pre-trained Keras model and the image to classify
model = tf.keras.applications.vgg16.VGG16()
image = np.random.random((image_size, image_size, 3))
img_tensor = preprocessing.image.img_to_array(image)
img_tensor = np.expand_dims(img_tensor, axis=0)
img_tensor = preprocess_input(img_tensor)
conv_layer = model.get_layer("block5_conv3")
heatmap_model = models.Model([model.inputs], [conv_layer.output, model.output])
# Get gradient of the winner class w.r.t. the output of the (last) conv. layer
with tf.GradientTape() as gtape:
conv_output, predictions = heatmap_model(img_tensor)
loss = predictions[:, np.argmax(predictions[0])]
grads = gtape.gradient(loss, conv_output)
pooled_grads = K.mean(grads, axis=(0, 1, 2))
heatmap = tf.reduce_mean(tf.multiply(pooled_grads, conv_output), axis=-1)
heatmap = np.maximum(heatmap, 0)
max_heat = np.max(heatmap)
if max_heat == 0:
max_heat = 1e-10
heatmap /= max_heat
print(heatmap.shape)

one more issue to view the heatmap
import matplotlib.pyplot as plt
hm=np.squeeze(heatmap)
hm.shape
(14, 14)
plt.imshow(hm)

Keras VGG extract features

I have loaded a pre-trained VGG face CNN and have run it successfully. I want to extract the hyper-column average from layers 3 and 8. I was following the section about extracting hyper-columns from here. However, since the get_output function was not working, I had to make a few changes:
Imports:
import matplotlib.pyplot as plt
import theano
from scipy import misc
import scipy as sp
from PIL import Image
import PIL.ImageOps
from keras.models import Sequential
from keras.layers.core import Flatten, Dense, Dropout
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.optimizers import SGD
import numpy as np
from keras import backend as K
Main function:
#after necessary processing of input to get im
layers_extract = [3, 8]
hc = extract_hypercolumn(model, layers_extract, im)
ave = np.average(hc.transpose(1, 2, 0), axis=2)
print(ave.shape)
plt.imshow(ave)
plt.show()
Get features function:(I followed this)
def get_features(model, layer, X_batch):
get_features = K.function([model.layers[0].input, K.learning_phase()], [model.layers[layer].output,])
features = get_features([X_batch,0])
return features
Hyper-column extraction:
def extract_hypercolumn(model, layer_indexes, instance):
layers = [K.function([model.layers[0].input],[model.layers[li].output])([instance])[0] for li in layer_indexes]
feature_maps = get_features(model,layers,instance)
hypercolumns = []
for convmap in feature_maps:
for fmap in convmap[0]:
upscaled = sp.misc.imresize(fmap, size=(224, 224),mode="F", interp='bilinear')
hypercolumns.append(upscaled)
return np.asarray(hypercolumns)
However, when I run the code, I'm getting the following error:
get_features = K.function([model.layers[0].input, K.learning_phase()], [model.layers[layer].output,])
TypeError: list indices must be integers, not list
How can I fix this?
NOTE:
In the hyper-column extraction function, when I use feature_maps = get_features(model,1,instance) or any integer in place of 1, it works fine. But I want to extract the average from layers 3 to 8.

It confused me a lot:
After layers = [K.function([model.layers[0].input],[model.layers[li].output])([instance])[0] for li in layer_indexes], layers is list of extracted feature.
And then you send that list into feature_maps = get_features(model,layers,instance).
In def get_features(model, layer, X_batch):, they second parameter, namely layer, is used to index in model.layers[layer].output.
What you want is:
feature_maps = get_features(model,layer_indexes,instance): passing layer indices rather than extracted features.
get_features = K.function([model.layers[0].input, K.learning_phase()], [model.layers[l].output for l in layer]): list cannot be used to indexing list.
Still, your feature abstracting function is horribly written. I suggest you to rewrite everything rather than mixing codes.

I rewrote your function for a single channel input image (W x H x 1). Maybe it will be helpful.
def extract_hypercolumn(model, layer_indexes, instance):
test_image = instance
outputs = [layer.output for layer in model.layers] # all layer outputs
comp_graph = [K.function([model.input]+ [K.learning_phase()], [output]) for output in outputs] # evaluation functions
feature_maps = []
for layerIdx in layer_indexes:
feature_maps.append(layer_outputs_list[layerIdx][0][0])
hypercolumns = []
for idx, convmap in enumerate(feature_maps):
# vv = np.asarray(convmap)
# print(vv.shape)
vv = np.asarray(convmap)
print('shape of feature map at layer ', layer_indexes[idx], ' is: ', vv.shape)
for i in range(vv.shape[-1]):
fmap = vv[:,:,i]
upscaled = sp.misc.imresize(fmap, size=(img_width, img_height),
mode="F", interp='bilinear')
hypercolumns.append(upscaled)
# hypc = np.asarray(hypercolumns)
# print('shape of hypercolumns ', hypc.shape)
return np.asarray(hypercolumns)