Hi I am getting weird results for the following code for the problem posted here (https://www.kaggle.com/c/titanic) -
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.advanced_activations import PReLU, LeakyReLU
from keras.layers.recurrent import SimpleRNN, SimpleDeepRNN
from keras.layers.embeddings import Embedding
from keras.layers.recurrent import LSTM, GRU
import pandas as pd
import numpy as np
from sklearn import preprocessing
np.random.seed(1919)
### Constants ###
data_folder = "/home/saj1919/Public/Data_Science_Mining_Study/submissions/titanic/data/"
out_folder = "/home/saj1919/Public/Data_Science_Mining_Study/submissions/titanic/output/"
batch_size = 4
nb_epoch = 10
### load train and test ###
train = pd.read_csv(data_folder+'train.csv', index_col=0)
test = pd.read_csv(data_folder+'test.csv', index_col=0)
print "Data Read complete"
Y = train.Survived
train.drop('Survived', axis=1, inplace=True)
columns = train.columns
test_ind = test.index
train['Age'] = train['Age'].fillna(train['Age'].mean())
test['Age'] = test['Age'].fillna(test['Age'].mean())
train['Fare'] = train['Fare'].fillna(train['Fare'].mean())
test['Fare'] = test['Fare'].fillna(test['Fare'].mean())
category_index = [0,1,2,4,5,6,8,9]
for i in category_index:
print str(i)+" : "+columns[i]
train[columns[i]] = train[columns[i]].fillna('missing')
test[columns[i]] = test[columns[i]].fillna('missing')
train = np.array(train)
test = np.array(test)
### label encode the categorical variables ###
for i in category_index:
print str(i)+" : "+str(columns[i])
lbl = preprocessing.LabelEncoder()
lbl.fit(list(train[:,i]) + list(test[:,i]))
train[:,i] = lbl.transform(train[:,i])
test[:,i] = lbl.transform(test[:,i])
### making data as numpy float ###
train = train.astype(np.float32)
test = test.astype(np.float32)
#Y = np.array(Y).astype(np.int32)
model = Sequential()
model.add(Dense(len(columns), 512))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(512, 1))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer="adam")
model.fit(train, Y, nb_epoch=nb_epoch, batch_size=batch_size, validation_split=0.20)
preds = model.predict(test,batch_size=batch_size)
pred_arr = []
for pred in preds:
pred_arr.append(pred[0])
### Output Results ###
preds = pd.DataFrame({"PassengerId": test_ind, "Survived": pred_arr})
preds = preds.set_index('PassengerId')
preds.to_csv(out_folder+'test.csv')
I am getting following results :
Train on 712 samples, validate on 179 samples
Epoch 0
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
Epoch 1
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
Epoch 2
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
Epoch 3
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
Epoch 4
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
Epoch 5
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
Epoch 6
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
Epoch 7
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
Epoch 8
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
Epoch 9
712/712 [==============================] - 0s - loss: -0.0000 - val_loss: -0.0000
I am trying to create a simple 3 layer network. Totally basic code.
I have tried these kind of classification problems before using keras on kaggle. But this time getting this error.
Is it overfitting due to less data.
What I am missing ? Can someone help ?
Old post, but answering anyway in case someone else attempts Titanic with Keras.
Your network may have too many parameters and too little regularization (e.g. dropout).
Call model.summary() right before the model.compile and it will show you how many parameters your network has. Just between your two Dense layers you should have 512 X 512 = 262,144 paramters. That's a lot for 762 examples.
Also you may want to use a sigmoid activation on the last layer and binary_cross entropy loss as you only have two output classes.
Related
I have made a model that tries to predict the chances of every piano key playing in a time step given all time steps before it. I tried making a GRU network with 88 outputs(one for every piano key)
input shape = (600,88,)
desired output/ label shape = (88, )
import numpy as np
import midi_processer
from keras import models
from keras import layers
x_train, x_test = np.load("samples.npy", mmap_mode='r'), np.load("test_samples.npy", mmap_mode='r')
y_train, y_test = np.load("labels.npy", mmap_mode='r'), np.load("test_labels.npy", mmap_mode='r')
def build_model():
model = models.Sequential()
model.add(layers.Input(shape=(600,88,)))
model.add(layers.GRU(512,activation='tanh',recurrent_activation='hard_sigmoid'))
model.add(layers.RepeatVector(600))
model.add(layers.GRU(512,activation='tanh', recurrent_activation='hard_sigmoid'))
model.add(layers.Dense(88, activation = 'sigmoid'))
return model
x_partial, x_val = x_train[:13000], x_train[13000:]
y_partial, y_val = y_train[:13000], y_train[13000:]
model = build_model()
model.compile(optimizer = 'adam',
loss = 'binary_crossentropy',
metrics = ['accuracy'])
history = model.fit(x_partial, y_partial, batch_size = 50, epochs = , validation_data= (x_val,y_val))
instead of learning normally my algorithm had stayed with constant accuracy throughout all of the epochs
Epoch 1/15
260/260 [==============================] - 998s 4s/step - loss: -0.1851 - accuracy: 0.0298 - val_loss: -8.8735 - val_accuracy: 0.0310
Epoch 2/15
260/260 [==============================] - 827s 3s/step - loss: -33.6520 - accuracy: 0.0382 - val_loss: -56.0122 - val_accuracy: 0.0310
Epoch 3/15
260/260 [==============================] - 844s 3s/step - loss: -78.6130 - accuracy: 0.0382 - val_loss: -98.2798 - val_accuracy: 0.0310
Epoch 4/15
260/260 [==============================] - 906s 3s/step - loss: -121.0963 - accuracy: 0.0382 - val_loss: -139.3440 - val_accuracy: 0.0310
Epoch 5/15
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I am training a CNN model with a 2D tensor of shape (400,22) as both input and output. I am using categorical_crossentropy both as loss and metric. However the loss/metrics values are very different.
My model is somewhat like this:
1. Using sample weights, and passing metrics with metrics= in model.compile.
# Imports
import tensorflow as tf
from tensorflow.keras.layers import *
from tensorflow.keras.optimizers import *
from tensorflow.keras.regularizers import *
from tensorflow.keras import *
import numpy as np
# Build the model
X_input = Input(shape=(400,22))
X = Conv1D(filters=32, kernel_size=2, activation='elu',
kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4),
padding='same')(X_input)
X = Dropout(0.2)(X)
X = Conv1D(filters=32, kernel_size=2, activation='elu',
kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4),
padding='same')(X)
X = Dropout(0.2)(X)
y = Conv1D(filters=22, kernel_size=1, activation='softmax',
kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4),
padding='same')(X)
model = Model(X_input, y, name='mymodel')
# Compile and train the model (with metrics=[])
model.compile(optimizer=Adam(1e-3),
loss=tf.keras.losses.categorical_crossentropy,
metrics=[tf.keras.losses.categorical_crossentropy])
Xtrain = np.random.rand(20,400,22)
ytrain = np.random.rand(20,400,22)
np.random.seed(0)
sample_weight = np.random.choice([0.01, 0.1, 1], size=20)
history = model.fit(x=Xtrain, y=ytrain, sample_weight=sample_weight, epochs=4)
Epoch 1/4
1/1 [==============================] - 0s 824us/step - loss: 10.2952 - categorical_crossentropy: 34.9296
Epoch 2/4
1/1 [==============================] - 0s 785us/step - loss: 10.2538 - categorical_crossentropy: 34.7858
Epoch 3/4
1/1 [==============================] - 0s 772us/step - loss: 10.2181 - categorical_crossentropy: 34.6719
Epoch 4/4
1/1 [==============================] - 0s 766us/step - loss: 10.1903 - categorical_crossentropy: 34.5797
From the results, it is evident that Keras is not using sample weights in the calculation of metrics, hence it is larger than the loss. If we change the sample weights to ones, we get the following:
2. Sample weights = ones, passing metrics with metrics= in `model.compile.
# Compile and train the model
model.compile(optimizer=Adam(1e-3),
loss=tf.keras.losses.categorical_crossentropy,
metrics=[tf.keras.losses.categorical_crossentropy])
Xtrain = np.random.rand(20,400,22)
ytrain = np.random.rand(20,400,22)
np.random.seed(0)
sample_weight = np.ones((20,))
history = model.fit(x=Xtrain, y=ytrain, sample_weight=sample_weight, epochs=4)
Epoch 1/4
1/1 [==============================] - 0s 789us/step - loss: 35.2659 - categorical_crossentropy: 35.2573
Epoch 2/4
1/1 [==============================] - 0s 792us/step - loss: 35.0647 - categorical_crossentropy: 35.0562
Epoch 3/4
1/1 [==============================] - 0s 778us/step - loss: 34.9301 - categorical_crossentropy: 34.9216
Epoch 4/4
1/1 [==============================] - 0s 736us/step - loss: 34.8076 - categorical_crossentropy: 34.7991
Now the metrics and loss are quite close with sample weights of ones. I understand that the loss is slightly larger than metrics due to the effects of dropout, regularization, and the fact that the metric is computed at the end of each epoch, whereas the loss is the average over the batches in the training.
How can I get the metrics to include the sample weights??
3. UPDATED: using sample weights, and passing metrics with weighted_metrics= in model.compile.
It was suggested that I used weighted_metrics=[...] instead of metrics=[...] in model.compile. However, Keras still does not include the sample weights in the evaluation of the metrics.
# Compile and train the model
model.compile(optimizer=Adam(1e-3),
loss=tf.keras.losses.categorical_crossentropy,
weighted_metrics=[tf.keras.losses.categorical_crossentropy])
Xtrain = np.random.rand(20,400,22)
ytrain = np.random.rand(20,400,22)
np.random.seed(0)
sample_weight = np.random.choice([0.01, 0.1, 1], size=20)
history = model.fit(x=Xtrain, y=ytrain, sample_weight=sample_weight, epochs=4)
Epoch 1/4
1/1 [==============================] - 0s 764us/step - loss: 10.2581 - categorical_crossentropy: 34.9224
Epoch 2/4
1/1 [==============================] - 0s 739us/step - loss: 10.2251 - categorical_crossentropy: 34.8100
Epoch 3/4
1/1 [==============================] - 0s 755us/step - loss: 10.1854 - categorical_crossentropy: 34.6747
Epoch 4/4
1/1 [==============================] - 0s 746us/step - loss: 10.1631 - categorical_crossentropy: 34.5990
What can be done to ensure that the sample weights are evaluated in the metrics?
Keras does not automatically include sample weights in the evaluation of metrics. That's why there is a huge difference between the loss and the metrics.
If you'll like to include sample weights when evaluating metrics, pass them as weighted_metrics rather than metrics.
model.compile(optimizer=Adam(1e-3),
loss=tf.keras.losses.categorical_crossentropy,
weighted_metrics=[tf.keras.losses.categorical_crossentropy]))
First of all, categorical cross-entropy is usually not used as a metric. Secondly, you are doing some type of seq2seq task, I hope you design the model with that intention.
Finally, in your setup, using sample_weight only works on the loss, it has no effect on the metrics or validation. There are other small bugs in your code too. Here is the fixed working code:
ref: TF 2.3.0 training keras model using tf dataset with sample weights does not apply to metrics (why sample_weight only works on loss)
import tensorflow as tf
from tensorflow.keras.layers import *
from tensorflow.keras.optimizers import *
from tensorflow.keras import *
import numpy as np
X_input = Input(shape=(400,22))
X = Conv1D(filters=32, kernel_size=2, activation='elu', kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4), padding='same')(X_input)
X = Dropout(0.2)(X)
X = Conv1D(filters=32, kernel_size=2, activation='elu', kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4), padding='same')(X)
X = Dropout(0.2)(X)
y = Conv1D(filters=22, kernel_size=1, activation='softmax', kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4), padding='same')(X)
model = Model(X_input, y, name='mymodel')
model.compile(optimizer=Adam(1e-3), loss=tf.keras.losses.categorical_crossentropy,
metrics=[tf.keras.losses.categorical_crossentropy])
Xtrain = np.random.rand(10,400,22)
ytrain = np.random.rand(10,400,22)
history = model.fit(Xtrain, ytrain, sample_weight=np.ones(10), epochs=10)
Epoch 1/10
1/1 [==============================] - 1s 719ms/step - loss: 35.4521 - categorical_crossentropy: 35.4437
Epoch 2/10
1/1 [==============================] - 0s 20ms/step - loss: 35.5138 - categorical_crossentropy: 35.5054
Epoch 3/10
1/1 [==============================] - 0s 19ms/step - loss: 35.5984 - categorical_crossentropy: 35.5900
Epoch 4/10
1/1 [==============================] - 0s 19ms/step - loss: 35.6617 - categorical_crossentropy: 35.6533
Epoch 5/10
1/1 [==============================] - 0s 19ms/step - loss: 35.7807 - categorical_crossentropy: 35.7723
Epoch 6/10
1/1 [==============================] - 0s 19ms/step - loss: 35.9045 - categorical_crossentropy: 35.8961
Epoch 7/10
1/1 [==============================] - 0s 18ms/step - loss: 36.0590 - categorical_crossentropy: 36.0505
Epoch 8/10
1/1 [==============================] - 0s 19ms/step - loss: 36.2040 - categorical_crossentropy: 36.1956
Epoch 9/10
1/1 [==============================] - 0s 18ms/step - loss: 36.4169 - categorical_crossentropy: 36.4084
Epoch 10/10
1/1 [==============================] - 0s 32ms/step - loss: 36.6622 - categorical_crossentropy: 36.6538
Here, if you use no sample_weight or 1 for each sample, you will get close/similar categorical cross-entropy.
Use weighted_metrics according to docs.
I am trying to build an image classifier that differentiates images into pumps, Turbines, and PCB classes. I am using transfer learning from Inception V3.
Below is my code to initialize InceptionV3
import os
from tensorflow.keras import layers
from tensorflow.keras import Model
!wget --no-check-certificate \
https://storage.googleapis.com/mledu-datasets/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5 \
-O /tmp/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5
from tensorflow.keras.applications.inception_v3 import InceptionV3
local_weights_file = '/tmp/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5'
pre_trained_model = InceptionV3(input_shape = (150, 150, 3),
include_top = False,
weights = None)
pre_trained_model.load_weights(local_weights_file)
for layer in pre_trained_model.layers:
layer.trainable = False
# pre_trained_model.summary()
last_layer = pre_trained_model.get_layer('mixed7')
print('last layer output shape: ', last_layer.output_shape)
last_output = last_layer.output
Next I connect my DNN to the pre-trained model:
from tensorflow.keras.optimizers import RMSprop
# Flatten the output layer to 1 dimension
x = layers.Flatten()(last_output)
# Add a fully connected layer with 1,024 hidden units and ReLU activation
x = layers.Dense(1024, activation='relu')(x)
# Add a dropout rate of 0.2
x = layers.Dropout(0.2)(x)
x = layers.Dense (3, activation='softmax')(x)
model = Model( pre_trained_model.input, x)
model.compile(optimizer = RMSprop(lr=0.0001),
loss = 'categorical_crossentropy',
metrics = ['accuracy'])
I feed in my images using ImageDataGenerator and train the model as below:
history = model.fit(
train_generator,
validation_data = validation_generator,
steps_per_epoch = 100,
epochs = 20,
validation_steps = 50,
verbose = 2)
However, the validation accuracy is not printed/generated after the first epoch:
Epoch 1/20
/usr/local/lib/python3.6/dist-packages/PIL/TiffImagePlugin.py:788: UserWarning: Corrupt EXIF data. Expecting to read 4 bytes but only got 0.
warnings.warn(str(msg))
/usr/local/lib/python3.6/dist-packages/PIL/Image.py:932: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
"Palette images with Transparency expressed in bytes should be "
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 50 batches). You may need to use the repeat() function when building your dataset.
100/100 - 43s - loss: 0.1186 - accuracy: 0.9620 - val_loss: 11.7513 - val_accuracy: 0.3267
Epoch 2/20
100/100 - 41s - loss: 0.1299 - accuracy: 0.9630
Epoch 3/20
100/100 - 39s - loss: 0.0688 - accuracy: 0.9840
Epoch 4/20
100/100 - 39s - loss: 0.0826 - accuracy: 0.9785
Epoch 5/20
100/100 - 39s - loss: 0.0909 - accuracy: 0.9810
Epoch 6/20
100/100 - 39s - loss: 0.0523 - accuracy: 0.9845
Epoch 7/20
100/100 - 38s - loss: 0.0976 - accuracy: 0.9835
Epoch 8/20
100/100 - 39s - loss: 0.0802 - accuracy: 0.9795
Epoch 9/20
100/100 - 39s - loss: 0.0612 - accuracy: 0.9860
Epoch 10/20
100/100 - 40s - loss: 0.0729 - accuracy: 0.9825
Epoch 11/20
100/100 - 39s - loss: 0.0601 - accuracy: 0.9870
Epoch 12/20
100/100 - 39s - loss: 0.0976 - accuracy: 0.9840
Epoch 13/20
100/100 - 39s - loss: 0.0591 - accuracy: 0.9815
Epoch 14/20
I am not understanding as to what is stopping the validation accuracy from being printed/generated. I get an error if the plot a graph on accuracy vs validation accuracy with a message as:
ValueError: x and y must have same first dimension, but have shapes (20,) and (1,)
what am I missing here?
It worked finally, posting my changes here in case if anybody faces issues like these.
So I changed the "weights" parameter in InceptionV3 from None to 'imagenet' and calculated my steps per epoch and validations steps as follows:
steps_per_epoch = np.ceil(no_of_training_images/batch_size)
validation_steps = np.ceil(no_of validation_images/batch_size)
As you see WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least ``steps_per_epoch * epochs`` batches (in this case, 50 batches). You may need to use the repeat() function when building your dataset.
To make sure that you have "at least steps_per_epoch * epochs batches", set the steps_per_epoch to:
steps_per_epoch = X_train.shape[0]//batch_size
I'm trying to make the most basic of basic neural networks to get familiar with functional API in Tensorflow 2.x.
Basically what I'm trying to do is the following with my simplified iris dataset (i.e. setosa or not)
Use the 4 features as input
Dense layer of 3
Sigmoid activation function
Dense layer of 2 (one for each class)
Softmax activation
Binary cross entropy / log-loss as my loss function
However, I can't figure out how to control one key aspect of the model. That is, how can I ensure that each feature from my input layer contributes to only one neuron in my subsequent dense layer? Also, how can I allow a feature to contribute to more than one neuron?
This isn't clear to me from the documentation.
# Load data
from sklearn.datasets import load_iris
import pandas as pd
iris = load_iris()
X, y = load_iris(return_X_y=True, as_frame=True)
X = X.astype("float32")
X.index = X.index.map(lambda i: "iris_{}".format(i))
X.columns = X.columns.map(lambda j: j.split(" (")[0].replace(" ","_"))
y.index = X.index
y = y.map(lambda i:iris.target_names[i])
y_simplified = y.map(lambda i: {True:1, False:0}[i == "setosa"])
y_simplified = pd.get_dummies(y_simplified, columns=["setosa", "not_setosa"])
# Traing test split
from sklearn.model_selection import train_test_split
seed=0
X_train,X_test, y_train,y_test= train_test_split(X,y_simplified, test_size=0.3, random_state=seed)
# Simple neural network
import tensorflow as tf
tf.random.set_seed(seed)
# Input[4 features] -> Dense layer of 3 neurons -> Activation function -> Dense layer of 2 (one per class) -> Softmax
inputs = tf.keras.Input(shape=(4))
x = tf.keras.layers.Dense(3)(inputs)
x = tf.keras.layers.Activation(tf.nn.sigmoid)(x)
x = tf.keras.layers.Dense(2)(x)
outputs = tf.keras.layers.Activation(tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs, name="simple_binary_iris")
model.compile(loss="binary_crossentropy", metrics=["accuracy"] )
model.summary()
history = model.fit(X_train, y_train, batch_size=64, epochs=10, validation_split=0.2)
test_scores = model.evaluate(X_test, y_test)
print("Test loss:", test_scores[0])
print("Test accuracy:", test_scores[1])
Results:
Model: "simple_binary_iris"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_44 (InputLayer) [(None, 4)] 0
_________________________________________________________________
dense_96 (Dense) (None, 3) 15
_________________________________________________________________
activation_70 (Activation) (None, 3) 0
_________________________________________________________________
dense_97 (Dense) (None, 2) 8
_________________________________________________________________
activation_71 (Activation) (None, 2) 0
=================================================================
Total params: 23
Trainable params: 23
Non-trainable params: 0
_________________________________________________________________
Epoch 1/10
2/2 [==============================] - 0s 40ms/step - loss: 0.6344 - accuracy: 0.6667 - val_loss: 0.6107 - val_accuracy: 0.7143
Epoch 2/10
2/2 [==============================] - 0s 6ms/step - loss: 0.6302 - accuracy: 0.6667 - val_loss: 0.6083 - val_accuracy: 0.7143
Epoch 3/10
2/2 [==============================] - 0s 7ms/step - loss: 0.6278 - accuracy: 0.6667 - val_loss: 0.6056 - val_accuracy: 0.7143
Epoch 4/10
2/2 [==============================] - 0s 7ms/step - loss: 0.6257 - accuracy: 0.6667 - val_loss: 0.6038 - val_accuracy: 0.7143
Epoch 5/10
2/2 [==============================] - 0s 7ms/step - loss: 0.6239 - accuracy: 0.6667 - val_loss: 0.6014 - val_accuracy: 0.7143
Epoch 6/10
2/2 [==============================] - 0s 7ms/step - loss: 0.6223 - accuracy: 0.6667 - val_loss: 0.6002 - val_accuracy: 0.7143
Epoch 7/10
2/2 [==============================] - 0s 7ms/step - loss: 0.6209 - accuracy: 0.6667 - val_loss: 0.5989 - val_accuracy: 0.7143
Epoch 8/10
2/2 [==============================] - 0s 7ms/step - loss: 0.6195 - accuracy: 0.6667 - val_loss: 0.5967 - val_accuracy: 0.7143
Epoch 9/10
2/2 [==============================] - 0s 7ms/step - loss: 0.6179 - accuracy: 0.6667 - val_loss: 0.5953 - val_accuracy: 0.7143
Epoch 10/10
2/2 [==============================] - 0s 7ms/step - loss: 0.6166 - accuracy: 0.6667 - val_loss: 0.5935 - val_accuracy: 0.7143
2/2 [==============================] - 0s 607us/step - loss: 0.6261 - accuracy: 0.6444
Test loss: 0.6261375546455383
Test accuracy: 0.644444465637207
how can I ensure that each feature from my input layer contributes to
only one neuron in my subsequent dense layer?
Have one input layer per feature and feed each input layer to a separate dense layer. Later you can concatenate the output of all the dense layers and proceed.
NOTE: One neuron can take any size input (in this case the input size is 1 as you want one feature to be used by the neuron) and the output size if always 1. A Dense layer with with n units will have n neurons and and so will have output size of n.
Working Sample
import tensorflow as tf
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
# Model architecutre
x1 = tf.keras.Input(shape=(1,))
x2 = tf.keras.Input(shape=(1,))
x3 = tf.keras.Input(shape=(1,))
x4 = tf.keras.Input(shape=(1,))
x1_ = tf.keras.layers.Dense(3, activation=tf.nn.relu)(x1)
x2_ = tf.keras.layers.Dense(3, activation=tf.nn.relu)(x2)
x3_ = tf.keras.layers.Dense(3, activation=tf.nn.relu)(x3)
x4_ = tf.keras.layers.Dense(3, activation=tf.nn.relu)(x4)
merged = tf.keras.layers.concatenate([x1_, x2_, x3_, x4_])
merged = tf.keras.layers.Dense(16, activation=tf.nn.relu)(merged)
outputs = tf.keras.layers.Dense(3, activation=tf.nn.softmax)(merged)
model = tf.keras.Model(inputs=[x1,x2,x3,x4], outputs=outputs)
model.compile(loss="sparse_categorical_crossentropy", metrics=["accuracy"] )
# Load and prepare data
iris = load_iris()
X = iris.data
y = iris.target
X_train,X_test, y_train,y_test= train_test_split(X,y, test_size=0.3)
# Fit the model
model.fit([X_train[:,0],X_train[:,1],X_train[:,2],X_train[:,3]], y_train, batch_size=64, epochs=100, validation_split=0.25)
# Evaluate the model
test_scores = model.evaluate([X_test[:,0],X_test[:,1],X_test[:,2],X_test[:,3]], y_test)
print("Test loss:", test_scores[0])
print("Test accuracy:", test_scores[1])
Output:
Epoch 1/100
2/2 [==============================] - 0s 75ms/step - loss: 1.6446 - accuracy: 0.4359 - val_loss: 1.6809 - val_accuracy: 0.5185
Epoch 2/100
2/2 [==============================] - 0s 10ms/step - loss: 1.4151 - accuracy: 0.6154 - val_loss: 1.4886 - val_accuracy: 0.5556
Epoch 3/100
2/2 [==============================] - 0s 9ms/step - loss: 1.2725 - accuracy: 0.6795 - val_loss: 1.3813 - val_accuracy: 0.5556
Epoch 4/100
2/2 [==============================] - 0s 9ms/step - loss: 1.1829 - accuracy: 0.6795 - val_loss: 1.2779 - val_accuracy: 0.5926
Epoch 5/100
2/2 [==============================] - 0s 10ms/step - loss: 1.0994 - accuracy: 0.6795 - val_loss: 1.1846 - val_accuracy: 0.5926
Epoch 6/100
.................. [ Truncated ]
Epoch 100/100
2/2 [==============================] - 0s 2ms/step - loss: 0.4049 - accuracy: 0.9333
Test loss: 0.40491223335266113
Test accuracy: 0.9333333373069763
Pictorial representation of the above model architecture
Dense layers in Keras/TF are fully connected layers. For example, when you use a Dense layer as follows
inputs = tf.keras.Input(shape=(4))
x = tf.keras.layers.Dense(3)(inputs)
all the 4 connected input neurons are connected to all the 3 output neurons.
There isn't any predefined layer in Keras/TF to specify how to connect input and output neurons. However, Keras/TF is very flexible in that it allows you to define your custom layers easily.
Borrowing the idea from this answer, you could define a CustomConnected layer as follows:
class CustomConnected(tf.keras.layers.Dense):
def __init__(self, units, connections, **kwargs):
self.connections = connections
super(CustomConnected, self).__init__(units, **kwargs)
def call(self, inputs):
self.kernel = self.kernel * self.connections
return super(CustomConnected, self).call(inputs)
Using this layer, you can then specify the connections between two layers through the connections argument. For example:
inputs = tf.keras.Input(shape=(4))
connections = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 0, 1]])
x = CustomConnected(3, connections)(inputs)
Here, the 1st, 2nd, and 3rd input neurons are connected to the 1st, 2nd, and 3rd output neurons, respectively. Additionally, the 4th input neuron is connected to the 3rd output neuron.
UPDATE: As discussed in the comments section, an adaptive approach (e.g. by using only the maximum weight for each output neuron) is also possible but not recommended. You could implement this via the following layer:
class CustomSparse(tf.keras.layers.Dense):
def __init__(self, units, **kwargs):
super(CustomSparse, self).__init__(units, **kwargs)
def call(self, inputs):
nb_in, nb_out = self.kernel.shape
argmax = tf.argmax(self.kernel, axis=0) # Shape=(nb_out,)
argmax_onehot = tf.transpose(tf.one_hot(argmax, depth=nb_in)) # Shape=(nb_in, nb_out)
kernel_max = self.kernel * argmax_onehot
# tf.print(kernel_max) # Uncomment this line to print the weights
out = tf.matmul(inputs, kernel_max)
if self.bias is not None:
out += self.bias
if self.activation is not None:
out = self.activation(out)
return out
The main issue of this approach is that you cannot propagate gradients through the argmax operation required to select the maximum weight. As a result, the network will only "switch input neurons" when the selected weight is no longer the maximum weight.
I am trying to predict the growth rate of a user using LSTM and Adam algo. But the predictions which I am getting from code is way far then accurate values. I am new to ML and just trying to learn how things are measured in ML. That what does units basically do in the LSTM model. I am reading values from CSV and trying to find the Growth rate of a user based on the amount he collected in 2 years. But my Predictions seem to be giving inaccurate values. Can anyone tell me how Can I find the correct prediction in order to get the Growth rate of a user?
Here my code:
import pymysql
import pandas as pd
import numpy as np
import csv
from datetime import datetime
import time
import json
import matplotlib.pyplot as plt
import seaborn as sns
import pprint
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.utils.np_utils import to_categorical
from keras.layers import Input
import os
os.environ['KERAS_BACKEND']='tensorflow'
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
from keras.layers.recurrent import LSTM
from matplotlib import style
from keras.layers import Activation, Dense, Dropout
df = pd.read_csv("trakop.csv")
print("="*50)
print("First Five Rows ","\n")
print(df.head(2),"\n")
dataset = df
dataset["Month"] = pd.to_datetime(df["timestamp"]).dt.month
dataset["Year"] = pd.to_datetime(df["timestamp"]).dt.year
dataset["Date"] = pd.to_datetime(df["timestamp"]).dt.date
dataset["Time"] = pd.to_datetime(df["timestamp"]).dt.time
dataset["Week"] = pd.to_datetime(df["timestamp"]).dt.week
dataset["Day"] = pd.to_datetime(df["timestamp"]).dt.day_name()
dataset["Hour"] = pd.to_datetime(df["timestamp"]).dt.hour
dataset = df.set_index("timestamp")
dataset.index = pd.to_datetime(dataset.index)
dataset.head(1)
print(df.Year.unique(),"\n")
print("Total Number of Unique Year", df.Year.nunique(), "\n")
NewDataSet = dataset.resample('D').mean()
# print(NewDataSet)
print("Old Dataset ",dataset.shape )
print("New Dataset ",NewDataSet.shape )
excludedValue = 5
TestData = NewDataSet.tail(10)
Training_Set = NewDataSet.iloc[:,0:1]
Training_Set = Training_Set[:-excludedValue]
print("Training Set Shape ", Training_Set.shape)
print("Test Set Shape ", TestData.shape)
Training_Set = Training_Set.values
sc = MinMaxScaler(feature_range=(0, 1))
Train = sc.fit_transform(Training_Set)
X_Train = []
Y_Train = []
# Range should be fromm 60 Values to END
for i in range(excludedValue, Train.shape[0]):
# X_Train 0-9
X_Train.append(Train[i- excludedValue:i])
# Y Would be 10 th Value based on past 10 Values
Y_Train.append(Train[i])
# Convert into Numpy Array
X_Train = np.array(X_Train)
Y_Train = np.array(Y_Train)
print(X_Train.shape)
print(Y_Train.shape)
X_Train = np.reshape(X_Train, newshape=(X_Train.shape[0], X_Train.shape[1], 1))
X_Train.shape
regressor = Sequential()
# Adding the first LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 1, return_sequences = True, input_shape = (X_Train.shape[1], 1)))
regressor.add(Dropout(0.4))
# Adding a second LSTM layer and some Dropout regularisation
regressor.add(LSTM(units=1, return_sequences = True))
regressor.add(Dropout(0.4))
# Adding a third LSTM layer and some Dropout regularisation
regressor.add(LSTM(units=1, return_sequences = True))
regressor.add(Dropout(0.4))
# Adding a fourth LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 1))
regressor.add(Dropout(0.4))
# Adding the output layer
regressor.add(Dense(units = 1))
# Compiling the RNN
regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error', metrics=['acc'])
regressor.fit(X_Train, Y_Train, epochs = 30, batch_size = 12,verbose=2)
Df_Total = pd.concat((NewDataSet[["amount"]], TestData[["amount"]]), axis=0)
Df_Total.shape
inputs = Df_Total[len(Df_Total) - len(TestData) - excludedValue:].values
# We need to Reshape
inputs = inputs.reshape(-1,1)
# Normalize the Dataset
inputs = sc.transform(inputs)
X_test = []
for i in range(excludedValue, inputs.shape[0]):
X_test.append(inputs[i- excludedValue:i])
# Convert into Numpy Array
X_test = np.array(X_test)
# Reshape before Passing to Network
X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))
# Pass to Model
predicted_raise = regressor.predict(X_test)
# Do inverse Transformation to get Values
predicted_raise = sc.inverse_transform(predicted_raise)
Predicted_Amount = predicted_raise
dates = TestData.index.to_list()
True_Amount = TestData["amount"].to_list()
Predicted_Amount = predicted_raise
dates = TestData.index.to_list()
growth_rate= (True_Amount-Predicted_Amount)/True_Amount*100
Machine_Df = pd.DataFrame(data={
"Date":dates,
"TrueAmount": True_Amount,
"PredictedAmount":[x[0] for x in Predicted_Amount ],
"Growthrate": [x[0] for x in growth_rate]
})
print(Machine_Df)
fig = plt.figure()
ax1= fig.add_subplot(111)
x = dates
y = True_Amount
y1 = Predicted_Amount
plt.plot(x,y, color="green")
plt.plot(x,y1, color="red")
# beautify the x-labels
plt.gcf().autofmt_xdate()
plt.xlabel('Dates')
plt.ylabel("Amount")
plt.title("Machine Learned the Pattern Predicting Future Values ")
plt.legend()
Here is what I am getting in my output:
('First Five Rows ', '\n')
( timestamp amount
0 2019-09-08 06:30:23 38.0
1 2019-09-08 06:36:48 19.0, '\n')
(array([2019, 2020]), '\n')
('Total Number of Unique Year', 2, '\n')
('Old Dataset ', (12492, 8))
('New Dataset ', (129, 5))
('Training Set Shape ', (124, 1))
('Test Set Shape ', (10, 5))
(119, 5, 1)
(119, 1)
Epoch 1/30
- 15s - loss: 0.0177 - acc: 0.0084
Epoch 2/30
- 1s - loss: 0.0165 - acc: 0.0084
Epoch 3/30
- 1s - loss: 0.0153 - acc: 0.0084
Epoch 4/30
- 1s - loss: 0.0167 - acc: 0.0084
Epoch 5/30
- 1s - loss: 0.0157 - acc: 0.0084
Epoch 6/30
- 1s - loss: 0.0158 - acc: 0.0084
Epoch 7/30
- 1s - loss: 0.0151 - acc: 0.0084
Epoch 8/30
- 1s - loss: 0.0153 - acc: 0.0084
Epoch 9/30
- 1s - loss: 0.0150 - acc: 0.0084
Epoch 10/30
- 1s - loss: 0.0160 - acc: 0.0084
Epoch 11/30
- 1s - loss: 0.0158 - acc: 0.0084
Epoch 12/30
- 1s - loss: 0.0155 - acc: 0.0084
Epoch 13/30
- 1s - loss: 0.0157 - acc: 0.0084
Epoch 14/30
- 1s - loss: 0.0155 - acc: 0.0084
Epoch 15/30
- 1s - loss: 0.0152 - acc: 0.0084
Epoch 16/30
- 1s - loss: 0.0153 - acc: 0.0084
Epoch 17/30
- 1s - loss: 0.0150 - acc: 0.0084
Epoch 18/30
- 1s - loss: 0.0151 - acc: 0.0084
Epoch 19/30
- 1s - loss: 0.0150 - acc: 0.0084
Epoch 20/30
- 1s - loss: 0.0151 - acc: 0.0084
Epoch 21/30
- 1s - loss: 0.0153 - acc: 0.0084
Epoch 22/30
- 1s - loss: 0.0151 - acc: 0.0084
Epoch 23/30
- 1s - loss: 0.0150 - acc: 0.0084
Epoch 24/30
- 1s - loss: 0.0153 - acc: 0.0084
Epoch 25/30
- 1s - loss: 0.0152 - acc: 0.0084
Epoch 26/30
- 1s - loss: 0.0151 - acc: 0.0084
Epoch 27/30
- 1s - loss: 0.0152 - acc: 0.0084
Epoch 28/30
- 1s - loss: 0.0151 - acc: 0.0084
Epoch 29/30
- 1s - loss: 0.0151 - acc: 0.0084
Epoch 30/30
- 1s - loss: 0.0151 - acc: 0.0084
Date Growthrate PredictedAmount TrueAmount
0 2020-01-05 1.695584 122.266731 124.375625
1 2020-01-06 1.691683 122.271584 98.166667
2 2020-01-07 1.682077 122.283531 120.892473
3 2020-01-08 1.690008 122.273666 84.863636
4 2020-01-09 1.694407 122.268196 94.673077
5 2020-01-10 1.706436 122.253235 99.140341
6 2020-01-11 1.700952 122.260056 124.580882
7 2020-01-12 1.701755 122.259056 56.390071
8 2020-01-13 1.696290 122.265854 78.746951
9 2020-01-14 1.698001 122.263725 49.423529
[100 rows x 3 columns]
Screenshot of Graph:
The CSV I am using:
https://drive.google.com/file/d/1nKHNqh7fJJJVvb2Qy-DxAO7c7HwNpEI0/view?usp=sharing
Any help would be greatly appreciated!!!
I have worked on your code. First of all please reduce the batch size because the size of dataset is small and change the optimizer from "adam" to "rmsprop". Because adam uses constant learning rate, that's why you are receiving the constant values in the prediction. I have also increased the dropout to 0.4.
For calculating the growth rate, I have used the formula,
growth rate= (True Amount- Predicted Amount)/True Amount *100
this formula, gives you the percentage score of growth in the predicted amount and true amount.
For full code, please follow the GitHub link
https://github.com/rohitnarain24/Optimizing-LSTM-model/blob/master/optimized%20lstm.txt