I've built a machine learning model with tensorflow and keras and when I test it with 'evaluate' I get pretty good scores. But then when I actually want to use the model I use 'predict' and then it predicts a lot wrong.
Here I have to add the information that for the evaluation I used some data I didn't use for training and when I use predict it even gets data wrong which I used for training.
So my question is, does 'evaluate' do something fundamentally different than using 'predict' and calculating the score of the predictions.
Sorry if this question is kinda stupid but it makes me mad that I get such good scores but then my model doesn't work.
Thank you in advance :)
PS: I think my whole code would be kinda overkill but I wanted to show you how i evaluate maybe I'm doing a mistake here:
At this point I have my test data in a dictionary called test_set where the keys are the categories (output of the model) and the values are the input for the model.
test_label = {}
for key in test_set:
test_set[key] = np.array(test_set[key]).reshape(-1,255)
test_label[key] = []
for entry in test_set[key]:
test_label[key].append(key)
test_label[key] = np.array(test_label[key]).reshape(-1,18)
for key in test_set:
model.evaluate(np.array(test_set[key]),test_label[key])
And this is my output:
1000/1000 [==============================] - 0s 124us/sample - loss: 0.0088
- binary_accuracy: 0.9970
1000/1000 [==============================] - 0s 72us/sample - loss: 0.0556 -
binary_accuracy: 0.9443
1000/1000 [==============================] - 0s 79us/sample - loss: 0.0556 -
binary_accuracy: 0.9444
1000/1000 [==============================] - ETA: 0s - loss: 0.0543 -
binary_accuracy: 0.912 - 0s 73us/sample - loss: 0.0543 - binary_accuracy:
0.9131
1000/1000 [==============================] - 0s 103us/sample - loss: 0.0045
- binary_accuracy: 0.9963
1000/1000 [==============================] - 0s 70us/sample - loss:
9.6615e-06 - binary_accuracy: 1.0000
1000/1000 [==============================] - 0s 79us/sample - loss: 0.0028 -
binary_accuracy: 0.9970
1000/1000 [==============================] - 0s 99us/sample - loss: 1.9372e-
05 - binary_accuracy: 1.0000
1000/1000 [==============================] - 0s 121us/sample - loss:
9.1933e-04 - binary_accuracy: 0.9993
1000/1000 [==============================] - 0s 82us/sample - loss: 0.0169 -
binary_accuracy: 0.9609
1000/1000 [==============================] - 0s 102us/sample - loss: 0.0098
- binary_accuracy: 0.9998
1000/1000 [==============================] - 0s 103us/sample - loss: 0.0561
- binary_accuracy: 0.9443
1000/1000 [==============================] - 0s 102us/sample - loss: 0.0495
- binary_accuracy: 0.9564
1000/1000 [==============================] - 0s 76us/sample - loss: 0.0590 -
binary_accuracy: 0.9444
1000/1000 [==============================] - 0s 83us/sample - loss: 0.0327 -
binary_accuracy: 0.9534
1000/1000 [==============================] - 0s 80us/sample - loss: 0.0303 -
binary_accuracy: 0.9561
1000/1000 [==============================] - 0s 86us/sample - loss: 0.0345 -
binary_accuracy: 0.9421
So i get pretty good scores for every category but then when I iterate over some data and use predict for every entry it gets in some categories every entry wrong.
Would be so glad if anyone knows something helpful.
Related
So I am trying to create an LSTM that can predict the next time step of a double pendulum. The data that I am trying to train with is a (2001, 4) numpy array. (i.e. the first 5 rows will look like:
array([[ 1.04719755, 0. , 1.04719755, 0. ],
[ 1.03659984, -0.42301933, 1.04717544, -0.00178865],
[ 1.00508218, -0.83475539, 1.04682248, -0.01551541],
[ 0.95354768, -1.22094052, 1.04514269, -0.05838011],
[ 0.88372305, -1.56345555, 1.04009056, -0.15443162]])
where each row is a unique representation of the state of the double pendulum.)
So I wanted to created an LSTM that could learn to predict the next state given the current one.
Here was my code for it so far (full_sol is the (2001, 4) matrix:
import numpy as np
from tensorflow import keras
import tensorflow as tf
# full_sol = np.random.rand(2001, 4)
full_sol = full_sol.reshape((full_sol.shape[0], 1, full_sol.shape[1]))
model = keras.Sequential()
model.add(keras.layers.LSTM(100, input_shape=(None, 4), return_sequences=True, dropout=0.2))
model.add(keras.layers.TimeDistributed(keras.layers.Dense(4, activation=tf.keras.layers.LeakyReLU(
alpha=0.3))))
model.compile(loss="mean_squared_error", optimizer="adam", metrics="accuracy")
history = model.fit(full_sol[:-1,:,:], full_sol[1:,:,:], epochs=20)
Then when I train, I get the following results:
Epoch 1/20
63/63 [==============================] - 3s 4ms/step - loss: 1.7181 - accuracy: 0.4200
Epoch 2/20
63/63 [==============================] - 0s 4ms/step - loss: 1.0481 - accuracy: 0.5155
Epoch 3/20
63/63 [==============================] - 0s 5ms/step - loss: 0.7584 - accuracy: 0.5715
Epoch 4/20
63/63 [==============================] - 0s 5ms/step - loss: 0.5134 - accuracy: 0.6420
Epoch 5/20
63/63 [==============================] - 0s 5ms/step - loss: 0.3944 - accuracy: 0.7260
Epoch 6/20
63/63 [==============================] - 0s 5ms/step - loss: 0.3378 - accuracy: 0.7605
Epoch 7/20
63/63 [==============================] - 0s 5ms/step - loss: 0.3549 - accuracy: 0.7825
Epoch 8/20
63/63 [==============================] - 0s 4ms/step - loss: 0.3528 - accuracy: 0.7995
Epoch 9/20
63/63 [==============================] - 0s 5ms/step - loss: 0.3285 - accuracy: 0.8020
Epoch 10/20
63/63 [==============================] - 0s 5ms/step - loss: 0.2874 - accuracy: 0.8030
Epoch 11/20
63/63 [==============================] - 0s 4ms/step - loss: 0.3072 - accuracy: 0.8135
Epoch 12/20
63/63 [==============================] - 0s 4ms/step - loss: 0.3075 - accuracy: 0.8035
Epoch 13/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2942 - accuracy: 0.8030
Epoch 14/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2637 - accuracy: 0.8170
Epoch 15/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2675 - accuracy: 0.8150
Epoch 16/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2644 - accuracy: 0.8085
Epoch 17/20
63/63 [==============================] - 0s 5ms/step - loss: 0.2479 - accuracy: 0.8200
Epoch 18/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2475 - accuracy: 0.8215
Epoch 19/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2243 - accuracy: 0.8340
Epoch 20/20
63/63 [==============================] - 0s 5ms/step - loss: 0.2430 - accuracy: 0.8240
So, quite high accuracy. But when I test it on the training set, the predictions aren't very good.
E.g. when I predict the first value:
model.predict(tf.expand_dims(full_sol[0], axis = 0))
I get array([[[ 1.0172144 , -0.3535697 , 1.1287913 , -0.23707283]]],dtype=float32)
Instead of array([[ 1.03659984, -0.42301933, 1.04717544, -0.00178865]]).
Where have I gone wrong?
I don't think you are doing anything wrong. What you are getting is still fairly close to the actual value. You can either change your choice of metric so it accurately represents the degree of error in your predictions, or you could try to increase the accuracy further.
I have implemented a complex neural network using Keras. The neural network trains normally and reaches up to 100% accuracy during training. However, during the evaluation, the results are very low and the output of the neural network is always 0.
First, I investigated the weights of the neural network and found some nan values. I replaced these values and retrained the neural network. After the training is finished, no nan values exist in the weights anymore. However, the evaluation results are still the same.
Out of desperation, I ran the following to train the network:
model.fit(dt[:200],labels[:200],epochs=100,batch_size=10,validation_data=(dt[:200],labels[:200]))
Despite all the nonsense in that code, I at least expected to receive identical values for the training and testing sets. But here is what I got:
Train on 200 samples, validate on 200 samples
Epoch 1/100
200/200 [==============================] - 21s 104ms/step - loss: 0.2230 - acc: 0.9650 - val_loss: 7.9712 - val_acc: 0.5000
Epoch 2/100
200/200 [==============================] - 21s 104ms/step - loss: 0.0302 - acc: 0.9850 - val_loss: 0.7086 - val_acc: 0.5000
Epoch 3/100
200/200 [==============================] - 21s 104ms/step - loss: 0.0353 - acc: 0.9800 - val_loss: 0.7096 - val_acc: 0.5000
Epoch 4/100
200/200 [==============================] - 21s 104ms/step - loss: 0.0349 - acc: 0.9900 - val_loss: 0.7088 - val_acc: 0.5000
Epoch 5/100
200/200 [==============================] - 21s 104ms/step - loss: 0.0298 - acc: 0.9900 - val_loss: 0.7097 - val_acc: 0.5000
Epoch 6/100
200/200 [==============================] - 21s 103ms/step - loss: 0.0457 - acc: 0.9800 - val_loss: 0.7065 - val_acc: 0.5000
Epoch 7/100
200/200 [==============================] - 21s 103ms/step - loss: 0.0060 - acc: 0.9950 - val_loss: 0.7080 - val_acc: 0.5000
Epoch 8/100
200/200 [==============================] - 20s 102ms/step - loss: 0.0337 - acc: 0.9900 - val_loss: 0.7045 - val_acc: 0.5000
Epoch 9/100
200/200 [==============================] - 21s 105ms/step - loss: 0.0048 - acc: 1.0000 - val_loss: 0.7049 - val_acc: 0.5000
Epoch 10/100
200/200 [==============================] - 21s 104ms/step - loss: 0.0020 - acc: 1.0000 - val_loss: 0.7059 - val_acc: 0.5000
The exact same sets are used for the training and evaluation but are returning different loss and accuracy values. Any ideas about what is going on!!!!?
I am using the following code to train a model in Keras:
model_A.fit(train_X, train_Y, epochs=20)
The code works fine and the outputs are like below:
Epoch 1/20
1800/1800 [==============================] - 0s 34us/step - loss: 0.2764 - acc: 0.9033
Epoch 2/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.2704 - acc: 0.9083
Epoch 3/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.2687 - acc: 0.9094
Epoch 4/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.2748 - acc: 0.9089
Epoch 5/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.2902 - acc: 0.8922
Epoch 6/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.2357 - acc: 0.9183
Epoch 7/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.2499 - acc: 0.9183
Epoch 8/20
1800/1800 [==============================] - 0s 33us/step - loss: 0.2286 - acc: 0.9228
Epoch 9/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.2325 - acc: 0.9194
Epoch 10/20
1800/1800 [==============================] - 0s 33us/step - loss: 0.2053 - acc: 0.9261
Epoch 11/20
1800/1800 [==============================] - 0s 33us/step - loss: 0.2256 - acc: 0.9161
Epoch 12/20
1800/1800 [==============================] - 0s 33us/step - loss: 0.2120 - acc: 0.9261
Epoch 13/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.2085 - acc: 0.9328
Epoch 14/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.1881 - acc: 0.9328
Epoch 15/20
1800/1800 [==============================] - 0s 31us/step - loss: 0.1835 - acc: 0.9344
Epoch 16/20
1800/1800 [==============================] - 0s 34us/step - loss: 0.1812 - acc: 0.9356
Epoch 17/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.1704 - acc: 0.9361
Epoch 18/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.1929 - acc: 0.9272
Epoch 19/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.1822 - acc: 0.9317
Epoch 20/20
1800/1800 [==============================] - 0s 32us/step - loss: 0.1713 - acc: 0.9417
I am wondering if there is a way to save the loss/accuracy values in an array, so I could plot them over epochs later.
The fit method returns a History object which contains information about the training process. For example:
# train the model
h = model.fit(...)
# loss values at the end of each epoch
h.history['loss']
# validation loss values per epoch (if you have used validation data)
h.history['val_loss']
# accuracy values at the end of each epoch (if you have used `acc` metric)
h.history['acc']
# validation accuracy values per epoch (if you have used `acc` metric and validation data)
h.history['val_acc']
# list of epochs number
h.epoch
Further, it's not necessary to store the History object in a variable (like h = model.fit(...)) because it could also be accessed using model.history.history (however, note that this history attribute would not be persisted when model is saved using model.save(...)).
I have a question about my NN model. I am using keras from python. My training consists of 1000 samples, each with 4320 features. There are 10 categories, and my Y contains numpy arrays of 10 elements with 0 on all the positions except one.
However, my NN doesn't learn from the first epoch and I probably have my model wrong, it's my first attempt of building a NN model and I must have got wrong a couple of things.
Epoch 1/150
1000/1000 [==============================] - 40s 40ms/step - loss: 6.7110 - acc: 0.5796
Epoch 2/150
1000/1000 [==============================] - 39s 39ms/step - loss: 6.7063 - acc: 0.5800
Epoch 3/150
1000/1000 [==============================] - 38s 38ms/step - loss: 6.7063 - acc: 0.5800
Epoch 4/150
1000/1000 [==============================] - 39s 39ms/step - loss: 6.7063 - acc: 0.5800
Epoch 5/150
1000/1000 [==============================] - 38s 38ms/step - loss: 6.7063 - acc: 0.5800
Epoch 6/150
1000/1000 [==============================] - 38s 38ms/step - loss: 6.7063 - acc: 0.5800
Epoch 7/150
1000/1000 [==============================] - 40s 40ms/step - loss: 6.7063 - acc: 0.5800
Epoch 8/150
1000/1000 [==============================] - 39s 39ms/step - loss: 6.7063 - acc: 0.5800
Epoch 9/150
1000/1000 [==============================] - 40s 40ms/step - loss: 6.7063 - acc: 0.5800
And this is part of my NN code:
model = Sequential()
model.add(Dense(4320, input_dim=4320, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(10, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X, Y, epochs=150, batch_size=10)
So, my X is a numpy array of length 1000 that contains other numpy arrays of 4320 elements. My Y is a numpy array of length 1000 that contains other numpy arrays of 10 elements (categories).
Am I doing something wrong or it just can't learn based on this training set? (On 1NN with manhattan distance I'm getting ~80% accuracy on this training set)
L.E.: After I've normalized the data, this is the output of my first 10 epochs:
Epoch 1/150
1000/1000 [==============================] - 41s 41ms/step - loss: 7.9834 - acc: 0.4360
Epoch 2/150
1000/1000 [==============================] - 41s 41ms/step - loss: 7.2943 - acc: 0.5080
Epoch 3/150
1000/1000 [==============================] - 39s 39ms/step - loss: 9.0326 - acc: 0.4070
Epoch 4/150
1000/1000 [==============================] - 39s 39ms/step - loss: 8.7106 - acc: 0.4320
Epoch 5/150
1000/1000 [==============================] - 40s 40ms/step - loss: 7.7547 - acc: 0.4900
Epoch 6/150
1000/1000 [==============================] - 44s 44ms/step - loss: 7.2591 - acc: 0.5270
Epoch 7/150
1000/1000 [==============================] - 42s 42ms/step - loss: 8.5002 - acc: 0.4560
Epoch 8/150
1000/1000 [==============================] - 41s 41ms/step - loss: 9.9525 - acc: 0.3720
Epoch 9/150
1000/1000 [==============================] - 40s 40ms/step - loss: 9.7160 - acc: 0.3920
Epoch 10/150
1000/1000 [==============================] - 39s 39ms/step - loss: 9.3523 - acc: 0.4140
Looks like it starts fluctuating so that seems to be good
It seems like your categories, classes are mutually exclusive since your target arrays are one-hot encoded (ie you never have to predict 2 classes at the same time). In that case, you should use softmax on your last layer to produce a distribution and train using categorical_crossentropy. If fact you can just set your targets as Y = [2,4,0,1] as your category indices and train with sparse_categorical_crossentropy which will save you the time of creating a 2 array of shape (samples, 10).
It seems like you have a lot of features, most likely the performance of your network will depend on how you pre-process your data. For continuous inputs, it's wise to normalise it and for discrete input encode it as one-hot to help the learning.
I am trying to run predictions on timeseries data using Keras/Theanos backend in python where I am taking factors from past few days to predict for next day.I am able to generate predictions using other algorithms like xgboost but wanted to try ANN but got stuck with index out of bounds error during prediction step
Part of code is something like this:
clfnn = Sequential()
clfnn.add(Dense(32, input_dim=9,init='uniform',activation='tanh'))
clfnn.add(Dense(9,init='uniform',activation='tanh'))
clfnn.add(Dense(1,activation='tanh'))
clfnn.compile(loss='mse', optimizer=sgd, metrics=['accuracy'])
imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(testsample[factorsnew])
testsample[factorsnew]=imp.transform(testsample[factorsnew])
validationsample[factorsnew]=imp.transform(validationsample[factorsnew])
models={'prediction':clfnn}
for key in models:
try:
models[key].fit(testsample[factorsnew].as_matrix(),testsample['returnranksnn'].as_matrix(),verbose=1)
validationsample[key]=models[key].predict_proba(validationsample[factorsnew].as_matrix(),verbose=1)[:,1]
except:
print sys.exc_info()[0]
print sys.exc_info()[1]
pass
The model somehow seems to fit without any problems but prediction step is giving an error. Output looks something like this:
Epoch 1/10
32240/32240 [==============================] - 0s - loss: 0.2506 - acc: 0.5980
Epoch 2/10
32240/32240 [==============================] - 0s - loss: 0.2504 - acc: 0.6054
Epoch 3/10
32240/32240 [==============================] - 0s - loss: 0.2504 - acc: 0.6069
Epoch 4/10
32240/32240 [==============================] - 0s - loss: 0.2505 - acc: 0.6028
Epoch 5/10
32240/32240 [==============================] - 0s - loss: 0.2504 - acc: 0.6015
Epoch 6/10
32240/32240 [==============================] - 0s - loss: 0.2503 - acc: 0.6067
Epoch 7/10
32240/32240 [==============================] - 0s - loss: 0.2504 - acc: 0.6020
Epoch 8/10
32240/32240 [==============================] - 0s - loss: 0.2505 - acc: 0.5999
Epoch 9/10
32240/32240 [==============================] - 0s - loss: 0.2504 - acc: 0.6040
Epoch 10/10
32240/32240 [==============================] - 0s - loss: 0.2505 - acc: 0.6024
32/40 [=======================>......] - ETA: 0s<type 'exceptions.IndexError'>
index 1 is out of bounds for axis 1 with size 1
Note: The data is normalized without any NaNs and predictor variable is int type having only two outcomes 0 or 1 and outcome should be just a probablity number
Tried changing settings of the optimizers, different factors in data but in vain. Majority of the samples get stuck at 32/40 or 32/*** as you see in the output. Any ideas on what I am missing here? Thanks