I am training and fitting a keras model using validation split:
self.model = Sequential()
self.model.add(LSTM(hidden_units, input_shape=(1, n_features), dropout=drp))
self.model.add(Dense(n_classes, activation='sigmoid'))
self.model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
self.model.build(X_train.shape)
self.history = self.model.fit(X, y.values, epochs=epochs, batch_size=32, validation_split=0.33, shuffle=True, callbacks=cb_list)
After fitting, I want to access the test set that was used.
How can I do this?
As per the FAQ in the documentation, the validation_split attribute makes the last % of samples to be your validation data.
In your case, you set the value of validation_split to be 0.33. This means that the last 33% of the samples in X get used as validation data.
So, you can just directly slice off the last 33% of X and use it.
Related
I am trying to predict a value that ranges from 3 to 7 with a regression nn model. I currently have 1000 samples with 1000 points each, corresponding to physiological data. However, predictions are always in the range of 5.6 to 5.9. I have tried to change the number of layers, adjust parameters etc.. but it only changes the basal value, not the range (i.e. predicts, for example, in the range of 6.6 to 6.9).
Here's the basic core of the algorithm (data is already normalized):
X_train, X_test, y_train, y_test = train_test_split(x_0, y_0, test_size=0.2, shuffle = False, stratify = None)
K.clear_session()
model = Sequential()
model.add(Dense(128, activation='relu', input_shape=(1000,)))
#model.add(Dropout(0.2))
model.add(Dense(128,activation='relu'))
model.add(Dense(128,activation='relu'))
model.add(Dense(128,activation='relu'))
model.add(Dense(1,activation='linear'))
model.compile(loss="mean_absolute_error", optimizer=Adam(lr=0.001), metrics=['mean_absolute_error'])
model.summary()
history = model.fit(X_train,y_train,batch_size=30, epochs=10,verbose=1, validation_split=0.2)
score = model.evaluate(X_test,y_test)
print("Test score:", score)
y_pred = model.predict(X_test)
Thank you for any help!
I'm guessing that your input is sorted, and I can see that you set shuffle=False in train_test_split. This means that validation values have never been seen by you neural network. And so it will not extrapolate during validation. Setting shuffle=True should work.
When training a model with TensorFlow 2 as shown below, should the validation data be separated from the training data before it is passed to the model's fit method, or can it be part of the training set? At the end of the code below show two options. I would believe option 1 is the correct, but as I have seen some sources using option 2 I want to make sure I understand it correctly.
X_train, X_test, y_train, y_test = train_test_split(df_x, series_y)
best_weight_path = 'best_weights.hdf5'
numpy_x = df_x.to_numpy()
numpy_y = series_y.to_numpy()
numpy_x_train = X_train.to_numpy()
numpy_y_train = y_train.to_numpy()
numpy_x_test = X_test.to_numpy()
numpy_y_test = y_test.to_numpy()
model = Sequential()
model.add(Dense(32, input_dim=x.shape[1], activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(2, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto')
checkpointer = ModelCheckpoint(filepath=best_weight_path, verbose=0, save_best_only=True)
option 1
model.fit(numpy_x_train, numpy_y_train, validation_data=(numpy_x_test, numpy_y_test), callbacks=[monitor, checkpointer], verbose=0, epochs=1000)
option 2
model.fit(numpy_x, numpy_y, validation_data=(numpy_x_test, numpy_y_test), callbacks=[monitor, checkpointer], verbose=0, epochs=1000)
the first option is correct... you split before the data and use your train to fit and evaluate on test/valid
the second option no... you are putting all your data to train while are passing a subpart of them for evaluation. Keras is not so clever to understand this. but to achieve what u are looking for in this second option u simply need validation_split = 0.xxx
model.fit(numpy_x, numpy_y, validation_split=0.xxx, callbacks=[monitor, checkpointer], verbose=0, epochs=1000)
in other words, u pass ALL your data in fit and then Keras operate a random split of them using a 0.xxx % for evaluation/testing
For a school project, I'm trying to predict data using the keras framework, but it's returning 'nan' loss and values when I try to get predicted data.
Source code :
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=5)
# create model
model = Sequential()
model.add(Dense(950, input_shape=(425,), activation='relu'))
model.add(Dense(425, activation='relu'))
model.add(Dense(200, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# Compile model
sgd = optimizers.SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='mean_squared_error', optimizer='sgd')
# Fit the model
model.fit(X_train, y_train, epochs=20, batch_size=1, verbose=1)
#evaluate the model
y_pred = model.predict(X_test)
score = model.evaluate(X_test, y_test,verbose=1)
print(score)
# calculate predictions
predictions = model.predict(X_pred)
Data :
X_train and X_test are (panda)dataframes of 5000 rows(nber of samples) * 425 columns (number of dimensions).
y_train and y_test look like :
array([ 1.17899644, 1.46080518, 0.9662137 , ..., 2.40157461,
0.53870386, 1.3192718 ])
Can you help me with that ?
Thank you for you help!
Usually, this means that something converges to infinity. As #desertnaut pointed out in the comment, reducing the learning rate might help.
But the root of the issue is your input data. What do these 425 data points mean? Are they from different sources, different features, different parameters? Finding outliners or normalizing the data, could help.
Your code looks fine otherwise.
Make sure your target output is in range (0, 1) as you have sigmoid in the last layer.
sigmoid has an output between zero and one so if the target output is not in this range then (a) change the activation function or (b) normalize outputs in the required range.
Make sure the purpose of this model is the regression.
After considering the above three points, play around with learning rate (decrease) and the optimiser (replace with any other).
Try changing your optimizer to 'Adam' instead of SGD
You initialized your SGD optimizer in variable sgd but you're not using it in compile
I'm working on a regression problem using Keras+Tensorflow. And I've found something interesting.
1) here are the two models, which are actually the same except that the first model is using a globally defined 'optimizer'.
optimizer = Adam() #as a global variable
def OneHiddenLayer_Model():
model = Sequential()
model.add(Dense(300 * inputDim, input_dim=inputDim, kernel_initializer='normal', activation=activationFunc))
model.add(Dense(1, kernel_initializer='normal'))
model.compile(loss='mean_squared_error', optimizer=optimizer)
return model
def OneHiddenLayer_Model2():
model = Sequential()
model.add(Dense(300 * inputDim, input_dim=inputDim, kernel_initializer='normal', activation=activationFunc))
model.add(Dense(1, kernel_initializer='normal'))
model.compile(loss='mean_squared_error', optimizer=Adam())
return model
2) Then, I use two schemes to train the datasets (training set(scaleX, Y); testing set(scaleTestX, testY)).
2.1) Scheme1. two successive fitting with the first model
numpy.random.seed(seed)
model = OneHiddenLayer_Model()
model.fit(scaleX, Y, validation_data=(scaleTestX, testY), epochs=250, batch_size=numBatch, verbose=0)
numpy.random.seed(seed)
model = OneHiddenLayer_Model()
history = model.fit(scaleX, Y, validation_data=(scaleTestX, testY), epochs=500, batch_size=numBatch, verbose=0)
predictY = model.predict(scaleX)
predictTestY = model.predict(scaleTestX)
2.2) Scheme2. one fitting with the second model
numpy.random.seed(seed)
model = OneHiddenLayer_Model2()
history = model.fit(scaleX, Y, validation_data=(scaleTestX, testY), epochs=500, batch_size=numBatch, verbose=0)
predictY = model.predict(scaleX)
predictTestY = model.predict(scaleTestX)
3). Finally, the results are plotted for each scheme, as shown below, (model loss history --> predict on scaleX --> predict on scaleTestX),
3.1) Scheme1
3.2) Scheme2 (with 500 epochs)
3.3) add one more test with Scheme2 and set epochs = 1000
From the images above, I've found that Scheme1 is better than Scheme2, even if Scheme2 is set with more epochs.
Can anyone help to explain why Scheme1 is better? Thanks a lot!!!
I have just started using Keras and was trying to train a model using Keras deep learning kit. Works till the epochs are runned but crashes just after it.
np.random.seed(1778) # for reproducibility
need_normalise=True
need_validataion=True
nb_epoch=2#8
#Creating model
model = Sequential()
model.add(Dense(512, input_shape=(dims,)))
model.add(PReLU())
model.add(BatchNormalization())
model.add(Dropout(0.5))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
opt=Adadelta(lr=1,decay=0.995,epsilon=1e-5)
model.compile(loss='binary_crossentropy', optimizer=opt)
auc_scores=[]
best_score=-1
best_model=None
print('Training model...')
if need_validataion:
for i in range(nb_epoch):
#early_stopping=EarlyStopping(monitor='val_loss', patience=0, verbose=1)
#model.fit(X_train, y_train, nb_epoch=nb_epoch,batch_size=256,validation_split=0.01,callbacks=[early_stopping])
model.fit(X_train, y_train, nb_epoch=2,batch_size=256,validation_split=0.15)
y_pre = model.predict_proba(X_valid)
scores = roc_auc_score(y_valid,y_pre)
auc_scores.append(scores)
print (i,scores)
if scores>best_score:
best_score=scores
best_model=model
plt.plot(auc_scores)
plt.show()
else:
model.fit(X_train, y_train, nb_epoch=nb_epoch, batch_size=256)
y_pre = model.predict_proba(X_test)[:,1]
print roc_auc_score(y_test,y_pre)
Error Recieved:
I have pasted it over here. Please have a look at it.
http://pastebin.com/dSw9ckkk
It looks like you have two classes, a positive class and a negative class, so that the positive class labels are 1 minus the negative class labels. In that case, you can discard the negative class labels and make it a single-class problem:
model.add(Dense(1), activation='sigmoid') # instead of Dense(nb_classes) and Activation('softmax')
Alternatively, you can still train the model on both classes and just use the positive class in the AUC calculation:
roc_auc_score(y_test[:, 1],y_pre[:, 1])