We Keep Coding

Confused about why something ran once but not another time

So I ran this code last night, and it worked fine it did plot the training loss as a fcn of epoch value. However, when I tried to run it today I changed the batch size from 1 to 8 and it gave me a 'plt not found' error. I then moved the plotting to below the matplotlib import line and it worked. This seems to suggest that line must come before the plotting, but how was I able to plot last night with the plot commands before the import?
This is just part of the complete code yes, but the rest wasn't relevant. This was in Jupyter notebook too, so perhaps I had ran the code before without the plot lines inside the tf.device block, and it saved the import or something?
with tf.device(device_name):
inputx = Input(shape=(7,))
x = Dense(4, activation='elu',name='x1')(inputx)
x = Dense(16, activation='elu',name='x2')(x)
x = Dense(25, activation='elu',name='x3')(x)
x = Dense(10, activation='elu',name='x4')(x)
xke = Dense(5,name='x5')(x)
model = Model(inputx, xke)
adam = optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=1e-6, amsgrad=False)
model.compile(optimizer=adam,
loss=['mean_squared_error','mean_squared_error','mean_squared_error','mean_squared_error','mean_squared_error'],
loss_weights=[1,1,1,1,1],)
model.summary()
history = model.fit(X_train, y_train, batch_size=1, epochs=30, verbose=1)
plt.plot(history.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend('train', loc='upper left')
plt.show()
from sklearn.metrics import mean_squared_error as mse
train_pred = model.predict(X_train)
train_rmse_sk = np.sqrt(mse(y_train, train_pred, multioutput= "raw_values"))
print("The training rmse value is: ", train_rmse_sk, "\n")
import matplotlib.pyplot as plt

How can I print the training and validation graphs, and training and validation loss graphs?

I need to plot the training and validation graphs, and trarining and validation loss for my model.
model.compile(loss=tf.keras.losses.binary_crossentropy,
optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),
metrics=['accuracy'])
history = model.fit(X_train, y_train,
batch_size=batch_size,
epochs=no_epochs,
verbose=verbosity,
validation_split=validation_split)
loss, accuracy = model.evaluate(X_test, y_test, verbose=1)

history object contains both accuracy and loss for both the training as well as the validation set. We can use matplotlib to plot from that.
In these plots x-axis is no_of_epochs and the y-axis is accuracy and loss value. Below is one basic implementation to achieve that, it can easily be customized according to requirements.
import matplotlib.pyplot as plt
def plot_history(history):
acc = history.history["accuracy"]
loss = history.history["loss"]
val_loss = history.history["val_loss"]
val_accuracy = history.history["val_accuracy"]
x = range(1, len(acc) + 1)
plt.figure(figsize=(12,5))
plt.subplot(1, 2, 1)
plt.plot(x, acc, "b", label="traning_acc")
plt.plot(x, val_accuracy, "r", label="traning_acc")
plt.title("Accuracy")
plt.subplot(1, 2, 2)
plt.plot(x, loss, "b", label="traning_acc")
plt.plot(x, val_loss, "r", label="traning_acc")
plt.title("Loss")
plot_history(history)
Plot would look like below:
Accuracy and Loss plot

Plot training and validation accuracy and loss

I am new to Python and trying to plot the training and validation accuracy and loss for my MLP Regressor, however, I am getting the following error, what am I doing wrong?
TypeError: fit() got an unexpected keyword argument 'validation_split'
mlp_new = MLPRegressor(hidden_layer_sizes=(18, 18,18),
max_iter = 10000000000,activation = 'relu',
solver = 'adam', learning_rate='constant',
alpha=0.05,validation_fraction=0.2,random_state=0,early_stopping=True)
mlp_new.fit(X_train, y_train)
mlp_new_y_predict = mlp_new.predict((X_test))
mlp_new_y_predict
import keras
from matplotlib import pyplot as plt
history = mlp_new.fit(X_train, y_train, validation_split = 0.1, epochs=50, batch_size=4)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()

Yes, you definitely can find a validation_split arg in the keras model .fit() method.
But:
The model you are going to use here is not that one.
Check the documentation below, Methods section:
method .fit(..) has only two args: X and y.
https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html#sklearn.neural_network.MLPRegressor.fit

How can I test real-time predictions in neural network? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
As I'm new here let me ask a question that would be fairly common. I wrote an MLP model of the neural network using deep learning. I'm using a standard data-set which I downloaded here enter link description here. from statistical point of view, the accuracy and f1-score of mine shows wonderfully output. now I need to test this program with real-time data. I will be glad to hear your suggestions for how can I perform real-time predictions in neural networks?
from pandas import pandas as pd
from pandas import DataFrame
from numpy import*
import numpy as np
from matplotlib import pyplot as plt
from sklearn.model_selection import GridSearchCV,train_test_split
from sklearn.metrics import confusion_matrix,accuracy_score,roc_curve,auc
from sklearn.neural_network import MLPClassifier
from sklearn import preprocessing
from sklearn.preprocessing import LabelEncoder
from keras.models import Sequential
from keras.layers import Dense
from keras.utils import to_categorical
from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import RandomForestClassifier
########################################################################################
db = pd.read_csv(r"C:\Users\cert 3\Desktop\Vasou\proposal\code\StackOverFlow\UDP-Flood-CSV.csv")
X = db.iloc[:, 0:4]
y = db.iloc[:, 4]
m, n = X.shape
MG = X
X = preprocessing.scale(X)
encoder = LabelEncoder()
encoder.fit(y)
encoded_y = encoder.transform(y)
y = to_categorical(encoded_y)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
########################################################################################
mlp = MLPClassifier()
parameter_space = {'hidden_layer_sizes': [(8,12,4), (5,5,2), (4,4,4)],
'activation': ['tanh', 'relu'],
'solver': ['sgd', 'adam'],
'alpha': [0.001,0.01, 0.05, 0.1],
'learning_rate': ['constant','adaptive'],
'max_iter':[20,50,100]
}
clf = GridSearchCV(mlp, parameter_space, n_jobs=-1, cv=3,return_train_score=True)
clf.fit(X_train, y_train)
print('Best parameters found:\n', clf.best_params_, clf.best_score_)
#######################################################################################
cvr = clf.cv_results_
df = DataFrame(cvr)
scores = df['mean_test_score']
h = df['param_hidden_layer_sizes']
alpha = df['param_alpha']
optim = df['param_solver']
l_rate = df['param_learning_rate']
activ = df['param_activation']
itr = df['param_max_iter']
dh = DataFrame({'Scores': scores, 'Itraction':itr, 'Hidden_Layers': h, 'alpha': alpha ,
'Solver':optim, 'Learning_Rate':l_rate, 'Activation':activ})
########################################################################################
model = Sequential()
model.add(Dense(8, input_dim=n, kernel_initializer='uniform', activation='tanh'))
model.add(Dense(12, activation='tanh'))
model.add(Dense(4, activation='tanh'))
model.add(Dense(2, activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
########################################################################################
hist = model.fit(X_train, y_train, batch_size = 10, epochs = 100, validation_split=0.5)
scoress = model.evaluate(X, y, verbose=0)
print("%s: %.2f%%" % (model.metrics_names[1], scoress[1]*100))
print(hist.history)
# save model and architecture to single file
model.save("model.h5")
model.save_weights("model_weight.h5")
print("saved model to disk")
# Plot training & validation accuracy values
plt.plot(hist.history['acc'])
plt.plot(hist.history['val_acc'])
plt.title('Training vs Test accuracy , DA')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Training acc', 'Validation acc'], loc='best')
#plt.show()
#plt.figure()
a = plt.savefig('Accuracy.png', dpi=300, bbox_inches='tight')
plt.close(a)
# Plot training & validation loss values
plt.plot(hist.history['loss'])
plt.plot(hist.history['val_loss'])
plt.title('Training vs Test Loss , DA')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Training loss', 'Validation loss'], loc='best')
#plt.show()
#plt.figure()
b = plt.savefig('Loss.png', dpi=300, bbox_inches='tight')
plt.close(b)
##########################################################################################
y_score = model.predict(X_test)
org = zeros((y_test.shape[0]))
prd = zeros((y_score.shape[0]))
def decode(datum):
return np.argmax(datum)
for i in range(y_score.shape[0]):
prd[i] = decode(y_score[i])
for j in range(y_test.shape[0]):
org[j] = decode(y_test[j])
confusion_matrix(org,prd)
print("Accuracy of MLP: ", "\n", confusion_matrix(org,prd))
f = open("output.txt", "a")
print('Accuracy Score : ' + str(accuracy_score(org,prd)), file=f)
f.close()
##########################################################################################
def generate_results(y_test, y_score):
fpr, tpr, _ = roc_curve(y_test, y_score)
roc_auc = auc(fpr, tpr)
plt.figure()
plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], 'k--')
plt.xlim([0.0, 1.05])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic curve')
#plt.show()
plt.savefig('False and True comparison.png', dpi=300, bbox_inches='tight')
print('AUC: %f' % roc_auc)
print('Generating results')
generate_results(y_test[:, 0], y_score[:, 0])
and this is my python code.

If you want to use this code on demand, you can use the < input stream arrow from shell, for example: python script.py < your_streamer.
At last you must choose special character at the end of packet to find out when you capture whole of packet.
In python input() is a good choose with \n seprator.
script.py
while True:
X = np.array(input().split(','), dtype=np.float)
y = model.predict([X])
print(X, y)
file.txt as streamer.
0.218,0.7451,0.7451,0.574
0.215,0.8854,0.7451,0.745
0.275,0.5744,0.7451,0.574
0.751,0.5744,0.2150,0.885
...
...
...
$ python script.py < file.txt

Issues with KeyError: 'val_acc'

This is rather a popular error, but I couldn't find a proper answer given my setup.
I found this tutorial code, but when running, I get this error:
val_acc = history.history['val_acc']
KeyError: 'val_acc'
The fit_generator() function unlike fit(), doesn't allow a validation split. So how to fix it?
Here is the code:
def plot_training(history):
print (history.history.keys())
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(len(acc))
plt.plot(epochs, acc, 'r.')
plt.plot(epochs, val_acc, 'r')
plt.title('Training and validation accuracy')
# plt.figure()
# plt.plot(epochs, loss, 'r.')
# plt.plot(epochs, val_loss, 'r-')
# plt.title('Training and validation loss')
plt.show()
plt.savefig('acc_vs_epochs.png')
#....
finetune_model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS, num_classes=len(class_list))
adam = Adam(lr=0.00001)
finetune_model.compile(adam, loss='categorical_crossentropy', metrics=['accuracy'])
filepath="./checkpoints/" + "ResNet50" + "_model_weights.h5"
checkpoint = ModelCheckpoint(filepath, monitor=["acc"], verbose=1, mode='max')
callbacks_list = [checkpoint]
history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=8,
steps_per_epoch=steps_per_epoch,
shuffle=True, callbacks=callbacks_list)
plot_training(history)

Hi writting my suggestions here because I'm not able to comment yet,
You are right the fuction fit_generator() dosen't have the validation split attribute.
Therefore you need to make your own validation dataset and feed it to the fit generator through validation_data=(val_X, val_y) as eg.:
history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=8, validation_data=(val_X, val_y),
steps_per_epoch=steps_per_epoch,
shuffle=True, callbacks=callbacks_list)
Hope this helps.
EDIT
To get a validation dataset from your data you can use the methode train_test_split() from sklearn. For example a split with 77% train and 33% validation data:
X_train, val_X, y_train, val_y= train_test_split(
X, y, test_size=0.33, random_state=42)
Look here for more information.
Alternatively you could write your own split methode :)
Edit 2
If you don't have the possibility to use train_test split and with the assumption you have a pandas dataframe called train_data with the features and labels together:
val_data=train_data.sample(frac=0.33,random_state=1)
This should creates a validation dataset with 33% of the data and a train dataset with 77% of the data.
Edit3
It turns out you are using ImageDataGenerator() to create your data. This is quite handy because you can set your validation percentage via validation_split= while you initialize the ImageDataGenerator() as seen in the documentation (here). This should look something like this:
train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input,
validation_split=0.33)
After this you need two "generated" datasets. One to train and one to make your validation. This should look as following:
train_generator = train_datagen.flow_from_directory(TRAIN_DIR,
target_size=(HEIGHT, WIDTH),
batch_size=BATCH_SIZE,subset="training")
validation_generator = train_datagen.flow_from_directory(TRAIN_DIR,
target_size=(HEIGHT, WIDTH),
batch_size=BATCH_SIZE,subset="validation")
Finally you can use both sets in your fit_generator as following:
history = finetune_model.fit_generator(train_generator,epochs=NUM_EPOCHS, workers=8,
validation_data=validation_generator, validation_steps = validation_generator.samples,steps_per_epoch=steps_per_epoch,
shuffle=True, callbacks=callbacks_list)
Let me know if this solves your problem :)