I accidentally ran a model where the training loss goes to "Cos_BN Loss", and the validation loss go to "Gen Cos_BN Loss", that means that I have two separate graphs for train and validation (for the same type of loss).
I want to see them together on the same plot. How?
NOTE: I know I can just run again the model by calling the validation loss - the same name as the training loss, BUT it took around 3 days to ran, on 3 GPUs and I really don't want to train it again all over.
Current state
Thank you
On the bottom there is a download data link that let's you save the data as a CSV file. You may need to enable it first (top left in following picture):
You can then import this e.g. using pandas and plot it to your liking using `matplotlib.
Related
I am running a deep learning model using the deepxde library link. I am trying to plot the losshistory (i.e MSE loss in Train, Test data vs epochs) in order to analyse my simulations. As I am performing a parametric study by varying many parameters, I am trying to save all the loss history while my simulation is running and then later on plot them one after the other at the end of the simulation!
Currently, the code is plotting them after every loop of each parameter (i.e if I have 10*5 parameters , then I am getting 50 plots periodically after each simulation (end of epochs)! This is becoming very crumble-some for me to wait and attend all the 50 plots one after the other during the run-time! That's why I want to look at these after all my simulations have finished! For that I enabled the issave=True. But was not bale to locate the plots!
I am not able to know where are my dde.saveplot files are being saved?
And what will be the name of these files? (Name or extension; such that I can find them easily in the folder)?
While keeping the isplot=True; Can there be anyway in which my terminal does not pauses to show the plots? That is, it can plot the error figures and also run in the background, hence all the plots are plotted without me going and closing each one of them one by one and then the program restarting?
I am using the deepxde module link for this simulation. It is based on tensorflow.
Code:
model = dde.Model(data, net)
model.compile("adam", lr=lr, metrics=[mean_squared_error])
checker = dde.callbacks.ModelCheckpoint(
"/Users/anshumansinha/Desktop/Project/model/"+save_str+"model.ckpt", save_better_only=False, period=100
)
print('check done 1')
# Training for different input points from 2^4 to 2^11.
# Trainng will be done for 10,000 epochs.
# isplot = True will generate 10 plots for each simulation.
losshistory, train_state = model.train(epochs=epochs, callbacks=[checker]) #Training Model batch_size = 10000
# For plotting the residuals and the training history: isplot=True will plot
dde.saveplot(losshistory, train_state, issave=True, isplot=True)
I have been using Keras/TF's CSVLogger in order to save the training and validation accuracies. Then I was plotting those data to check the trajectory of training and validation accuracy or loss.
Yesterday, I read this link.
Here, they used model.evaluate() to plot the accuracy metric.
What is the difference between these two approaches?
Here are the big key differences between the two:
model.evaluate() gives a different loss on training data from the one in the training process, while CSVLogger tracks metrics on the validation set after every epoch in the process.
Log and plot the results
The CSVLogger will save the results of the training in a CSV file.
The output of fit, normally called history, will have the same results ready in a variable, without needing a CSV file.
Their data should be exactly the same.
Evaluate
The evaluate method is not a log. It's a method that you call any time you want, and will output a single value for each loss and metric you have in compile for the "current" model.
This is not registered anywhere during training, and this should usually not be called during training, unless you create a custom training loop and manually call evaluate inside this loop every epoch (this is unnecessary and inconvenient for standard use).
You can use evaluate with any data you want, including data that was neither in training nor validation data.
The link you shared
Notice that what they plot is history, and history is the output of fit. It has nothing to do with evaluate.
plot.plot(history.history['acc'])
plot.plot(history.history['val_acc'])
See in the previous page of the link:
history = model.fit(...)
The link doesn't show exactly "where" they call evaluate, and they're probably doing it "only once after training", just to see the final results of the model (not a log of the training)
I am trying to add data for my scikit-learn model after it has already been trained. For example, I have the data that I used in the beginning (there are about 250 of them). After that, I need to train this model one more time by calling the function, and so on. The only thing that came to my mind was to add new values to the existing data array every time and train the model again, but this is very resource-intensive and takes more time.
Is there another way to train the machine learning model?
model = LinearRegression().fit(test, result)
reg.predict(task)
### and here I want to add some data, for example one or two examples like:
model.addFit(one_test, one_result)
The short answer in your case (using the sklearn.linear_model.LinearRegression model) is no, it is not possible to add one or two more examples and train without adding this to the original training set and fitting it all at the same time. Under the hood, the model is simply using Ordinary Least Squares (described here) which requires the complete matrix of training data on which to fit your model. However, this algorithm is very fast and in the case of ~ hundreds of training examples, it would be very quick to re-calculate the parameters of the model with each new couple examples.
I am training a fairly large keras unet model with some obscure data to do semantic image segmentation. The val_loss decreases initially but after some hours it starts to shoot up. It was likely overfitting and I put in more data only to delay the same behavior.
I want to observe what is going on in the validation stage. Is it possible to output a "per sample accuracy and loss" after each epoch since val_loss and val_accuracy is computed after each epoch not batch?
To put it another way, I want to find out the sample (let's assume image or numpy array) that gives the highest accuracy and the lowest accuracy and visualize them. I already have written a custom data generator from which I can output the file/array name/number that is being processed but what I am looking to do has been explained above.
Quite likely, I need some sort of a callback.
Clarification (after comment): by sample I mean a datapoint (or an image in this context) .
The problem at hand is a semantic segmentation problem (with three classes) not a classification one.
So basically, on a high level, I am trying to ask the question: in my validation set, which image is being quantified to have high accuracy/low loss, so I can do a sanity check to see if the functions are making sense?
This may be possible without a custom callback using Tensorboard with the update_freq option.
logdir = "./logs/" + datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = keras.callbacks.TensorBoard(log_dir=logdir, update_freq=
'batch')
model.fit(..., callbacks=[tensorboard_callback])
And open TensorBoard with
%tensorboard --logdir logs/scalars
I would like to run multiple experiments, then report model accuracy per experiment.
I'm training a toy MNIST example with pytorch (v1.1.0), but the goal is, once I can compare performance for the toy problem, to have it integrated with the actual code base.
As I understand the TRAINS python package, with the "two lines of code" all my hyper-parameters are already logged (Command line argparse in my case).
What do I need to do in order to report a final scalar and then be able to sort through all the different training experiments (w/ hyper-parameters) in order to find the best one.
What I'd like to get, is a graph/s where on the X-axis I have hyper-parameter values and on the Y-axis I have the validation accuracy.
I assume you are referring to: https://pypi.org/project/trains/ (https://github.com/allegroai/trains),
which I'm one of the maintainers.
You can manually create a plot with a single point X-axis for the hyper-parameter value, and Y-Axis for the accuracy.
number_layers = 10
accuracy = 0.95
Task.current_task().get_logger().report_scatter2d(
"performance", "accuracy", iteration=0,
mode='markers', scatter=[(number_layers, accuracy)])
Assuming your hyper-parameter is "number_layers" with current value 10, and the accuracy for the trained model is 0.95.
Then when you compare the experiments you get something like that: