To find the loss during training a model we can use cntk.squared_error() function, like this:
loss = cntk.squared_error(z, l)
But I am interested in finding the loss in terms of absolute error. The below code doesn't work:
loss = cntk.absolute_error(z, l)
It gives error as:
AttributeError: module 'cntk' has no attribute 'absolute_error'
Is there any inbuilt function in CNTK toolkit to find the absolute error? I am new to deep learning so I don't know much. Thanks for help!
There's no out-of-the-box L1 loss function in CNTK, but you can provide a custom one:
def absolute_error(z, l):
return cntk.reduce_mean(cntk.abs(z - l))
Related
I am a interested in GAN.
I tried to adjust the DCGAN's discriminator by this method below:
https://github.com/vasily789/adaptive-weighted-gans/blob/main/aw_loss.py
which name is aw method.
So I find a DCGAN code in kaggle(https://www.kaggle.com/vatsalmavani/deep-convolutional-gan-in-pytorch) and try to edit the discriminator by class the aw_loss.
Here is my code:
https://colab.research.google.com/drive/1AsZztd0Af0UMzBXXkI9QKQZhAUoK01bk?usp=sharing
it seems like I can not class the aw loss correctly. Because the discriminator's loss is still 0 when I training.
Any one can help me. Please!
In the code you provided, it does display the correct error when trying to use aw_method(), you should first instance the class as shown below after which you should be able to call the method.
aw_instance = aw_method()
aw_loss = aw_instance.aw_loss(D_real_loss, D_fake_loss, D_opt, D)
Notice that we are using default parameters for the class, not so familiar with aw loss to tell you if you should tweak that.
Regarding your discriminator's loss, correct code relies on aw_cost to work. It doesn't seem like your providing both losses from real and fake, so the discriminator is only learning to output 1's or 0's (which can be easily verified by printing those values or monitoring with wandb or similar tools). Again didn't go deep enough into the algorithm of the aw loss, so check this specifically.
Also could try to test as a linear combination of your normal D_loss = (D_fake_loss + D_real_loss + aw_loss) / 3.
I'm struggling to implement a custom metric in Keras (2.4.3 with the tensorflow backend) such that I can trigger an early stopping mechanic. Essentially, I want to have Keras stop training a model should there be too big a decrease in the training loss function. To do this, I am using the following code:
def custom_metric(y_true,y_pred):
y=keras.losses.CategoricalCrossentropy(y_true,y_pred)
z=1.0/(1.0-y.numpy())
return z
model.compile(loss='categorical_crossentropy',
optimizer=opt,
metrics=['categorical_accuracy',custom_metric])
custom_stop = EarlyStopping(monitor='custom_metric',min_delta=0,patience=2,
verbose=1,mode='min',restore_best_weights=True)
I'm getting errors along the lines of AttributeError: 'CategoricalCrossentropy' object has no attribute 'numpy', which I understand is due to the definition of z, but I can't get something equivalent to work using by replacing the floats in the definition of z with tf.constants or anything like that. Does anyone have any suggestions?
Thanks a lot
Use this instead, mind the spelling:
keras.losses.categorical_crossentropy(y_true,y_pred)
This should work:
def custom_metric(y_true,y_pred):
y=keras.losses.categorical_crossentropy(y_true,y_pred)
z=1.0/(1.0-y)
return z
I want to implement a custom loss which can be calculated by using each sample.
Calculation of the loss is a little complicated and requires me to use an external python file for this (or one can assume that we give the inputs to a function).
How can I implement this?
Is it possible to use #tf.function annotation and make it a graph?
This is how it is supposed to look
def loss(input,output):
loss = 0
for x, y in zip(input, output):
sim = Class(x)
a = sim.GetA()
b = sim.GetB()
loss = loss + np.linalg.norm(np.dot(a,b)+y)
return loss
An implementation of the same via PyTorch was possible as it supports dynamic computational graph
I am quite new to tensorflow and in order to learn to use it I am currently trying to implement a very simple DNNRegressor that predicts the movement of an object in 2D but I can't seem to the the predict function to work.
for this purpose I have some Input data - x and y coordinates of the object in a number of previous time steps. I want the output to a reasonable estimation of the location the object if it continues to move in the same direction with the same speed.
I am using tensorflow version 1.8.0
My regressor is defined like this:
CSV_COLUMN_NAMES = ['X_0', 'X_1', 'X_2', 'X_3', 'X_4', 'Y_0', 'Y_1', 'Y_2', 'Y_3', 'Y_4', 'Y_5']
my_feature_columns = []
for key in columnNames:
my_feature_columns.append(tf.feature_column.numeric_column(key=key))
regressor = estimator.DNNRegressor(feature_columns=my_feature_columns,
label_dimension=1,
hidden_units=hidden_layers,
model_dir=MODEL_PATH,
dropout=dropout,
config=test_config)
my input is, like the one in the tensorflow tutorial on premade estimators, a dict with the column as key.
An example for this input can be seen here.
regressor.train(arguments) and regressor.evaluate(arguments) seem to work just fine, but predict does not.
parallel to the code on the tensorflow site I tried to do this:
y_pred = regressor.predict(input_fn=eval_input_fn(X_test, labels=None, batch_size=1))
and it seems like that works as well.
The problem I'm facing now is that I can't get anything from that y_pred object.
when I enter print(y_pred) I get <generator object Estimator.predict at 0x7fd9e8899888> which would suggest to me that should be able to iterate over it but
for elem in y_pred:
print(elem)
results in TypeError: unsupported callable
Again, I'm quite new to this and I am sorry if the answer is obvious but I would be very grateful if someone could tell me what I'm doing wrong here.
The input_fn to regressor.predict should be a function. See the definition:
input_fn: A function that constructs the features.
You need to change your code to:
y_pred = regressor.predict(input_fn=eval_input_fn)
Problem
I'm running a Deep Neural Network on the MNIST where the loss defined as follow:
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, label))
The program seems to run correctly until I get a nan loss in the 10000+ th minibatch. Sometimes, the program runs correctly until it finished. I think tf.nn.softmax_cross_entropy_with_logits is giving me this error.
This is strange, because the code just contains mul and add operations.
Possible Solution
Maybe I can use:
if cost == "nan":
optimizer = an empty optimizer
else:
...
optimizer = real optimizer
But I cannot find the type of nan. How can I check a variable is nan or not?
How else can I solve this problem?
I find a similar problem here TensorFlow cross_entropy NaN problem
Thanks to the author user1111929
tf.nn.softmax_cross_entropy_with_logits => -tf.reduce_sum(y_*tf.log(y_conv))
is actually a horrible way of computing the cross-entropy. In some samples, certain classes could be excluded with certainty after a while, resulting in y_conv=0 for that sample. That's normally not a problem since you're not interested in those, but in the way cross_entropy is written there, it yields 0*log(0) for that particular sample/class. Hence the NaN.
Replacing it with
cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv + 1e-10))
Or
cross_entropy = -tf.reduce_sum(y_*tf.log(tf.clip_by_value(y_conv,1e-10,1.0)))
Solved nan problem.
The reason you are getting NaN's is most likely that somewhere in your cost function or softmax you are trying to take a log of zero, which is not a number. But to answer your specific question about detecting NaN, Python has a built-in capability to test for NaN in the math module. For example:
import math
val = float('nan')
val
if math.isnan(val):
print('Detected NaN')
import pdb; pdb.set_trace() # Break into debugger to look around
Check your learning rate. The bigger your network, more parameters to learn. That means you also need to decrease the learning rate.
I don't have your code or data. But tf.nn.softmax_cross_entropy_with_logits should be stable with a valid probability distribution (more info here). I assume your data does not meet this requirement. An analogous problem was also discussed here. Which would lead you to either:
Implement your own softmax_cross_entropy_with_logits function, e.g. try (source):
epsilon = tf.constant(value=0.00001, shape=shape)
logits = logits + epsilon
softmax = tf.nn.softmax(logits)
cross_entropy = -tf.reduce_sum(labels * tf.log(softmax), reduction_indices=[1])
Update your data so that it does have a valid probability distribution