I am training a CNN LSTM model using Keras, and after the training was done, I tried to evaluate the model on the testing data like I did when I fine-tuned my CNN, however an error appears this time.
After training was done, I tried to following piece of code to evaluate on my testing set:
x, y = zip(*(testgenerator[i] for i in range(len(testgenerator))))
x_test, y_test = np.vstack(x), np.vstack(y)
loss, acc = Bi_LSTM.evaluate(x_test, y_test, batch_size=9)
print("Accuracy: " ,acc)
print("Loss: ", loss)
I have used this code before to evaluate my fine tuned model and it had no issue, but now I get the following error:
TypeError: object of type 'generator' has no len()
I have tried few solutions online like using len(list(generator)) but it did not work. Is it because I am using a custom generator? How can I do to evaluate model in this case ?
I think this line is the problem
x, y = zip(*(testgenerator[i] for i in range(len(testgenerator))))
because you call len on generator object.
The solution may be if you just create some counter, increment it and use it as index in testgenerator[i]
The way I solved this is by using a different method. In this case I do not need to extract values for x,y:
loss, acc = Bi_LSTM.evaluate_generator(testgenerator, batch_size=9)
Related
I trained a set of LinearRegression models using the following GridSearchCV
MAX_COLUMNS=list(range(2, len(house_df.columns)))
X = house_df.drop(columns=['SalePrice'])
y = house_df.loc[:, 'SalePrice']
column_list = MAX_COLUMNS
# Box-cox transform the target
reg_strategy = TransformedTargetRegressor()
bcox_transformer = PowerTransformer(method='box-cox')
model_pipeline = Pipeline([("std_scaler", StandardScaler()),
('feature_selector', SelectKBest()),
('regress', reg_strategy)])
parameter_grid = [{'feature_selector__k' : column_list,
'feature_selector__score_func' : [f_regression, mutual_info_regression],
'regress__regressor' : [LinearRegression()],
'regress__regressor__fit_intercept' : [True],
'regress__transformer' : [None, bcox_transformer]}]
score_types = {'MSE' : 'neg_mean_squared_error', 'r2' : 'r2'}
gs = GridSearchCV(estimator=model_pipeline, param_grid=parameter_grid, scoring=score_types, refit='MSE', cv=5, n_jobs=5, verbose=1)
gs.fit(X, y)
PATH = './datasets/processed_data/'
gridsearch_result_filename = 'pfY_np10_nt2_rfS_ct0_8_st1_orY_ccY_LR1_GS.pkl'
full_path = PATH + gridsearch_result_filename
with open(full_path, 'wb') as file:
pickle.dump(gs, file)
I then load the trained GridSearch and can make predictions using the best estimator as follows:
with open(MODEL_PATH, 'rb') as file:
gs_results = pickle.load(file)
predictions = gs_results.predict(test_df)
The problem I am facing is that since the Box-Cox transform was applied during GridSearch, all of my predictions are in the Box-Cox transformed distributions domain (huge values).
I need to use the PowerTransformers inverse_transform() method on my predictions, but I am not sure how to access it.
I can get the entire pipeline for the best estimator like this
gs_results.best_estimator_
I can then access the TransformedTargetRegressor inside the pipeline like this:
Taking a step further, we get all the way to the PowerTransformer inside the TransformedTargetRegressor like this:
After making it here, I foolishly thought I had made it where I needed to be, and simply needed to use the PowerTransformers inverse_transform method to make predictions that would be back in the original units. However, much to my disappointment, an error is thrown:
The error seems pretty clear, telling me I cannot use the inverse_transform method because the PowerTransformer has not been fit.
This is where I am stumped. It doesn't make sense to say the PowerTransformer has not been fit, when clearly it was fit during the GridSearch process.
This makes me think I am simply accessing the PowerTransformer incorrectly, which is my current question.
Based on the set up above, does anyone know the correct way to take the inverse transform of my predictions so they are in the original units rather than the Box-Cox distributions units?
I have been banging my head against the wall for this and have searched all over for a similar question. Thank you so much in advance!
-Braden
Much like here, the attribute transformer is the unfitted initialization attribute; you need the fitted transformer_ attribute.
However, I'm not sure why predict doesn't already do what you want; the documentation for TransformedTargetRegressor.predict says
Predict using the base regressor, applying inverse.
The regressor is used to predict and the inverse_func or inverse_transform is applied before returning the prediction.
I'm struggling to implement a custom metric in Keras (2.4.3 with the tensorflow backend) such that I can trigger an early stopping mechanic. Essentially, I want to have Keras stop training a model should there be too big a decrease in the training loss function. To do this, I am using the following code:
def custom_metric(y_true,y_pred):
y=keras.losses.CategoricalCrossentropy(y_true,y_pred)
z=1.0/(1.0-y.numpy())
return z
model.compile(loss='categorical_crossentropy',
optimizer=opt,
metrics=['categorical_accuracy',custom_metric])
custom_stop = EarlyStopping(monitor='custom_metric',min_delta=0,patience=2,
verbose=1,mode='min',restore_best_weights=True)
I'm getting errors along the lines of AttributeError: 'CategoricalCrossentropy' object has no attribute 'numpy', which I understand is due to the definition of z, but I can't get something equivalent to work using by replacing the floats in the definition of z with tf.constants or anything like that. Does anyone have any suggestions?
Thanks a lot
Use this instead, mind the spelling:
keras.losses.categorical_crossentropy(y_true,y_pred)
This should work:
def custom_metric(y_true,y_pred):
y=keras.losses.categorical_crossentropy(y_true,y_pred)
z=1.0/(1.0-y)
return z
I am using tensorflow V2.2 and run into TyepError when I do model.evaluate. Can someone advise what the issues may be? A screenshot of the execution and error message is shown below.
you need to define a metric when you compile the model model.compile('adam', 'binary_crossentropy', metrics='accuracy')
in this way during evaluation, loss and accuracy are returned
Actually, if you don't have metric='accuracy' in the model.compile, don't try to extract it in model.evaluate.
So, you can change:
loss, acc= model.evaluate(df, df_y, verbose=0)
to
loss = model.evaluate(df, df_y, verbose=0)
I'm reading Object Detection API source code and I wonder how to use TFSlim to train model?
More specifically, when we use Tensorflow to train the model, we use something like this:
parameters = model(X_train, Y_train, X_test, Y_test)
# Returns: parameters -- parameters learnt by the model.
# They can then be used to predict.
And to predict the result, we use something like:
y_image_prediction = predict(my_image, parameters)
But in file trainer.py, we don't have something like above, we only get:
slim.learning.train(
train_tensor,
logdir=train_dir,
master=master,
is_chief=is_chief,
session_config=session_config,
startup_delay_steps=train_config.startup_delay_steps,
init_fn=init_fn,
summary_op=summary_op,
number_of_steps=(
train_config.num_steps if train_config.num_steps else None),
save_summaries_secs=120,
sync_optimizer=sync_optimizer,
saver=saver)
And there are no return about this slim.learning.train function. So I wonder what is the using of slim.learning.train function, and how do we get the parameters -- that can be used to predict the result?
HERE is source code of trainer.py.
The train function does not return a value because it modifies the actual parameters of the model. The function does that by running the train_tensor which is: "A Tensor that, when executed, will apply the gradients and return the loss value." as written in the function documentation.
The tensor the documentation talks about what you get when you tell an optimizer to optimize some cost function. It is opt_op in the following example:
opt = GradientDescentOptimizer(learning_rate=0.1)
opt_op = opt.minimize(cost)
Find more in the optimizer documentation.
I am quite new to tensorflow and in order to learn to use it I am currently trying to implement a very simple DNNRegressor that predicts the movement of an object in 2D but I can't seem to the the predict function to work.
for this purpose I have some Input data - x and y coordinates of the object in a number of previous time steps. I want the output to a reasonable estimation of the location the object if it continues to move in the same direction with the same speed.
I am using tensorflow version 1.8.0
My regressor is defined like this:
CSV_COLUMN_NAMES = ['X_0', 'X_1', 'X_2', 'X_3', 'X_4', 'Y_0', 'Y_1', 'Y_2', 'Y_3', 'Y_4', 'Y_5']
my_feature_columns = []
for key in columnNames:
my_feature_columns.append(tf.feature_column.numeric_column(key=key))
regressor = estimator.DNNRegressor(feature_columns=my_feature_columns,
label_dimension=1,
hidden_units=hidden_layers,
model_dir=MODEL_PATH,
dropout=dropout,
config=test_config)
my input is, like the one in the tensorflow tutorial on premade estimators, a dict with the column as key.
An example for this input can be seen here.
regressor.train(arguments) and regressor.evaluate(arguments) seem to work just fine, but predict does not.
parallel to the code on the tensorflow site I tried to do this:
y_pred = regressor.predict(input_fn=eval_input_fn(X_test, labels=None, batch_size=1))
and it seems like that works as well.
The problem I'm facing now is that I can't get anything from that y_pred object.
when I enter print(y_pred) I get <generator object Estimator.predict at 0x7fd9e8899888> which would suggest to me that should be able to iterate over it but
for elem in y_pred:
print(elem)
results in TypeError: unsupported callable
Again, I'm quite new to this and I am sorry if the answer is obvious but I would be very grateful if someone could tell me what I'm doing wrong here.
The input_fn to regressor.predict should be a function. See the definition:
input_fn: A function that constructs the features.
You need to change your code to:
y_pred = regressor.predict(input_fn=eval_input_fn)