I created a Sequential model using tf.keras as follows:
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(8, input_dim=4))
model.add(tf.keras.layers.Dense(3, activation=tf.nn.softmax))
opt = tf.train.AdamOptimizer(learning_rate=0.001)
model.compile(optimizer=opt, loss="categorical_crossentropy", metrics=["accuracy"])
model.summary()
After that, I created a training process using train_on_batch:
EPOCHS=50
for epoch in range(EPOCHS):
for metrics, labels in dataset:
# Calculate training loss and accuracy
tr_loss, tr_accuracy = model.train_on_batch(metrics, labels)
When I try to save the model, I receive a warning. I can't understand why, because I included the optimizer as part of the model.compile:
tf.keras.models.save_model(
model,
"./model/iris_model.h5",
overwrite=True,
include_optimizer=True
)
WARNING:tensorflow:TensorFlow optimizers do not make it possible to access optimizer attributes or optimizer state after instantiation. As a result, we cannot save the optimizer as part of the model save file.You will have to compile your model again after loading it. Prefer using a Keras optimizer instead (see keras.io/optimizers).
The TF version I used is 1.9.0-rc2.
As the warning says the Tensorflow optimizers cannot be saved when saving the model. Instead, use the optimizers provided by Keras:
opt = tf.keras.optimizers.Adam(lr=0.001)
Optimizer like tf.keras.optimizers.Adam() will be saved, and tf.train.AdamOptimizer() will not be saved on model.save().
As the time as writing this some official tutorials on TensorFlow use tf.train.* optimizer while I strongly believe selecting the tf.keras.optimizers.* is the best way to go.
Related
I have been trying to save the weights of my neural network model so that I could use a few of its layers for another neural network model to be trained on another dataset.
pre-trained model:
model = Sequential()
model.add(tf.keras.layers.Dense(100, input_shape=(X_train_orig_sm.shape)))
model.add(tf.keras.layers.Activation('relu'))
model.add(tf.keras.layers.Dropout(0.2))
model.add(tf.keras.layers.Dense(10))
model.add(tf.keras.layers.Activation('relu'))
model.add(tf.keras.layers.Dropout(0.2))
model.add(tf.keras.layers.Dense(1))
model.add(tf.keras.layers.Activation('sigmoid'))
model.summary()
# need sparse otherwise shape is wrong. check why
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
print('Fitting the data to the model')
batch_size = 20
epochs = 10
history = model.fit(X_train_orig_sm, Y_train_orig_sm, batch_size=batch_size, epochs=epochs, verbose=1, validation_split=0.2)
print('Evaluating the test data on the model')
How I saved the weights of neural network:
model.save_weights("dnn_model.h5")
How I try to load the weights of neural network:
dnn_model=model.load_weights("dnn_model.h5")
dnn_model.layers[5]
While trying to load the model, I get the following error:
AttributeError: 'NoneType' object has no attribute 'layers'
I dont seem to understand why the layers of the neural network are not recognised even though the pre-trained neural network is trained before the model was saved. Any advice, solution or direction will be highly appreciated. Thank you.
When you call model.save_weights("dnn_model.h5"), you only save the "weights" of the model. You do not save the actual structure of the model. That's why you cannot access the layers etc.
To save the actual model, you can call the below.
# save
model.save('dnn_model') # save as pb
model.save('dnn_model.h5') # save as HDF5
# load
dnn_model = tf.keras.models.load_model('dnn_model') # load as pb
dnn_model = tf.keras.models.load_model('dnn_model.h5') # load as HDF5
Note: You do not need to add an extension to the name to save as pb.
Source: https://www.tensorflow.org/tutorials/keras/save_and_load
What actually happens on model creation and on model compile? Why compile is not inside Model? What is happening in terms of tensorflow graph, session?
Example code:
# model creation
model = Model(inputs, outputs)
# model compile
model.compile(optimizer='adadelta', loss='binary_crossentropy')
model = Model(inputs, outputs)
The above statement groups the various layers together and defines the flow of data between the various layers. The optimizers, loss functions or metrics to be used for evaluation are not specified here.
model.compile(optimizer='adadelta', loss='binary_crossentropy')
The above statement, model.compile() is basically used to config the model with losses and metrics.
So this question might stem from a lack of knowledge about tensorflow. But I am trying to build a multilayer perceptron with tensorflow 2.0, but without Keras.
The reason being that it is a requirement for my machine learning course that we do not use keras. Why you might ask? I am not sure.
I already have implemented our model in tensorflow 2.0 with Keras ease, and now I want to do the exact same thing without keras.
model = Sequential()
model.add(Dense(64, activation='relu', input_dim=784))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(5, activation='softmax'))
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
optimizer=Adam(),
metrics=['accuracy'])
X_train = X[:7000]
y_train = tf.keras.utils.to_categorical(y[:7000], num_classes=5)
X_dev = X[7000:]
y_dev = tf.keras.utils.to_categorical(y[7000:], num_classes=5)
model.fit(X_train, y_train,
epochs=100,
batch_size=128)
score = model.evaluate(X_dev, y_dev, batch_size=128)
print(score)
Here is my problem. Whenever I look up the documentation on Tensorflow 2.0, then even the guides on custom training are using Keras.
As placeholders and sessions are a thing of the past in tensorflow 2.0, as I understand it, then I am a bit unsure of how to structure it.
I can make tensor objects. I have the impression that I need to use eager execution and use gradient tape. But I still am unsure of how to put these things together.
Now my question is. Where should I look to get a better understanding? Which direction has the greatest descent?
Please do tell me if I am doing this stack overflow post wrong. It is my first time here.
As #Daniel Möller stated, there are these tutorials for custom training and custom layers on the official TensorFlow page. As stated on the custom training page:
This tutorial used tf.Variable to build and train a simple linear model.
There is also this blog that creates custom layers and training without Keras API. You can check this code on Google Colab, which uses Cifar-10 with custom layers and training in the same manner.
The question:
With a keras model (partly) specified such as this:
# create model
model = Sequential()
model.add(Dense(12, input_dim=8, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Fit the model
model.fit(X, Y, epochs=150, batch_size=10)
Is it in any way possible to save all details in the model for later use?
The details:
I've been following an example from machinelearningmastery.com and trying to modify and add traits / arguments of the model such as
activation='relu'
activation='sigmoid'
metrics=['accuracy']
And as the question suggests, I'd like to store model settings for later use.
I understand that these arguments are parts of different functions, but shouldn't it be possible all the same?
What I've tried:
1. model.save() and model.load()
Only returns
AttributeError: 'Sequential' object has no attribute 'load'
2. model.get_config()
Here I've been able to find some of the settings such as:
[{'class_name': 'Dense', 'config': {'activation': 'relu',
But I haven't found a way to load that config as a standalone model, and more often than not, I can't seem to find all settings.
3. I've also checked other posts such as Keras - Reuse weights from a previous layer - converting to keras tensor, but all aspects of the model don't seem to be covered.
Any suggestions?
Instead of trying model.load() try using load_model() provided by keras to load the model you saved using model.save()
from keras.models import load_model
load_model(filepath)
Also you can save the model as json using model.to_json() and load from json using model_from_json()
You can see more ways to save and load a model in Keras Documentation
here
model.save() will do the trick to save the model, to load it use from keras.models import load_model and use model=load_model(model_name) to load the model
I'm training a modified InceptionV3 model with the multi_gpu_model in Keras, and I use model.save to save the whole model.
Then I closed and restarted the IDE and used load_model to reinstantiate the model.
The problem is that I am not able to resume the training exactly where I left off.
Here is the code:
parallel_model = multi_gpu_model(model, gpus=2)
parallel_model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
history = parallel_model.fit_generator(generate_batches(path), steps_per_epoch = num_images/batch_size, epochs = num_epochs)
model.save('my_model.h5')
Before the IDE closed, the loss is around 0.8.
After restarting the IDE, reloading the model and re-running the above code, the loss became 1.5.
But, according to the Keras FAQ, model_save should save the whole model (architecture + weights + optimizer state), and load_model should return a compiled model that is identical to the previous one.
So I don't understand why the loss becomes larger after resuming the training.
EDIT: If I don't use the multi_gpu_model and just use the ordinary model, I'm able to resume exactly where I left off.
When you call multi_gpu_model(...), Keras automatically sets the weights of your model to some default values (at least in the version 2.2.0 which I am currently using). That's why you were not able to resume the training at the same point as it was when you saved it.
I just solved the issue by replacing the weights of the parallel model with the weights from the sequential model:
parallel_model = multi_gpu_model(model, gpus=2)
parallel_model.layers[-2].set_weights(model.get_weights()) # you can check the index of the sequential model with parallel_model.summary()
parallel_model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
history = parallel_model.fit_generator(generate_batches(path), steps_per_epoch = num_images/batch_size, epochs = num_epochs)
I hope this will help you.
#saul19am When you compile it, you can only load the weights and the model structure, but you still lose the optimizer_state. I think this can help.