Related
I’m tuning hyperparameters of a keras deep learning model with the help of a config.json file containing hyperparameters.
{ “opt: “Adam”,
“lr”: 0.01,
“grad_clip”: 0.5
}
Keras allows to specify an optimizer in two ways:
As string argument in call to function without additional parameters.
model.compile(loss='categorical_crossentropy’,
optimizer=’Adam’,
metrics=['mse'])
As eponymous function with additional parameters.
model.compile(loss='categorical_crossentropy',
optimizer=optimizers.Adam(lr=0.01, clipvalue=0.5),
metrics=['mse'])
My question is: how to pass the optimizer (SGD, Adam etc.) as argument from config file along with the subparameters and employ the keras.optimizers.optimizer() function call as in (2)?
from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed, Bidirectional
from keras import optimizers
def train(X,y, opt, lr, clip):
model = Sequential()
model.add(Bidirectional(LSTM(100, return_sequences=True), input_shape=(500, 300)))
model.add(TimeDistributed(Dense(5, activation='sigmoid')))
model.compile(loss='categorical_crossentropy',
optimizer=optimizers.opt(lr=lr, clipvalue=clip),
metrics=['mse'])
model.fit(X, y, epochs=100, batch_size=1, verbose=2)
return(model)
When I try to pass parameters from my config file to the above train() function, I get the following error:
AttributeError: module 'keras.optimizers' has no attribute 'opt'
How do I parse the optimizer in string from as a function?
You could use a class that constructs an optimizer like so:
class Optimizer:
def __init__(self, lr, clip):
self.lr=lr
self.clip = clip
def get_opt(self, opt):
"""Dispatch method"""
method_name = 'opt_' + str(opt)
# Get the method from 'self'. Default to a lambda.
method = getattr(self, method_name, lambda: "Invalid optimizier")
# Call the method as we return it
return method()
def opt_Adam(self):
return optimizers.Adam(lr=self.lr, clipvalue=self.clip)
def opt_example(self):
return optimizers.example(lr=self.lr, clipvalue=self.clip)
#and so on for how many cases you would need
then you can call it as:
a=Optimizer(lr, clip)
model.compile(loss='categorical_crossentropy',
optimizer=a.get_opt(opt=opt),
metrics=['mse'])
You can initialize a json configuration file which contains the initialization of the optimizers:
eg:
"Adam": {
"lr":0.001,
"beta_1":0.9,
"beta_2":0.999,
"epsilon":None,
"decay":0.0,
"amsgrad":False
}
Then you can parse it from the configuration using the following lines:
with open('configuration.json') as json_data_file:
data = json.load(json_data_file)
in the data structure you will find the parameters setup of the optimizer:
optimizer = data["Adam"]
After all, you can access all the parameters of the chosen optimizer:
lr = data["lr"]
beta_1 = data["beta_1"]
etc...
An other way is to access the configuration of the optimizer using only the configuration file. Using Keras, you are able to compile your neural network with a specific optimizer chosen from a configuration file using an optimizer dispatcher:
optimizer= {"Adam": keras.optimizers.Adam(**config}
keep in mind that the keras optimizer name should be the the same in the config file.
Make sure the keys of your csl object (the config object) actually match those arguments of the classes. Then, the following will create the optimizer object, search for the appropriate arguments from the configuration object, and pass them to it
csl = { "opt": "Adam",
"lr": 0.01,
"grad_clip": 0.5}
optimizer = eval(f"keras.optimizers.{csl["opt"]}")()
optimizer = optimizer.from_config({k:v for k,v in csl.items() if hasattr(optimizer, k)})
I am building a simple neural network using Keras. It has activity regularization so that the output of the only hidden layer is forced to have small values. Here is the code:
import numpy as np
import math
import keras
from keras.models import Model, Sequential
from keras.layers import Input, Dense, Activation
from keras import regularizers
from keras import backend as K
a=1
def my_regularizer(inputs):
means=K.mean((inputs),axis=1)
return a*K.sum(means)**2
x_train=np.random.uniform(low=-1,high=1,size=(200,2))
model=Sequential([
Dense(20,input_shape=(2,),activity_regularizer=my_regularizer),
Activation('tanh'),
Dense(2,),
Activation('linear')
])
model.compile(optimizer='adam',loss='mean_squared_error')
model.fit(x_train,x_train,epochs=20,validation_split=0.1)
Questions:
1) Currently, parameter a is set at the beginning and it does not change. How can I change the code such that the parameter a is updated after each iteration such that
a_new=f(a_old,input)
where input is the values at the hidden layer and f(.) is an arbitrary function.
2) I want my activity regularizer to be applied after the first activation function tanh is applied. Have I written my code correctly? The term "activity_regularizer=my_regularizer" in
Dense(20,input_sahpe=(2,),activity_regularizer=my_regularizer)
makes me feel that the regularizer is being applied to values before the activation function tanh.
You can - but first, you need a valid Keras Regularizer object (your function won't work):
class MyActivityRegularizer(Regularizer):
def __init__(self, a=1):
self.a = K.variable(a, name='a')
# gets called at each train iteration
def __call__(self, x): # your custom function here
means = K.mean(x, axis=1)
return self.a * K.sum(means)**2
def get_config(self): # required class method
return {"a": float(K.get_value(self.a))}
Next, to work with .fit, you need a custom Keras Callback object (see alternative at bottom):
class ActivityRegularizerScheduler(Callback):
""" 'on_batch_end' gets automatically called by .fit when finishing
iterating over a batch. The model, and its attributes, are inherited by
'Callback' (except at __init__) and can be accessed via, e.g., self.model """
def __init__(self, model, update_fn):
self.update_fn=update_fn
self.activity_regularizers=_get_activity_regularizers(model)
def on_batch_end(self, batch, logs=None):
iteration = K.get_value(self.model.optimizer.iterations)
new_activity_reg = self.update_fn(iteration)
# 'activity_regularizer' references model layer's activity_regularizer (in this
# case 'MyActivityRegularizer'), so its attributes ('a') can be set directly
for activity_regularizer in self.activity_regularizers:
K.set_value(activity_regularizer.a, new_activity_reg)
def _get_activity_regularizers(model):
activity_regularizers = []
for layer in model.layers:
a_reg = getattr(layer,'activity_regularizer',None)
if a_reg is not None:
activity_regularizers.append(a_reg)
return activity_regularizers
Lastly, you'll need to create your model within the Keras CustomObjectScope - see in full ex. below.
Example usage:
from keras.layers import Dense
from keras.models import Sequential
from keras.regularizers import Regularizer
from keras.callbacks import Callback
from keras.utils import CustomObjectScope
from keras.optimizers import Adam
import keras.backend as K
import numpy as np
def make_model(my_reg):
return Sequential([
Dense(20, activation='tanh', input_shape=(2,), activity_regularizer=my_reg),
Dense(2, activation='linear'),
])
my_reg = MyActivityRegularizer(a=1)
with CustomObjectScope({'MyActivityRegularizer':my_reg}): # required for Keras to recognize
model = make_model(my_reg)
opt = Adam(lr=1e-4)
model.compile(optimizer=opt, loss='mse')
x = np.random.randn(320,2) # dummy data
y = np.random.randn(320,2) # dummy labels
update_fn = lambda x: .5 + .4*np.cos(x) #x = number of train updates (optimizer.iterations)
activity_regularizer_scheduler = ActivityRegularizerScheduler(model, update_fn)
model.fit(x,y,batch_size=32,callbacks=[activity_regularizer_scheduler],
epochs=4,verbose=1)
To TRACK your a and make sure it's changing, you can get its value at, e.g., each epoch end via:
for epoch in range(4):
model.fit(x,y,batch_size=32,callbacks=[activity_regularizer_scheduler],epochs=1)
print("Epoch {} activity_regularizer 'a': {}".format(epoch,
K.get_value(_get_activity_regularizers(model)[0].a)))
# My output:
# Epoch 0 activity_regularizer 'a': 0.7190816402435303
# Epoch 1 activity_regularizer 'a': 0.4982417821884155
# Epoch 2 activity_regularizer 'a': 0.2838689386844635
# Epoch 3 activity_regularizer 'a': 0.8644570708274841
Regarding (2), I'm afraid you're right - the 'tanh' outputs won't be used; you'll need to pass activation='tanh' instead.
Lastly, you can do it without a callback, via train_on_batch - but a drawback is, you'll need to feed data to the model yourself (and shuffle it, etc):
activity_regularizers = _get_activity_regularizers(model)
for iteration in range(100):
x, y = get_data()
model.train_on_batch(x,y)
iteration = K.get_value(model.optimizer.iterations)
for activity_regularizer in activity_regularizers:
K.set_value(activity_regularizer, update_fn(iteration))
I've keras model defined as follow
class ConvLayer(Layer) :
def __init__(self, nf, ks=3, s=2, **kwargs):
self.nf = nf
self.grelu = GeneralReLU(leak=0.01)
self.conv = (Conv2D(filters = nf,
kernel_size = ks,
strides = s,
padding = "same",
use_bias = False,
activation = "linear"))
super(ConvLayer, self).__init__(**kwargs)
def rsub(self): return -self.grelu.sub
def set_sub(self, v): self.grelu.sub = -v
def conv_weights(self): return self.conv.weight[0]
def build(self, input_shape):
# No weight to train.
super(ConvLayer, self).build(input_shape) # Be sure to call this at the end
def compute_output_shape(self, input_shape):
output_shape = (input_shape[0],
input_shape[1]/2,
input_shape[2]/2,
self.nf)
return output_shape
def call(self, x):
return self.grelu(self.conv(x))
def __repr__(self):
return f'ConvLayer(nf={self.nf}, activation={self.grelu})'
class ConvModel(tf.keras.Model):
def __init__(self, nfs, input_shape, output_shape, use_bn=False, use_dp=False):
super(ConvModel, self).__init__(name='mlp')
self.use_bn = use_bn
self.use_dp = use_dp
self.num_classes = num_classes
# backbone layers
self.convs = [ConvLayer(nfs[0], s=1, input_shape=input_shape)]
self.convs += [ConvLayer(nf) for nf in nfs[1:]]
# classification layers
self.convs.append(AveragePooling2D())
self.convs.append(Dense(output_shape, activation='softmax'))
def call(self, inputs):
for layer in self.convs: inputs = layer(inputs)
return inputs
I'm able to compile this model without any issues
>>> model.compile(optimizer=tf.keras.optimizers.Adam(lr=lr),
loss='categorical_crossentropy',
metrics=['accuracy'])
But when I query the summary for this model, I see this error
>>> model = ConvModel(nfs, input_shape=(32, 32, 3), output_shape=num_classes)
>>> model.summary()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-220-5f15418b3570> in <module>()
----> 1 model.summary()
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py in summary(self, line_length, positions, print_fn)
1575 """
1576 if not self.built:
-> 1577 raise ValueError('This model has not yet been built. '
1578 'Build the model first by calling `build()` or calling '
1579 '`fit()` with some data, or specify '
ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.
I'm providing input_shape for the first layer of my model, why is throwing this error?
The error says what to do:
This model has not yet been built. Build the model first by calling build()
model.build(input_shape) # `input_shape` is the shape of the input data
# e.g. input_shape = (None, 32, 32, 3)
model.summary()
There is a very big difference between keras subclassed model and other keras models (Sequential and Functional).
Sequential models and Functional models are datastructures that represent a DAG of layers. In simple words, Functional or Sequential model are static graphs of layers built by stacking one on top of each other like LEGO. So when you provide input_shape to first layer, these (Functional and Sequential) models can infer shape of all other layers and build a model. Then you can print input/output shapes using model.summary().
On the other hand, subclassed model is defined via the body (a call method) of Python code. For subclassed model, there is no graph of layers here. We cannot know how layers are connected to each other (because that's defined in the body of call, not as an explicit data structure), so we cannot infer input / output shapes. So for a subclass model, the input/output shape is unknown to us until it is first tested with proper data. In the compile() method, we will do a deferred compile and wait for a proper data. In order for it to infer shape of intermediate layers, we need to run with a proper data and then use model.summary(). Without running the model with a data, it will throw an error as you noticed. Please check GitHub gist for complete code.
The following is an example from Tensorflow website.
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
class ThreeLayerMLP(keras.Model):
def __init__(self, name=None):
super(ThreeLayerMLP, self).__init__(name=name)
self.dense_1 = layers.Dense(64, activation='relu', name='dense_1')
self.dense_2 = layers.Dense(64, activation='relu', name='dense_2')
self.pred_layer = layers.Dense(10, name='predictions')
def call(self, inputs):
x = self.dense_1(inputs)
x = self.dense_2(x)
return self.pred_layer(x)
def get_model():
return ThreeLayerMLP(name='3_layer_mlp')
model = get_model()
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 784).astype('float32') / 255
x_test = x_test.reshape(10000, 784).astype('float32') / 255
model.compile(loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=keras.optimizers.RMSprop())
model.summary() # This will throw an error as follows
# ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.
# Need to run with real data to infer shape of different layers
history = model.fit(x_train, y_train,
batch_size=64,
epochs=1)
model.summary()
Thanks!
Another method is to add the attribute input_shape() like this:
model = Sequential()
model.add(Bidirectional(LSTM(n_hidden,return_sequences=False, dropout=0.25,
recurrent_dropout=0.1),input_shape=(n_steps,dim_input)))
# X is a train dataset with features excluding a target variable
input_shape = X.shape
model.build(input_shape)
model.summary()
Make sure you create your model properly. A small typo mistake like the following code may also cause a problem:
model = Model(some-input, some-output, "model-name")
while the correct code should be:
model = Model(some-input, some-output, name="model-name")
If your Tensorflow, Keras version is 2.5.0 then just add Tensorflow when you import Keras package
Not this:
from tensorflow import keras
from keras.models import Sequential
import tensorflow as tf
Like this:
from tensorflow import keras
from tensorflow.keras.models import Sequential
import tensorflow as tf
Version issues of your Tensorflow, Keras, can be the reason for this.
Same problem I encountered during training for the LSTM model for regression.
Error:
ValueError: This model has not yet been built. Build the model first
by calling build() or by calling the model on a batch of data.
Earlier:
from tensorflow.keras.models import Sequential
from tensorflow.python.keras.models import Sequential
Corrected:
from keras.models import Sequential
I was also facing same error, so I have removed model.summary(). Then issue is resolved. As it arises if model of summary is defined before the model is built.
Here is the LINK for description which states that
Raises:
ValueError: if `summary()` is called before the model is built.**
I have a Tensorflow code for classifying images which I want to convert to Keras code. But I'm having trouble with the higher level API not having all codes which I desire. The problem which I have been stuck at is:
#net = get_vgg_model() <- got tf.VGG16 model
net = tf.keras.applications.VGG16()
g1 = tf.Graph()
with tf.Session(graph=g1, config=config) as sess, g1.device('/cpu:0'):
tf.import_graph_def(net['graph_def'], name='vgg')
this code gives the error:
Traceback (most recent call last):
File "app.py", line 16, in <module>
from modules.xvision import Xvision
File "/app/modules/xvision.py", line 84, in <module>
tf.import_graph_def(net['graph_def'], name='vgg')
TypeError: 'Model' object has no attribute '__getitem__'
Could someone help me with this graph?
Getting the graph
You can get the graph from Keras with:
import keras.backend as K
K.get_session().graph
You can probably pass it to import_graph_def, but I suspect it's already Tensorflow's default graph, since in the link below, the creator of Keras says there is only one graph.
More in: https://github.com/keras-team/keras/issues/3223
Working suggestion
I don't know what you're trying to achieve, but if the idea is using Keras regularly, you'd probably never need to grab the graph.
In Keras, once you created your model with net = tf.keras.applications.VGG16(), you'd start using Keras methods from this model, such as:
#compile for training
net.compile(optimizer=someKerasOptimizer, loss=someKerasLoss, metrics=[m1,m2])
#training
net.fit(trainingInputs, trainingTargets, epochs=..., batch_size=..., ...)
net.fit_generator(someGeneratorThatLoadsBatches, steps_per_epoch=...., ....)
#predicting
net.predict(inputs)
net.predict_generator(someGeneratorThatLoadsInputImages, steps=howManyBatches)
Accessing weights and layers would be done by:
layer = net.layers[index]
layer = net.get_layer('layer_name')
weights = layer.get_weights()
layer.set_weights(someWeightsList)
allWeights = net.get_weights()
net.set_weights(listWithAllWeights)
If what you are trying to do is import trained tensorflow to keras, First you have to consider that the naming of each keras model's variable matches with tensorflows model's variables. To show this I've created simple model with single input and single output without hidden layer. To deal with naming issue I've created the model using keras's layers and used keras to train the model.
inputs = tf.keras.layers.Input(shape=(1,), name="inputs")
outputs = tf.keras.layers.Dense(1, activation="linear", name="outputs")(inputs)
model = tf.keras.models.Model(inputs=inputs, outputs=outputs)
Now fit the network with dummy dataset, then save the model using tensorflow
model.compile(loss="mse", optimizer=tf.keras.optimizers.Adam(1e-1))
x = np.random.randn(1000) * 1000
y = x * .5 + 3
model.fit(x, y, epochs=20, batch_size=32)
with tf.keras.backend.get_session() as sess:
saver = tf.train.Saver()
meta_graph_def = tf.train.export_meta_graph(filename='./model.meta')
save_path = saver.save(sess, "./model.ckpt")
Now you can create the same model using keras and load weights using tensorflow as follows.
inputs = tf.keras.layers.Input(shape=(1,), name="inputs")
outputs = tf.keras.layers.Dense(1, activation="linear", name="outputs")(inputs)
model = tf.keras.models.Model(inputs=inputs, outputs=outputs)
sess = tf.keras.backend.get_session()
saver = tf.train.Saver()
saver.restore(sess, "./model.ckpt")
Now you can use your model for predicting or what ever you want.
print(model.predict([10, 2,4,5,6]))
# [[8.000007 ]
# [4.0000067]
# [5.0000067]
# [5.5000067]
# [6.0000067]]
I am using keras with tensor flow to implement a deep auto-encoder with CNN:
So basically the model would be similar to:
input_data = Input(shape=(40,500,1))
#encoder
x= Conv2D(32,kernel_size=(3,3), padding="same",activation='linear')(input_data)
encoded= Conv2D(15,kernel_size=(1,2), strides=(1,2), padding="same",activation='linear')(x)
#decoder
x= Conv2DTranspose(15,kernel_size=(1,2), padding="same",activation='linear')(encoded)
x= Conv2DTranspose(32,kernel_size=(3,3), padding="same",activation='linear')(x)
decoded = Conv2DTranspose(1, (3, 3), activation=activationfuntion, padding="same")(x)
autoencoder = Model(inputs=input_data,outputs=decoded)
encoder = Model(inputs=input_data,outputs=encoded)
In order to save the best model weights during training, I am using ModelCheckpoint:
autoencoder.compile(loss='mean_squared_error', optimizer='rmsprop');
checkpoint=ModelCheckpoint('bestweight.best.hdf5',monitor='val_loss',verbose=1,save_best_only=True,mode='min');
callbacks_list=[checkpoint]
history_info =autoencoder.fit(x_train, x_train,
batch_size=batch_size,
epochs=50,
validation_data=(x_validation,x_validation),
callbacks=callbacks_list,
shuffle=True)
and then later to test on the testdataset:
autoencoder.load_weights('bestweight.best.hdf5');
autoencoder.predict(test_data);
My question is:
I know how to save the best weights of the whole auto-encoder, but is there a way to just save the best training weights of the encoder part so I can use it later for testing.
so I can use it in this way:
encoder.load_weights('encoderbestweight.best.hdf5');
encoder.predict(test_data);
Before trying to answer your question, I would like to make a quick remark about your use of the ModelCheckpoint callback. Let's have a look at the default parameters :
keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', period=1)
The save_weights_only parameter's default value is False which means what you are actually saving is not only the model's weights but the entire architecture ! Thus, when loading the weights of your model you can either redefine the model and use load_weights. Or you can directly load your model from the file, using the load_model function.
Now, to save only the encoder, I would write a new checkpoint callback, like this :
class CustomCheckpoint(Callback):
def __init__(self, filepath, encoder):
self.monitor = 'val_loss'
self.monitor_op = np.less
self.best = np.Inf
self.filepath = filepath
self.encoder = encoder
def on_epoch_end(self, epoch, logs=None):
current = logs.get(self.monitor)
if self.monitor_op(current, self.best):
self.best = current
# self.encoder.save_weights(self.filepath, overwrite=True)
self.encoder.save(self.filepath, overwrite=True) # Whichever you prefer
As an alternative, since you already have the save file for the entire network, you can separate your encoder from the decoder like this :
from keras.models import load_model
autoencoder = load_model("path_to_file")
encoder = Model(autoencoder.layers[0].input, autoencoder.layers[1].output)
The encoder part is the first two layers. So after "autoencoder.fit()" try this
encoder = Model(input_data, autoencoder.layers[2].output)
for more "https://www.kaggle.com/marlesson/autoencoder-embedding-for-food"