I have a subclassed model that instantiates a few custom layers via subclassing. I tried using keras.utils.plot_model() but all it does is print the model block, none of the layers appeared.
Can a Tensorflow expert comment on this? Will this feature ever be implemented in the future? If not, what is the next best alternative to examine the computation graph? Note that model.summary() only gives a summary of the parameters of the custom layer, within which contains two dense layers. Ideally, I like to see all the computations, if that is not asking too much...
Update: I dug into the source, looks like plot_model() first check for the _is_graph_network attribute. Graph Networks are used in Functional and Sequential APIs. From the source:
Two types of Networks exist: Graph Networks and Subclass Networks. Graph
networks are used in the Keras Functional and Sequential APIs. Subclassed
networks are used when a user subclasses the Model class. In general,
more Keras features are supported with Graph Networks than with Subclassed
Networks, specifically:
Model cloning (keras.models.clone())
Serialization (model.get_config()/from_config(), model.to_json()/to_yaml())
Whole-model saving (model.save())
(custom graph component)
Naturally, I like to know if I can build a graph network component, so my subclassed model/layer can work with these features. Does that involve a lot of effort?
(tf.function graph visualization)
Can someone let me know if graph visualization via Tensorboard works with Tensorflow2 tf.functions? In Tensorflow 1.x, one defines a name scope for a logical group of ops (e.g. generator/discriminator in GAN, encoder/decoder in VAE and loss/metrics), they are then displayed as a high-level block in the graph visualization. Can I define something similar for tf.functions?
Accodring to the officical documentation https://www.tensorflow.org/tensorboard/graphs,
you can
use TensorFlow Summary Trace API to log autographed functions for
visualization in TensorBoard
Here is a simple example for visuliazing a subclassed model:
import tensorflow as tf
from tensorflow.keras import layers
class MyModel(tf.keras.Model):
def __init__(self, num_classes=10):
super(MyModel, self).__init__(name='my_model')
self.num_classes = num_classes
self.dense_1 = layers.Dense(32, activation='relu')
self.dense_2 = layers.Dense(num_classes, activation='sigmoid')
def call(self, inputs):
x = self.dense_1(inputs)
return self.dense_2(x)
model = MyModel(num_classes=10)
model.compile(optimizer=tf.keras.optimizers.RMSprop(0.001),
loss='categorical_crossentropy',
metrics=['accuracy'])
data = np.random.random((1000, 32))
labels = np.random.random((1000, 10))
#tf.function
def trace():
data = np.random.random((1, 32))
model(data)
logdir = "trace_log"
writer = tf.summary.create_file_writer(logdir)
tf.summary.trace_on(graph=True, profiler=True)
# Forward pass
trace()
with writer.as_default():
tf.summary.trace_export(name="model_trace", step=0, profiler_outdir=logdir)
Then you can use Tensorboard to examine the computation graph:
tensorboard --logdir trace_log
Related
I'm working on supporting automatic model detection/logging for Keras/Tensorflow models for our Machine learning platform https://iko.ai and I have some questions:
What are the different ways we can define a tf/keras model ?
tf.keras.Model
tf.Estimator
tensorflow_estimator
Any other ways I'm not aware of? Why are there so many ways to do the same thing?
What are the proper functions to save/load them?
How could we differentiate TF/Keras model instances from other non-model objects? I want to be able to write a function that checks if an object is a TF/Keras model, something like
def is_tf_or_keras_model(obj):
# check somehow if the obj is a TF/Keras model
pass
Regarding questions 1 and 2:
Another way to represent a neural network model is by using tf.keras.Sequential. It allows you to easily create a model that follows a sequential structure, for example:
import tensorflow as tf
# tf.__version__ == 2.4
model_seq = tf.keras.Sequential()
model_seq.add(tf.keras.Input(shape=(64,1)))
model_seq.add(tf.keras.layers.Conv1D(32, 2, activation='relu'))
model_seq.add(tf.keras.layers.Dense(16, activation='relu'))
model_seq.add(tf.keras.layers.Dense(4, activation='softmax'))
This is the same as using the tf.keras functional API. This allows to build more complex networks that do not follow a sequential structure, but of course you can build sequential models anyway:
i = tf.keras.Input(shape=(64,1))
x = tf.keras.layers.Conv1D(32, 2, activation='relu')(i)
x = tf.keras.layers.Dense(16, activation='relu')(x)
x = tf.keras.layers.Dense(4, activation='softmax')(x)
model_func = tf.keras.Model(inputs=i, outputs=x)
tf.estimator.Estimator is used to train prebuilt networks, so you only have to define some hyperparameter values and train out of the box model.
In conclusion, the method used to build a model depends on how complex/ad-hoc the network is.
Regarding question 3:
The tf.keras implementation allows you to either save the weights of a model (model.save_weights) or save the whole model (model.save). To load the model weights afterwards you need to create the model first, following the same structure of the model used to train those weights.
Regarding question 4:
model_classes = [tf.keras.Model, keras.Model, tf.estimator.Estimator]
def is_keras_or_tf_model(obj, model_classes):
return isinstance(obj, model_clases)
All I have said here is in the TensorFlow documentation.
I want to train a model in a sequential manner. That is I want to train the model initially with a simple architecture and once it is trained, I want to add a couple of layers and continue training. Is it possible to do this in Keras? If so, how?
I tried to modify the model architecture. But until I compile, the changes are not effective. Once I compile, all the weights are re-initialized and I lose all the trained information.
All the questions in web and SO I found are either about loading a pre-trained model and continuing training or modifying the architecture of pre-trained model and then only test it. I didn't find anything related to my question. Any pointers are also highly appreciated.
PS: I'm using Keras in tensorflow 2.0 package.
Without knowing the details of your model, the following snippet might help:
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, Input
# Train your initial model
def get_initial_model():
...
return model
model = get_initial_model()
model.fit(...)
model.save_weights('initial_model_weights.h5')
# Use Model API to create another model, built on your initial model
initial_model = get_initial_model()
initial_model.load_weights('initial_model_weights.h5')
nn_input = Input(...)
x = initial_model(nn_input)
x = Dense(...)(x) # This is the additional layer, connected to your initial model
nn_output = Dense(...)(x)
# Combine your model
full_model = Model(inputs=nn_input, outputs=nn_output)
# Compile and train as usual
full_model.compile(...)
full_model.fit(...)
Basically, you train your initial model, save it. And reload it again, and wrap it together with your additional layers using the Model API. If you are not familiar with Model API, you can check out the Keras documentation here (afaik the API remains the same for Tensorflow.Keras 2.0).
Note that you need to check if your initial model's final layer's output shape is compatible with the additional layers (e.g. you might want to remove the final Dense layer from your initial model if you are just doing feature extraction).
I would like to train an InceptionV3 Neural Network from scratch. I already have an implementation running which utilizes this TensorFlow Hub module: https://tfhub.dev/google/imagenet/inception_v3/feature_vector/1 and performs fine-tuning utilizing the included pre-trained weights.
I now would like to use the same TensorFlow Hub module but discard the provided weights and use my own kernel initializer (e.g. tf.initializers.truncated_normal, tf.initializers.he_normal, etc...).
How can I modify the trainable variables in the TFHub module to use a custom initializer? Just to be clear, I want to replace the pre-trained weights at runtime, and just keep the model architecture. Please let me know if I really should be using TFSlim, or the model zoo.
Here is what I have so far:
import tensorflow as tf
import tensorflow_hub as hub
tfhub_module_url = 'https://tfhub.dev/google/imagenet/inception_v3/feature_vector/1'
initializer = tf.truncated_normal
def main(_):
_graph = tf.Graph()
with _graph.as_default():
module_spec = hub.load_module_spec(tfhub_module_url)
height, width = hub.get_expected_image_size(module_spec)
resized_input_tensor = tf.placeholder(tf.float32, [None, height, width, 3], name='resized_input_tensor')
m = hub.Module(module_spec, trainable=True)
bottleneck_tensor = m(resized_input_tensor)
trainable_vars = tf.trainable_variables()
# TODO: This fails, because this probably isn't how this is supposed to be done:
for trainable_var in trainable_vars:
trainable_var.initializer = tf.initializers.he_normal
with tf.Session(graph=_graph) as sess:
print(trainable_vars)
tf.logging.set_verbosity(tf.logging.INFO)
tf.app.run()
What is the correct way of doing this?
There is no straightforward way to do what you want to do, because TF Hub modules are really built to represent pre-trained model pieces. If you only want the graph, you could use tensorflow_models/slim code directly. (Or you could patch the tensorflow_hub library code to not rewire variable initializers with restore ops in the first place.)
EDIT 2019-04-15: See also tensorflow_hub issue #267: In TF2, the notion of initializers is going away, so the TF Hub Authors don't want to start depending on it for the TF1 API.
I currently have a custom LSTM model that I have saved as a .h5 file using save(). I am loading this model using load_model() during a tensorflow graph construction, and want to construct a part of the graph using the LSTM model's prediction output (which I therefore need in the form of a tensor). I have established the same session for the tensorflow graph and the keras backend graph, but I am having trouble connecting the output into my tensorflow code graph. Using the standard predict() seems to attempt to run the keras model's session, and I have scoured the internet for something other than hideously converting it to a .pb file and messing with it like that. It seems like it should be easy, considering I am using tensorflow as the Keras backend...Any ideas on how to achieve this?
I will show how to import saved keras model into tensorflow graph. I will show this using simple single layer feed forward model.
inputs = tf.keras.layers.Input(shape=(1,), name="inputs")
outputs = tf.keras.layers.Dense(1, activation="linear", name="outputs")(inputs)
model = tf.keras.models.Model(inputs=inputs, outputs=outputs)
model.compile(loss="mse", optimizer="adam")
model.save("model.h5")
Now let's load the model using load_model method of keras and use it in tensorflow to multiply the output of the model with new placeholder tensor.
model = tf.keras.models.load_model("model.h5")
model_output = model.output
new_tensor_ph = tf.placeholder(tf.float32, [None, 1])
new_output = tf.multiply(model_output, new_tensor_ph)
sess = tf.keras.backend.get_session()
prediction = sess.run(new_output, feed_dict={model.input:[[3]],new_tensor_ph :[[4]]})
## This works without error
I'm trying to implement an adversarial loss in keras.
The model consists of two networks, one auto-encoder (the target model) and one discriminator. The two models share the encoder.
I created the adversarial loss of the auto-encoder by setting a keras variable
def get_adv_loss(d_loss):
def loss(y_true, y_pred):
return some_loss(y_true, y_pred) - d_loss
return loss
discriminator_loss = K.variable()
L = get_adv_loss(discriminator_loss)
autoencoder.compile(..., loss=L)
and during training I interleave train_on_batch of discriminator and autoencoder to update discriminator_loss
d_loss = disciminator.train_on_batch(x, y_domain)
discriminator_loss.assign(d_loss)
a_loss, ... = self.segmenter.train_on_batch(x, y_target)
However, I found out that the value of these variables is frozen when the model is compiled. I tried to recompile the model during training but that raise the error
Node 'IsVariableInitialized_13644': Unknown input node
'training_12/Adam/Variable'
which I guess it means i cant recompile during training? any suggestion on how i can inject the discriminator loss in the autoencoder?
Keras model supports multiple outputs. So just include your discirminator into your keras model and freeze the discrminator layers, if the discriminator should not be trained.
The next question would be how to combine autoencoder loss and discriminator loss. Luckily keras model.compile supports loss weights. If autoencoder is your first output and discriminator is your second you could do something like loss_weights=[1, -1]. So a better discriminator is worse for the autoencoder.
Edit: Here is an example, how to implement an Adversary Network:
# Build your architecture
auto_encoder_input = Input((5,))
auto_encoder_net = Dense(10)(auto_encoder_input)
auto_encoder_output = Dense(5)(auto_encoder_net)
discriminator_net = Dense(20)(auto_encoder_output)
discriminator_output = Dense(5)(discriminator_net)
# Define outputs of your model
train_autoencoder_model = Model(auto_encoder_input, [auto_encoder_output, discriminator_output])
train_discriminator_model = Model(auto_encoder_input, discriminator_output)
# Compile the models (compile the first model and then change the trainable attribute for the second)
for layer_index, layer in enumerate(train_autoencoder_model.layers):
layer.trainable = layer_index < 3
train_autoencoder_model.compile('Adam', loss=['mse', 'mse'], loss_weights=[1, -1])
for layer_index, layer in enumerate(train_discriminator_model.layers):
layer.trainable = layer_index >= 3
train_discriminator_model.compile('Adam', loss='mse')
# A simple example how a training can look like
for i in range(10):
auto_input = np.random.sample((10,5))
discrimi_output = np.random.sample((10,5))
train_discriminator_model.fit(auto_input, discrimi_output, steps_per_epoch=5, epochs=1)
train_autoencoder_model.fit(auto_input, [auto_input, discrimi_output], steps_per_epoch=1, epochs=1)
As you can see there is no much magic behind building an Adversary Model with keras.
Unless you decide to go deep in the keras source code, I don't think you can do this easily. Before writing your own adversarial module, you should check the existing works carefully. As far as I know, keras-adversarial is still used by many people. Of course, it only supports old keras versions, e.g. 2.0.8.
Several other things:
be careful when you freeze your model weights. If you first compile a model and then freeze some weights, these weights are still trainable, because when the train function is generated during compiling. So you should freeze weights first then compile.
keras-adversarial does this job in a more elegant way. Instead of making two models, shared weights but freeze some weights in different ways, it creates two train functions, one for each player.