What is the difference between tf.keras.layers versus tf.layers?
E.g. both of them have Conv2d, do they provide different outputs?
Is there any benefits if you mix them (something like a tf.keras.layers.Conv2d in one hidden layer and in the next, tf.layers.max_pooling2d)?
Since TensorFlow 1.12, tf.layers are merely wrappers around tf.keras.layers.
A few examples:
Convolutional tf.layers just inherit from the convolutional tf.keras.layers, see source code here:
#tf_export('layers.Conv2D')
class Conv2D(keras_layers.Conv2D, base.Layer):
The same is true for all core tf.layers, e.g.:
#tf_export('layers.Dense')
class Dense(keras_layers.Dense, base.Layer):
With the integration of Keras into TensorFlow, it would make little sense to maintain several different layer implementations. tf.keras is becoming the de-facto high-level API for TensorFlow, therefore tf.layers are now just wrappers around tf.keras.layers.
tf.keras.layers.Conv2d is a tensorflow-keras layer while tf.layers.max_pooling2d is a tensorflow 'native layer'
You cannot use a native layer directly within a Keras model, as it will be missing certain attributes required by the Keras API.
However, it is possible to use native layer if wrapped within a tensorflow-keras Lambda layer. A link to the documentation for this is below.
https://www.tensorflow.org/api_docs/python/tf/keras/layers/Lambda
tf.layers module is Tensorflow attempt at creating a Keras like API whereas tf.keras.layers is a compatibility wrapper. In fact, most of the implementation refers back to tf.layers, for example the tf.keras.layers.Dense inherits the core implementation:
#tf_export('keras.layers.Dense')
class Dense(tf_core_layers.Dense, Layer):
# ...
Because the tf.keras compatibility module is checked into the Tensorflow repo separately, it might lack behind what Keras actually offers. I would use Keras directly or tf.layers but not necessarily mix them.
Related
In some tf. keras tutorials, I've seen them instantiated their model class like this:
model = tf.keras.Sequential()
While in some places, they use something like this:
model = tf.keras.Model(inputs=input, outputs=output)
But seeing here in the docs, they do seem the same, but I am not sure nor is it explicitly mentioned. What are the differences between the two?
There are two class API to define a model in tf. keras. According to the doc
Sequential class: Sequential groups a linear stack of layers into a tf. keras.Model.
Model class: Model group's layers into an object with training and inference features.
An Sequential model is the simplest type of model, a linear stack of layers. But there are some flaws in using the sequential model API, it's limited in certain points. We can't build complex networks such as multi-input or multi-output networks using this API.
But using Model class, we can instantiate a Model with the Functional API (and also with Subclassing the Model class) that allows us to create arbitrary graphs of layers. From this, we can get more flexibility and easily define models where each layer can connect not just with the previous and next layers but also share feature information with other layers in the model, for example, model-like ResNet, EfficientNet.
In fact, most of the SOTA model that you can get from tf.keras.applications is basically implemented using the Functional API. However, in subclassing API, we define our layers in __init__ and we implement the model's forward pass in the call method.
Generally speaking, all the model definitions using Sequential API, can be achieved in Functional API or Model Subclassing API. And in Functional API or Model Subclassing API, we can create complex layers that not possible to achieve in Sequential API. If you wondering which one to choose, the answer is, it totally depends on your need. However, check out the following blog post where we have discussed the various model strategies in tf. keras with more examples. Model Sub-Classing and Custom Training Loop from Scratch in TensorFlow 2
It's because they are from different versions of tensorflow. According to the documentation (TensorFlow 2.0), tf.keras.Sequential is the most recent way of calling the function. If you go to the documentation, and click on "View aliases", you can see the different aliases used in older version of Tensorflow for that function.
In Tensorflow 2.0, the main "tensors" we see are in fact EagerTensors (tensorflow.python.framework.ops.EagerTensor to be more precise):
x = [[2.]]
m = tf.matmul(x, x)
type(m)
# returns tensorflow.python.framework.ops.EagerTensor
But, in some cases, we have the symbolic Tensor object (tensorflow.python.framework.ops.Tensor), as in TF1.X.
For example in keras:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
model = Sequential()
model.add(Dense(units=64, activation='relu', input_dim=100))
model.add(Dense(units=10, activation='softmax'))
type(model.outputs[0])
# returns tensorflow.python.framework.ops.Tensor
So, what are the use for these symbolic:tensorflow.python.framework.ops.Tensor in Tensorflow:
In the TF library internals: Keras is at least using these tensors, but is it used at other places (which are using a graph, like tf.function, or tf.data.Dataset)?
In the API: is there an actual use for end-users of these?
In the TF library internals: Keras is at least using these tensors, but is it used at other places (which are using a graph, like tf.function, or tf.data.Dataset)?
Well, yes. Your instinct is correct here. EagreTensor represents a tensor who's value has been calculated in eager mode whereas Tensor represents a tensor node in a graph that may not yet have been calculated.
In the API: is there an actual use for end-users of these?
Well, on some level we use them all the time. We create keras models, tf.data.Dataset pipelines etc. but, really, for the vast majority of use-cases we don't tend to instantiate or interact directly with the Tensor object itself though so probably just want to not worry about the object type and consider them to be an implementation detail internal to tensorflow.
Tensorflow has some docs for subclassing (tf) Keras Model and Layer.
However, it is unclear which to use for "modules" or "blocks" (e.g. several layers collectively).
Since it is technically several layers, I feel that subclassing Layer would be deceiving, and while subclassing Model works, I am unsure if there are any negative penalties for doing so.
e.g.
x = inputs
a = self.dense_1(x) # <--- self.dense_1 = tf.keras.Dense(...)
b = self.dense_2(a)
c = self.add([x, b])
which is appropriate to use?
(Please note that this answer is old, later, Keras changed to allow and use subclassing regularly)
Initially, there is no need to sublass anything with Keras. Unless you have a particular reason for that (which is not building, training, predicting), you don't subclass for Keras.
Buiding a Keras model:
Either using Sequential (the model is ready already, just add layers), or using Model (create a graph with layers and finally call Model(inputs, outputs)), you don't need to subclass.
The moment you create an instance of Sequential or Model, you have a fully defined model, ready to use in all situations.
This model can even be used as parts of other models, its layers can be easily accessed to get intermetiate outputs and create new branches in your graph.
So, I don't see any reason at all to subclass Model, unless you are using some additional framework that would require this (but I don't think so). This seems to be something from PyTorch users (because this kind of model building is typical for PyTorch, create a subclass for Module and add layers and a call method). But Pytorch doesn't offer the same ease as Keras does for getting intermediate results.
The main advantage of using Keras is exactly this: you can easily access layers from blocks and models and instantly start branching from that point without needing to rebuild any call methods or adding any extra code for that in the models. So, when you subclass Model, you just defeat the purpose of Keras making it all more difficult.
The docs you mentioned say:
Model subclassing is particularly useful when eager execution is enabled since the forward pass can be written imperatively.
I don't really understand what "imperatively" means, and I don't see how it would be easier than just building a model with regular layers.
Another quote from the docs:
Key Point: Use the right API for the job. While model subclassing offers flexibility, it comes at a cost of greater complexity and more opportunities for user errors. If possible, prefer the functional API.
Well... it's always possible.
Subclassing layers
Here, there may be good reasons to do so. And these reasons are:
You want a layer that performs custom calculations that are not available with regular layers
This layer must have persistent weights.
If you don't need "both" things above, you don't need to subclass a layer. If you just want "custom calculations" without weights, a Lambda layer is enough.
I wan to re-write TensorFlow code into Keras. I just wonder if you can use for this purpose the tf.keras.layers to just replace the tf.layers?
Like
tf.layers.max_pooling2d()
to:
tf.keras.layers.max_pooling2d()
Can I re-write TensorFlow to Keras in this way?
Does this define a proper Keras model where you can use the model.fit method?
First of all, I think you meant tf.keras.layers.MaxPool2D, which is a class, not a function. If I got your point, it shouldn't be an issue. There are some minor difference in syntax, but nothing serious. Besides, tf.keras.layers is a direct substitute for tf.layers. As per official docs, tf.layers are wrappers around tf.keras.layers. For example, convolutional layers in Layers API inherit from tf.keras.layers.
#tf_export('layers.Conv1D')
class Conv1D(keras_layers.Conv1D, base.Layer):
"""1D convolution layer (e.g. temporal convolution).
Even more so, Layers API is deprecated and will be removed from TF 2.0.
I'm trying to use a tensorflow op inside a Keras model. I previously tried to wrap it with a Lambda layer but I believe this disables that layers' backpropagation.
More specifically, I'm trying to use the layers from here in a Keras model, without porting it to Keras layers (I hope to deploy to tensorflow later on). I can compile these layers in a shared library form and load these into python. This gives me tensorflow ops and I don't know how to combine this in a Keras model.
A simple example of a Keras MNIST model, where for example one Conv2D layer is replaced by a tf.nn.conv2d op, would be exactly what I'm looking for.
I've seen this tutorial but it appears to do the opposite of what I am looking for. It seems to insert Keras layers into a tensorflow graph. I'm looking to do the exact opposite.
Best regards,
Hans
Roughly two weeks have passed and it seems I am able to answer my own question now.
It seems like tensorflow can look up gradients if you register them using this decorator. As of writing, this functionality is not (yet) available in C++, which is what I was looking for. A workaround would be to define a normal op in C++ and wrap it in a python method using the mentioned decorator. If these functions with corresponding gradients are registered with tensorflow, backpropagation will happen 'automagically'.