How do I implement product AND in Theano? Mathematically that is the equivalent of multiply all of the previous layer (no weights). I think my code works for a batch size of 1 but I want it to work for batches.
Here is what I have tried. Note, I have no clue what I am doing.
Product AND function
def prod_and(result, k):
elif k[0][0] == 7:
return theano.tensor.stack([
[result[i][0] * result[i][1]*\
result[i][2] * result[i][3]*\
result[i][4] * result[i][5]*\
result[i][6]] for i in np.arange(1)
])
Product AND Layer
class ProdAnd(layers.BaseLayer):
# Begin by initializing.
def initialize(self,):
super(ProdAnd, self).initialize()
# Create output from input.
def output(self, *input_values):
return prod_and(input_values[0], self.input_shape)
I think that my problem arises from my inability to understand the relationship between neupy types (example what is connecting neupy layers {layers.0}). Also I don't think that I understand how Theano implements batch differently than stochastic.
Ideally the best answer would include the fix to the product and function as well as an explanation on how inputs and outputs work in this particular example.
Demonstration
An example where you would use this is a fuzzy-neuro network that utilizes Takagi-Sugeno style fuzzy inference (look at layer 5).
Im trying to make the problem easy so I broke it down into simply multiplying all of the input of the previous layer.
The equation is given by:
where x is the input layer and y is the output layer. Differentiating with respect to each variables leaves the other variable from the previous layer (so it is not a difficult computation).
Related
What does the following line of code do? How to interprete?
model.add(tf.keras.layers.Lambda(lambda x: x * 200))
My interpretation:
Lambda is like a function.
>>> f = lambda x: x + 1
>>> f(3)
4
In the second example the function is called using f(3). But what is the purpose of model.add?
The model.add method adds a layer to the associated Keras model. Now, the argument of this method usually is a Keras layer. In your case, it is a special kind of layer called Lambda. You are right that lambda is a function. In principle, lambda is common syntactic sugar that allows you to declare a simple function without naming it. It would be just like:
def my_func(x):
return x*200
model.add(tf.keras.layers.Lambda(my_func))
As you can see, this is way more code for a very basic functionality. Coming back to the Lambda layer, this just applies the given function to all of the nodes of the previous layer. If you don't understand what a Keras model is or how machine learning works, at least in a broad sense, you may want to start with some tutorials on that instead of looking into what the individual lines of code do. This way you could become productive way faster.
I bet it is used a a last layer. Normally, you can just a have a Dense layer output. However, you can help the training by scaling up the output to around the same figures as your labels. This will depend on the activation functions you used in your model. LSTM or SimpleRNN use tanh by default and that has an output range of [-1,1]. You will use this Lambda() layer to scale the output by 200 before it adjusts the layer weights.
I want to extract features of a optical image and save them into numpy array . I've seen similar questions , also can be seen here : https://keras.io/getting_started/faq/#how-can-i-obtain-the-output-of-an-intermediate-layer-feature-extraction , but don't know how to go about it .
Keras documentation does exaclty specify how to do that. If you have defined your model model_full you can create another one, that is just a part of it - from the input layer, to the one you're interested in.
model_part = Model(
inputs=model_full.input,
outputs=model_full.get_layer("intermed_layer").output)
Then you should be able to obtain output from intermediate layer using:
intermed_output = model_part(data)
In order to do that, you just need a model_full defined, which I assume you already have.
2nd approach
You can also use built-in Keras function, which I guess you already saw in documentation as well. It may look kind of complicated at first, but it's just creating a function with bound values i.e.
from keras import backend as K
get_3rd_layer_output = K.function(
[model.layers[0].input], # param 1 will be treated as layer[0].output
[model.layers[3].output]) # and this function will return output from 3rd layer
# here X is param 1 (input) and the function returns output from layers[3]
output = get_3rd_layer_output([X])[0]
Clearly, again model has to be defined. Not sure if there are any other requirements apart from that.
I'm trying to implement an INN (invertible neural network) with the structure as described in this paper.
I was wondering if it is possible to create a block (as proposed in the paper) as a custom keras layer with two different call functions.
The basic structur would look as follows:
import tensorflow as tf
import tensorflow.keras.layers as layers
class INNBlock(tf.keras.Model):
#inheriting from model instead of keras.layers.Layer, because I want manage the
#underlying layer as well
def __init__(self, size):
super(INNBlock, self).__init__(name='innblock')
#define layers
self.denseL1 = layers.Dense(size,activation='relu')
def call(self, inputs):
#define the relationship between the layers for a foward call
out = self.denseL1(inputs)
return out
def inverse_call(self, inputs):
#define inverse relationship between the layer
out = -self.denseL1(inputs) #use the same weights as the foward call
return out
class INN(tf.keras.Model):
def __init__(self,kenel_size,input_dim,min_clip,max_clip):
super(INN, self).__init__()
self.block_1 = INNBlock(size)
self.block_2 = INNBlock(size)
def call(self, inputs):
x = self.block_1(inputs)
x = self.block_2.inverse_call(y)
x = self.block_1.inverse_call(x)
return (y,x)
Solutions I already thought of (but don't particulary like):
Creating new layers for the inverse call and give them the same weights as the layers in the forward call.
Adding another dimension to inputs and have a variable in there, that determines whether or not the inverse call or the foward call is to be executed (but I don't know if this would even be allowed by keras)
I hope someone knows, if there is a way to implement this.
Thank you in advance :)
There is nothing wrong with your code. You can try it and it will run normally.
The call method is the standard method for when you simply do model_instance(input_tensor) or layer_instance(input_tensor).
But there is nothing wrong if you define another method and use that method inside the model's call method. What will happen is just:
If you use the_block(input_tensor), it will use the_block.call(input_tensor).
If you use the_block.inverse_call(input_tensor) somewhere outside a layer/model, it will fail to build a Keras model (nothing can be outside a layer)
If you use the_block.inverse_call(input_tensor) inside a layer/model (that's what you're doing), it is exactly the same as just writing the operations directly. You just wrapped it inside another function.
For Keras/Tensorflow, there will be nothing special about inverse_call. You can use it anywhere you could use any other keras/tensorflow function.
Will the gradients be updated twice?
Not exactly twice, but the operation will certainly be counted in. When the system calculates the gradient of the loss with relation to the weights, if the loss was built with inverse_call in the way, then it will participate in the gradient calculation.
But the update will be once per batch, as usual.
I just recently started playing around with Keras and got into making custom layers. However, I am rather confused by the many different types of layers with slightly different names but with the same functionality.
For example, there are 3 different forms of the concatenate function from https://keras.io/layers/merge/ and https://www.tensorflow.org/api_docs/python/tf/keras/backend/concatenate
keras.layers.Concatenate(axis=-1)
keras.layers.concatenate(inputs, axis=-1)
tf.keras.backend.concatenate()
I know the 2nd one is used for functional API but what is the difference between the 3? The documentation seems a bit unclear on this.
Also, for the 3rd one, I have seen a code that does this below. Why must there be the line ._keras_shape after the concatenation?
# Concatenate the summed atom and bond features
atoms_bonds_features = K.concatenate([atoms, summed_bond_features], axis=-1)
# Compute fingerprint
atoms_bonds_features._keras_shape = (None, max_atoms, num_atom_features + num_bond_features)
Lastly, under keras.layers, there always seems to be 2 duplicates. For example, Add() and add(), and so on.
First, the backend: tf.keras.backend.concatenate()
Backend functions are supposed to be used "inside" layers. You'd only use this in Lambda layers, custom layers, custom loss functions, custom metrics, etc.
It works directly on "tensors".
It's not the choice if you're not going deep on customizing. (And it was a bad choice in your example code -- See details at the end).
If you dive deep into keras code, you will notice that the Concatenate layer uses this function internally:
import keras.backend as K
class Concatenate(_Merge):
#blablabla
def _merge_function(self, inputs):
return K.concatenate(inputs, axis=self.axis)
#blablabla
Then, the Layer: keras.layers.Concatenate(axis=-1)
As any other keras layers, you instantiate and call it on tensors.
Pretty straighforward:
#in a functional API model:
inputTensor1 = Input(shape) #or some tensor coming out of any other layer
inputTensor2 = Input(shape2) #or some tensor coming out of any other layer
#first parentheses are creating an instance of the layer
#second parentheses are "calling" the layer on the input tensors
outputTensor = keras.layers.Concatenate(axis=someAxis)([inputTensor1, inputTensor2])
This is not suited for sequential models, unless the previous layer outputs a list (this is possible but not common).
Finally, the concatenate function from the layers module: keras.layers.concatenate(inputs, axis=-1)
This is not a layer. This is a function that will return the tensor produced by an internal Concatenate layer.
The code is simple:
def concatenate(inputs, axis=-1, **kwargs):
#blablabla
return Concatenate(axis=axis, **kwargs)(inputs)
Older functions
In Keras 1, people had functions that were meant to receive "layers" as input and return an output "layer". Their names were related to the merge word.
But since Keras 2 doesn't mention or document these, I'd probably avoid using them, and if old code is found, I'd probably update it to a proper Keras 2 code.
Why the _keras_shape word?
This backend function was not supposed to be used in high level codes. The coder should have used a Concatenate layer.
atoms_bonds_features = Concatenate(axis=-1)([atoms, summed_bond_features])
#just this line is perfect
Keras layers add the _keras_shape property to all their output tensors, and Keras uses this property for infering the shapes of the entire model.
If you use any backend function "outside" a layer or loss/metric, your output tensor will lack this property and an error will appear telling _keras_shape doesn't exist.
The coder is creating a bad workaround by adding the property manually, when it should have been added by a proper keras layer. (This may work now, but in case of keras updates this code will break while proper codes will remain ok)
Keras historically supports 2 different interfaces for their layers, the new functional one and the old one, that requires model.add() calls, hence the 2 different functions.
For the TF -- their concatenate() functions does not do everything that required for Keras to work, hence, the additional calls to make ._keras_shape variable correct and not to upset Keras that expects that variable to have some particular value.
This is a generic question. I found that in the tensorflow, after we build the graph, fetch data into the graph, the output from graph is a tensor. but in many cases, we need to do some computation based on this output (which is a tensor), which is not allowed in tensorflow.
for example, I'm trying to implement a RNN, which loops times based on data self property. That is, I need use a tensor to judge whether I should stop (I am not using dynamic_rnn since in my design, the rnn is highly customized). I find tf.while_loop(cond,body.....) might be a candidate for my implementation. But the official tutorial is too simple. I don't know how to add more functionalities into the 'body'. Can anyone give me few more complex example?
Also, in such case that if the future computation is based on the tensor output (ex: the RNN stop based on the output criterion), which is very common case. Is there an elegant way or better way instead of dynamic graph?
What is stopping you from adding more functionality to the body? You can build whatever complex computational graph you like in the body and take whatever inputs you like from the enclosing graph. Also, outside of the loop, you can then do whatever you want with whatever outputs you return. As you can see from the amount of 'whatevers', TensorFlow's control flow primitives were built with much generality in mind. Below is another 'simple' example, in case it helps.
import tensorflow as tf
import numpy as np
def body(x):
a = tf.random_uniform(shape=[2, 2], dtype=tf.int32, maxval=100)
b = tf.constant(np.array([[1, 2], [3, 4]]), dtype=tf.int32)
c = a + b
return tf.nn.relu(x + c)
def condition(x):
return tf.reduce_sum(x) < 100
x = tf.Variable(tf.constant(0, shape=[2, 2]))
with tf.Session():
tf.global_variables_initializer().run()
result = tf.while_loop(condition, body, [x])
print(result.eval())