In my input_shape there is a variable x that goes from 0 to 100, and I want to separate the neural network such that for values smaller than 50 it uses some weights and for values bigger than 50 it uses other weights. I imagine something like this:
inputs = keras.Input(shape=(3,)) # let's say x is the first of the three variables
if inputs[0] < 50:
x = layers.Dense(10)(inputs)
output_small = layers.Dense(10)(x)
else:
x = layers.Dense(10)(inputs)
output_big = layers.Dense(10)(x)
model1 = keras.Model(inputs, output_small, name="small_values")
model2 = keras.Model(inputs, output_big, name="big_values")
I necessarily need the if inside the model because I later want to use output_small and output_big in a same Dense layer, therefore I can't run the two samples separately. Would this work or is it possible in another way using Keras? Otherwise is it possible using some other tool?
Yes you should be able to manage conditional branching and merging of layers in a model with functional API.
Link to guide on functional API: https://keras.io/guides/functional_api/
Related
I have to create a network with keras like in the picture below, where NN - individual neural networks.
The problem is, that they all must have same weights
I can't use shared layers (at least to my understanding), because then one network will get all the inputs in it and I need each to get specially one
Is there any way of doing this?
Use the functional api. You can reuse a layer for different inputs. For example:
inp1 = Input(...)
inp2 = Input(...)
layer1 = Dense(...)
a1 = layer1(inp1)
a2 = layer1(inp2)
layer1 will be applied on inp1 and on inp2. It is just one layer instance, the same weights will be used for inp1 and inp2.
I am trying to test many ML models using keras.models.Sequential.
My idea is that once I have an iterator that looks like [num_layers, num_units_per_layers], for example [(1, 64),(2, (64,128))], to create a script using a kind of for loop running the iterator to be able to create a keras sequential model with the number of layers and units in each step of the iterator.
This is what I am trying:
it = [[(1, 128),(2, (64,128)), (3,(128,64,256))]]
for layers, units in it:
model = keras.Sequential([
layers.Dense(units[0])
#How to get another layers here when layers > 1.
])
But I am stuck when adding new layers automatically. To sum up, what I want in each step of the iterator is the keras model represented by its values.
Is there any way to do this?
For example, when layers = 2 and units = (64,128) the code should look like:
model = keras.Sequential([
layers.Dense(64),
layers.Dense(128)
])
If layers = 1 and units = 128 the code must be:
model = keras.Sequential([
layers.Dense(128)
])
Well the first issue is the way you set up it. The way you're doing it makes it a single list, where you want a list of n lists (here n is 3). If you define it as follows, you can extract layers, units the way you are looking for.
it = [[1,[128]],[2,(64,128)],[3,(128,64,256)]]
If you want a model with one layer, you need to put the number of units in brackets, or it won't work well with the other architectures (because of indexing). Next, there are some necessary tweaks to the code that I would suggest. First I would use a different way to build a Sequential model (shown below). Then, you would need to define your input shape otherwise the model will not know how to build. Finally, just create an output layer for each model outside the hidden layer generator loop.
I wrote this toy problem to fit your idea of iterating through models for 10 training samples and one input dimension and one output dimension.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
import numpy as np
x = np.random.rand(10,1)
y = np.random.rand(10,1)
it = [[1,[128]],[2,(64,128)],[3,(128,64,256)]]
for layers, units in it:
model = Sequential()
for i in range(layers):
model.add(Dense(units[i],input_shape=(1,)))
model.add(Dense(1))
model.summary()
model.compile(loss='mse',optimizer='Adam')
model.fit(x,y,batch_size=1,epochs=1)
Is there a way to pass a feature to a keras model as an input only to be accessed by a custom loss function without affecting the model as an input feature? I only need the feature to calculate the loss, not to feed-forward through the hidden layers in the network. (Basically what I want is to feed the feature in as an input and extract it as it is as an output along with y_pred to be accessed in the loss function).
A worked example would be much appreciated.
If you are writing your custom loss, you could use pass the feature as an input, and then using a Lambda layer, you can make it bypass the network and directly concatenate at the end. Something like the following -
from tensorflow.keras import layers, Model, utils
inp = layers.Input((11,))
x = layers.Lambda(lambda x: x[:,:-1])(inp)
o2 = layers.Lambda(lambda x: x[:,-1:])(inp)
x = layers.Dense(20)(x)
x = layers.Dense(20)(x)
o1 = layers.Dense(1)(x)
out = layers.concatenate([o1, o2])
model = Model(inp, out)
def custom_loss(outputs, actuals):
...
utils.plot_model(model, show_shapes=True, show_layer_names=False)
Here the first 10 features are the ones you want to pass via the network, and the last feature is the one you just want as is, for the custom loss. The final output is going to just be a concatenation of your expected output for the first 10 features via the network + the untouched feature.
If you want to know how to write a custom loss, please check this excellent SO post that explains it.
Is there a way in which we can enforce constraint on the prediction of sequences?
Say, if my modeling is as follows:
model = Sequential()
model.add(LSTM(150, input_shape=(n_timesteps_in, n_features)))
model.add(RepeatVector(n_timesteps_in))
model.add(LSTM(150, return_sequences=True))
model.add(TimeDistributed(Dense(n_features, activation='linear')))
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['acc'])
Can I somehow capture the constraint that model.pred(x) <= x
The docs shows that we can add constraints to the network weights. However, they do not mention how to map relationship or constraints between input and output.
Never heard of it.... but there are quite a few ways you can implement that yourself using a functional API model and custom functions.
Below, there is a possible answer to this, but first, is this really the best to do??
If you're trying to create an autoencoder, you should not care about limiting the outputs. Otherwise, your model will not really learn much.
Maybe the best to do is simply normalizing the inputs first (between -1 and +1), and using the tanh activation at the end.
Funtional API model to preserve the input:
inputTensor = Input(n_timesteps_in, n_features)
out = LSTM(150, input_shape=)(inputTensor)
out = RepeatVector(n_timesteps_in)(out) #this line sounds funny in your model...
out = LSTM(150, return_sequences=True)(out)
out = TimeDistributed(Dense(n_features))(out)
out = Activation(chooseOneActivation)(out)
out = Lambda(chooseACustomFunction)([out,inputTensor])
model = Model(inputTensor,out)
model.compile(...)
Custom limit options
There are infinite ways of doing this, here are some examples that may or may not be what you need. But from this you are free to develop anything similar.
The options below limits the individual outputs to the respective individual inputs. But you may prefer to use all outputs confined to the maximum input instead.
If so, use this below: maxInput = max(originalInput, axis=1, keepdims=True)
1 - A simple stretched 'tanh':
You can simply define both top and bottom limits by using a tanh (that ranges from -1 to +1) and multiplying it by the inputs.
Use the Activation('tanh') layer, and the following custom function in the Lambda layer:
import keras.backend as K
def stretchedTanh(x):
originalOutput = x[0]
originalInput = x[1]
return K.abs(originalInput) * originalOutput
I'm not totally sure this will be a healthy option. If the idea is to create an autoencoder, this model will easily find a solution of outputing all tanh activations as close to 1 as possible, without really looking at the inputs.
2 - Modified 'relu'
First, you could simply clip your outputs based on the inputs, changing a relu activation. Use the Activation('relu')(out) in your model above, and the following custom function in the Lambda layer:
def modifiedRelu(x):
negativeOutput = (-1) * x[0] #ranging from -infinite to 0
originalInput = x[1]
#ranging from -infinite to originalInput
return negativeOutput + originalInput #needs the same shape between input and output
This may have a downside when everything goes above the limit and backpropagation gets unable to return. (A problem that might happen with 'relu').
3 - Half linear, half modified tanh
In this case, you don't need the Activation layer, or you can use it as 'linear'.
import keras.backend as K
def halfTanh(x):
originalOutput = x[0]
originalInput = x[1] #assuming all inputs are positive
#find the positive outputs and get a tensor with 1's at their positions
positiveOutputs = K.greater(originalOuptut,0)
positiveOutputs = K.cast(positiveOutputs,K.floatx())
#now the 1's are at the negative positions
negativeOutputs = 1 - positiveOutputs
tanhOutputs = K.tanh(originalOutput) #function limited to -1 or +1
tanhOutputs = originalInput * sigmoidOutputs #raises the limit from 1 to originalInput
#use the conditions above to select between the negative and the positive side
return positiveOutputs * tanhOutputs + negativeOutputs * originalOutputs
Keras provides an easy way to handle such trivial constraints. We could write out = Minimum()([out, input_tensor])
Complete example
import keras
from keras.layers.merge import Maximum, Minimum
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot
n_timesteps_in = 20
n_features=2
input_tensor = keras.layers.Input(shape = [n_timesteps_in, n_features])
out = keras.layers.LSTM(150)(input_tensor)
out = keras.layers.RepeatVector(n_timesteps_in)(out)
out = keras.layers.LSTM(150, return_sequences=True)(out)
out = keras.layers.TimeDistributed(keras.layers.Dense(n_features))(out)
out = Minimum()([out, input_tensor])
model = keras.Model(input_tensor, out )
SVG(model_to_dot(model, show_shapes=True, show_layer_names=True, rankdir='HB').create(prog='dot', format='svg'))
Here's the network structure of the model. It shows how the input and the output are used to compute clamped output.
I'm new in Keras and Neural Networks. I'm writing a thesis and trying to create a SimpleRNN in Keras as it is illustrated below:
As it is shown in the picture, I need to create a model with 4 inputs + 2 outputs and with any number of neurons in the hidden layer.
This is my code:
model = Sequential()
model.add(SimpleRNN(4, input_shape=(1, 4), activation='sigmoid', return_sequences=True))
model.add(Dense(2))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(data, target, epochs=5000, batch_size=1, verbose=2)
predict = model.predict(data)
1) Does my model implement the graph?
2) Is it possible to specify connections between neurons Input and Hidden layers or Output and Input layers?
Explanation:
I am going to use backpropagation to train my network.
I have input and target values
Input is a 10*4 array and target is a 10*2 array which I then reshape:
input = input.reshape((10, 1, 4))
target = target.reshape((10, 1, 2))
It is crucial for to able to specify connections between neurons as they can be different. For instance, here you can have an example:
1) Not really. But I'm not sure about what exactly you want in that graph. (Let's see how Keras recurrent layers work below)
2) Yes, it's possible to connect every layer to every layer, but you can't use Sequential for that, you must use Model.
This answer may not be what you're looking for. What exactly do you want to achieve? What kind of data you have, what output you expect, what is the model supposed to do? etc...
1 - How does a recurrent layer work?
Documentation
Recurrent layers in keras work with an "input sequence" and may output a single result or a sequence result. It's recurrency is totally contained in it and doesn't interact with other layers.
You should have inputs with shape (NumberOrExamples, TimeStepsInSequence, DimensionOfEachStep). This means input_shape=(TimeSteps,Dimension).
The recurrent layer will work internally with each time step. The cycles happen from step to step and this behavior is totally invisible. The layer seems to work just like any other layer.
This doesn't seem to be what you want. Unless you have a "sequence" to input. The only way I know if using recurrent layers in Keras that is similar to you graph is when you have a segment of a sequence and want to predict the next step. If that's the case, see some examples by searching for "predicting the next element" in Google.
2 - How to connect layers using Model:
Instead of adding layers to a sequential model (which will always follow a straight line), start using the layers independently, starting from an input tensor:
from keras.layers import *
from keras.models import Model
inputTensor = Input(shapeOfYourInput) #it seems the shape is "(2,)", but we must see your data.
#A dense layer with 2 outputs:
myDense = Dense(2, activation=ItsAGoodIdeaToUseAnActivation)
#The output tensor of that layer when you give it the input:
denseOut1 = myDense(inputTensor)
#You can do as many cycles as you want here:
denseOut2 = myDense(denseOut1)
#you can even make a loop:
denseOut = Activation(ItsAGoodIdeaToUseAnActivation)(inputTensor) #you may create a layer and call it with the input tensor in just one line if you're not going to reuse the layer
#I'm applying this activation layer here because since we defined an activation for the dense layer and we're going to cycle it, it's not going to behave very well receiving huge values in the first pass and small values the next passes....
for i in range(n):
denseOut = myDense(denseOut)
This kind of usage allows you to create any kind of model, with branches, alternative ways, connections from anywhere to anywhere, provided you respect the shape rules. For a cycle like that, inputs and outputs must have the same shape.
At the end, you must define a model from one or many inputs to one or many outputs (you must have training data to match all inputs and outputs you choose):
model = Model(inputTensor,denseOut)
But notice that this model is static. If you want to change the number of cycles, you will have to create a new model.
In this case, it would be as simple as repeating the loop step denseOut = myDense(denseOut) and creating another model2=Model(inputTensor,denseOut).
3 - Trying to create something like the image below:
I am supposing C and F will participate in all iterations. If not,
Since there are four actual inputs, and we are going to treat them all separately, let's create 4 inputs instead, all like (1,).
Your input array should be divided in 4 arrays, all being (10,1).
from keras.models import Model
from keras.layers import *
inputA = Input((1,))
inputB = Input((1,))
inputC = Input((1,))
inputF = Input((1,))
Now the layers N2 and N3, that will be used only once, since C and F are constant:
outN2 = Dense(1)(inputC)
outN3 = Dense(1)(inputF)
Now the recurrent layer N1, without giving it the tensors yet:
layN1 = Dense(1)
For the loop, let's create outA and outB. They start as actual inputs and will be given to the layer N1, but in the loop they will be replaced
outA = inputA
outB = inputB
Now in the loop, let's do the "passes":
for i in range(n):
#unite A and B in one
inputAB = Concatenate()([outA,outB])
#pass through N1
outN1 = layN1(inputAB)
#sum results of N1 and N2 into A
outA = Add()([outN1,outN2])
#this is constant for all the passes except the first
outB = outN3 #looks like B is never changing in your image....
Now the model:
finalOut = Concatenate()([outA,outB])
model = Model([inputA,inputB,inputC,inputF], finalOut)