I am doing a multiple embedding, and need to concatenate all the embedded layers together for training. However, I keep getting the indices[1,0] = 7 is not in [0.7) error.
Here is what I did:
models = []
i0 = Input(shape=(1,),name='model_store')
model_store = Embedding(1115,10,input_length=1)(i0)
model_store = Reshape(target_shape=(10,))(model_store)
models.append(model_store)
i1 = Input(shape=(1,),name='model_dow')
model_dow = Embedding(7,6,input_length=1)(i1)
model_dow = Reshape(target_shape=(6,))(model_dow)
models.append(model_dow)
i2 = Input(shape=(1,),name='model_promo')
model_promo = Dense(1,input_dim=1)(i2)
models.append(model_promo)
# there are 8 embedding and 3 dense layers in models.
# then, I do:
net = Concatenate()(models)
net = Dense(1000,kernel_initializer='uniform',activation='relu')(net)
# another dense layer
output = Dense(1,activation='relu')(net)
model = Model(inputs = [i0,i1,i2,...i10],outputs = output)
model.compile(loss='mean_absolute_error',optimizer='adam')
but when I do model.fit(), i get the indices[] = something not in [) error.
The inputs that go into i0,i1,...,i10 are like array([[1],[2],[3],...]), all length 1 inputs.
I have also tried to replace the Reshape() layers with Flatten() layers, but got the same error.
Someone,please help.
Well, i found the problem.
It's that I didn't feed the data in correct shape. In the Sequential API, the input data for multiplue network inputs should be a list of ndarrays (dict may also work). While it says a list of ndarrays should still work for Functional API, it didn't work in my case, probably due to some order issue. I used dictionary of ndarrys with input names ('model_store','model_dow'...) as keys, and it works.
Related
I am trying to test many ML models using keras.models.Sequential.
My idea is that once I have an iterator that looks like [num_layers, num_units_per_layers], for example [(1, 64),(2, (64,128))], to create a script using a kind of for loop running the iterator to be able to create a keras sequential model with the number of layers and units in each step of the iterator.
This is what I am trying:
it = [[(1, 128),(2, (64,128)), (3,(128,64,256))]]
for layers, units in it:
model = keras.Sequential([
layers.Dense(units[0])
#How to get another layers here when layers > 1.
])
But I am stuck when adding new layers automatically. To sum up, what I want in each step of the iterator is the keras model represented by its values.
Is there any way to do this?
For example, when layers = 2 and units = (64,128) the code should look like:
model = keras.Sequential([
layers.Dense(64),
layers.Dense(128)
])
If layers = 1 and units = 128 the code must be:
model = keras.Sequential([
layers.Dense(128)
])
Well the first issue is the way you set up it. The way you're doing it makes it a single list, where you want a list of n lists (here n is 3). If you define it as follows, you can extract layers, units the way you are looking for.
it = [[1,[128]],[2,(64,128)],[3,(128,64,256)]]
If you want a model with one layer, you need to put the number of units in brackets, or it won't work well with the other architectures (because of indexing). Next, there are some necessary tweaks to the code that I would suggest. First I would use a different way to build a Sequential model (shown below). Then, you would need to define your input shape otherwise the model will not know how to build. Finally, just create an output layer for each model outside the hidden layer generator loop.
I wrote this toy problem to fit your idea of iterating through models for 10 training samples and one input dimension and one output dimension.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
import numpy as np
x = np.random.rand(10,1)
y = np.random.rand(10,1)
it = [[1,[128]],[2,(64,128)],[3,(128,64,256)]]
for layers, units in it:
model = Sequential()
for i in range(layers):
model.add(Dense(units[i],input_shape=(1,)))
model.add(Dense(1))
model.summary()
model.compile(loss='mse',optimizer='Adam')
model.fit(x,y,batch_size=1,epochs=1)
def generator_model(self):
input_images = Input(shape=[64,64,1])
layer1= Conv2D(self.filter_size,self.kernel_size,(2,2),padding='same',use_bias=False,kernel_initializer='random_uniform')(input_images)
layer1=LeakyReLU(0.2)(layer1)
layer2= Conv2D(self.filter_size*2,self.kernel_size,(2,2),padding='same',use_bias=False,kernel_initializer='random_uniform')(layer1)
layer2=BatchNormalization()(layer2)
layer2=LeakyReLU(0.2)(layer2)
layer3=Conv2D(self.filter_size*4,self.kernel_size,(2,2),padding='same',use_bias=False,kernel_initializer='random_uniform')(layer2)
layer3=BatchNormalization()(layer3)
layer3=LeakyReLU(0.2)(layer3)
layer4=Conv2D(self.filter_size*8,self.kernel_size,(2,2),padding='same',use_bias=False,kernel_initializer='random_uniform')(layer3)
layer4=BatchNormalization()(layer4)
layer4=LeakyReLU(0.2)(layer4)
layer5=Conv2D(self.filter_size*16,self.kernel_size,(2,2),padding='same',use_bias=False,kernel_initializer='random_uniform')(layer4)
layer5=BatchNormalization()(layer5)
layer5=LeakyReLU(0.2)(layer5)
up_layer5 = Conv2DTranspose(self.filter_size*8,self.kernel_size,strides = (2,2),padding='same',use_bias=False)(layer5)
up_layer5=BatchNormalization()(up_layer5)
up_layer5=LeakyReLU(0.2)(up_layer5)
#shape = 4*4*512
up_layer5_concat = tf.concat([up_layer5,layer4],0)
up_layer6 = Conv2DTranspose(self.filter_size*4,self.kernel_size,strides = (2,2),padding='same',use_bias=False)(up_layer5_concat)
up_layer6 =BatchNormalization()(up_layer6)
up_layer6 =LeakyReLU(0.2)(up_layer6)
up_layer_6_concat = tf.concat([up_layer6,layer3],0)
up_layer7 = Conv2DTranspose(self.filter_size*2,self.kernel_size,strides = (2,2),padding='same',use_bias=False)(up_layer_6_concat)
up_layer7 =BatchNormalization()(up_layer7)
up_layer7 =LeakyReLU(0.2)(up_layer7)
up_layer_7_concat = tf.concat([up_layer7,layer2],0)
up_layer8 = Conv2DTranspose(self.filter_size,self.kernel_size,strides = (2,2),padding='same',use_bias=False)(up_layer_7_concat)
up_layer8 =BatchNormalization()(up_layer8)
up_layer8 =LeakyReLU(0.2)(up_layer8)
up_layer_8_concat = tf.concat([up_layer8,layer1],0)
output = Conv2D(3,self.kernel_size,strides = (1,1),padding='same',use_bias=False)(up_layer_8_concat)
final_output = LeakyReLU(0.2)(output)
model = Model(input_images,output)
model.summary()
return model
This is how my generator_model looks like, and I have followed a research paper to make the architecture. But, I am in problem with the error. I have checked the other solutions to given problem here in SO, but none of them worked for me as they are little bit different maybe. My guess, the problem is there with the tf.concat() function which should be put as tensorflow keras layer of Lambda, but I tried that too and of no help. Any help regarding this issue? Bugging me for 2 days now.
When you define a model using the Keras functional API, you must use the Keras Layers to build your model.
Therefore you are right, the problem is in your tf.concat invocation.
In the tf.keras.layers package, however, you can find the Concatenate layer, that uses the functional API too.
Thus, you can replace your concat layers from:
up_layer5_concat = tf.concat([up_layer5,layer4],0)
to
up_layer5_concat = tf.keras.layers.Concatenate()([up_layer5, layer4])
And so on for every other tf.concat invocation in your network
My model is a simple fully connected network like this:
inp=Input(shape=(10,))
d=Dense(64, activation='relu')(inp)
d=Dense(128,activation='relu')(d)
d=Dense(256,activation='relu')(d) #want to give input here, layer3
d=Dense(512,activation='relu')(d)
d=Dense(1024,activation='relu')(d)
d=Dense(128,activation='linear')(d)
So, after saving the model I want to give input to layer 3. What I am doing right now is this:
model=load_model('blah.h5') #above described network
print(temp_input.shape) #(16,256), which is equal to what I want to give
index=3
intermediate_layer_model = Model(inputs=temp_input,
outputs=model.output)
End_output = intermediate_layer_model.predict(temp_input)
But it isn't working, i.e. I am getting errors like incompatible input, inputs should be tuple etc. The error message is:
raise TypeError('`inputs` should be a list or tuple.')
TypeError: `inputs` should be a list or tuple.
Is there any way I can pass my own inputs in middle of network and get the output instead of giving an input at the start and getting output from the end? Any help will be highly appreciated.
First you must learn that in Keras when you apply a layer on an input, a new node is created inside this layer which connects the input and output tensors. Each layer may have multiple nodes connecting different input tensors to their corresponding output tensors. To build a model, these nodes are traversed and a new graph of the model is created which consists all the nodes needed to reach output tensors from input tensors (i.e. which you specify when creating a model: model = Model(inputs=[...], outputs=[...]).
Now you would like to feed an intermediate layer of a model and get the output of the model. Since this is a new data-flow path, we need to create new nodes for each layer corresponding to this new computational graph. We can do it like this:
idx = 3 # index of desired layer
input_shape = model.layers[idx].get_input_shape_at(0) # get the input shape of desired layer
layer_input = Input(shape=input_shape) # a new input tensor to be able to feed the desired layer
# create the new nodes for each layer in the path
x = layer_input
for layer in model.layers[idx:]:
x = layer(x)
# create the model
new_model = Model(layer_input, x)
Fortunately, your model consists of one-branch and we could simply use a for loop to construct the new model. However, for more complex models it may not be easy to do so and you may need to write more codes to construct the new model.
Here is another method for achieving the same result. Initially create a new input layer and then connect it to the lower layers(with weights).
For this purpose, first re-initialize these layers(with same name) and reload the corresponding weights from the parent model using
new_model.load_weights("parent_model.hdf5", by_name=True)
This will load the required weights from the parent model.Just make sure you name your layers properly beforehand.
idx = 3
input_shape = model.layers[idx].get_input_shape_at(0) layer
new_input = Input(shape=input_shape)
d=Dense(256,activation='relu', name='layer_3')(new_input)
d=Dense(512,activation='relu', name='layer_4'))(d)
d=Dense(1024,activation='relu', name='layer_5'))(d)
d=Dense(128,activation='linear', name='layer_6'))(d)
new_model = Model(new_input, d)
new_model.load_weights("parent_model.hdf5", by_name=True)
This method will work for complex models with multiple inputs or branches.You just need to copy the same code for required layers, connect the new inputs and finally load the corresponding weights.
You can easily use keras.backend.function for this purpose:
import numpy as np
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K
inp=Input(shape=(10,))
d=Dense(64, activation='relu')(inp)
d=Dense(128,activation='relu')(d)
d=Dense(256,activation='relu')(d) #want to give input here, layer3
d=Dense(512,activation='relu')(d)
d=Dense(1024,activation='relu')(d)
d=Dense(128,activation='linear')(d)
model = Model(inp, d)
foo1 = K.function(
[inp],
model.layers[2].output
)
foo2 = K.function(
[model.layers[2].output],
model.output
)
X = np.random.rand(1, 10)
X_intermediate = foo1([X])
print(np.allclose(foo2([X_intermediate]), model.predict(X)))
Sorry for ugly function naming - do it best)
I was having the same problem and the proposed solutions worked for me but I was looking for something more explicit, so here it is for future reference:
d1 = Dense(64, activation='relu')
d2 = Dense(128,activation='relu')
d3 = Dense(256,activation='relu')
d4 = Dense(512,activation='relu')
d5 = Dense(1024,activation='relu')
d6 = Dense(128,activation='linear')
inp = Input(shape=(10,))
x = d1(inp)
x = d2(x)
x = d3(x)
x = d4(x)
x = d5(x)
x = d6(x)
full_model = tf.keras.Model(inp, x)
full_model.summary()
intermediate_input = Input(shape=d3.get_input_shape_at(0)) # get shape at node 0
x = d3(intermediate_input)
x = d4(x)
x = d5(x)
x = d6(x)
partial_model = tf.keras.Model(intermediate_input, x)
partial_model.summary()
Reference:
https://keras.io/guides/functional_api/#shared-layers
I have a dataset with each data point having 4 images (different pixel sizes for each) that are correlated to each other. I want to do convolutions on them separately, and then combine the information for the 4 images and feed it to 1 dense network. How can I do this in keras functional API?
I also have 10 other features that are not images. I plan to feed it directly to the dense end of the network.
So what I want is:
4 independent conv layers
flatten
concatenate
Dense layers
1 Output
How can I provide the input to keras in such a way?
According to the description you provided, I think this is what you are looking for:
input_im1 = Input(...)
input_im2 = Input(...)
input_im3 = Input(...)
input_im4 = Input(...)
conv_im1 = Conv2D(...)(input_im1)
conv_im2 = Conv2D(...)(input_im2)
conv_im3 = Conv2D(...)(input_im3)
conv_im4 = Conv2D(...)(input_im4)
concat_conv = concatenate([conv_im1,conv_im2,conv_im3,conv_im4])
flatten_conv = Flatten()(concat_conv)
input_feat = Input(...)
concat_conv_feat = concatenate([flatten_conv, input_feat])
output = Dense(...)(concat_conv_feat)
model = Model([input_im1,input_im2,input_im3,input_im4,input_feat], output)
Though, I am not aware of the sizes of the input images and the parameters for each of the convolution layers. So you may need to modify the code above to adjust it to your exact requirements.
I'm new in Keras and Neural Networks. I'm writing a thesis and trying to create a SimpleRNN in Keras as it is illustrated below:
As it is shown in the picture, I need to create a model with 4 inputs + 2 outputs and with any number of neurons in the hidden layer.
This is my code:
model = Sequential()
model.add(SimpleRNN(4, input_shape=(1, 4), activation='sigmoid', return_sequences=True))
model.add(Dense(2))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(data, target, epochs=5000, batch_size=1, verbose=2)
predict = model.predict(data)
1) Does my model implement the graph?
2) Is it possible to specify connections between neurons Input and Hidden layers or Output and Input layers?
Explanation:
I am going to use backpropagation to train my network.
I have input and target values
Input is a 10*4 array and target is a 10*2 array which I then reshape:
input = input.reshape((10, 1, 4))
target = target.reshape((10, 1, 2))
It is crucial for to able to specify connections between neurons as they can be different. For instance, here you can have an example:
1) Not really. But I'm not sure about what exactly you want in that graph. (Let's see how Keras recurrent layers work below)
2) Yes, it's possible to connect every layer to every layer, but you can't use Sequential for that, you must use Model.
This answer may not be what you're looking for. What exactly do you want to achieve? What kind of data you have, what output you expect, what is the model supposed to do? etc...
1 - How does a recurrent layer work?
Documentation
Recurrent layers in keras work with an "input sequence" and may output a single result or a sequence result. It's recurrency is totally contained in it and doesn't interact with other layers.
You should have inputs with shape (NumberOrExamples, TimeStepsInSequence, DimensionOfEachStep). This means input_shape=(TimeSteps,Dimension).
The recurrent layer will work internally with each time step. The cycles happen from step to step and this behavior is totally invisible. The layer seems to work just like any other layer.
This doesn't seem to be what you want. Unless you have a "sequence" to input. The only way I know if using recurrent layers in Keras that is similar to you graph is when you have a segment of a sequence and want to predict the next step. If that's the case, see some examples by searching for "predicting the next element" in Google.
2 - How to connect layers using Model:
Instead of adding layers to a sequential model (which will always follow a straight line), start using the layers independently, starting from an input tensor:
from keras.layers import *
from keras.models import Model
inputTensor = Input(shapeOfYourInput) #it seems the shape is "(2,)", but we must see your data.
#A dense layer with 2 outputs:
myDense = Dense(2, activation=ItsAGoodIdeaToUseAnActivation)
#The output tensor of that layer when you give it the input:
denseOut1 = myDense(inputTensor)
#You can do as many cycles as you want here:
denseOut2 = myDense(denseOut1)
#you can even make a loop:
denseOut = Activation(ItsAGoodIdeaToUseAnActivation)(inputTensor) #you may create a layer and call it with the input tensor in just one line if you're not going to reuse the layer
#I'm applying this activation layer here because since we defined an activation for the dense layer and we're going to cycle it, it's not going to behave very well receiving huge values in the first pass and small values the next passes....
for i in range(n):
denseOut = myDense(denseOut)
This kind of usage allows you to create any kind of model, with branches, alternative ways, connections from anywhere to anywhere, provided you respect the shape rules. For a cycle like that, inputs and outputs must have the same shape.
At the end, you must define a model from one or many inputs to one or many outputs (you must have training data to match all inputs and outputs you choose):
model = Model(inputTensor,denseOut)
But notice that this model is static. If you want to change the number of cycles, you will have to create a new model.
In this case, it would be as simple as repeating the loop step denseOut = myDense(denseOut) and creating another model2=Model(inputTensor,denseOut).
3 - Trying to create something like the image below:
I am supposing C and F will participate in all iterations. If not,
Since there are four actual inputs, and we are going to treat them all separately, let's create 4 inputs instead, all like (1,).
Your input array should be divided in 4 arrays, all being (10,1).
from keras.models import Model
from keras.layers import *
inputA = Input((1,))
inputB = Input((1,))
inputC = Input((1,))
inputF = Input((1,))
Now the layers N2 and N3, that will be used only once, since C and F are constant:
outN2 = Dense(1)(inputC)
outN3 = Dense(1)(inputF)
Now the recurrent layer N1, without giving it the tensors yet:
layN1 = Dense(1)
For the loop, let's create outA and outB. They start as actual inputs and will be given to the layer N1, but in the loop they will be replaced
outA = inputA
outB = inputB
Now in the loop, let's do the "passes":
for i in range(n):
#unite A and B in one
inputAB = Concatenate()([outA,outB])
#pass through N1
outN1 = layN1(inputAB)
#sum results of N1 and N2 into A
outA = Add()([outN1,outN2])
#this is constant for all the passes except the first
outB = outN3 #looks like B is never changing in your image....
Now the model:
finalOut = Concatenate()([outA,outB])
model = Model([inputA,inputB,inputC,inputF], finalOut)