We Keep Coding

How to make separate Keras model with dependent weights?

I have to create a network with keras like in the picture below, where NN - individual neural networks.
The problem is, that they all must have same weights
I can't use shared layers (at least to my understanding), because then one network will get all the inputs in it and I need each to get specially one
Is there any way of doing this?

Use the functional api. You can reuse a layer for different inputs. For example:
inp1 = Input(...)
inp2 = Input(...)
layer1 = Dense(...)
a1 = layer1(inp1)
a2 = layer1(inp2)
layer1 will be applied on inp1 and on inp2. It is just one layer instance, the same weights will be used for inp1 and inp2.

Realization of time dependent neural network in keras using for-loop

I have a recurrent model in keras that explicitly realized a for-loop over sequence. See model bellow [edited]
I have a large matrix input of size [samples~1M x time ~1k) and I need to use the same model which take as input a segment of that time neighbors ~ (-5,+5).
That writing seems:
Non-natural in term of Keras usage
for-loop might be inefficient and it is hard to tell the reason.
The question is: Should I and can I replace the use of for-loop within the model with something more built-in in keras layer?
Will that also help in terms of speed?
If so, how will that being done, is the TimeDistributed keras model the correct approach? Couldn't figure out how to take running time segment with that model.
Thanks
def my_model(signal_length = 1000 ,pre = 5 ,post = 5, num_classes=4):
# Define the input
X = Input(shape=(signal_length,))
# output as a list of size (slightly less) as X
outputs = []
# sliding window on X, considering each segment as input
for t in range(pre,signal_length-post):
# per time unit consider the input signal of the neighboring ~10 signals:
p1 = t - pre
p2 = t + post
Xt = Lambda(lambda x: X[:,p1:p2])(X)
# Connect to network:
Xt = Dense(20, activation='relu')(Xt)
Xt = Dense(20, activation='relu')(Xt)
Xt = Dense(num_classes, activation='softmax')(Xt)
#output per unit time in a list:
outputs.append(Xt)
model = Model(inputs=X, outputs=outputs)
mode.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy'])
return model

keras concatenate embedding layers gets index error

I am doing a multiple embedding, and need to concatenate all the embedded layers together for training. However, I keep getting the indices[1,0] = 7 is not in [0.7) error.
Here is what I did:
models = []
i0 = Input(shape=(1,),name='model_store')
model_store = Embedding(1115,10,input_length=1)(i0)
model_store = Reshape(target_shape=(10,))(model_store)
models.append(model_store)
i1 = Input(shape=(1,),name='model_dow')
model_dow = Embedding(7,6,input_length=1)(i1)
model_dow = Reshape(target_shape=(6,))(model_dow)
models.append(model_dow)
i2 = Input(shape=(1,),name='model_promo')
model_promo = Dense(1,input_dim=1)(i2)
models.append(model_promo)
# there are 8 embedding and 3 dense layers in models.
# then, I do:
net = Concatenate()(models)
net = Dense(1000,kernel_initializer='uniform',activation='relu')(net)
# another dense layer
output = Dense(1,activation='relu')(net)
model = Model(inputs = [i0,i1,i2,...i10],outputs = output)
model.compile(loss='mean_absolute_error',optimizer='adam')
but when I do model.fit(), i get the indices[] = something not in [) error.
The inputs that go into i0,i1,...,i10 are like array([[1],[2],[3],...]), all length 1 inputs.
I have also tried to replace the Reshape() layers with Flatten() layers, but got the same error.
Someone,please help.

Well, i found the problem.
It's that I didn't feed the data in correct shape. In the Sequential API, the input data for multiplue network inputs should be a list of ndarrays (dict may also work). While it says a list of ndarrays should still work for Functional API, it didn't work in my case, probably due to some order issue. I used dictionary of ndarrys with input names ('model_store','model_dow'...) as keys, and it works.

extracting Bottleneck features using pretrained Inceptionv3 - differences between Keras' implementation and Native Tensorflow implementation

(Apologies for the long post)
All,
I want to use the bottleneck features from a pretrained Inceptionv3 model to predict classification for my input images. Before training a model and predicting classification, I tried 3 different approaches for extracting the bottleneck features.
My 3 approaches yielded different bottleneck features (not just in values but even the size was different).
Size of my bottleneck features from Approach 1 and 2: (number of input images) x 3 x 3 x 2048
Size of my bottleneck features from Approach 3: (number of input images) x 2048
Why are the sizes different between the Keras based Inceptionv3 model and the native Tensorflow model? My guess is that when I say include_top=False in Keras, I'm not extracting the 'pool_3/_reshape:0' layer. Is this correct? If yes, how do I extract the 'pool_3/_reshape:0' layer in Keras? If my guess is incorrect, what 'am I missing?
I compared the bottleneck feature values from Approach 1 and 2 and they were significantly different. I think I'm feeding it the same input images because I resize and rescale my images before I even read it as input for my script. I have no options for my ImageDataGenerator in Approach 1 and according to the documentation for that function all the default values do not change my input image. I have set shuffle to false so I assumed that predict_generator and predict are reading images in the same order. What 'am I missing?
Please note:
My inputs images are in RGB format (so number of channels = 3) and I resized all of them to 150x150. I used the preprocess_input function in inceptionv3.py to preprocess all my images.
def preprocess_input(image):
image /= 255.
image -= 0.5
image *= 2.
return image
Approach 1: Used Keras with tensorflow as backend, an ImageDataGenerator to read my data and model.predict_generator to compute bottleneck features
I followed the example (Section Using the bottleneck features of a pre-trained network: 90% accuracy in a minute) from Keras' blog. Instead of VGG model listed there I used Inceptionv3. Below is the snippet of code I used
(code not shown here but what i did before the code below) : read all input images, resize to 150x150x3, rescale according to the preprocessing_input function mentioned above, save the resized and rescaled images
train_datagen = ImageDataGenerator()
train_generator = train_datagen.flow_from_directory(my_input_dir, target_size=(150,150),shuffle=False, batch_size=16)
# get bottleneck features
# use pre-trained model and exclude top layer - which is used for classification
pretrained_model = InceptionV3(include_top=False, weights='imagenet', input_shape=(150,150,3))
bottleneck_features_train_v1 = pretrained_model.predict_generator(train_generator,len(train_generator.filenames)//16)
Approach 2: Used Keras with tensorflow as backend, my own reader and model.predict to compute bottleneck features
Only difference between this approach and earlier one is that I used my own reader to read the input images.
(code not shown here but what i did before the code below) : read all input images, resize to 150x150x3, rescale according to the preprocessing_input function mentioned above, save the resized and rescaled images
# inputImages is a numpy array of size <number of input images x 150 x 150 x 3>
inputImages = readAllJPEGsInFolderAndMergeAsRGB(my_input_dir)
# get bottleneck features
# use pre-trained model and exclude top layer - which is used for classification
pretrained_model = InceptionV3(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))
bottleneck_features_train_v2 = pretrained_model.predict(trainData.images,batch_size=16)
Approach 3: Used tensorflow (NO KERAS) compute bottleneck features
I followed retrain.py to extract bottleneck features for my input images. Please note that that the weights from that script can be obtained from (http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz)
As mentioned in that example, I used the bottleneck_tensor_name = 'pool_3/_reshape:0' as the layer to extract and compute bottleneck features. Similar to the first 2 approaches, I used resized and rescaled images as input to the script and I called this feature list bottleneck_features_train_v3
Thank you so much

Different results between 1 and 2
Since you haven't shown your code, I (maybe wrongly) suggest that the problem is that you might not have used preprocess_input when declaring ImageDataGenerator ?
from keras.applications.inception_v3 import preprocess_input
train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
Make sure, though, that your saved image files range from 0 to 255. (Bit depth 24).
Different shapes between 1 and 3
There are three possible types of model in this case:
include_top = True -> this will return classes
include_top = False (only) -> this implies in pooling = None (no final pooling layer)
include_top = False, pooling='avg' or ='max' -> has a pooling layer
So, your declared model without an explicit pooling=something doesn't have the final pooling layer in keras. Then the outputs will still have the spatial dimensions.
Solve that simply by adding a pooling at the end. One of these:
pretrained_model = InceptionV3(include_top=False, pooling = 'avg', weights='imagenet', input_shape=(img_width, img_height, 3))
pretrained_model = InceptionV3(include_top=False, pooling = 'max', weights='imagenet', input_shape=(img_width, img_height, 3))
Not sure which one the model in the tgz file is using.
As an alternative, you can also get another layer from the Tensorflow model, the one coming immediately before 'pool_3'.

You can look into the Keras implementation of inceptionv3 here:
https://github.com/keras-team/keras/blob/master/keras/applications/inception_v3.py
so, the default parameter is:
def InceptionV3(include_top=True,
weights='imagenet',
input_tensor=None,
input_shape=None,
pooling=None,
classes=1000):
Notice that default for pooling=None, then when building the model, the code is:
if include_top:
# Classification block
x = GlobalAveragePooling2D(name='avg_pool')(x)
x = Dense(classes, activation='softmax', name='predictions')(x)
else:
if pooling == 'avg':
x = GlobalAveragePooling2D()(x)
elif pooling == 'max':
x = GlobalMaxPooling2D()(x)
# Ensure that the model takes into account
# any potential predecessors of `input_tensor`.
if input_tensor is not None:
inputs = get_source_inputs(input_tensor)
else:
inputs = img_input
# Create model.
model = Model(inputs, x, name='inception_v3')
So if you do not specify the pooling the bottleneck feature is extracted without any pooling, you need to specify if you want to get an average pooling or max pooling on top of these feature.

How to verify structure a neural network in keras model?

I'm new in Keras and Neural Networks. I'm writing a thesis and trying to create a SimpleRNN in Keras as it is illustrated below:
As it is shown in the picture, I need to create a model with 4 inputs + 2 outputs and with any number of neurons in the hidden layer.
This is my code:
model = Sequential()
model.add(SimpleRNN(4, input_shape=(1, 4), activation='sigmoid', return_sequences=True))
model.add(Dense(2))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(data, target, epochs=5000, batch_size=1, verbose=2)
predict = model.predict(data)
1) Does my model implement the graph?
2) Is it possible to specify connections between neurons Input and Hidden layers or Output and Input layers?
Explanation:
I am going to use backpropagation to train my network.
I have input and target values
Input is a 10*4 array and target is a 10*2 array which I then reshape:
input = input.reshape((10, 1, 4))
target = target.reshape((10, 1, 2))
It is crucial for to able to specify connections between neurons as they can be different. For instance, here you can have an example:

1) Not really. But I'm not sure about what exactly you want in that graph. (Let's see how Keras recurrent layers work below)
2) Yes, it's possible to connect every layer to every layer, but you can't use Sequential for that, you must use Model.
This answer may not be what you're looking for. What exactly do you want to achieve? What kind of data you have, what output you expect, what is the model supposed to do? etc...
1 - How does a recurrent layer work?
Documentation
Recurrent layers in keras work with an "input sequence" and may output a single result or a sequence result. It's recurrency is totally contained in it and doesn't interact with other layers.
You should have inputs with shape (NumberOrExamples, TimeStepsInSequence, DimensionOfEachStep). This means input_shape=(TimeSteps,Dimension).
The recurrent layer will work internally with each time step. The cycles happen from step to step and this behavior is totally invisible. The layer seems to work just like any other layer.
This doesn't seem to be what you want. Unless you have a "sequence" to input. The only way I know if using recurrent layers in Keras that is similar to you graph is when you have a segment of a sequence and want to predict the next step. If that's the case, see some examples by searching for "predicting the next element" in Google.
2 - How to connect layers using Model:
Instead of adding layers to a sequential model (which will always follow a straight line), start using the layers independently, starting from an input tensor:
from keras.layers import *
from keras.models import Model
inputTensor = Input(shapeOfYourInput) #it seems the shape is "(2,)", but we must see your data.
#A dense layer with 2 outputs:
myDense = Dense(2, activation=ItsAGoodIdeaToUseAnActivation)
#The output tensor of that layer when you give it the input:
denseOut1 = myDense(inputTensor)
#You can do as many cycles as you want here:
denseOut2 = myDense(denseOut1)
#you can even make a loop:
denseOut = Activation(ItsAGoodIdeaToUseAnActivation)(inputTensor) #you may create a layer and call it with the input tensor in just one line if you're not going to reuse the layer
#I'm applying this activation layer here because since we defined an activation for the dense layer and we're going to cycle it, it's not going to behave very well receiving huge values in the first pass and small values the next passes....
for i in range(n):
denseOut = myDense(denseOut)
This kind of usage allows you to create any kind of model, with branches, alternative ways, connections from anywhere to anywhere, provided you respect the shape rules. For a cycle like that, inputs and outputs must have the same shape.
At the end, you must define a model from one or many inputs to one or many outputs (you must have training data to match all inputs and outputs you choose):
model = Model(inputTensor,denseOut)
But notice that this model is static. If you want to change the number of cycles, you will have to create a new model.
In this case, it would be as simple as repeating the loop step denseOut = myDense(denseOut) and creating another model2=Model(inputTensor,denseOut).
3 - Trying to create something like the image below:
I am supposing C and F will participate in all iterations. If not,
Since there are four actual inputs, and we are going to treat them all separately, let's create 4 inputs instead, all like (1,).
Your input array should be divided in 4 arrays, all being (10,1).
from keras.models import Model
from keras.layers import *
inputA = Input((1,))
inputB = Input((1,))
inputC = Input((1,))
inputF = Input((1,))
Now the layers N2 and N3, that will be used only once, since C and F are constant:
outN2 = Dense(1)(inputC)
outN3 = Dense(1)(inputF)
Now the recurrent layer N1, without giving it the tensors yet:
layN1 = Dense(1)
For the loop, let's create outA and outB. They start as actual inputs and will be given to the layer N1, but in the loop they will be replaced
outA = inputA
outB = inputB
Now in the loop, let's do the "passes":
for i in range(n):
#unite A and B in one
inputAB = Concatenate()([outA,outB])
#pass through N1
outN1 = layN1(inputAB)
#sum results of N1 and N2 into A
outA = Add()([outN1,outN2])
#this is constant for all the passes except the first
outB = outN3 #looks like B is never changing in your image....
Now the model:
finalOut = Concatenate()([outA,outB])
model = Model([inputA,inputB,inputC,inputF], finalOut)