I was testing with Keras Functional API because I need to migrate a model to Tensorflow LITE.
I built a model with 3 inputs and 3 outputs.
The model works if all inputs have the same number of observations. I don't understand that point because the are independent.
ValueError: All input arrays (x) should have the same number of samples. Got array shapes: [(10, 5), (20, 5), (30, 5)
I would like to build a model with several inputs with different number of observations. Is that possible?
import numpy as np
from keras.layers import Input, Dense
from keras.models import Model
capas = 3
data = [ np.random.random(size=(50,5)) for i in range(3)]
labels = [ np.random.random(size=(50,2)) for i in range(3)]
visible=[]
preds=[]
for i in range( capas):
visible.append(Input(shape=(5,)))
x=Dense(5, activation='relu')(visible[i])
x=Dense(10, activation='relu')(x)
preds.append( Dense(2)(x))
model = Model(inputs=visible,output=preds)
model.compile(optimizer='adam',
loss='mean_squared_error',
metrics=['accuracy'])
model.fit(data, labels,epochs=50)
It does not matter if the sub-models are each independent, because if you make a multi-input multi-output model, it is trained by combining (weighting) the losses of each model into a single loss from where gradient descent is performed, and this requires the same number of samples in each input and output.
Since you say that the models are each independent, you can train them independently, and then make a new model that combines the three models (and their trained weights) with multiple inputs and outputs.
Related
I'm trying to implement an LSTM model for DNA sequence classification, but at the moment it is unusable because of how long it takes to train (25 seconds per epoch over 6.5K sequences, about 4ms per sample, and we need to train several versions of the model over 100s of thousands of sequences).
DNA sequence can be represented as a string of A, C, G, and T, e.g. "ACGGGTGACAT" could be an example of a single DNA sequence. Each sequence belongs to one of two categories that I am trying to predict and each sequence contains 1000 characters.
Initially, my model did not include an Embedding layer and instead I manually converted each sequence into a one-hot encoded matrix (4 rows by 1000 columns) and the model didn't work great but was incredibly fast. At this point though I had seen online that using an embedding layer has clear advantages. So I added an embedding layer and instead of using the one-hot encoded matrix I converted the sequences into integers with each character represented by a different integer.
Indeed the model works much better now, but it is about 30 times slower and impossible to work with. Is there something I can do here to speed up the embedding layer?
Here are the functions for constructing and fitting my model:
from tensorflow.keras.layers import Embedding, Dense, LSTM, Activation
from tensorflow.keras import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical
def build_model():
# initialize a sequential model
model = Sequential()
# add embedding layer
model.add(Embedding(5, 1, input_length=1000, mask_zero=True))
# Add LSTM layer
model.add(
LSTM(5)
)
# Add Dense NN layer
model.add(
Dense(units=2)
)
model.add(Activation('softmax'))
optimizer = Adam(clipnorm=1.)
model.compile(
loss="categorical_crossentropy", optimizer=optimizer, metrics=['accuracy']
)
return model
def train_model(X_train, y_train, epochs, batch_size):
model = build_model()
# y_train is initially a list of zeroes and ones, needs to be converted to categorical
y_train = to_categorical(y_train)
history = model.fit(
X_train, y_train, epochs=epochs, batch_size=batch_size
)
return model, history
Any help will be greatly appreciated - after much googling and trial-and-error, I can't seem to speed this up.
A possible suggestion is to use a "cheaper" RNN, such as the SimpleRNN instead of LSTM. It has less parameters to train. In some simple testing, I got a ~3x speed up over LSTM, with the same Embedding processing as you currently have. Not sure if you can reduce the sequence length from 1000 to a lower number, but that might be a direction to explore as well. I hope this helps.
In my TensorFlow model I have some data that I feed into a stack of CNNs before it goes into a few fully connected layers. I have implemented that with Keras' Sequential model. However, I now have some data that should not go into the CNN and instead be fed directly into the first fully connected layer because that data contains some values and labels that are part of the input data but that data should not undergo convolutions as it is not image data.
Is such a thing possible with tensorflow.keras or should I do that with tensorflow.nn instead? As far as I understand Keras' sequential models is that the input goes in one end and comes out the other with no special wiring in the middle.
Am I correct that to do this I have to use tensorflow.concat on the data from the last CNN layer and the data that bypasses the CNNs before feeding it into the first fully connected layer?
Here is an simple example in which the operation is to sum the activations from different subnets:
import keras
import numpy as np
import tensorflow as tf
from keras.layers import Input, Dense, Activation
tf.reset_default_graph()
# this represents your cnn model
def nn_model(input_x):
feature_maker = Dense(10, activation='relu')(input_x)
feature_maker = Dense(20, activation='relu')(feature_maker)
feature_maker = Dense(1, activation='linear')(feature_maker)
return feature_maker
# a list of input layers, of course the input shapes can be different
input_layers = [Input(shape=(3, )) for _ in range(2)]
coupled_feature = [nn_model(input_x) for input_x in input_layers]
# assume you take the sum of the outputs
coupled_feature = keras.layers.Add()(coupled_feature)
prediction = Dense(1, activation='relu')(coupled_feature)
model = keras.models.Model(inputs=input_layers, outputs=prediction)
model.compile(loss='mse', optimizer='adam')
# example training set
x_1 = np.linspace(1, 90, 270).reshape(90, 3)
x_2 = np.linspace(1, 90, 270).reshape(90, 3)
y = np.random.rand(90)
inputs_x = [x_1, x_2]
model.fit(inputs_x, y, batch_size=32, epochs=10)
You can actually plot the model to gain more intuition
from keras.utils.vis_utils import plot_model
plot_model(model, show_shapes=True)
The model of the above code looks like this
With a little remodeling and the functional API you can:
#create the CNN - it can also be a sequential
cnn_input = Input(image_shape)
cnn_output = Conv2D(...)(cnn_input)
cnn_output = Conv2D(...)(cnn_output)
cnn_output = MaxPooling2D()(cnn_output)
....
cnn_model = Model(cnn_input, cnn_output)
#create the FC model - can also be a sequential
fc_input = Input(fc_input_shape)
fc_output = Dense(...)(fc_input)
fc_output = Dense(...)(fc_output)
fc_model = Model(fc_input, fc_output)
There is a lot of space for creativity, this is just one of the ways.
#create the full model
full_input = Input(image_shape)
full_output = cnn_model(full_input)
full_output = fc_model(full_output)
full_model = Model(full_input, full_output)
You can use any of the three models in any way you want. They share the layers and the weights, so internally they are the same.
Saving and loading the full model might be quirky. I'd probably save the other two separately and when loading create the full model again.
Notice also that if you save two models that share the same layers, after loading they will probably not share these layers anymore. (Another reason for saving/loading only fc_model and cnn_model, while creating full_model again from code)
I am very new to Keras, neural networks and machine learning having just started to learn yesterday. I decided to try predicting the experience over an hour (0 to 23) (for a game and my own generated data-set) that a user would earn. Currently running what I have the predictions seem to be very low and very poor. I have tried a relu activation, which produced predictions all to be zero and from a bit of research, LeakyReLU.
This is the code I have for the prediction model so far:
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LeakyReLU
import numpy
numpy.random.seed(7)
dataset = numpy.loadtxt("experience.csv", delimiter=",")
X = dataset[: ,0]
Y = dataset[: ,1]
model = Sequential()
model.add(Dense(12, input_dim = 1, activation=LeakyReLU(0.3)))
model.add(Dense(8, activation=LeakyReLU(0.3)))
model.add(Dense(1, activation=LeakyReLU(0.3)))
model.compile(loss = 'mean_absolute_error', optimizer='adam', metrics = ['accuracy'])
model.fit(X, Y, epochs=120, batch_size=10, verbose = 0)
predictions = model.predict(X)
rounded = [round(x[0]) for x in predictions]
print(rounded)
I have also tried playing around with the hidden levels of the network, but honestly have no idea how many there should be or a good way to justify an amount.
If it helps here is the data-set I have been using:
https://raw.githubusercontent.com/NightShadeII/xpPredictor/master/experience.csv
Thankyou for any help
Looking at your data it does not seem like a classification problem.
You have two options:
-> Look at the second column and bucket them depending on the ranges and make classes that can be predicted, for instance: 0, 1, 2 etc. Now it tries to train but does not have enough examples for millions of classes that it thinks you are trying to predict.
-> If you want real valued output and not classes, try using linear regression.
I am using Tensorflow 1.12 which has Keras integrated together with Python 3.6.x
I wish to use Keras for its simplicity of model building, but also would like to use data on the intermediate layer for visualization of feature maps and kernels to better understand how machine learning works(even though this is admittedly not so evident)
I am using the mnist data base and a very basic Keras model to try to do what I want to do.
Here is the code
import tensorflow as tf
from tensorflow.keras import layers
from tensorflow import keras
print(tf.VERSION)
print(tf.keras.__version__)
tf.keras.backend.clear_session()
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train_shaped = np.expand_dims(x_train, axis=3) / 255.0
x_test_shaped = np.expand_dims(x_test, axis=3) / 255.0
def create_model():
model = tf.keras.models.Sequential([
keras.layers.Conv2D(32, kernel_size=(4, 4),strides=(1,1),activation='relu', input_shape=(28,28,1)),
keras.layers.Dropout(0.5),
keras.layers.MaxPooling2D(pool_size=(2,2), strides=(2,2)),
keras.layers.Conv2D(24, kernel_size=(8, 8),strides=(1,1)),
keras.layers.Flatten(),
keras.layers.Dropout(0.5),
keras.layers.Dense(128, activation=tf.nn.relu),
keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer=tf.keras.optimizers.Adam(),
loss=tf.keras.losses.sparse_categorical_crossentropy,
metrics=['accuracy'])
return model
The above sets up the dataset and the model
Next I define my session for Tensorflow and do the training.
This all works fine but now I want to get my data for the, as example, the first layer out as ideally a numpy array on which I can do the visualization.
My model.layers[0].output gives me a Tensor of (?,25,25,32) as expected and now I try to do a eval() and thenafter a .numpy() method to get my result.
The error message is
You must feed a value for placeholder tensor 'conv2d_6_input' with dtype float and shape [?,28,28,1]
I am looking for help on how to get my data (32 feature maps of 25x25 pixels) out as numpy array for visualization.
sess = tf.Session(graph=tf.get_default_graph())
tf.keras.backend.set_session(sess)
with sess.as_default():
model = create_model()
model.summary()
model.fit(x_train_shaped[:10000], y_train[:10000], epochs=2,
batch_size=64, validation_split=.2,)
model.layers[0].output
print(model.layers[0].output.shape)
my_array = model.layers[0].output
my_array.eval()
tf.keras.backend.clear_session()
sess.close()
First of all, you must note that getting the output of a model or a layer only makes sense when you feed the input layers with some data. You get the model something (i.e. input data), you get something in return (i.e. output or feature map or activation map). That's why it would produce the following error:
You must feed a value for placeholder tensor 'conv2d_6_input'
You haven't fed the baby, so it would cry :)
Now, the idea of building a new Keras model is counterproductive. When you have a large model in the first place, one would like to plug in some kind of ready-made code that can get the output of the feature maps and visualize them. So this route seems not really interesting.
I think you are mistakenly thinking that when you construct a new model out of the layers of another model, a whole new model is cloned. That's not the case since the parameters of the layers would be shared.
Concretely, what you are looking for can be achieved like this:
viz_conv = Model(model.input, model.layers[0].output)
conv_active = viz_conv(my_input_data) # my_input_data is a numpy array of shape `(num_samples,28,28,1)`
All the parameters of viz_conv are shared with model and they have not been copied either. Under the hood they are using the same weight Tensors.
Alternatively, you could define a backend function to do this:
from tensorflow.keras import backend as K
viz_func = K.function([model.input], [any layer(s) you would like in the model])
output = viz_func([my_input_data])
This has been covered in Keras documentation and I highly recommend to read that as well.
I am learning my data set with the following code.
import numpy as np
from keras.models import Sequential
from keras.layers import Dense
np.random.seed(5)
dataset = np.loadtxt('path to dataset', delimeter=',')
x_train=dataset[:700,0:3]
y_train=dataset[:700,3]
x_test=dataset[700:,0:3]
y_test=dataset[700:,3]
model =Sequential()
model.add(Dense(12, input_dim=3, activate='relu'))
model.add(Dense(8, activate='relu'))
model.add(Dense(1, activate='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer = 'adam', metrics = ['acuuracy'])
model.fit(x_train,y_train,epochs = 100, batch_size =24)
score = model.evaluate(x_test,y_test)
The above learning data consists of three attributes and two classes.
I would like to use the learned model above to determine data that the class does not contain.
For example, the learning data is (1, 34, 23, 0).
And the data I put in the input is (3,56,33), so I want to classify the class
using the learned model with these property information.
I would be very grateful if you could give me a concrete method.
You can use model.predict() function to determine the class of any new datapoint. As in your case, if you want to predict the class of (3,56,33), you can write this after fitting the model,
model.fit(x_train,y_train,epochs = 100, batch_size =24)
prediction_prob = model.predict(np.array([3,56,33]))
Since, you are using sigmoid activation in the final layer, your output will be a single probability value between 0 and 1. You can then threshold the prediction to get the final class value.
prediction_prob = model.predict(np.array([3,56,33]))
class_name = int(round(prediction_prob[0]))
print(class_name)
Remember, that model.predict() takes a numpy array as input and outputs another numpy array of predictions.