Note: I already read keras forward pass with tensorflow variable as input but it did not help.
I'm training an auto-encoder unsupervised neural-network with Keras with the MNIST database:
import keras, cv2
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(60000, 784).astype('float32') / 255.0
x_test = x_test.reshape(10000, 784).astype('float32') / 255.0
model = Sequential()
model.add(Dense(100, activation='sigmoid', input_shape=(784,)))
model.add(Dense(10, activation='sigmoid'))
model.add(Dense(100, activation='sigmoid'))
model.add(Dense(784, activation='sigmoid'))
model.compile(loss='mean_squared_error', optimizer='sgd')
history = model.fit(x_train, x_train, batch_size=1, epochs=1, verbose=0)
Then I would like to get the output vector when the input vector is x_test[i]:
for i in range(100):
x = x_test[i]
a = model(x)
cv2.imshow('img', a.reshape(28,28))
cv2.waitKey(0)
but I get this error:
All inputs to the layer should be tensors.
How should I modify this code to do a forward pass of an input vector in the neural network, and get a vector in return?
Also how to get the activation after, say, the 2nd layer? i.e. don't propagate until the last layer, but get the output after the 2nd layer.
Example: input: vector of size 784, output: vector of size 10
To run a model after you've finished training it you need to use keras predict(). This will evaluate the graph, given your input data. Note that the input data must be the same dimensions as the specified model inputs, which in your case looks to be [None, 784]. Keras does not require you to specify the batch dimension but you still need a 2D array going in. Do something like..
x = x_test[5]
x = x[numpy.newaxis,:]
out_val = model.predict(x)[0]
if you just want to process a single value.
The numpy.newaxis is required to make a 2D array and thus match your input size. You can skip this if you pass in an array of values to evaluate all at once.
With Keras/Tensorflow, your model is a graph/function, not standard python procedural code. You can't call it with data directly. You need to create functions and then call the functions. To get the output from an intermediate layer you can do something like..
OutFunc = K.function([model.input], [model.layers[2].output])
out_val = OutFunc([x])[0]
again, keep in mind there is a batch dimension on the input which will be produced in the output. There's a number of posts on getting data from intermediate layers if you need some additional examples. For instance see Keras, How to get the output of each layer?
An other way to do this than the accepted answer: when x is just a (784,) or (784,1) numpy array, we can use this:
model.predict([[x]])
with a double [[...]].
Related
I would like to build a Recurrent Network using the functional API in Keras but change the output shape. For now, the output shape is (n,1), where n is the number of input vectors, and if I understood correctly the additional dimension represents the number of batches. I would like model.predict to have an output that is of shape (n,) (so that it has the same shape as y_test). I know how to reshape the output after running model.predict, but is there a way to change the network so that running model.predict already has the desired shape?
I also tried to use a Reshape layer but this did not change the output shape.
Would really appreciate any idea & help! Here is a toy example:
#random toy example
X=np.random.rand(100,3)
y=np.random.rand(100)
X_train, X_test = np.vsplit(X,[80])
y_train, y_test = np.split(y,[80])
#define model
inputs=Input(shape=(3,1))
h=Conv1D(filters=64,kernel_size=2, activation='relu')(inputs)
h=MaxPooling1D(pool_size=2)(h)
h=Flatten()(h)
h=Dense(50, activation='relu')(h)
outputs=Dense(1)(h)
model=Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam', loss='mse')
#fit model
model.fit(X_train, y_train,verbose=0)
print(model.predict(X_test).shape)
which prints the shape
(20, 1)
Using a Reshape Layer does not change the output shape. Am I using it wrong?
inputs=Input(shape=(3,1))
h=Conv1D(filters=64,kernel_size=2, activation='relu')(inputs)
h=MaxPooling1D(pool_size=2)(h)
h=Flatten()(h)
h=Dense(50, activation='relu')(h)
h=Dense(1)(h)
outputs=Reshape(target_shape=(1,))(h)
model=Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam', loss='mse')
#fit model
model.fit(X_train, y_train,verbose=0)
print(model.predict(X_test).shape)
As you understood, most of the tf.keras.layers layers work with the undefined batch dimension of size None. If you need to reshape your output this way, you need to use the Lambda layer.
outputs_1d = Lambda(lambda x: tf.squeeze(x))(outputs)
and,
model=Model(inputs=inputs, outputs=outputs_1d)
I need help with calculating derivatives for model output wrt inputs in Keras.
I want to add a regularization functional to the loss function. The regularizer contains the derivative of the classifier function. So I tried to take the derivative of model output. The model is a MLP with one hidden layer. The dataset is MNIST. When I compile the model and take the derivative, I get [None] as the result instead of the derivative function.
I have seen a similar post, but didn't get answer there either:
Taking derivative of Keras model wrt to inputs is returning all zeros
Here is my code. Please help me to solve the problem.
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense
from keras import backend as K
num_hiddenNodes = 1024
num_classes = 10
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(-1, 28 * 28)
X_train = X_train.astype('float32')
X_train /= 255
y_train = keras.utils.to_categorical(y_train, num_classes)
model = Sequential()
model.add(Dense(num_hiddenNodes, activation='softplus', input_shape=(784,)))
model.add(Dense(num_classes, activation='softmax'))
# Compile the model
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
logits = model.output
# logits = model.layers[-1].output
print(logits)
X = K.identity(X_train)
# X = tf.placeholder(dtype=tf.float32, shape=(None, 784))
print(X)
print(K.gradients(logits, X))
Here is the output for the code. The two parameters are Tensors. The gradients function returns None.
Tensor("dense_2/Softmax:0", shape=(?, 10), dtype=float32)
Tensor("Identity:0", shape=(60000, 784), dtype=float32)
[None]
You are computing the gradients respect to X_train, which is not an input variable to the computation graph. Instead you need to get the symbolic input tensor to the model, so try something like:
grads = K.gradients(model.output, model.input)
I have the following code,
from keras.models import Sequential
from keras.layers import Dense
import numpy as np
# load dataset
dataset = np.loadtxt("data.csv", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:, 0:8]
Y = dataset[:, 8]
# create model
model = Sequential()
model.add(Dense(8, activation="relu", input_dim=8, kernel_initializer="uniform"))
model.add(Dense(12, activation="relu", kernel_initializer="uniform"))
model.add(Dense(1, activation="sigmoid", kernel_initializer="uniform"))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Fit the model
model.fit(X, Y, epochs=150, batch_size=10, verbose=2)
# calculate predictions
test = np.array([6,148,72,35,0,33.6,0.627,50])
predictions = model.predict(test)
# round predictions
rounded = [round(x[0]) for x in predictions]
print(rounded)
When I run the program, it gives me the following error.
ValueError: Error when checking : expected dense_1_input to have
shape (None, 8) but got array with shape (8,1)
I know there are a lot of duplicates to this question, I have tried all of them but it still gives me the same errors. How do I solve it?
Eventhough we don't see the full error trace, I think that the model learns and the error comes at the line :
predictions = model.predict(test)
Please confirm this.
The prediction fails because what you should always feed the network with is a numpy array of shape (number_of_samples_to_predict, input_shape).
There is always an additionnal dimension at the beginning, this is where you pile up all of the samples that you want to predict. When there is only one sample, you still have to feed a [1, input_shape] array.
To fix this use define your test input like this :
test = np.array([[6,148,72,35,0,33.6,0.627,50]])
now test has shape (1,8) which should run as the model expects (?,8).
well i think that this is diabetes dataset, you will get desired output by doing
this
model.predict(np.array([[6,148,72,35,0,33.6,0.627,50]])) > 0.5
test = test.transpose
And now you will have test of desired shape.
I'm very new to keras and also to python.
I have a time series dataset with different sequence lengths (for example 1st sequence is 484000x128, 2nd sequence is 563110x128, etc)
I've put the sequences in 3D array.
My question is how to define the input shape, because I'm confused. I was using DL4J but the concept is different in defining the network configuration.
Here is my first trial code:
import numpy as np
from keras.models import Sequential
from keras.layers import Embedding,LSTM,Dense,Dropout
## Loading dummy data
sequences = np.array([[[1,2,3],[1,2,3]], [[4,5,6],[4,5,6],[4,5,6]]])
y = np.array([[[0],[0]], [[1],[1],[1]]])
x_test=np.array([[2,3,2],[4,6,7],[1,2,1]])
y_test=np.array([0,1,1])
n_epochs=40
# model configration
model = Sequential()
model.add(LSTM(100, input_shape=(3,1), activation='tanh', recurrent_activation='hard_sigmoid')) # 100 num of LSTM units
model.add(LSTM(100, activation='tanh', recurrent_activation='hard_sigmoid'))
model.add(Dense(1, activation='softmax'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
print(model.summary())
## training with batches of size 1 (each batch is a sequence)
for epoch in range(n_epochs):
for seq, label in zip(sequences, y):
model.train(np.array([seq]), [label]) # train a batch at a time..
scores=model.evaluate(x_test, y_test) # evaluate batch at a time..
Here is the docs on input shapes for LSTMs:
Input shapes
3D tensor with shape (batch_size, timesteps, input_dim), (Optional) 2D
tensors with shape (batch_size, output_dim).
Which implies that you you're going to need timesteps with a constant size for each batch.
The canonical way of doing this is padding your sequences using something like keras's padding utility
then you can try:
# let say timestep you choose: is 700000 and dimension of the vectors are 128
timestep = 700000
dims = 128
model.add(LSTM(100, input_shape=(timestep, dim),
activation='tanh', recurrent_activation='hard_sigmoid'))
I edited the answer to remove the batch_size argument. With this setup the batch size is unspecified, you could set that when you fitting the model (in model.fit()).
I wanted to train a single layered neural network using the cifar data set and the keras framework. Since each image for the data set is 32 by 32 by 3 I wasn't quite sure how to process the image using a single layered network with no convolution. I think that flattening each image to have a data set of shape N by 32*32*3 is the right thing to do. Thus I did the following:
#Flatten
X_train = X_train.reshape((X_train.shape[0],32*32*3))
X_test = X_test.reshape((X_test.shape[0],32*32*3))
then I just made a single layered network that matched the input dimension:
model.add(Dense(units_single_layer, input_shape=(32*32*3,)))
model.add(Activation('relu'))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
the code seems to compile fine and everything. Thus, is this the correct way to train a single layered Neural Network on an image data set without convolution? I guess the main thing that is throwing me off is that the image is a 3D tensor but a singled layer net would just treat it as a 1D vector regardless of its shape. Right?
Also, since Keras provided a flatten() function it seemed unclear to me if that was a preferred method to use due to efficiency or some other reason. However, I wasn't able to make that one work.
Also, this goes without saying but the softmax layer doesn't really count as another layer. Right? I want it to be single layer.
whole code:
from __future__ import print_function
from keras.datasets import cifar10
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
batch_size = 32
nb_classes = 10
units_single_layer = 100
nb_epoch = 200
data_augmentation = False
# input image dimensions
img_rows, img_cols = 32, 32
# The CIFAR10 images are RGB.
img_channels = 3
# The data, shuffled and split between train and test sets:
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
#Flatten
X_train = X_train.reshape((X_train.shape[0],32*32*3))
X_test = X_test.reshape((X_test.shape[0],32*32*3))
# Convert class vectors to binary class matrices.
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
model = Sequential()
#model.add( Flatten(input_shape=32*32*3) )
#model.add(Flatten())
#model.add(Flatten(100, input_shape=(32*32*3,)))
model.add(Dense(units_single_layer, input_shape=(32*32*3,)))
model.add(Activation('relu'))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
# Let's train the model using RMSprop
model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print('Not using data augmentation.')
model.fit(X_train, Y_train,
batch_size=batch_size,
nb_epoch=nb_epoch,
validation_data=(X_test, Y_test),
shuffle=True)
You're doing everything right, this is the way if you only want one dense layer. It accepts only 1D tensors so you did well to reshape.
The softmax isn't considered as a layer because it doesnt have any weights or parameters to train.
Just out of curiosity, why do you use relu on the inputs? Aren't they already supposed to be between 0 and 1?