I loaded a keras model and used its middle output as follows:
pretrain_model = load_model(path)
model = Model(inputs=pretrain_model.input, outputs=pretrain_model.layers[-2].output)
When I run:
>>model.output
<tf.Tensor 'pretrain_variable/dropout_2/cond/Merge:0' shape=(?, ?) dtype=float32>
>>model.output_shape
(None, 2000)
This influences my next step:
input = tf.placeholder(tf.float32,shape=(None,X_all.shape[1],X_all.shape[2]),name='X')
pretrain_output = pretrain_model(input) # pretrain_output shape should be none*2000, however it is none*none
output_y = Dense(units=y.shape[1])(pretrain_output) # this works
output_y = tf.keras.layers.Dense(units=y.shape[1])(pretrain_output) # won't work, cause pretrain_output shape is (none,none)
I cannot use tf.keras.layers.Dense directly due to Dimension equals none.
Can anyone teach me how to get the middle layer output with correct shape?
Related
I am trying to take out a single element out of one dimension, while keeping the shapes the same.
The shape of the tensor is: (BATCH_SIZE, N_STEPS, NUM_FEATURES)
I want to create a new tensor that is (BATCH_SIZE, 1, NUM_FEATURES), where 1 is the final step.
The input tensor shape is (None, 128,16)
I tried to create a new tensor with the following:
X = X[:,-1,:]
X's shape becomes (None, 16) , but I need this to be (None, 1,16)
Update: I got this to work with the following code:
s = tf.shape(X)
X = tf.reshape(X[:,-1,:],shape=[s[0],1,s[2]])
I want to reshape and resize an image in the first layers before using Conv2D and other layers. The input will be a flattend array. Here is my code:
#Create flat example image:
img_test = np.zeros((120,160))
img_test_flat = img_test.flatten()
reshape_model = Sequential()
reshape_model.add(tf.keras.layers.InputLayer(input_shape=(img_test_flat.shape)))
reshape_model.add(tf.keras.layers.Reshape((120, 160,1)))
reshape_model.add(tf.keras.layers.experimental.preprocessing.Resizing(28, 28, interpolation='nearest'))
result = reshape_model(img_test_flat)
result.shape
Unfortunately this code results in the error I added down below. What is the issue and how do I correctly reshape and resize the flattend array?
WARNING:tensorflow:Model was constructed with shape (None, 19200) for input Tensor("input_13:0", shape=(None, 19200), dtype=float32), but it was called on an input with incompatible shape (19200,).
InvalidArgumentError: Input to reshape is a tensor with 19200 values, but the requested shape has 368640000 [Op:Reshape]
EDIT:
I tried:
reshape_model = Sequential()
reshape_model.add(tf.keras.layers.InputLayer(input_shape=(None, img_test_flat.shape[0])))
reshape_model.add(tf.keras.layers.Reshape((120, 160,1)))
reshape_model.add(tf.keras.layers.experimental.preprocessing.Resizing(28, 28, interpolation='nearest'))
Which gave me:
WARNING:tensorflow:Model was constructed with shape (None, None, 19200) for input Tensor("input_19:0", shape=(None, None, 19200), dtype=float32), but it was called on an input with incompatible shape (19200,).
EDIT2:
I recieve the input in C++ from a 1D array and pass it with
// Copy value to input buffer (tensor)
for (size_t i = 0; i < fb->len; i++){
model_input->data.i32[i] = (int32_t) (fb->buf[i]);
so what I pass on to the model is a flat array.
Your use of shapes simply doesn't make sense here. The first dimension of your input should be the number of samples. Is it supposed to be 19,200, or 1 sample?
input_shape should omit the number of samples, so if you want 1 sample, input shape should be 19,200. If you have 19,200 samples, shape should be 1.
The reshaping layer also omits the number of samples, so Keras is confused. What exactly are you trying to do?
This seems to be roughly what you're trying to achieve but I would personally resize the image outside of the neural network:
import numpy as np
import tensorflow as tf
img_test = np.zeros((120,160)).astype(np.float32)
img_test_flat = img_test.reshape(1, -1)
reshape_model = tf.keras.Sequential()
reshape_model.add(tf.keras.layers.InputLayer(input_shape=(img_test_flat.shape[1:])))
reshape_model.add(tf.keras.layers.Reshape((120, 160,1)))
reshape_model.add(tf.keras.layers.Lambda(lambda x: tf.image.resize(x, (28, 28))))
result = reshape_model(img_test_flat)
print(result.shape)
TensorShape([1, 28, 28, 1])
Feel free to use the Resizing layer instead of the Lambda layer, I can't use it due to my Tensorflow version.
I am trying to train a RNN by batches.
The input input size
(10, 70, 3075),
where 10 is the batch size, 70 the time dimension, 3075 are the frequency dimension.
There are three outputs whose size is
(10, 70, 1025)
each, basically 10 spectrograms with size (70,1025).
I would like to train this RNN by regression, whose structure is
input_img = Input(shape=(70,3075 ) )
x = Bidirectional(LSTM(n_hid,return_sequences=True, dropout=0.5, recurrent_dropout=0.2))(input_img)
x = Dropout(0.2)(x)
x = Bidirectional(LSTM(n_hid, dropout=0.5, recurrent_dropout=0.2))(x)
x = Dropout(0.2)(x)
o0 = ( Dense(1025, activation='sigmoid'))(x)
o1 = ( Dense(1025, activation='sigmoid'))(x)
o2 = ( Dense(1025, activation='sigmoid'))(x)
The problem is that output dense layers cannot take into account three dimensions, they want something like (None, 1025), which I don't know how to provide, unless I concatenate along the time dimension.
The following error occurs:
ValueError: Cannot feed value of shape (10, 70, 1025) for Tensor u'dense_2_target:0', which has shape '(?, ?)'
Would be the batch_shape option useful in the input layer? I have actually tried it, but I've got the same error.
In this instance the second RNN is collapsing the sequence to a single vector because by default return_sequences=False. To make the model return sequences and run the Dense layer over each timestep separately just add return_sequences=True to the second RNN as well:
x = Bidirectional(LSTM(n_hid, return_sequences=True, dropout=0.5, recurrent_dropout=0.2))(x)
The Dense layers automatically apply to the last dimension so no need to reshape afterwards.
To get the right output shape, you can use the Reshape layer:
o0 = Dense(70 * 1025, activation='sigmoid')(x)
o0 = Reshape((70, 1025)))(o0)
This will output (batch_dim, 70, 1025). You can do exactly the same for the other two outputs.
...coming from TensorFlow, where pretty much any shape and everything is defined explicitly, I am confused about Keras' API for recurrent models. Getting an Elman network to work in TF was pretty easy, but Keras resists to accept the correct shapes...
For example:
x = k.layers.Input(shape=(2,))
y = k.layers.Dense(10)(x)
m = k.models.Model(x, y)
...works perfectly and according to model.summary() I get an input layer with shape (None, 2), followed by a dense layer with output shape (None, 10). Makes sense since Keras automatically adds the first dimension for batch processing.
However, the following code:
x = k.layers.Input(shape=(2,))
y = k.layers.SimpleRNN(10)(x)
m = k.models.Model(x, y)
raises an exception ValueError: Input 0 is incompatible with layer simple_rnn_1: expected ndim=3, found ndim=2.
It works only if I add another dimension:
x = k.layers.Input(shape=(2,1))
y = k.layers.SimpleRNN(10)(x)
m = k.models.Model(x, y)
...but now, of course, my input would not be (None, 2) anymore.
model.summary():
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) (None, 2, 1) 0
_________________________________________________________________
simple_rnn_1 (SimpleRNN) (None, 10) 120
=================================================================
How can I have an input of type batch_size x 2 when I just want to feed vectors with 2 values to the network?
Furthermore, how would I chain RNN cells?
x = k.layers.Input(shape=(2, 1))
h = k.layers.SimpleRNN(10)(x)
y = k.layers.SimpleRNN(10)(h)
m = k.models.Model(x, y)
...raises the same exception with incompatible dim sizes.
This sample here works:
x = k.layers.Input(shape=(2, 1))
h = k.layers.SimpleRNN(10, return_sequences=True)(x)
y = k.layers.SimpleRNN(10)(h)
m = k.models.Model(x, y)
...but then layer h does not output (None, 10) anymore, but (None, 2, 10) since it returns the whole sequence instead of just the "regular" RNN cell output.
Why is this needed at all?
Moreover: where are the states? Do they just default to 1 recurrent state?
The documentation touches on the expected shapes of recurrent components in Keras, let's look at your case:
Any RNN layer in Keras expects a 3D shape (batch_size, timesteps, features). This means you have timeseries data.
The RNN layer then iterates over the second, time dimension of the input using a recurrent cell, the actual recurrent computation.
If you specify return_sequences then you collect the output for every timestep getting another 3D tensor (batch_size, timesteps, units) otherwise you only get the last output which is (batch_size, units).
Now returning to your questions:
You mention vectors but shape=(2,) is a vector so this doesn't work. shape=(2,1) works because now you have 2 vectors of size 1, these shapes exclude batch_size. So to feed vectors of size to you need shape=(how_many_vectors, 2) where the first dimension is the number of vectors you want your RNN to process, the timesteps in this case.
To chain RNN layers you need to feed 3D data because that what RNNs expect. When you specify return_sequences the RNN layer returns output at every timestep so that can be chained to another RNN layer.
States are collection of vectors that a RNN cell uses, LSTM uses 2, GRU has 1 hidden state which is also the output. They default to 0s but can be specified when calling the layer using initial_states=[...] as a list of tensors.
There is already a post about the difference between RNN layers and RNN cells in Keras which might help clarify the situation further.
I'm trying to implement a tensorflow Estimator, and getting a shape mismatch error I don't know how to debug. I think I may be misunderstanding how to specify the tf.feature_column's shape. My intention is to create a model with 6010 inputs. Any suggestions would be appreciated.
def train_input_fn():
with np.load(TRAIN_NN_FEATURES) as train:
train_features = train['features']
train_labels = train['labels']
train_dataset = tf.data.Dataset.from_tensor_slices(
({'all_features': train_features}, train_labels))
train_iterator = train_dataset.make_one_shot_iterator()
return train_iterator.get_next()
all_features = tf.feature_column.numeric_column(
'all_features',
shape=(6010,),
dtype=tf.float64
)
estimator = tf.estimator.DNNClassifier(
feature_columns=[all_features],
hidden_units=[1024, 512, 256]
)
estimator.train(input_fn=train_input_fn)
When I run this, I get the following error:
InvalidArgumentError (see above for traceback): Input to reshape
is a tensor with 6010 values, but the requested shape has 36120100
[[Node: dnn/input_from_feature_columns/input_layer/all_features/Reshape =
Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]
(dnn/input_from_feature_columns/input_layer/all_features/ToFloat,
dnn/input_from_feature_columns/input_layer/all_features/Reshape/shape)]]
The shape of the data is as I expect, but the feature_column seems to be expecting its square.
>>> train_features.shape
(10737, 6010)
>>>train_labels.shape
(10737, 1)
>>> 36120100./6010
6010.0
My understanding is that Dataset.from_tensor_slices takes slices along axis 0 of the given tensor, which corresponds with the error message "Input to reshape is a tensor with 6010 values." But why is a shape with 36120100 values being requested?
I'd still like to know why the above wasn't working, or how to debug though.
The problem is with the tensor size produced from the train_iterator.get_next(). If the batch size is not specified, the iterator returns:
({'all_features': <tf.Tensor 'IteratorGetNext:0' shape=(6010,) dtype=float64>},
<tf.Tensor 'IteratorGetNext:1' shape=(1,) dtype=float64>)
... tuple. As you can see the features tensor shape is (6010,), which DNNClassifier interprets as the batch_size=6010 (by convention the first dimension is the batch size), and it still expects 6010 features. Hence the error: it can't reshape (6010,) to (6010, 6010).
In order to make it working you have to either reshape this tensor manually, or simply set the batch size by calling:
train_dataset = train_dataset.batch(16)
Even batch size 1 will do fine, as it will force the get_next tensor to be:
({'all_features': <tf.Tensor 'IteratorGetNext:0' shape=(?, 6010) dtype=float64>},
<tf.Tensor 'IteratorGetNext:1' shape=(?, 1) dtype=float64>)
... but you clearly want to set it bigger for efficiency.