I want to create a RNN in Tensorflow that classifies short texts analyzing them on per-letter basis. For that I created a numpy 2D array, where each piece of text was either padded or truncated, where each element is a character code. An output is just vector of clasess represented as one-hot encoded numpy 2D-array.
Here is an example:
train_x.shape, train_y.shape
((91845, 50), (91845, 5))
Input consists of 90K rows 50 chars each, output is 90K rows with 5 classes. Next, I want to build a network shown in a figure below.
The structure looks trivial, but I deffinetelly lack knowledge in Tensorflow and run in all kinds of problems trying to at least do training. Here is the part of code I use to build the network
chars = sequence_categorical_column_with_identity('chars', params['domain_size']+1)
chars_emb = tf.feature_column.embedding_column(chars, dimension=10)
columns = [chars_emb]
input_layer, sequence_length = sequence_input_layer(features, columns)
hidden_units = 32
lstm = tf.nn.rnn_cell.LSTMCell(hidden_units, state_is_tuple=True)
rnn_outputs, state = tf.nn.dynamic_rnn(lstm,
inputs = input_layer,
sequence_length=sequence_length,
dtype=tf.float32)
output = rnn_outputs[:,-1,:]
logits = tf.layers.dense(output, params['n_classes'], activation=tf.nn.tanh)
# apply projection to every timestep.
# Compute predictions.
predicted_classes = tf.nn.softmax(logits)
# Compute loss.
loss = tf.nn.softmax_cross_entropy_with_logits_v2(labels=labels, logits=logits)
# Compute evaluation metrics.
accuracy = tf.metrics.accuracy(labels=labels,
predictions=predicted_classes,
name='acc_op')
But I get an error
InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 8 values, but the requested shape has 1
[[Node: Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](softmax_cross_entropy_with_logits, sequence_input_layer/chars_embedding/assert_equal/Const)]]
A fuller minimal example you can find here. Most likely you would need Tensorflow 1.8.0.
Adding
loss = tf.reduce_mean(loss)
now allows to train the network, but the results are underwhelming.
Related
I notice the layer LogisticEndpoint in https://www.tensorflow.org/guide/keras/train_and_evaluate#automatically_setting_apart_a_validation_holdout_set . The document build a model like this :
import numpy as np
inputs = keras.Input(shape=(3,), name="inputs")
targets = keras.Input(shape=(10,), name="targets")
logits = keras.layers.Dense(10)(inputs)
predictions = LogisticEndpoint(name="predictions")(logits, targets)
model = keras.Model(inputs=[inputs, targets], outputs=predictions)
model.compile(optimizer="adam") # No loss argument!
data = {
"inputs": np.random.random((3, 3)),
"targets": np.random.random((3, 10)),
}
model.fit(data)
My question is that how to use this model when inference , since we don't know the target when we use model.predict
The "LogisticEndpoint" is actually a layer. It takes the prediction and target as input, it can calculate the loss tracked by add_loss(), and calculate the precision scalar tracked by add_metric().
The target, if I am not wrong, is actually the ground truth data.
When you make the inference (testing stage, neither training nor validating), you do not need the ground truth data.
Just pass the input to the model, then take the output as the prediction.
To make the prediction of the multi-input:
First, convert the multi-input into one array (maybe a big one).
Then make sure the shape of the array (or precisely, tensor) matches the size of the input layer.
There is a part of the code in TF2 I am using for multi-input.
...
for view_id in range(1,9):
...
img = self.transform(img).float()
pose_2d = self.transform(pose_2d).float()
img_con.append(img)
pose_2d_con.append(pose_2d)
# then make the img_con and pose_2d_con two tensor.
I want to implement a GRU for sentiment analysis and this is what I have so far:
epochs = 20
batch_size = 25
embedding_size = 50
layers = 1
max_label = 2 # only valid target labels are 0 and 1
# one word is fed in at any time instance
embedding_matrix = tf.Variable(tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
embeddings = tf.nn.embedding_lookup(embedding_matrix, x)
# number of neurons in a single LSTM cell is equal to the embedding size
cell = tf.contrib.rnn.GRUCell(embedding_size)
cell = tf.contrib.rnn.DropoutWrapper(cell=cell, output_keep_prob=0.75)
# encoding fed into softmax prediction layer
output, states = tf.nn.dynamic_rnn(cell, embeddings, dtype=tf.float32)
logits = tf.layers.dense(output, max_label, activation=None)
cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y)
loss = tf.reduce_mean(cross_entropy)
# prediction is accurate if predicted label is equal to the actual label
prediction = tf.equal(tf.argmax(logits, 1), tf.cast(y, tf.int64))
accuracy = tf.reduce_mean(tf.cast(prediction, tf.float32))
But I get this error:
ValueError: Rank mismatch: Rank of labels (received 1) should equal rank of logits minus 1 (received 3).
I tried changing the value of max_label to 2 and 1 and tried to make logits=logits-1 but the error is still there. How can I fix it?
NB: This is without knowing the size of your label vector y nor which line the error message is coming from. Please include this important information in future questions.
According to the tensorflow 1.3 docs, the usage of nn.sparse_softmax_cross_entropy_with_logits is:
"... to have logits of shape [batch_size, num_classes] and labels of shape [batch_size]. But higher dimensions are supported."
From my experience with using RNN's in tensorflow, the size convention of your model will be [Batch, Length of Sequence, Feature Size]. So, your output will be 3 dimensional. According to the quote I've attached above, your output should then be of size [Batch, Length of Sequence], with each value being the index of your label (so if position (X,Y) is class 30, then, in your code, y[X,Y] = 30). This is different (from my memory) to the non-sparse version, which takes in one-hot (or multiple-hot) encodings.
I'm building an RNN to analyze motion capture (MoCap) data using TensorFlow, Pandas, and Keras.
About my data:
Data is obtained through pandas.read_csv and has a shape of
(832, 165)
Each row denotes a whole frame of data in a movement sequence (832 frames)
Each column denotes the rotational data for a joint (165 joints total)
I'm attempting to feed the data in one row at a time. The output should be the next frame in the movement sequence. I keep running into different types of errors when running model.fit.
I've attached a series of photos representing the different attempts to make the model work. If someone could provide some guidance as to why it's not working and how to fix I'd greatly appreciate it.
As a side note, each version of my code is different. I'm okay with using any as long as it ends up working, so when providing feedback if you could identify which version of my code you're talking about?
Uses tf.data.Dataset as input
Version 1 Code / Output
Version 2 Code / Output
Version 3: [Code] [Output]
Uses pandas arrays for input and target
Version 4 Code / Output
Version 5 Code / Output
Using Code 4 as a basis for troubleshooting, I noticed that you are passing incompatible shapes to the layers.
This line model.add(keras.layers.InputLayer(input_shape = (N_TIMESTEPS, N_FEATURES))) expects your data to have the same shape.
Whereas your data have (832, 165), which is the N_SAMPLES on the first index and the N_FEATURES, the N_TIMESTEPS is missing.
First, you should create a modified dataset that will generate a shape of (N_SAMPLES, N_TIMESTEPS, N_FEATURES).
Here is an example to generate a dummy dataset:
data = tf.random.normal((N_SAMPLES, N_TIMESTEPS, N_FEATURES))
target = tf.random.normal((N_SAMPLES, N_TIMESTEPS, N_FEATURES))
The N_TIMESTEPS in your data is important in LSTM as it determines how many TIME_STEPS to consider per update.
Here is the complete code used to simulate successful execution in Google Colab.
%tensorflow_version 2.x # To ensure latest Tensorflow version in Google Colab
import tensorflow as tf
import tensorflow.keras as keras
print(tf.__version__) # Tensorflow 2.2.0-rc3
BATCH_SIZE = 1
N_TIMESTEPS = 10
#Data is obtained through pandas.read_csv and has a shape of (832, 165)
#Each row denotes a whole frame of data in a movement sequence (832 frames)
#Each column denotes the rotational data for a joint (165 joints total)
# N_SAMPLES = data.values.shape[0]
# N_FEATURES = data.values.shape[1]
N_SAMPLES = 832
N_FEATURES = 165
def get_compiled_model():
model = keras.Sequential()
model.add(keras.layers.InputLayer(input_shape = (N_TIMESTEPS, N_FEATURES)))
model.add(keras.layers.LSTM(35, activation = 'relu', return_sequences = True))
model.add(keras.layers.LSTM(35, activation = 'relu', return_sequences = True))
model.add(keras.layers.Dense(165, activation = 'tanh'))
model.compile(optimizer = 'adam',
loss = 'mse',
metrics = ['accuracy'])
return model
model = get_compiled_model()
model.summary()
data = tf.random.normal((N_SAMPLES, N_TIMESTEPS, N_FEATURES))
target = tf.random.normal((N_SAMPLES, N_TIMESTEPS, N_FEATURES))
model.fit(data, target, epochs = 15, batch_size = BATCH_SIZE, shuffle = False)
Hope this helps you.
You could read more about the Tensorflow Keras Guide using RNN in this link.
I'm using the tf.data.Dataset.from_generator() function to create a datset for ASR with audio wav_file, length of audio wav_file, transcript and transcript_len. For the ML model I need audio wav_file and length to be zero padded and therefore I used .padded_batch() already. Now I need something else than .batch() as this needs the tensors to be in the same shape but without zero padding to batch my dataset.
I want to use the CTC Loss function tf.nn.ctc_loss_v2 which needs transcript and transcript_len tensors not be padded with zeros but batched. Is there a possibility to batch a dataset with tensors included in different shapes?
def generate_values():
for _, row in df.iterrows():
yield row.wav_filename, row.transcript, len(row.transcript)
def entry_to_features(wav_filename, transcript, transcript_len):
features, features_len = audiofile_to_features(wav_filename)
return features, features_len, transcript, transcript_len
def batch_fn(features, features_len, transcripts, transcript_len):
features = tf.data.Dataset.zip((features, features_len))
features = features.padded_batch(batch_size,
padded_shapes=([None, Config.n_input], []))
trans=tf.data.Dataset.zip((transcripts,
transcript_len)).batch(batch_size) ###PROBLEM:
#### ONLY WORKING WITH BATCH_SIZE=1
return tf.data.Dataset.zip((features, trans))
dataset = tf.data.Dataset.from_generator(generate_values,
output_types=(tf.string,tf.int64, tf.int64))
dataset= dataset.map(entry_to_features)
dataset= dataset.window(batch_size, drop_remainder=True)
dataset= dataset.flat_map(batch_fn)
InvalidArgumentError (see above for traceback): Cannot batch tensors with different shapes in component 0. First element had shape [36] and element 2 had shape [34]
If you want to train a seq2seq model and use features, transcript as training examples dataset.window is not what you gonna use.
dataset = tf.data.Dataset.from_generator(generate_values,
output_types=(tf.string, tf.int64, tf.int64))
dataset = dataset.map(entry_to_features)
dataset = dataset.padded_batch(batch_size, padded_shapes=([None, Config.n_input], [], [None], []))
later you can use the dataset as follows:
for features, feature_length, labels, label_length in dataset.take(30):
logits, logit_length = model(features, feature_length)
loss = tf.nn.ctc_loss_v2(labels, tf.cast(logits, tf.float32),
label_length, logit_length, logits_time_major=False)
I'm new in keras and deep learning field. In fact, I want to make a dense vector for each document in my data so that i built a simple autoencoder using keras library.
The input data are normalized using Word2vec with 200 as embedding size and all features are between -1 and 1. I prepared a 3D tensor that contains 137 samples (number of document) with 469 columns (maximum numbers of words) and the third dimension is the embedding size.I used mse loss function and GRU as recurrent neural network. I am having the same vector for all documents as the autoencoder prediction output while loss start with a very low value and became constant after a few number of epochs.
I tried different number of epochs but I got the same thing. I tried also to change the batch size but no change. Can any one help me find the problem please.
input = Input(shape=(469,200))
encoder = GRU(120,activation='sigmoid',dropout=0.2)(input)
neck = Dense(20)(encoder)
decoder1 = RepeatVector(469)(neck)
decoder1 = GRU(120,return_sequences=True,activation='sigmoid',dropout=0.2)(decoder1)
decoder1 = TimeDistributed(Dense(200,activation='tanh'))(decoder1)
model = Model(inputs=input, outputs=decoder1)
model.compile(optimizer='adam', loss='mse')
history = model.fit(x_train, x_train,validation_data=(x_test,x_test) ,epochs=10, batch_size=8)
this is the input data "x_train" :
print(model.predict(x_train)) return this values (same vectors):
Why "model.predict(x_train)" return the same vector for the 137 samples ?
Thank you in advance.