I'm pretty new with NLP and I want to classify different words depending on their language (basically my model should tell me if a word is french, or english, or spanish and so on).
When I fit the following model I get a dimension error. The "dataset" contains the words, it's a padded tensor of size (1550, 19) and the "y" contains the different languages, it's also a padded tensor of size (1550, 10).
np.random.seed(42)
tf.random.set_seed(42)
from tensorflow.keras.layers import LSTM, GRU, Input, Embedding, Dense
input = Input(shape=[None])
z = Embedding(max_id + 1, 128, input_shape=[None], mask_zero=True)(input)
z = GRU(128)(z)
output = Dense(18, activation='softmax')(z)
model = keras.models.Model(input, output)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
h = model.fit(dataset, y, epochs=5)
ValueError: Shapes (None, 10) and (None, 18) are incompatible
Do you see where the problem is?
Thanks!
The message tells you that the shapes are not compatible, they need to match. I would have put this as a comment, but I can't due to my reputation, so that's why I answered directly, however I am not sure if it works, have you tried:
output = Dense(10, activation='softmax')(z)
Related
The following code gives me an input error and i cannot figure it out.
import tensorflow as tf
import neural_structured_learning as nsl
.
.
.
b_size = 132
m = tf.keras.Sequential()
m.add(tf.keras.layers.Dense(980, activation = 'relu', input_shape = (2206,2,)))
m.add(tf.keras.layers.Dense(560, activation = 'relu'))
m.add(tf.keras.layers.Dense(10, activation = 'softmax'))
adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.5)
adv_model = nsl.keras.AdversarialRegularization(m, adv_config=adv_config)
adv_model.compile(optimizer = "adam",
loss = "sparse_categorical_crossentropy",
metrics = ['accuracy'])
adv_model.fit({"feature" : x_Train, "label" : y}, epochs = 50, batch_size=b_size)
My x_Train has shape (5002, 2206, 2) (5002 samples of size (2206,2)). I have tried to add a Flatten() layer at the beginning but it gives me a object of type 'NoneType' has no len() error, even though this works perfectly with tf.keras. I also have tried different shapes for the input but none of them work. So it throws me one of the following errors
KeyError: 'dense_115_input'
ValueError: Input 0 of layer sequential_40 is incompatible with the layer: expected axis -1 of input shape to have value 2206 but received input with shape [None, 2206, 2]
TypeError: object of type 'NoneType' has no len()
To train an NSL model with an input dictionary (like your {"feature" : x_Train, "label" : y}), the base model has to know which feature(s) in the dictionary to look at.
One way to specify the feature names is to add an Input layer:
m = tf.keras.Sequential()
m.add(tf.keras.Input(name="feature", shape=(2206, 2)))
Also as this answer pointed out, the input feature has to be flatten before passing to dense layers:
m.add(tf.keras.layers.Flatten())
m.add(tf.keras.layers.Dense(...))
If you want to use a dense layer the input should be (5002, 2206*2), i.e a matrix.
Maybe the simplest solution is to reshape your input x_train before the "fit".
Alternatively, you can use a TimeDistributed layer (see here), but the usage of this kind of layer depends on the physical meaning behind the input dimensions. Basically, TimeDistributed applies a certain operation many times, in your case twice.
Hoping this can help you.
Following a paper, I'm using word embeddings as a feature vector for entity recognition.
I've attempted to architect the network using Keras but have run into a dimensionality problem I cannot seem to resolve.
Take the following example sentence:
["I went to the shop"]
The sentence has 5 words, and after computing the feature matrix, I am left with a matrix of dimension: (1, 120, 1000) == (#examples, sequence_length, embedding).
Note that sequence_length appends 0. padding when not complete. In this example, the actual sequence_length would be 5.
My network architecture is as follows:
enc = encode()
claims_input = Input(shape=(120, 1000), dtype='float32', name='claims')
x = Masking(mask_value=0., input_shape=(120, 1000))(claims_input)
x = Bidirectional(LSTM(units=512, return_sequences=True, recurrent_dropout=0.2, dropout=0.2))(x)
x = Bidirectional(LSTM(units=512, return_sequences=True, recurrent_dropout=0.2, dropout=0.2))(x)
out = TimeDistributed(Dense(8, activation="softmax"))(x)
model = Model(inputs=claims_input, output=out)
model.compile(loss="sparse_categorical_crossentropy", optimizer='adam', metrics=["accuracy"])
model.fit(enc, y)
The architecture is straight forward, I mask specific time steps, run two bidirectional LSTMs, followed by a softmax output. My y variable in this case, is a (9,8) one-hot-encoded matrix corresponding to the gold label of each word.
When trying to fit() this model, I am running into a dimensionality problem relating to the TimeDistributed() layer and I'm unsure how to resolve, or even begin to debug this.
Error: ValueError: Error when checking target: expected time_distributed_1 to have 3 dimensions, but got array with shape (9, 8)
Any help would be appreciated.
You are doing entity recognition. So each element in your input sequence will be assigned an entity (probably some of them as null). If your model takes an input sample of shape (120, n_features), then the output must also be a sequence of length of 120, i.e. one entity for each element. Therefore, the labels, i.e. y, you provide to the model must have a shape of (n_samples, 120, n_entities) (or (n_samples, 120, 1) if you are using sparse labeling).
Side note: There is no difference between TimeDistributed(Dense(...)) and Dense(...), as the Dense layer is applied on the last axis.
Trying to set up a Conv1D layer to be the input layer in keras.
The dataset is 1000 timesteps, and each timestep has 1 feature.
After reading a bunch of answers I reshaped my dataset to be in the following format of (n_samples, timesteps, features), which corresponds to the following in my case:
train_data = (78968, 1000, 1)
test_data = (19742, 1000, 1)
train_target = (78968,)
test_target = (19742,)
I later create and compile the code using the following lines
model = Sequential()
model.add(Conv1D(64, (4), input_shape = (1000,1) ))
model.add(MaxPooling1D(pool_size=2))
model.add(Dense(1))
optimizer = opt = Adam(decay = 1.000-0.999)
model.compile(optimizer=optimizer,
loss='mean_squared_error',
metrics=['mean_absolute_error','mean_squared_error'])
Then I try to fit, note, train_target and test_target are pandas series so i'm calling DataFrame.values to convert to numpy array, i suspect there might be an issue there?
training = model.fit(train_data,
train_target.values,
validation_data=(test_data, test_target.values),
epochs=epochs,
verbose=1)
The model compiles but I get an error when I try to fit
Error when checking target: expected dense_4 to have 3 dimensions,
but got array with shape (78968, 1)
I've tried every combination of reshaping the data and can't get this to work.
I've used keras with dense layers only before for a different project where the input_dimension was specificied instead of the input_shape, so I'm not sure what I'm doing wrong here. I've read almost every stack overflow question about data shape issues and I'm afraid the problem is elsewhere, any help is appreciated, thank you.
Under the line model.add(MaxPooling1D(pool_size=2)), add one line model.add(Flatten()), your problem will be solved. Flatten function will help you convert your data into correct shape, please see this site for more information https://www.tensorflow.org/api_docs/python/tf/keras/layers/Flatten
I'm trying to assign one of two classes (positive/nagative) to audio using CNN with Keras. My model should accept varied lengths of input (frames) in which each frame contains 41 features but I struggle with the input size. Bear in mind that I haven't acquired full dataset so I just mocked some meaningless data just to check if network works at all.
According to documentation https://keras.io/layers/convolutional/ and my best understanding Conv1D can tackle varied lengths if first element of input_shape tuple is None. Shape of variable containing input data X_train.shape is (4, 497, 41).
data = pd.read_csv('output_file.csv', sep=';')
featureCount = data.values.shape[1]
#mocks because full data is not available yet
Y_train = np.asarray([1, 0, 1, 0])
X_train = np.asarray(
[np.array(data.values, copy=True), np.array(data.values, copy=True), np.array(data.values, copy=True),
np.array(data.values, copy=True)])
# variable length with 41 features
model = keras.models.Sequential()
model.add(keras.layers.Conv1D(100, 5, activation='relu', input_shape=(None, featureCount)))
model.add(keras.layers.GlobalMaxPooling1D())
model.add(keras.layers.Dense(10, activation='relu'))
model.add(keras.layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
model.summary()
model.fit(X_train, Y_train, epochs=10, verbose=False, validation_data=(np.array(data.values, copy=True), [1]))
This code produces error
ValueError: Error when checking input: expected conv1d_input to have 3 dimensions, but got array with shape (497, 41). So it appears like the first dimension was cut out as it contains training samples (it seems correct to me) what bothered me is the required dimensionality, why is it 3?
After searching for the answer I stumbled onto Dimension of shape in conv1D and followed it by adding last dimension (using X_train = np.expand_dims(X_train, axis=3)) that contains only single digit but I ended up with another, similar error:
ValueError: Error when checking input: expected conv1d_input to have 3 dimensions, but got array with shape (4, 497, 41, 1) now it seems that first dimension that previously was treated as sample "list" is now part of actual data.
I also tried fiddling with input_shape parameter but to no avail and using Reshape layer but ended up fighting with size the
What should I do to satisfy required shape? How to prepare data for processing?
My input data consists of 10 samples, each of which has 200 time steps, while each time step is described by a vector of 30 dimensions.
In addition, each time step consists of a 3 dimensional vector (one hot encoding) which describes the action which has been taken at that particular time step. With that being said, I am trying to build a model which get fed in all previous actions and then predicts which action would be the best to take next.
I tried to get this working with tflearn and tensorflow but with limited success so far.
Simple sample code:
import numpy as np
import operator
import tflearn
from tflearn import regression
from tflearn.layers.core import input_data, dropout, fully_connected
from tflearn.layers.embedding_ops import embedding
from tflearn.layers.recurrent import bidirectional_rnn, BasicLSTMCell
from tflearn.data_utils import to_categorical, pad_sequences
SAMPLES = 10
TIME_STEPS = 200
DATA_DIMENSIONS = 30
LABEL_CLASSES = 3
x = []
y = []
# Generate fake data.
for i in range(SAMPLES):
sequences = []
outputs = []
for i in range(TIME_STEPS):
d = []
for i in range(DATA_DIMENSIONS):
d.append(1)
sequences.append(d)
outputs.append([0,0,1])
x.append(sequences)
y.append(outputs)
print("X1:", len(x), ", X2:", len(x[0]), ", X3:", len(x[0][0]))
print("Y1:", len(y), ", Y2:", len(y[0]), ", Y3:", len(y[0][0]))
# Define model
net = tflearn.input_data([None, TIME_STEPS, DATA_DIMENSIONS], name='input')
net = tflearn.lstm(net, 128, dropout=0.8, return_seq=True)
net = tflearn.fully_connected(net, LABEL_CLASSES, activation='softmax')
net = tflearn.regression(net, optimizer='adam', loss='categorical_crossentropy', name='targets')
model = tflearn.DNN(net)
# Fit model.
model.fit({'input': x}, {'targets': y},
n_epoch=1,
snapshot_step=1000,
show_metric=True, run_id='test', batch_size=32)
Error
ValueError: Cannot feed value of shape (10, 200, 3) for Tensor
'targets/Y:0', which has shape '(?, 3)'
As far as I understand, the input_data should be correct. However, the output data is apparently wrong, at least, Tensorflow throws an error. That is probably because my model expects one label per sample rather than one label per time step.
Can I even achieve my goal with an LSTM, and if so, how do I have to set up my model?
Thanks,
Robert
As the error suggests, there is a shape mismatch between the expected size of your targets tensor, and the one of the data you actually provide for it. Let us break it down.
From what I understand, you have labeled action for every timestep of your sequences. This means that the labels that you provide should have a shape (10, 200, 3). This seems to be the case from the error message. Good.
So we now know the error comes from what the network generates.
=================
Input data -> (10, 200, 30)
LSTM -> (10, 128) (because return_seq=False)
FullyConnected -> (10, 3).
=================
So that explains the second part of the error message, your network indeed produces an output with shape (10, 3) which mismatches the one of your data.
I think you missed the return_seq argument of the LSTM. As is usually the case with RNN implementations, you have a parameter telling if you want the layer to return outputs for the whole sequence, or only for the last timestep. Here by default it is the second option, that is why you don't get an output with the expected shape. Use return_seq=True.