So, I am creating an AI which predicts how long a user will take to finish exercises. I previously created a NN with Sklearn, but I want to integrate Tensorflow.
I have 6 features as input and 1 output, which is a number.
I tried this but it does not seem to be willing to work:
# Train data
X_train = X[:1500]
y_train = y[:1500]
# Test data
X_test = X[1500:]
y_test = y[1500:]
# Create the TF model
model = tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=(6,)),
tf.keras.layers.Dense(256, activation='softmax'),
tf.keras.layers.Dense(128, activation='softmax'),
tf.keras.layers.Dense(64, activation='softmax'),
tf.keras.layers.Dense(1)
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
model.fit(X_train, y_train, epochs=10)
With that, it used to work with a simple MLPClassifier.
I also managed to get this nice error which does not seem to be fixed by changing the layers:
Received a label value of 1209638408 which is outside the valid range of [0, 1).
So I changed it a bit and came up with this:
features_train = features[:1500]
output_train = output[:1500]
features_test = features[1500:]
output_test = output[1500:]
classifier = Sequential()
classifier.add(Dense(units = 16, activation = 'relu', input_dim = 6))
classifier.add(Dense(units = 128, activation = 'relu'))
classifier.add(Dense(units = 64, activation = 'relu'))
classifier.add(Dense(units = 32, activation = 'relu'))
classifier.add(Dense(units = 8, activation = 'relu'))
classifier.add(Dense(units = 2, activation = 'relu'))
classifier.add(Dense(units = 1))
classifier.compile(optimizer='rmsprop', loss='binary_crossentropy')
classifier.fit(features_train, output_train, batch_size = 1, epochs = 10)
But now I get a loss of 100%.
You should use a smaller network. Try with fewer Dense layers, 2 or 3 maximum. If you use the binary_crossentropy loss, use a sigmoid activation in the last Dense layer. You can also pass metrics=['accuracy'] when compiling the model to monitor the accuracy.
Related
Problem: I have S sequences of T timesteps each and each timestep contains F features so collectively, a dataset of
(S x T x F) and each s in S is described by 2 values (Target_1 and Target_2)
Goal: Model/Train an architecture using LSTMs in order to learn/achieve a function approximator model M and given a sequence s, to predict Target_1 and Target_2 ?
Something like this:
M(s) ~ (Target_1, Target_2)
I'm really struggling to find a way, below is a Keras implementation of an example that probably does not work. I made 2 models one for the first Target value and 1 for the second.
model1 = Sequential()
model1.add(Masking(mask_value=-10.0))
model1.add(LSTM(1, input_shape=(batch, timesteps, features), return_sequences = True))
model1.add(Flatten())
model1.add(Dense(hidden_units, activation = "relu"))
model1.add(Dense(1, activation = "linear"))
model1.compile(loss='mse', optimizer=Adam(learning_rate=0.0001))
model1.fit(x_train, y_train[:,0], validation_data=(x_test, y_test[:,0]), epochs=epochs, batch_size=batch, shuffle=False)
model2 = Sequential()
model2.add(Masking(mask_value=-10.0))
model2.add(LSTM(1, input_shape=(batch, timesteps, features), return_sequences=True))
model2.add(Flatten())
model2.add(Dense(hidden_units, activation = "relu"))
model2.add(Dense(1, activation = "linear"))
model2.compile(loss='mse', optimizer=Adam(learning_rate=0.0001))
model2.fit(x_train, y_train[:,1], validation_data=(x_test, y_test[:,1]), epochs=epochs, batch_size=batch, shuffle=False)
I want to make somehow good use of LSTMs time relevant memory in order to achieve good regression.
IIUC, you can start off with a simple (naive) approach by using two output layers:
import tensorflow as tf
timesteps, features = 20, 5
inputs = tf.keras.layers.Input((timesteps, features))
x = tf.keras.layers.Masking(mask_value=-10.0)(inputs)
x = tf.keras.layers.LSTM(32, return_sequences=False)(x)
x = tf.keras.layers.Dense(32, activation = "relu")(x)
output1 = Dense(1, activation = "linear", name='output1')(x)
output2 = Dense(1, activation = "linear", name='output2')(x)
model = tf.keras.Model(inputs, [output1, output2])
model.compile(loss='mse', optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001))
x_train = tf.random.normal((500, timesteps, features))
y_train = tf.random.normal((500, 2))
model.fit(x_train, [y_train[:,0],y_train[:,1]] , epochs=5, batch_size=32, shuffle=False)
I have started training a basic MLP model on MNIST data taken from here. Below is my code for implementing the model.
train = pd.read_csv(r"train.csv")
test = pd.read_csv(r"test.csv")
train_img_path = "./Images/train/"
test_img_path = "./Images/test/"
train_img = []
for img in train['filename']:
img_path = train_img_path+img
image = imread(img_path)
image = image/255
train_img.append(image)
train_img = np.array(train_img)
batch_size = 64
y_train = train['label']
from tensorflow.keras.utils import to_categorical
#y_train = to_categorical(y_train)
model = Sequential()
model.add(Dense(10, activation = 'relu'))
model.add(Dense(10, activation = 'relu'))
model.add(Dense(10, activation = 'softmax'))
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(train_img, y_train, epochs=20, batch_size=batch_size)
While trying to fit my model on this data I get error InvalidArgumentError: logits and labels must have the same first dimension, got logits shape [50176,10] and labels shape [64] with loss='sparse_categorical_crossentropy'.
There were suggestions to try with loss='categorical_crossentropy' after having one-hot encoded values and that also gives error ValueError: Shapes (None, 10) and (None, 28, 28, 10) are incompatible
I am confused on how I am getting the shape [50176,10] (though examples are 49000) in the error.
I guess I am missing something on shape. Can someone guide me where I am doing wrong and how to solve this.
Edit: I have modified my code as below to pick the data from keras for_from_dataframe. But I still get the same error.
train_datagen = ImageDataGenerator(rescale=1./255)
test_datagen = ImageDataGenerator(rescale=1./255)
train_data = train_datagen.flow_from_dataframe(
dataframe=train,
directory='./Images/train',
x_col='filename',
y_col='label',
weight_col=None,
target_size=(28,28),
color_mode='grayscale',
class_mode='categorical',
batch_size=64
)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#model.summary()
model.fit(train_data, epochs=20)
The main problem is in your model building code:
model = Sequential()
model.add(Dense(10, activation = 'relu'))
model.add(Dense(10, activation = 'relu'))
model.add(Dense(10, activation = 'softmax'))
You are trying to feed images and its label to ANN which obviously gives error. Also there is no any inputs given in your model.
For images, CNN should be used instead of ANN.
import tensorflow as tf
model = Sequential()
model.add(tf.keras.layers.Conv2D(32, activation = 'relu', input_shape=(28,28,3)))
model.add(tf.keras.layers.MaxPooling2D((2,2))
model.add(tf.keras.layers.Conv2D(64, activation = 'relu'))
model.add(tf.keras.layers.MaxPooling2D((2,2))
model.add(tf.keras.layers.Conv2D(128, activation = 'relu'))
model.add(tf.keras.layers.MaxPooling2D((2,2))
model.add(tf.keras.layers.Flatten())
model.add(Dense(10, activation = 'relu'))
model.add(Dense(20, activation = 'relu'))
model.add(Dense(10, activation = 'softmax'))
If you have one-hot encoded your labels, use categorical_crossentropy. If your labels are numbers then use sparse_categorical_crossentropy
I'm having problem with keras sequential().predict(x_test).
Btw getting the same output using sequential().predict_proba(x_test) as I found that these two are indifferent in sequential now.
My data has two classes: 0 or 1, I believe predict(x_test) should give two columns, where the first column is the prob for getting 0 and the second is prob of getting 1. However I only have one column with this.
In [85]:y_train.value_counts()
Out[85]:
0 616751
1 11140
Name: _merge, dtype: int64
There should be no problem with my data as I used the same x_train, y_train, x_test, y_test for both LogisticRegression model and neural network model, it works perfect in LogisticRegression.
In [87]:y_pred_LR
Out[87]:
array([[ 9.96117151e-01, 3.88284921e-03],
[ 9.99767583e-01, 2.32417329e-04],
[ 9.87375774e-01, 1.26242258e-02],
...,
[ 9.72159138e-01, 2.78408623e-02],
[ 9.97232916e-01, 2.76708432e-03],
[ 9.98146985e-01, 1.85301489e-03]])
but I only get 1 column in neural network model.
So I guess there is some problem with the NN model setting up? Here is my codes
NN = Sequential()
NN.add(Dense(40, input_dim = 65, kernel_initializer = 'uniform', activation = 'relu'))
NN.add(Dense(20, kernel_initializer = 'uniform', activation = 'relu'))
NN.add(Dense(1, kernel_initializer = 'uniform', activation = 'sigmoid'))
NN.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
NN.fit(x_train, y_train, batch_size = 50, epochs=5)
y_pred_NN = NN.predict(x_test)
print(y_pred_NN)
In [86]: print(y_pred_NN)
[[ 0.00157279]
[ 0.0010451 ]
[ 0.03178826]
...,
[ 0.01030775]
[ 0.00584918]
[ 0.00186538]]
Actually it looks like it's the prob of getting 1?
Any help is appreciated!
Btw the shapes of my predictions in both models are as follows
In [91]:y_pred_LR.shape
Out[91]: (300000, 2)
In [90]:y_pred_NN.shape
Out[90]: (300000, 1)
If you want to output two probabilities, you will have to replace your y_train with to_categorical(y_train) and then adjust the network accordingly:
from keras.utils import to_categorical
NN = Sequential()
NN.add(Dense(40, input_dim = 65, kernel_initializer = 'uniform', activation = 'relu'))
NN.add(Dense(20, kernel_initializer = 'uniform', activation = 'relu'))
NN.add(Dense(2, activation='sigmoid'))
NN.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
NN.fit(x_train, to_categorical(y_train), batch_size = 50, epochs=5)
Consult here: https://keras.io/utils/#to_categorical
The last line of your model defines the output layer.
NN.add(Dense(1, kernel_initializer = 'uniform', activation = 'sigmoid'))
Your last line has one node, with a sigmoid activation. This means that your output will be a single number (per input sample) between 0 and 1, which I believe you can interpret as P(y=1), as per logistic regression.
With more than two classes you would have n_classes in the output layer:
NN.add(Dense(n_classes, kernel_initializer = 'uniform', activation = 'softmax'))
The softmax ensures that the probabilities sum to unity. Of course to do this you would have to one-hot encode your training y values.
In your case you can choose between either approach. I favour the second, as it allows you to add more classes in the future.
I keep receiving this error:
Error when checking target: expected dense_256 to have shape (1,) but got array with shape (10,)
I have check my X_train variable and I get a shape of (576,10). So, I have 576 samples, each with 10 features (all of which have been scaled already).
I now try this:
classifier = Sequential()
classifier.add(Dense(units = 5, kernel_initializer='uniform', activation = 'relu', input_shape=(10,)))
classifier.add(Dense(units = 5, kernel_initializer='uniform', activation = 'relu'))
classifier.add(Dense(units = 5, kernel_initializer='uniform', activation = 'relu'))
classifier.add(Dense(units = 5, kernel_initializer='uniform', activation = 'relu'))
classifier.add(Dense(units = 1, kernel_initializer='uniform', activation = 'relu'))
classifier.compile(optimizer = 'adam', loss='mean_squared_error', metrics=['mse', 'mae', 'mape'])
classifier.fit(X_train, y_train, batch_size = 10, epochs=100)
Which is when I get the input_shape error referenced above.
So, my question is, when defining input_shape, how do I set this correctly?
I am trying a NN model using this example. I am fitting a list of values to a NN model. However, I am getting an AttributeError. This has been asked before and has been answered. Unfortunately, it is not working for me. As shown in the example, I created the following,
from keras.models import Sequential
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from keras.layers import Dense
def neuralnetmodel():
#Crete model
model = Sequential()
model.add(Dense(13, input_dim = 13, kernel_initializer = 'normal', activation = 'relu'))
model.add(Dense(1, kernel_initializer = 'normal', activation = 'relu'))
model.add(Dense(1, kernel_initializer = 'normal', activation = 'relu'))
## Output layer
model.add(Dense(1, kernel_initializer = 'normal'))
#Compile model
model.compile(loss = 'mean_squared_error', optimizer = 'adam')
return model
fit training data,
NNmodelList = []
for i,j in zip(X_train_scaled,y_train):
nn_model = KerasRegressor(build_fn= neuralnetmodel, nb_epoch = 50, batch_size = 10, verbose = 0)
NNmodelList.append(nn_model.fit(i,j))
predict from test data,
PredList = []
for val in X_test_scaled:
for mod in NNmodelList:
pred = mod.predict(val)
PredList.append(pred)
Now, I am getting the error:
AttributeError: 'History' object has no attribute 'predict'
In previous threads , it seems to be the train set was not fit to the model before predict. However, in mine, I fit them in the second code snippet. Any ideas what other possible mistakes I am making?
model.fit() does not return the Keras model, but a History object containing loss and metric values of your training. So in this code:
NNmodelList.append(nn_model.fit(i,j))
you're creating a list of History objects, not models. A simple fix would be:
NNmodelList.append(nn_model)
nn_model.fit(i,j)
enter code here
def build_regressor() :
regressor = Sequential()
regressor.add(Dense(units = 64, kernel_initializer = 'uniform', activation = 'relu', input_dim = 5))
regressor.add(Dense(units =64 , kernel_initializer = 'uniform', activation = 'relu'))
regressor.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'relu'))
regressor.compile(optimizer='adam', loss='mean_squared_error')
return regressor
regressor=KerasRegressor(build_fn=build_regressor, batch_size=32,epochs=200)
results=regressor.fit(X_train,y_train)
regressor.history
print(results.coef_)
but this shows the following error:
AttributeError: 'KerasRegressor' object has no attribute 'history'