I'm trained a model in Keras, only Dense layers. However, when i try to predict it gives me the same answer all the time even with different values.
import numpy
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout
from keras.layers.embeddings import Embedding
from keras.optimizers import Adam
import pandas as pd
import tensorflow as tf
tf.python.control_flow_ops = tf
df = pd.read_csv('/home/sam/Documents/data.csv')
dfX = df[['Close']]
dfY = df[['Y']]
bobX = dfX.as_matrix()
boby = dfY.as_matrix()
model = Sequential()
model.add(Dense(200, input_dim=1))
model.add(Activation('sigmoid'))
model.add(Dense(75))
model.add(Activation('sigmoid'))
model.add(Dense(10))
model.add(Activation('sigmoid'))
model.add(Dense(1))
adam = Adam(lr=0.1)
model.compile(loss='mse', optimizer= adam)
print(model.summary())
model.fit(bobX, boby, nb_epoch=2500, batch_size=500, verbose=0)
model.predict(np.array([[210.99]]))
Your learning rate is WAY to high for Adam. Actually 0.1 is too high for most optimizers I have used. You should use 1e-3 or 1e-4 as the learning rate. These usually work well for me. When you use that high of a learning rate the model will fail to converge. From my experience it often just settles for the constant average value of the problem.
Related
I'm currently using keras to create a neural net in python. I have a basic model and the code looks like this:
from keras.layers import Dense
from keras.models import Sequential
model = Sequential()
model.add(Dense(23, input_dim=23, kernel_initializer='normal', activation='relu'))
model.add(Dense(500, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation="relu"))
model.compile(loss='mean_squared_error', optimizer='adam')
It works well and gives me good predictions for my use case. However, I would like to be able to use a Variational Gaussian Process layer to give me an estimate for the prediction interval as well. I'm new to this type of layer and am struggling a bit to implement it. The tensorflow documentation on it can be found here:
https://www.tensorflow.org/probability/api_docs/python/tfp/layers/VariationalGaussianProcess
However, I'm not seeing that same layer in the keras library. For further reference, I'm trying to do something similar to what was done in this article:
https://blog.tensorflow.org/2019/03/regression-with-probabilistic-layers-in.html
There seems to be a bit more complexity when you have 23 inputs vs one that I'm not understanding. I'm also open to other methods to achieving the target objective. Any examples on how to do this or insights on other approaches would be greatly appreciated!
tensorflow_probability is a separate library but suitable to use with Keras and TensorFlow. You can add those custom layers in your code and change it to a probabilistic model. If your goal is just to get a prediction interval it would be simpler to use the DistributionLambda layer. So your code would be as follows:
from keras.layers import Dense
from keras.models import Sequential
from sklearn.datasets import make_regression
import tensorflow_probability as tfp
import tensorflow as tf
tfd = tfp.distributions
# Sample data
X, y = make_regression(n_samples=100, n_features=23, noise=4.0, bias=15)
# loss function Negative log likelyhood
negloglik = lambda y, p_y: -p_y.log_prob(y)
# Model
model = Sequential()
model.add(Dense(23, input_dim=23, kernel_initializer='normal', activation='relu'))
model.add(Dense(500, kernel_initializer='normal', activation='relu'))
model.add(Dense(2))
model.add(tfp.layers.DistributionLambda(
lambda t: tfd.Normal(loc=t[..., :1],
scale=1e-3 + tf.math.softplus(0.05 * t[..., 1:]))))
model.compile(loss=negloglik, optimizer='adam')
model.fit(X,y, epochs=250, verbose=None)
After training your model, you can get your prediction distribution with the following lines:
yhat = model(X) # make predictions
means = yhat.mean() # prediction means
stds = yhat.stddev() # prediction standard deviation
i tried to optimize my DL model using the grid search algorithm
it runs and when it complete it arise the following error - note that i used both tensorflow and keras - cause i didn't understand the reason of the error.
i try to import just each of them , but i doesnt work also
import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import InputLayer
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from keras.layers import Dropout
from tensorflow.keras import regularizers
from tensorflow.keras.models import load_model
from tensorflow.keras.layers import concatenate
from tensorflow.keras.models import Model
from numpy import argmax
from sklearn.model_selection import GridSearchCV, KFold
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.wrappers.scikit_learn import KerasClassifier
from tensorflow.keras import optimizers
from keras.optimizers import Adam
from keras.models import Sequential
from keras.layers import Dense,Dropout
#Grid search optimization
def create_model(learn_rate, dropout_rate):
# Create model
model = keras.Sequential()
model.add(layers.Dense(260, input_dim=265, activation='relu'))
model.add(layers.Dense(240, activation='relu',kernel_regularizer=regularizers.l2(0.01)))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dropout(0.1))
model.add(layers.Dense(56, activation='relu'))
model.add(layers.Dense(16, activation='relu'))
model.add(layers.Dropout(0.1))
model.add( layers.Dense(1, activation='sigmoid', name='class'))
# Compile the model
adam = Adam(lr=learn_rate)
model.compile(loss='binary_crossentropy', optimizer=adam, metrics=['accuracy'])
return model
# Create the model
model = KerasClassifier(build_fn=create_model, verbose=1)
learn_rate = [0.001, 0.02, 0.2]
dropout_rate = [0.0, 0.2, 0.4]
batch_size = [10, 20, 30]
epochs = [1, 5, 10]
seed = 42
# Make a dictionary of the grid search parameters
param_grid = dict(learn_rate=learn_rate, dropout_rate=dropout_rate, batch_size=batch_size, epochs=epochs )
# Build and fit the GridSearchCV
grid = GridSearchCV(estimator=model, param_grid=param_grid,
cv=KFold(random_state=seed), verbose=10)
grid_results = grid.fit(X_train, Y_train)
when i run grid.fit
it runs then show the following error
ValueError: Could not interpret optimizer identifier: <keras.optimizers.Adam object at 0x0000027A2BE4BAC8>
any help will be appreciated
I'm trying to search Hyperparameters for a model using Keras Tuner, but I'm getting this error when I run the code: "RuntimeError: Model-building function did not return a valid Keras Model instance, found < keras.engine.sequential.Sequential object at 0x000001E9C2903F28 >"
I've searched on Internet but didn't found anything that could help, also I've followed the tutorial in the Keras Tuner gitHub page (https://github.com/keras-team/keras-tuner), but it dind't work either.
Here is my code:
class MyHyperModel(HyperModel):
def __init__(self, num_classes):
self.num_classes = num_classes
def build(self, hp):
model=Sequential()
model.add(Dense(units=hp.Int('units_0', 30, 900, step=30),
activation=hp.Choice('act_0', ['relu', 'tanh']),
input_dim=12))
for i in range(hp.Int('layers', 3, 9)):
model.add(Dense(units=hp.Int('units_' + str(i), 30, 900, step=30),
activation=hp.Choice('act_' + str(i), ['relu', 'tanh'])))
model.add(Dense(6, activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer=hp.Choice('optimizer', ['adam', 'sgd']),
metrics=['categorical_accuracy'])
return model
hypermodel = MyHyperModel(num_classes=6)
tuner = kt.tuners.bayesian.BayesianOptimization(
hypermodel,
objective='val_accuracy',
max_trials=5,
executions_per_trial=3,
seed=(np.random.seed(1)),
directory='Tests',
project_name='test')
tuner.search_space_summary()
tuner.search(data[:200], labels[:200],
verbose=2,
epochs=3,
validation_data=(data[200:], labels[200:]))
models = tuner.get_best_models(num_models=2).summary()
tuner.get_best_hyperparameters()[0].values
tuner.results_summary()
data is an list of 300 vector with 12 values and on labels there are 6 classes which was converted to tensor with the function tensorflow.convert_to_tensor().
I appreciate any help.
If you import module members from keras, you must import from tensorflow.keras instead of keras. For example, if you write:
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.optimizers import Adam
Then change them to:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Dropout
from tensorflow.keras.optimizers import Adam
I know what's wrong and is not the code, my model have 6 neurons on the last layer and I've used the loss as 'categorical_crossentropy', but this only works when the labels are 0 and 1, so I've changed the loss to 'sparse_categorical_crossentropy' and metrics to 'accuracy' and it worked.
Thanks everyone for the reply, I appreciate the help.
I'm trying to get a very (over) simplified Keras binary classifier neural network running without success. The LOSS just stays constant. I've played around with Optimizers (SGD, Adam, RMSProp), Learningrates, Weight-Initializations, Batch Size and input data normalization so far.
Nothing changes at all. Am I doing something fundamentally wrong? Here is the code:
from tensorflow import keras
from keras import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
data = np.array(
[
[100,35,35,12,0],
[101,46,35,21,0],
[130,56,46,3412,1],
[131,58,48,3542,1]
]
)
x = data[:,1:-1]
y_target = data[:,-1]
x = x / np.linalg.norm(x)
model = Sequential()
model.add(Dense(3, input_shape=(3,), activation='softmax', kernel_initializer='lecun_normal',
bias_initializer='lecun_normal'))
model.add(Dense(1, activation='softmax', kernel_initializer='lecun_normal',
bias_initializer='lecun_normal'))
model.compile(optimizer=SGD(learning_rate=0.1),
loss='binary_crossentropy',
metrics=['accuracy'])
model.fit(x, y_target, batch_size=2, epochs=10,
verbose=1)
Softmax definition is:
exp(a) / sum(exp(a)
so when you use with a single neuron you will get:
exp(a) / exp(a) = 1
That is why your classifier doesn't work with a single neuron.
You can use sigmoid instead in this special case:
exp(a) / (exp(a) + 1)
Furthermore sigmoid function is for two class classifiers. Softmax is an extension of sigmoid for multiclass classifers.
For the first layer you should use relu or sigmoid function instead of softmax.
This is the working solution based on the feedback I got
from tensorflow import keras
from keras import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.utils import to_categorical
data = np.array(
[
[100,35,35,12,0],
[101,46,35,21,0],
[130,56,46,3412,1],
[131,58,48,3542,1]
]
)
x = data[:,1:-1]
y_target = data[:,-1]
x = x / np.linalg.norm(x)
model = Sequential()
model.add(Dense(3, input_shape=(3,), activation='sigmoid'))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer=SGD(learning_rate=0.1),
loss='binary_crossentropy',
metrics=['accuracy'])
model.fit(x, y_target, epochs=1000,
verbose=1)
I have a code to train MNIST dataset to work on the street view house number project, but when I run the code I have acc = 0,1
Import libraries and modules
import numpy as np
np.random.seed(123) # for reproducibility
from keras import backend as K
K.set_image_dim_ordering('th')
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten
from tensorflow.keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
from keras.datasets import mnist
from keras.models import load_model
from keras.utils import CustomObjectScope
from keras.initializers import glorot_uniform
4. Load pre-shuffled MNIST data into train and test sets
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
X_train = keras.utils.normalize(X_train,axis=1)
X_test = keras.utils.normalize(X_test, axis=1)
7. Define model architecture
model = Sequential()
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(10, activation='softmax'))
8. Compile model
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
9. Fit model on training data
model.fit(np.array(X_train), np.array(Y_train), batch_size=32, epochs=3,verbose=1)
In step 4, are you normalizing the data correctly? If I recall correctly X_train has shape batch, width, height. I don't really know what you would want to normalize, but axis=1 doesn't seem it's supposed to be there. I think you should the normalization.
If you still have low accuracy, try training more epochs than 3. 3 epochs are not that many.
There are several reasons why you have an accuracy like that.
Your data is not normalized correctly.
You are trying to do image recognition with 3 dense layers in 3 epochs, which will not work.
There is no optimization in your code.
Look at the https://keras.io/examples/mnist_cnn/ . It is the Keras documentation on working with the MNIST data using neural network.