We Keep Coding

Why am I getting horizontal line (almost zero) from neural network instead of the desired curve?

I am trying to use neural network for my regression problem in python but the output of the neural network is a straight horizontal line which is zero. I have one input and obviously one output.
Here is my code:
def baseline_model():
# create model
model = Sequential()
model.add(Dense(1, input_dim=1, kernel_initializer='normal', activation='relu'))
model.add(Dense(4, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal'))
# Compile model
model.compile(loss='mean_squared_error',metrics=['mse'], optimizer='adam')
model.summary()
return model
# evaluate model
estimator = KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=64,validation_split = 0.2, verbose=1)
kfold = KFold(n_splits=10)
results = cross_val_score(estimator, X_train, y_train, cv=kfold)
Here are the plots of NN prediction vs. target for both training and test data.
Training Data
Test Data
I have also tried different weight initializers (Xavier and He) with no luck!
I really appreciate your help

First of all correct your syntax while adding dense layers in model remove the double equal == with single equal = with kernal_initilizer like below
model.add(Dense(1, input_dim=1, kernel_initializer ='normal', activation='relu'))
Then to make the performance better do the followong
Increase the number of hidden neurons in the hidden layers
Increase the number of hidden layers.
If still you have same problem then try to change the optimizer and activation function. Tuning the hyperparameters may help you in converging to the solution
EDIT 1
You also have to fit the estimator after cross validation like below
estimator.fit(X_train, y_train)
and then you can test on the test data as follow
prediction = estimator.predict(X_test)
from sklearn.metrics import accuracy_score
accuracy_score(Y_test, prediction)

Why the accuracy and loss plot is wavy at the end of training?

I was training a word-level sentence generation model and as its training started to reach towards the end of total iterations, the accuracy started to go down and up multiple times and formed a wavy pattern in the history plot. I am unable to understand why is this happening. Is it because of overfitting? Do I need to add some dropout layers to my model?
My model:
def rnn_model():
model = Sequential()
model.add(Embedding(uniq_vals, 50, input_length=s_len))
model.add(SimpleRNN(25, return_sequences=True))
model.add(SimpleRNN(25))
model.add(Dense(50, activation='relu'))
model.add(Dense(uniq_vals, activation='softmax'))
return model
model = rnn_model()
model.compile(loss = 'categorical_crossentropy',optimizer = 'adam', metrics = ['accuracy'])
model.summary()
Also I have set the batch_size = 128 and epochs = 150
Accuracy and loss plot:

LSTM Model accuracy is stuck in a certain level and doesn't change

I'm working on a multivariate timeseries prediction using LSTM. I'm trying to get a better match between my actual and predicted values, but no matter what my hyperparamters are, the accuracy won't change. I was wondering if you can give me few insight on how to increase my model accuracy...
I have 3 inputs (time, two rates) and one output (pressure).
enter image description here
This is the LSTM section of my code:
model = Sequential()
model.add(LSTM(units=4,
activation='tanh',
recurrent_activation='hard_sigmoid',
use_bias=True,
unit_forget_bias=True,
dropout=0,
recurrent_dropout=0.3,
input_shape=(look_back, 3)))
model.add(Dense(units=1,
activation='linear',
use_bias=True))
model.compile(loss='mean_squared_error', optimizer='Adam', metrics=['mae','accuracy'])
hist = model.fit(x_train, y_train,
epochs=50,
batch_size=20,
validation_split=0.0,
verbose=2,
shuffle=False)

Keras LSTM fit underfitting

I have time series training data of about 5000 numbers. For each 100 numbers, I am trying to predict the 101st. At the end of the series, I would put in the predicted numbers back into the model to predict ahead of the time series.
The attached graph shows the training data, the test data and the prediction output. Currently, the model seems to be under-fitting. I would like to know what hyperparameters should be changed, or if I need to re-structure my input and output data.
I am using the following LSTM network.
model = Sequential()
model.add(LSTM(128, input_shape=([bl,1]), activation='relu', return_sequences=True))
model.add(Dropout(0.1))
model.add(LSTM(128, return_sequences=True))
model.add(Dropout(0.1))
model.add(Flatten())
model.add(Dense(20,activation='relu'))
model.add(Dense(1))
model.compile(optimizer=adam(lr=0.0001), loss='mean_squared_error', metrics=['accuracy'])
model.fit(y_ba_tr_in, y_ba_tr_out,
epochs=20,
batch_size=5,shuffle=False,verbose=2)
y_ba_tr_in.shape = (4961, 100, 1)
y_ba_tr_out.shape = (4961, 1)

Something you could try is taking return_sequences=True out of your last LSTM layer. I believe this is generally the approach when you intend to predict for the next timestep.
After that modification, you also shouldn't need the subsequent Flatten() and Dense() layers.

How can I create a LSTM model with dynamic outputs in Python with Keras?

I am creating a LSTM model in Keras with Python and I have the next context:
I have a regression problem, and I want a LSTM model of X different layers that inputs a sequence of 40 elements and outputs the next Y values. Those Y values should be dynamic. This is: maybe I want to predict the next 10 elements for a sequence of 40 elements in one case, but in other case I want to predict the next 100 values.
My question is: is it possible?
I have the next code:
model = Sequential()
model.add(LSTM(200, activation='relu', input_shape=(trainx.shape[1], trainx.shape[2])))
model.add(RepeatVector(outputs))
model.add(LSTM(200, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(100, activation='relu')))
model.add(TimeDistributed(Dense(1)))
model.compile(loss='mse', optimizer='adam')
# fit network
history = model.fit(trainx, trainy, epochs=100, batch_size=300, verbose=1)
# make predictions
trainPredict = model.predict(trainx)