I've tried following tutorials on implementing this but I keep getting dimension errors on the LSTM layer.
ValueError: Input 0 of layer LSTM is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 2]
import random
import numpy as np
import tensorflow as tf
from tensorflow import feature_column as fc
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, DenseFeatures, Reshape
from sklearn.model_selection import train_test_split
def df_to_dataset(features, target, batch_size=32):
return tf.data.Dataset.from_tensor_slices((dict(features), target)).batch(batch_size)
# Reset randomization seeds
np.random.seed(0)
tf.random.set_random_seed(0)
random.seed(0)
# Assume 'frame' to be a dataframe with 3 columns: 'optimal_long_log_return', 'optimal_short_log_return' (independent variables) and 'equilibrium_log_return' (dependent variable)
X = frame[['optimal_long_log_return', 'optimal_short_log_return']][:-1]
Y = frame['equilibrium_log_return'].shift(-1)[:-1]
X_train, _X, y_train, _y = train_test_split(X, Y, test_size=0.5, shuffle=False, random_state=1)
X_validation, X_test, y_validation, y_test = train_test_split(_X, _y, test_size=0.5, shuffle=False, random_state=1)
train = df_to_dataset(X_train, y_train)
validation = df_to_dataset(X_validation, y_validation)
test = df_to_dataset(X_test, y_test)
feature_columns = [fc.numeric_column('optimal_long_log_return'), fc.numeric_column('optimal_short_log_return')]
model = Sequential()
model.add(DenseFeatures(feature_columns, name='Metadata'))
model.add(LSTM(256, name='LSTM'))
model.add(Dense(1, name='Output'))
model.compile(loss='logcosh', metrics=['mean_absolute_percentage_error'], optimizer='Adam')
model.fit(train, epochs=10, validation_data=validation, verbose=1)
loss, accuracy = model.evaluate(test, verbose=0)
print(f'Target Error: {accuracy}%')
After seeing this issue elsewhere I've tried setting input_shape=(None, *X_train.shape), input_shape=X_train.shape, neither works. I also tried inserting a Reshape layer model.add(Reshape(X_train.shape)) before the LSTM layer and it fixed the issue but I got another issue in its place:
InvalidArgumentError: Input to reshape is a tensor with 64 values, but the requested shape has 8000
...and I'm not even sure adding the Reshape layer is doing what I think it is doing. After all, why would reshaping the data to its own shape fix things? Something is happening with my data that I just don't understand.
Also, I'm using this for time series analysis (stock returns), so I would think that the LSTM model should be stateful and temporal. Would I need to move the timestamp index into its own column in the pandas database before converting to a tensor?
Unfortunately I'm obligated to use tensorflow v1.15 as this is being developed on the QuantConnect platform and they presumably won't be updating the library any time soon.
EDIT: I've made a bit of progress by using TimeseriesGenerator, but now I'm getting the following error (which returns no results on Google):
KeyError: 'No key found for either mapped or original key. Mapped Key: []; Original Key: []'
Code below (I'm sure I'm using the input_shape arguments incorrectly):
train = TimeseriesGenerator(X_train, y_train, 1, batch_size=batch_size)
validation = TimeseriesGenerator(X_validation, y_validation, 1, batch_size=batch_size)
test = TimeseriesGenerator(X_test, y_test, 1, batch_size=batch_size)
model = Sequential(name='Expected Equilibrium Log Return')
model.add(LSTM(256, name='LSTM', stateful=True, batch_input_shape=(1, batch_size, X_train.shape[1]), input_shape=(1, X_train.shape[1])))
model.add(Dense(1, name='Output'))
model.compile(loss='logcosh', metrics=['mean_absolute_percentage_error'], optimizer='Adam', sample_weight_mode='temporal')
print(model.summary())
model.fit_generator(train, epochs=10, validation_data=validation, verbose=1)
loss, accuracy = model.evaluate_generator(test, verbose=0)
print(f'Model Accuracy: {accuracy}')
Turns out this specific issue relates to a patch that Quantconnect made to pandas dataframes which interfered with the older version of tensorflow/keras.
Related
I have written the following code for a neural network to perform regression on a dataset, but I am getting a ValueError. I have looked up to different answers and they suggested to use df = df.values to get a numpy array. I tried it but it still produced the same error. How to fix this?
CODE
from keras import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.optimizers import Adam
from sklearn.model_selection import train_test_split
#Define Features and Label
features = ['posted_by', 'under_construction', 'rera', 'bhk_no.', 'bhk_or_rk',
'square_ft', 'ready_to_move', 'resale', 'longitude',
'latitude']
X=train[features].values
y=train['target(price_in_lacs)'].values
#Train Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state = 23, shuffle = True)
#Model
model = Sequential()
model.add(Dense(10, activation='relu', kernel_initializer='random_normal', input_dim = 10))
model.add(Dense(1, activation = 'relu', kernel_initializer='random_normal'))
#Compiling the neural network
model.compile(optimizer = Adam(learning_rate=0.1) ,loss='mean_squared_logarithmic_error', metrics =['mse'])
#Fitting the data to the training dataset
model.fit(X_train,y_train, batch_size=256, epochs=100, verbose=0)
ERROR
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int).
I'm new to Python (although not to programming - I'm usually programming in JavaScript) and I'm very interested in AI development.
Recently I've been trying to develop a deep learning algorithm by following this article.
My goal is to predict a set of 7 numbers, based on a CSV file that contains a large list, with each row having 7 numbers as well. The order of the list matters.
I ended up having the following code:
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import train_test_split
from numpy import loadtxt, random
random.seed(seed)
dataset = loadtxt("data/Lotto.csv", delimiter=",", skiprows=1)
X = dataset[:, 0:7]
Y = dataset[:, 6]
(X_train, X_test, Y_train, Y_test) = train_test_split(X, Y, test_size=0.33, random_state=4)
model = Sequential()
model.add(Dense(8, input_dim=7, kernel_initializer="uniform", activation="relu"))
model.add(Dense(6, kernel_initializer="uniform", activation="relu"))
model.add(Dense(1, kernel_initializer="uniform", activation="sigmoid"))
model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=100, batch_size=5, shuffle=False)
scores = model.evaluate(X_test, Y_test)
print("Accuracy: %.2f%%" %(scores[1] * 100))
After running it in Google Colaboratory, while I'm not getting any errors - I noticed that for each epoch, the loss result doesn't change, and as a result, I keep getting low accuracy (~6%).
What am I doing wrong?
Try changing optimizer to RMSprop with learning rate of around 0.0001.
RMSprop is usually better than most optimizers and gives better accuracy and lesser loss than others. You could alternatively try SGD, which is also a good optimizer.
Also increase number of parameters, as more trainable parameters leads to the model being trained with more precision and gives a much accurate prediction.
You could update the code to tensorflow 2.x and change the code to :
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from sklearn.model_selection import train_test_split
from numpy import loadtxt, random
#Rest of the code
.......
.......
.......
model = Sequential()
model.add(Dense(64, input_shape=(7,), activation='relu', kernel_initializer='uniform'))
model.add(Dense(64, kernel_initializer="uniform", activation="relu"))
model.add(Dense(1, kernel_initializer="uniform", activation="sigmoid"))
model.compile(loss="binary_crossentropy", optimizer=tf.keras.optimizers.RMSprop(learning_rate=0.0001), metrics=["accuracy"])
model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=100, batch_size=5, shuffle=False)
scores = model.evaluate(X_test, Y_test)
print("Accuracy: %.2f%%" %(scores[1] * 100))
Correct me if I'm wrong, but by the looks of it your input is a list of 7 numbers, and you want to output the 7th number in that list. Now by using a sigmoid activation in your last layer, you're restricting your model output to the interval (0,1). Are you sure that your data is in this interval?
Also, your model is way to complicated for that task. You really need only one dense layer without an activation or bias to be able to do this.
I was following the basic classification tutorial of tensorflow. Due to proxy reasons, I have to use the dataset offline. So, instead of using the fashion_mnist database, I am using mnist dataset.
from __future__ import absolute_import, division,
print_function, unicode_literals
# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras
# Helper libraries
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.layers import Flatten, Dense
# Noting class names
class_names = ['Zero', 'One', 'Two', 'Three', 'Four', 'Five', 'Six', 'Seven', 'Eight', 'Nine']
# Load dataset
mnist = keras.datasets.mnist
path = 'C:/projects/VirtualEnvironment/MyScripts/load/mnist.npz'
(train_x, train_y), (test_x, test_y) = mnist.load_data(path)
# Scale, so that training and testing set is preprocessed in the same way
train_x = train_x / 255.0
test_x = test_y / 255.0
train_y = tf.expand_dims(train_y, axis = -1)
test_y = tf.expand_dims(test_y, axis = -1)
#Build the model
#1. Setup the layers
model = keras.Sequential()
model.add(Flatten(input_shape = (28, 28)))
model.add(Dense(128, activation=tf.nn.relu))
model.add(Dense(10, activation=tf.nn.softmax))
#2. Compile the model
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
# Train the model
model.fit(train_x, train_y, epochs=1)
print("Finished Training")
# Evaluate how the model performs on the test dataset
test_loss, test_acc = model.evaluate(test_x, test_y, verbose=2)
I am getting the following error: ValueError: Error when checking input: expected flatten_input to have 3 dimensions, but got array with shape (10000, 1). I know very little about tensorflow, so if someone could guide me in the direction of a useful webpage, or could explain to me what the error means, I'd be much obliged
This is caused by a typo in your code
change this
test_x = test_y / 255.0
to
test_x = test_x / 255.0
This works for me:
(train_x, train_y), (test_x, test_y) = tf.keras.datasets.mnist.load_data()
# Scale, so that training and testing set is preprocessed in the same way
train_x = train_x / 255.0
test_x = test_x / 255.0
model = tf.keras.Sequential()
model.add(tf.keras.layers.Flatten(input_shape = (28, 28)))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))
#2. Compile the model
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
# Train the model
model.fit(train_x, train_y, epochs=1)
# Evaluate the model
test_loss, test_acc = model.evaluate(test_x, test_y, verbose=2)
The error you get means that you have incorrectly reshaped your input somewhere in your code.
I'm not sure about the array shape problem however i can help you out with the proxy so you can properly download the dataset. Assuming you have all clear from your IT for using the tools you are you can set pip proxy by exporting at the terminal level:
Let's say your login credential is COMPANY\username
export http_proxy=http://COMPANY%5Cusername:password#proxy_ip:proxy_port
export https_proxy=http://COMPANY%5Cusername:password#proxy_ip:proxy_port
If you're using a conda environment, .condarc at C:\Users\username and edit as:
channels:
- defaults
# Show channel URLs when displaying what is going to be downloaded and
# in 'conda list'. The default is False.
show_channel_urls: True
allow_other_channels: True
proxy_servers:
http: http://COMPANY\username:password#proxy_ip:proxy_port
https: https://COMPANY\username:password#proxy_ip:proxy_port
ssl_verify: False
Hope it helps. For debugging the array shape i suggest you print train_y and train_x shapes with train_y.shape() and train_x.shape() after expanding the dimensions. The error specifies that its getting a 10000D object with 1 value which shouldn't be the case.
Today I've ran into some very strange behavior of Keras. When I try to do a classification run on the iris-dataset with a simple model, keras version 1.2.2 gives me +- 95% accuracy, whereas a keras version of 2.0+ predicts the same class for every training example (leading to an accuracy of +- 35%, as there are three types of iris). The only thing that makes my model predict +-95% accuracy is downgrading keras to a version below 2.0:
I think it is a problem with Keras, as I have tried the following things, all do not make a difference;
Switching activation function in the last layer (from Sigmoid to softmax).
Switching backend (Theano and Tensorflow both give roughly same performance).
Using a random seed.
Varying the number of neurons in the hidden layer (I only have 1 hidden layer in this simple model).
Switching loss-functions.
As the model is very simple and it runs on it's own (You just need the easy-to-obtain iris.csv dataset) I decided to include the entire code;
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from keras.utils import np_utils
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import LabelEncoder
#Load data
data_frame = pd.read_csv("iris.csv", header=None)
data_set = data_frame.values
X = data_set[:, 0:4].astype(float)
Y = data_set[:, 4]
#Encode class values as integers
encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)
# convert integers to dummy variables (i.e. one hot encoded)
dummy_y = np_utils.to_categorical(encoded_Y)
def baseline_model():
#Create & Compile model
model = Sequential()
model.add(Dense(8, input_dim=4, init='normal', activation='relu'))
model.add(Dense(3, init='normal', activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
return model
#Create Wrapper For Neural Network Model For Use in scikit-learn
estimator = KerasClassifier(build_fn=baseline_model, nb_epoch=200, batch_size=5, verbose=0)
#Create kfolds-cross validation
kfold = KFold(n_splits=10, shuffle=True)
#Evaluate our model (Estimator) on dataset (X and dummy_y) using a 10-fold cross-validation procedure (kfold).
results = cross_val_score(estimator, X, dummy_y, cv=kfold)
print("Accuracy: {:2f}% ({:2f}%)".format(results.mean()*100, results.std()*100))
if anyone wants to replicate the error here are the dependencies I used to observe the problem:
numpy=1.16.4
pandas=0.25.0
sk-learn=0.21.2
theano=1.0.4
tensorflow=1.14.0
In Keras 2.0, many parameters changed names, there is compatibility layer to keep things working, but somehow it did not apply when using KerasClassifier.
In this part of the code:
estimator = KerasClassifier(build_fn=baseline_model, nb_epoch=200, batch_size=5, verbose=0)
You are using the old name nb_epoch instead of the modern name of epochs. The default value is epochs=1, meaning that your model was only being trained for one epoch, producing very low quality predictions.
Also note that here:
model.add(Dense(3, init='normal', activation='sigmoid'))
You should be using a softmax activation instead of sigmoid, as you are using the categorical cross-entropy loss:
model.add(Dense(3, init='normal', activation='softmax'))
I've managed to isolate the issue, if you change nb_epoch to epochs, (All else being exactly equal) the model predicts very good again, in keras 2 as well. I don't know if this is intended behavior or a bug.
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = tf.keras.utils.normalize(x_train, axis=1)
x_test = tf.keras.utils.normalize(x_test, axis=1)
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128,activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(128,activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10,activation=tf.nn.softmax))
model.compile(optimizer ='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=3)
When I tried to save the model
model.save('epic_num_reader.model')
I get a NotImplementedError:
NotImplementedError Traceback (most recent call last)
<ipython-input-4-99efa4bdc06e> in <module>()
1
----> 2 model.save('epic_num_reader.model')
NotImplementedError: Currently `save` requires model to be a graph network. Consider using `save_weights`, in order to save the weights of the model.
So how can I save the model defined in the code?
You forgot the input_shape argument in the definition of the first layer, which makes the model undefined, and saving undefined models has not been implemented yet, which triggers the error.
model.add(tf.keras.layers.Flatten(input_shape = (my, input, shape)))
Just add the input_shape to the first layer and it should work fine.
For those who still have not solved the problem even did as Matias suggested, you can consider using tf.keras.models.save_model() and load_model(). In my case, it worked.
tf.keras.models.save_model
Works here (tensorflow 1.12.0) (even when the input_shape is unspecified)
Reason For the Error:
I was getting the same error and tried the above answers but got errors. But I find a solution to the problem that I will share below:
Check whether you passed input_shape at the time of defining the input layer of the model if not you will get an error at the time of saving and loading the model.
How to define input_shape?
Lets consider the one example If you use minst dataset:
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
It consists of images of handwritten digits 0-9 of size 28 x 28 resolution each.
For this, we can define input shape as (28,28) without mentioning batch size as follows:
model.add(tf.keras.layers.Flatten(input_shape=(28,28)))
In this way, you can give input shape by looking at your input training dataset.
Save your trained model:
Now after training and testing the model we can save our model. Following code worked for me it did not change the accuracy as well after reloading the model:
by using save_model()
import tensorflow as tf
tf.keras.models.save_model(
model,
"your_trained_model.model",
overwrite=True,
include_optimizer=True
)
by using .save()
your_trained_model.save('your_trained_model.model')
del model # deletes the existing model
Now load the model which we saved :
model2 = tf.keras.models.load_model("your_trained_model.model")
For more details refer to this link: Keras input explanation: input_shape, units, batch_size, dim, etc
<!-- Success, please check -->
import tensorflow as tf
import matplotlib.pyplot as plt
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
plt.imshow(x_train[0], cmap=plt.cm.binary)
x_train = tf.keras.utils.normalize(x_train, axis=1)
x_test = tf.keras.utils.normalize(x_test, axis=1)
plt.imshow(x_train[0], cmap=plt.cm.binary)
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten(input_shape=x_train[0].shape))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=3)
val_loss, val_acc = model.evaluate(x_test, y_test)
print(val_loss)
print(val_acc)
model.save('epic_num_reader.model')