We Keep Coding

How to convert to Keras code from MATLAB Deep learning model

I am making the binary sound classification model by Keras on Python3.7. I have been make the sound classification model on MATLAB however some specifically layer is not installed on MATLAB (ex. GRU). So I try to convert to Keras deep learning model from MATLAB deep learning model.
The original MATLAB code is shown bellow:
inputsize=[31,69]
layers = [ ...
sequenceInputLayer(inputsize(1))
bilstmLayer(200,'OutputMode','last')
fullyConnectedLayer(2)
softmaxLayer
classificationLayer
]
options = trainingOptions('adam', ...
'MaxEpochs',30, ...
'MiniBatchSize', 200, ...
'InitialLearnRate', 0.01, ...
'GradientThreshold', 1, ...
'ExecutionEnvironment',"auto",...
'plots','training-progress', ...
'Verbose',false);
This model get to the accuracy is 0.955.
The Keras code based on MATLAB code is shown below:
# traindatasize=(86400,31,69)
inputsize=(31,69)
batchsize=200
epochs=30
model = Sequential()
model.add(Bidirectional(LSTM(200, input_shape=inputsize)))
model.add(Dense(2, activation='softmax'))
model.compile(optimizer=RMSprop(), loss='binary_crossentropy', metrics=['accuracy'])
model.fit(traindata, trainlabel, batch_size=batchsize, epochs=epochs, verbose=1)
This model get to the accuracy is 0.444
I don't understand what is the effect.
The traindata used same data from STFT and normalize before train those model using standard deviation and mean average.
Please some comments.
Python 3.7 on Anaconda
Keras 2.2.4

I think that's because the MATLAB code uses the Adam optimizer for training, and you defined RMSprop instead in:
model.compile(optimizer=RMSprop(),loss='binary_crossentropy',metrics=['accuracy'])
instead, use:
from keras import optimizers
adam = optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False)
...
model.compile(optimizer=adam,loss='binary_crossentropy',metrics=['accuracy'])
check if this improves the answer.

batch normalization, yes or no?

I use Tensorflow 1.14.0 and Keras 2.2.4. The following code implements a simple neural network:
import numpy as np
np.random.seed(1)
import random
random.seed(2)
import tensorflow as tf
tf.set_random_seed(3)
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.layers import Input, Dense, Activation
x_train=np.random.normal(0,1,(100,12))
model = Sequential()
model.add(Dense(8, input_shape=(12,)))
# model.add(tf.keras.layers.BatchNormalization())
model.add(Activation('linear'))
model.add(Dense(12))
model.add(Activation('linear'))
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x_train, x_train,epochs=20, validation_split=0.1, shuffle=False,verbose=2)
The final val_loss after 20 epochs is 0.7751. When I uncomment the only comment line to add the batch normalization layer, the val_loss changes to 1.1230.
My main problem is way more complicated, but the same thing occurs. Since my activation is linear, it does not matter if I put the batch normalization after or before the activation.
Questions: Why batch normalization cannot help? Is there anything I can change so that the batch normalization improves the result without changing the activation functions?
Update after getting a comment:
An NN with one hidden layer and linear activations is kind of like PCA. There are tons of papers on this. For me, this setting gives minimal MSE among all combinations of activation functions for the hidden layer and output.
Some resources that state linear activations mean PCA:
https://arxiv.org/pdf/1702.07800.pdf
https://link.springer.com/article/10.1007/BF00275687
https://www.quora.com/How-can-I-make-a-neural-network-to-work-as-a-PCA

Yes.
The behavior you're observing is a bug - and you don't need BN to see it; plot to the left is for #V1, to the right is for #V2:
#V1
model = Sequential()
model.add(Dense(8, input_shape=(12,)))
#model.add(Activation('linear')) <-- uncomment == #V2
model.add(Dense(12))
model.compile(optimizer='adam', loss='mean_squared_error')
Clearly nonsensical, as Activation('linear') after a layer with activation=None (=='linear') is an identity: model.layers[1].output.name == 'activation/activation/Identity:0'. This can be confirmed further by fetching and plotting intermediate layer outputs, which are identical for 'dense' and 'activation' - will omit here.
So, the activation does literally nothing, except it doesn't - somewhere along the commit chain between 1.14.0 and 2.0.0, this was fixed, though I don't know where. Results w/ BN using TF 2.0.0 w/ Keras 2.3.1 below:
val_loss = 0.840 # without BN
val_loss = 0.819 # with BN
Solution: update to TensorFlow 2.0.0, Keras 2.3.1.
Tip: use Anaconda w/ virtual environment. If you don't have any virtual envs yet, run:
conda create --name tf2_env --clone base
conda activate tf2_env
conda uninstall tensorflow-gpu
conda uninstall keras
conda install -c anaconda tensorflow-gpu==2.0.0
conda install -c conda-forge keras==2.3.1
May be a bit more involved than this, but that's subject of another question.
UPDATE: importing from keras instead of tf.keras also solves the problem.
Disclaimer: BN remains a 'controversial' layer in Keras, yet to be fully fixed - see Relevant Git; I plan on investigating it myself eventually, but for your purposes, this answer's fix should suffice.
I also recommend familiarizing yourself with BN's underlying theory, in particular regarding its train vs. inference operation; in a nutshell, batch sizes under 32 is a pretty bad idea, and dataset should be sufficiently large to allow BN to accurately approximate test-set gamma and beta.
Code used:
x_train=np.random.normal(0, 1, (100, 12))
model = Sequential()
model.add(Dense(8, input_shape=(12,)))
#model.add(Activation('linear'))
#model.add(tf.keras.layers.BatchNormalization())
model.add(Dense(12))
model.compile(optimizer='adam', loss='mean_squared_error')
W_sum_all = [] # fit rewritten to allow runtime weight collection
for _ in range(20):
for i in range(9):
x = x_train[i*10:(i+1)*10]
model.train_on_batch(x, x)
W_sum_all.append([])
for layer in model.layers:
if layer.trainable_weights != []:
W_sum_all[-1] += [np.sum(layer.get_weights()[0])]
model.evaluate(x[-10:], x[-10:])
plt.plot(W_sum_all)
plt.title("Sum of weights (#V1)", weight='bold', fontsize=14)
plt.legend(labels=["dense", "dense_1"], fontsize=14)
plt.gcf().set_size_inches(7, 4)
Imports/pre-executions:
import numpy as np
np.random.seed(1)
import random
random.seed(2)
import tensorflow as tf
if tf.__version__[0] == '2':
tf.random.set_seed(3)
else:
tf.set_random_seed(3)
import matplotlib.pyplot as plt
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.layers import Input, Dense, Activation

keras load model error trying to load a weight file containing 17 layers into a model with 0 layers

I am currently working on vgg16 model with keras.
I fine tune vgg model with some of my layer.
After fitting my model (training), I save my model with model.save('name.h5').
It can be saved without problem.
However, when I try to reload the model with load_model function, it shows the error:
You are trying to load a weight file containing 17 layers into a model
with 0 layers
Did anyone meet this problem before?
My keras verion is 2.2.
Here is part of my code ...
from keras.models import load_model
vgg_model = VGG16(weights='imagenet',include_top=False,input_shape=(224,224,3))
global model_2
model_2 = Sequential()
for layer in vgg_model.layers:
model_2.add(layer)
for layer in model_2.layers:
layer.trainable= False
model_2.add(Flatten())
model_2.add(Dense(128, activation='relu'))
model_2.add(Dropout(0.5))
model_2.add(Dense(2, activation='softmax'))
model_2.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model_2.fit(x=X_train,y=y_train,batch_size=32,epochs=30,verbose=2)
model_2.save('name.h5')
del model_2
model_2 = load_model('name.h5')
Actually I do not delete the model and then load_model immediately,
just for showing my problem.

It seems that this problem is related with the input_shape parameter of the first layer. I had this problem with a wrapper layer (Bidirectional) which did not have an input_shape parameter set. In code:
model.add(Bidirectional(LSTM(units=units, input_shape=(None, feature_size)), merge_mode='concat'))
did not work for loading my old model because the input_shape is only defined for the LSTM layer not the outer one. Instead
model.add(Bidirectional(LSTM(units=units), input_shape=(None, feature_size), merge_mode='concat'))
worked because the wrapper Birectional layer now has an input_shape parameter. Maybe you should check if the VGG net input_shape parameter is set or not or you should add a single input_layer to your model with the correct input_shape parameter.

I spent 6 hours looking around for a solution.. to apply me trained model.
finally i tried VGG16 as model and using h5 weights i´ve trained on my own and Great!
weights_model='C:/Anaconda/weightsnew2.h5' # my already trained weights .h5
vgg=applications.vgg16.VGG16()
cnn=Sequential()
for capa in vgg.layers:
cnn.add(capa)
cnn.layers.pop()
for layer in cnn.layers:
layer.trainable=False
cnn.add(Dense(2,activation='softmax'))
cnn.load_weights(weights_model)
def predict(file):
x = load_img(file, target_size=(longitud, altura))
x = img_to_array(x)
x = np.expand_dims(x, axis=0)
array = cnn.predict(x)
result = array[0]
respuesta = np.argmax(result)
if respuesta == 0:
print("Gato")
elif respuesta == 1:
print("Perro")

In case anyone is still wondering about this error:
I had the same Problem and spent days figuring out, whats causing it. I have a copy of my whole code and dataset on another system on which it worked. I noticed that it is something about the training, because without training my model, saving and loading was no problem.
The only difference between my systems was, that I was using tensorflow-gpu on my main system and for this reason, the tensorflow base version was a little bit lower (1.14.0 instead of 2.2.0). So all I had to do was using
model.fit_generator()
instead of
model.fit()
before saving it. And it works

"Could not interpret optimizer identifier" error in Keras

I got this error when I tried to modify the learning rate parameter of SGD optimizer in Keras. Did I miss something in my code or my Keras was not installed properly?
Here is my code:
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import Dense, Flatten, GlobalAveragePooling2D, Activation
import keras
from keras.optimizers import SGD
model = Sequential()
model.add(Dense(64, kernel_initializer='uniform', input_shape=(10,)))
model.add(Activation('softmax'))
model.compile(loss='mean_squared_error', optimizer=SGD(lr=0.01), metrics= ['accuracy'])*
and here is the error message:
Traceback (most recent call last): File
"C:\TensorFlow\Keras\ResNet-50\test_sgd.py", line 10, in
model.compile(loss='mean_squared_error', optimizer=SGD(lr=0.01), metrics=['accuracy']) File
"C:\Users\nsugiant\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\keras_impl\keras\models.py",
line 787, in compile
**kwargs) File "C:\Users\nsugiant\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\keras_impl\keras\engine\training.py",
line 632, in compile
self.optimizer = optimizers.get(optimizer) File "C:\Users\nsugiant\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\keras_impl\keras\optimizers.py",
line 788, in get
raise ValueError('Could not interpret optimizer identifier:', identifier) ValueError: ('Could not interpret optimizer identifier:',
<keras.optimizers.SGD object at 0x000002039B152FD0>)

The reason is you are using tensorflow.python.keras API for model and layers and keras.optimizers for SGD. They are two different Keras versions of TensorFlow and pure Keras. They could not work together. You have to change everything to one version. Then it should work.

I am bit late here, Your issue is you have mixed Tensorflow keras and keras API in your code. The optimizer and the model should come from same layer definition. Use Keras API for everything as below:
from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM, BatchNormalization
from keras.callbacks import TensorBoard
from keras.callbacks import ModelCheckpoint
from keras.optimizers import adam
# Set Model
model = Sequential()
model.add(LSTM(128, input_shape=(train_x.shape[1:]), return_sequences=True))
model.add(Dropout(0.2))
model.add(BatchNormalization())
# Set Optimizer
opt = adam(lr=0.001, decay=1e-6)
# Compile model
model.compile(
loss='sparse_categorical_crossentropy',
optimizer=opt,
metrics=['accuracy']
)
I have used adam in this example. Please use your relevant optimizer as per above code.
Hope this helps.

This problem is mainly caused due to different versions. The tensorflow.keras version may not be same as the keras.
Thus causing the error as mentioned by #Priyanka.
For me, whenever this error arises, I pass in the name of the optimizer as a string, and the backend figures it out.
For example instead of
tf.keras.optimizers.Adam
or
keras.optimizers.Adam
I do
model.compile(optimizer= 'adam' , loss= keras.losses.binary_crossentropy, metrics=['accuracy'])

from tensorflow.keras.optimizers import SGD
This works well.
Since Tensorflow 2.0, there is a new API available directly via tensorflow:
https://www.pyimagesearch.com/2019/10/21/keras-vs-tf-keras-whats-the-difference-in-tensorflow-2-0/
Solution works for tensorflow==2.2.0rc2, Keras==2.2.4 (on Win10)
Please also note that the version above uses learning_rate as parameter and no longer lr.

For some libraries (e.g. keras_radam) you'll need to set up an environment variable before the import:
import os
os.environ['TF_KERAS'] = '1'
import tensorflow
import your_library

In my case it was because I missed the parentheses. I am using tensorflow_addons so my code was like
model.compile(optimizer=tfa.optimizers.LAMB, loss='binary_crossentropy',
metrics=['binary_accuracy'])
And it gives
ValueError: ('Could not interpret optimizer identifier:', <class tensorflow_addons.optimizers.lamb.LAMB'>)
Then I changed my code into:
model.compile(optimizer=tfa.optimizers.LAMB(), loss='binary_crossentropy',
metrics=['binary_accuracy'])
and it works.

recently, in the latest update of Keras API 2.5.0 , importing Adam optimizer shows the following error:
from keras.optimizers import Adam
ImportError: cannot import name 'Adam' from 'keras.optimizers'
instead use the following for importing optimizers (i.e. Adam) :
from keras.optimizers import adam_v2
optimizer = adam_v2.Adam(learning_rate=lr, decay=lr/epochs)
Model.compile(loss='--', optimizer=optimizer , metrics=['--'])

Running the Keras documentaion example https://keras.io/examples/cifar10_cnn/
and installing the latest keras and tensor flow versions
(at the time of this writing
tensorflow 2.0.0a0 and Keras version 2.2.4 )
I had to import explicitly the optimizer the keras the example is using,specifically the line on top of the example :
opt = tensorflow.keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)
was replaced by
from tensorflow.keras.optimizers import RMSprop
opt = RMSprop(lr=0.0001, decay=1e-6)
In the recent version the api "broke" and keras.stuff in a lot of cases became tensorflow.keras.stuff.

Use one style in one kernel, try not to mix
from keras.optimizers import sth
with
from tensorflow.keras.optimizers import sth

I tried the following and it worked for me:
from keras import optimizers
sgd = optimizers.SGD(lr=0.01)
model.compile(loss='mean_squared_error', optimizer=sgd)

use
from tensorflow.keras import optimizers
instead of
from keras import optimizers

Try changing your import lines to
from keras.models import Sequential
from keras.layers import Dense, ...
Your imports seem a little strange to me. Maybe you could elaborate more on that.

I have misplaced parenthesis and got this error,
Initially it was
x=Conv2D(filters[0],(3,3),use_bias=False,padding="same",kernel_regularizer=l2(reg),x))
The corrected version was
x=Conv2D(filters[0],(3,3),use_bias=False,padding="same",kernel_regularizer=l2(reg))(x)

Just give
optimizer = 'sgd' / 'RMSprop'

I got the same error message and resolved this issue, in my case, by replacing the assignment of optimizer:
optimizer=keras.optimizers.Adam
with its instance instead of the class itself:
optimizer=keras.optimizers.Adam()

I tried everything in this thread to fix it but they didn't work. However, I managed to fix it for me. For me, the issue was that calling the optimizer class, ie. tensorflow.keras.optimizers.Adam caused the error, but calling the optimizer as a function, ie. tensorflow.keras.optimizers.Adam() worked. So my code looks like:
model.compile(
loss=tensorflow.keras.losses.categorical_crossentropy(),
optimizer=tensorflow.keras.optimizers.Adam()
)
Looking at the tensorflow github, I am not the only one with this error where calling the function rather than the class fixed the error.

Keras model load_weights for Neural Net

I'm using the Keras library to create a neural network in python. I have loaded the training data (txt file), initiated the network and "fit" the weights of the neural network. I have then written code to generate the output text. Here is the code:
#!/usr/bin/env python
# load the network weights
filename = "weights-improvement-19-2.0810.hdf5"
model.load_weights(filename)
model.compile(loss='categorical_crossentropy', optimizer='adam')
My problem is: on execution the following error is produced:
model.load_weights(filename)
NameError: name 'model' is not defined
I have added the following but the error still persists:
from keras.models import Sequential
from keras.models import load_model
Any help would be appreciated.

you need to first create the network object called model, compile it and only after call the model.load_weights(fname)
working example:
from keras.models import Sequential
from keras.layers import Dense, Activation
def build_model():
model = Sequential()
model.add(Dense(output_dim=64, input_dim=100))
model.add(Activation("relu"))
model.add(Dense(output_dim=10))
model.add(Activation("softmax"))
# you can either compile or not the model
model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
return model
model1 = build_model()
model1.save_weights('my_weights.model')
model2 = build_model()
model2.load_weights('my_weights.model')
# do stuff with model2 (e.g. predict())
Save & Load an Entire Model
in Keras we can save & load the entire model like this (more info here):
from keras.models import load_model
model1 = build_model()
model1.save('my_model.hdf5')
model2 = load_model('my_model.hdf5')
# do stuff with model2 (e.g. predict()