I am trying to create my own custom activation function in keras, which would return 0 if x < 0 and 1 if x >= 0
from keras.layers import Dense
from keras.models import Sequential
from keras.layers import Activation
import tensorflow as tf
def hard_lim(x):
zero = tf.convert_to_tensor(0., x.dtype.base_dtype)
one = tf.convert_to_tensor(1., x.dtype.base_dtype)
sess = tf.Session()
if sess.run(tf.greater_equal(x, zero)):
return one
else:
return zero
model = Sequential()
model.add(Dense(4, input_dim=2, activation=Activation(hard_lim))
model.add(Dense(2, activation=Activation(hard_lim))
model.add(Dense(1, activation=Activation(hard_lim))
It's giving me this error
InvalidArgumentError (see above for traceback): You must feed a value
for placeholder tensor '1_input' with dtype float and shape [?,2]
How can I fix it?
Warning: this operation you want has no gradients and will not allow any weights before it to be trainable. You will see error messages like "an operation has None for gradient" or something like "None type not supported".
As a workaround for your activation, I believe the 'relu' activation would be the closest and best option, with the advantage of being very popular and used in most models.
In Keras, you don't usually run sessions. For custom operations, you create a function using backend functions.
So, you'd use a Lambda layer:
import keras.backend as K
def hardlim(x):
return K.cast(K.greater_equal(x,0), K.floatx())
You can then use activation=hardlim in layers.
Related
I am developing an LSTM autoencoder model for anomaly detection. I have my keras model setup as below:
from keras.models import Sequential
from keras import Model, layers
from keras.layers import Layer, Conv1D, Input, Masking, Dense, RNN, LSTM, Dropout, RepeatVector, TimeDistributed, Masking, Reshape
def create_RNN_with_attention():
x=Input(shape=(X_train_dt.shape[1], X_train_dt.shape[2]))
RNN_layer_1 = LSTM(units=64, return_sequences=False)(x)
attention_layer = attention()(RNN_layer_1)
dropout_layer_1 = Dropout(rate=0.2)(attention_layer)
repeat_vector_layer = RepeatVector(n=X_train_dt.shape[1])(dropout_layer_1)
RNN_layer_2 = LSTM(units=64, return_sequences=True)(repeat_vector_layer)
dropout_layer_1 = Dropout(rate=0.2)(RNN_layer_2)
output = TimeDistributed(Dense(X_train_dt.shape[2], trainable=True))(dropout_layer_1)
model=Model(x,output)
model.compile(loss='mae', optimizer='adam')
return model
Notice the attention layer that I added, attention_layer. Before adding this, the model compiled perfectly, however after adding this attention_layer - the model is throwing out the following error: ValueError: Input 0 is incompatible with layer repeat_vector_40: expected ndim=2, found ndim=1
My attention layer is setup as follows:
import keras.backend as K
class attention(Layer):
def __init__(self,**kwargs):
super(attention,self).__init__(**kwargs)
def build(self,input_shape):
self.W=self.add_weight(name='attention_weight', shape=(input_shape[-1],1),
initializer='random_normal', trainable=True)
self.b=self.add_weight(name='attention_bias', shape=(input_shape[1],1),
initializer='zeros', trainable=True)
super(attention, self).build(input_shape)
def call(self,x):
# Alignment scores. Pass them through tanh function
e = K.tanh(K.dot(x,self.W)+self.b)
# Remove dimension of size 1
e = K.squeeze(e, axis=-1)
# Compute the weights
alpha = K.softmax(e)
# Reshape to tensorFlow format
alpha = K.expand_dims(alpha, axis=-1)
# Compute the context vector
context = x * alpha
context = K.sum(context, axis=1)
return context
The idea of the attention mask is to allow the model to focus on more prominent features as is trains.
Why am I getting the error above and how can I fix this?
I think that the problem lies in this line:
RNN_layer_1 = LSTM(units=64, return_sequences=False)(x)
This layer outputs a tensor of shape (batch_size, 64). So this means that you output a vector and then run attention mechanism on w.r.t. to the batch dimension instead of a sequential dimension. This also means that you output with a squeezed batch dimension that is not acceptable for any keras layer. This is why the Repeat layer raises error as it expects vector of at least shape (batch_dimension, dim).
If you want to run attention mechanism over a sequence then you should switch the line mentioned above to:
RNN_layer_1 = LSTM(units=64, return_sequences=True)(x)
This is a very simple problem that I cannot get around. I am new to tensorflow and this is the second time I am facing this problem.
from tensorflow.keras.layers import Dense, Conv2D, MaxPooling2D, Dropout, Flatten, Input
from tensorflow.keras.models import Model
import numpy as np
x = tf.keras.Input(shape=(128, 128, 4))
conv = Conv2D(30, (3, 3), activation='relu',input_shape=(128, 128, 4))(x)
conv = Conv2D(12, (5,5))(conv)
conv = MaxPooling2D(pool_size=(2,2))(conv)
print(conv[2])
conv = np.array(conv[2]) # <---- here is the problem
input_mean = np.mean(conv[1:], axis=0)
input_std = np.std(conv, axis=0)
conv = (conv - input_mean) / input_std
conv = Flatten()(conv)
conv = Dense(157, activation='relu')(conv)
model = Model(inputs = x, outputs = conv)
#model.summary()
The error that I am getting is,
Cannot convert a symbolic Keras input/output to a numpy array. This error may indicate that you're trying to pass a symbolic value to a NumPy call, which is not supported. Or, you may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.
My question is, How would I take the Output from my Maxpooling layer and take the mean and standard deviation for each incoming channel? The output of the mean and std would be a tensor where each channel is separately normalized. I would then flatten this output and send it to my fully connected dense layer.
Thanks in advance.
I obtained a similar error and I performed the following:
del model
Before:
model = Model(inputs = x, outputs = conv)
It resolved my issue.
I am eager to know if it solves your issue too:) .
I have a few tensors in my code and and need to get the values of those tensors. This is one them. How to print the values of tensor OA?
Input:OA
Output: <tf.Tensor 'Sum_1:0' shape=(1, 600) dtype=float32>
Input:type(OA)
Output: tensorflow.python.framework.ops.Tensor
I have tried all the available functions like tf.print(), eval(), tensor.numpy(). None of them worked for me in Tensorflow 2.0. It seems they work only for 'EagerTensor' and not for 'ops.Tensor'.
1) OA.eval(session=sess)
Error: ValueError: Cannot use the given session to evaluate tensor: the tensor's graph is different from the session's graph.
2) tf.print(OA)
Output:
3) print (OA.numpy())
Output: AttributeError: 'Tensor' object has no attribute 'numpy'
Is there any way to convert ops.Tensor to EagerTensor to try the above functions? Or is there any other option to print the values of ops.Tensor. Please advise.
--Adding the minimal code to reproduce the example ops.Tensor in TF2.0.
!pip install tensorflow==2.0.0
tf.__version__
import tensorflow as tf
from keras.layers import Dense, Conv1D, MaxPooling1D, Flatten, Dropout, Input, Embedding, Bidirectional, LSTM
from tensorflow.keras import regularizers
EMBEDDING_DIM = 300
max_length = 120
batch_size = 512
vocab_size = 1000
units = 300
from keras.layers import Dense, Conv1D, MaxPooling1D, Flatten, Dropout, Input, Embedding, Bidirectional, LSTM
from tensorflow.keras import regularizers
input_text = tf.keras.Input(shape= (max_length), batch_size=batch_size)
embedding_layer = tf.keras.layers.Embedding(vocab_size, EMBEDDING_DIM, input_length =max_length, name="Embedding_Layer_1")
embedding_sequence = embedding_layer(input_text)
HQ = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(units,recurrent_dropout=0.5,kernel_regularizer=regularizers.l2(0.001),return_sequences=True,name='Bidirectional_1'))(embedding_sequence)
HQ = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(units,recurrent_dropout=0.5,kernel_regularizer=regularizers.l2(0.001),name='Bidirectional_2'))(HQ)
print (HQ)
Output: Tensor("bidirectional_3/concat:0", shape=(512, 600), dtype=float32)
type(HQ)
Output: tensorflow.python.framework.ops.Tensor
How to check the actual values of this tensor?
Your graph is not complete at the point you are printing HQ. You need to complete the model creation. Presumably something like
output = tf.keras.layers.xyz()(HQ)
model = tf.keras.models.Model(input_text, output)
The trick to print an intermediate layer is to just make it an output. You can make it an additional output of your existing model temporarily, or just make a new model.
inspection_model = tf.keras.models.Model(input_text, [output, HQ])
now run inference on your inspection_model to get the value of the intermediate activation HQ.
print(inspection_model(xyz))
Directly you cannot print the values of tensors the way you are trying to do. Tensorflow 2.x by default runs in eager mode and also you have given no input to your incomplete model.
The way to do it is by using the custom training loop. Suppose the layers mentioned in your code stack up to create a model my_model.
from tensorflow.keras import Model
my_model = Model(input_text, outputs=HQ)
with tf.GradientTape() as t:
HQ_predictions = my_model(input_data)
print(HQ_predictions)
Use .numpy() attribute like :
your_tensor.numpy()
I am fairly new to tensorflow and I was following the answer to the question below in order to build a custom loss function in Keras that considers only the top 20 predictions.
How can I sort the values in a custom Keras / Tensorflow Loss Function?
However, when I try to compile my model using this code I get the following error about dimensions
InvalidArgumentError: input must have last dimension >= k = 20 but is 1 for 'loss_21/dense_65_loss/TopKV2' (op: 'TopKV2') with input shapes: [?,1], [] and with computed input tensors: input[1] = <20>.
A simplified version of the code that re-produces the error is the following.
import tensorflow as tf
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import Dense
from tensorflow.python.keras.optimizers import SGD
top = 20
def top_loss(y_true, y_pred):
y_pred_top_k, y_pred_ind_k = tf.nn.top_k(y_pred, top)
loss_per_sample = tf.reduce_mean(tf.reduce_sum(y_pred_top_k,
axis=-1))
return loss_per_sample
model = Sequential()
model.add(Dense(50, input_dim=201))
model.add(Dense(1))
sgd = SGD(lr=0.01, decay=0, momentum=0.9)
model.compile(loss=top_loss, optimizer=sgd)
and the error is thrown at the following line of the top_loss function when the model is compiled.
y_pred_top_k, y_pred_ind_k = tf.nn.top_k(y_pred, top)
It seems that y_pred in compile time is by default of shape [?,1] while the tf.nn.top_k function expects dimension at least higher than 'k` (i.e. 20).
Do I have to cast y_pred to something so that tf.nn.top_k knows it is of the correct dimensions?
Use:
y_pred_top_k, y_pred_ind_k = tf.nn.top_k(y_pred[:,0], top)
y_pred[:,0] gets the predicted values of the full batch as a rank 1 tensor.
Another Problem:
However, you will still end up with problem with the last batch. Say your batch size is 32 and your train data is of size 100 then the last batch will be of size less then 20 and so tf.nn.top_k will result in a run time error for the last batch. Just make sure your last batch size is >= 20 to avoid this issue. However a much better way is to check if the current batch is less then 20 and if so adjust your k value to be used in the top_k
Code
import tensorflow as tf
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import Dense
from tensorflow.python.keras.optimizers import SGD
top = tf.constant(20)
def top_loss(y_true, y_pred):
result = tf.cond(tf.math.greater(top_, tf.shape(y_true)[0]),
lambda: tf.shape(y_true)[0], lambda: top)
y_pred_top_k, y_pred_ind_k = tf.nn.top_k(y_pred[:,0], result)
loss_per_sample = tf.reduce_mean(tf.reduce_sum(y_pred_top_k,
axis=-1))
return loss_per_sample
model = Sequential()
model.add(Dense(50, input_dim=201))
model.add(Dense(1))
sgd = SGD(lr=0.01, decay=0, momentum=0.9)
model.compile(loss=top_loss, optimizer=sgd)
I'm new to Keras and python, now I'm working on Keras to find a model of data and use that model.predict for optimization, however the model.predict can only take input as numpy array of at least 2 elements.
My code is
import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
import numpy as np
x = np.arange(-2,3.0,0.01)
y = x**2 - 2*x + 1
model = Sequential()
model.add(Dense(50, activation='sigmoid',
input_dim=1, init='uniform'))
model.add(Dense(1, activation='linear'))
sgd = SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=False)
model.compile(loss='mean_squared_error',
optimizer='sgd',
metrics=['accuracy'])
model.fit(x,y,nb_epoch=300, batch_size = 5,verbose = 0)
The code can fit fine, but if I try to use model.predict for a scalar number it gives me error
(Pdb) model.predict(0.0)
*** Exception: Error when checking : data should be a Numpy array, or list/dict of Numpy arrays. Found: 0.0...
I force it to be numpy array but still failed, and it said the input needs to be 2 dimensions!!!
(Pdb) model.predict(np.asarray(0.0))
*** Exception: Error when checking : expected dense_input_1 to have 2 dimensions, but got array with shape ()
but if I input two numbers then it gives me the answer
(Pdb) model.predict([0.0,0.0])
array([[ 1.07415712],
[ 1.07415712]], dtype=float32)
I need the model.predict to take single number as input to use for optimization. I'm not sure any setting I use wrong. Please help, thanks.
Try:
model.predict(np.asarray(0.0).reshape((1,1)))
In Keras first dimension is always connected with example number, so it must be provided.