I am trying to make my own activation function which is f(X) = 2/pi * arctan(x) when x<0 and x/1+abs(x) when x >=0
Iam trying to do it in keras and i used this code :
from keras import backend as k
def fun8(y):
return ((2/k.pi) * k.atan2(y)) if y<0 else (y/1+k.abs(y))
and i used it in my model :
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), kernel_regularizer=regularizers.l2(w_l2),
input_shape=input_shape, kernel_initializer='he_normal'))
model.add(BatchNormalization())
model.add(Activation(fun8))
model.add(Conv2D(64, (3, 3), kernel_regularizer=regularizers.l2(w_l2),
kernel_initializer='he_normal'))
model.add(BatchNormalization())
model.add(Activation(fun8))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, kernel_regularizer=regularizers.l2(w_l2), kernel_initializer='he_normal'))
model.add(BatchNormalization())
model.add(Activation(fun8))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adam(),
metrics=['accuracy'])
model.summary()
and it gave me that error:
TypeError: Using a `tf.Tensor` as a Python `bool` is not allowed. Use `if t is not None:` instead of `if t:` to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.
The problem is in the expression if y<0. y<0 is a vector of Booleans; if requires a scalar argument. What you're trying to do is a filter type of operation; you can't do it with this simple syntax. Instead, you'll need to use y<0 as a tensor, and adjust you expression accordingly. #Marat already gave you one possibility (although missing the "equals" clause on one side or the other).
Related
So basically, I trained a model with a resize layer. Here is my model:
model = Sequential()
model.add(keras.layers.Lambda(
lambda image: tf.image.resize(
image,
(470,470),
method = tf.image.ResizeMethod.BICUBIC,
preserve_aspect_ratio = True
)
))
model.add(Conv2D(32, (3, 3), activation='relu', input_shape=shape))
model.add(MaxPool2D((2, 2)))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPool2D((2, 2)))
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(MaxPool2D((2, 2)))
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.33))
model.add(Dense(128, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
Before using the first layer, I could save and load the model, but now when loading I get this error:
ValueError: The channel dimension of the inputs should be defined. The input_shape received is (None, None, None, None), where axis -1 (0-based) is the channel dimension, which found to be `None`.
I loaded my data first like this:
model2 = load_model('catsanddogs.h5')
and also tried some solutions from a github issue that made it like this model =load_model('catsanddogs.h5',custom_objects={"tf":tf})
Does anyone know how to properly load this model?
To anyone that has this issue in the future, instead of using a lambda layer, use a keras resizing layer. Also the h5 file that wouldn't load still works on huggingface.
I am trying to produce a CNN using Keras, and wrote the following code:
batch_size = 64
epochs = 20
num_classes = 5
cnn_model = Sequential()
cnn_model.add(Conv2D(32, kernel_size=(3, 3), activation='linear',
input_shape=(380, 380, 1), padding='same'))
cnn_model.add(Activation('relu'))
cnn_model.add(MaxPooling2D((2, 2), padding='same'))
cnn_model.add(Conv2D(64, (3, 3), activation='linear', padding='same'))
cnn_model.add(Activation('relu'))
cnn_model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
cnn_model.add(Conv2D(128, (3, 3), activation='linear', padding='same'))
cnn_model.add(Activation('relu'))
cnn_model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
cnn_model.add(Flatten())
cnn_model.add(Dense(128, activation='linear'))
cnn_model.add(Activation('relu'))
cnn_model.add(Dense(num_classes, activation='softmax'))
cnn_model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adam(), metrics=['accuracy'])
I want to use Keras's LeakyReLU activation layer instead of using Activation('relu'). However, I tried using LeakyReLU(alpha=0.1) in place, but this is an activation layer in Keras, and I get an error about using an activation layer and not an activation function.
How can I use LeakyReLU in this example?
All advanced activations in Keras, including LeakyReLU, are available as layers, and not as activations; therefore, you should use it as such:
from keras.layers import LeakyReLU
# instead of cnn_model.add(Activation('relu'))
# use
cnn_model.add(LeakyReLU(alpha=0.1))
Sometimes you just want a drop-in replacement for a built-in activation layer, and not having to add extra activation layers just for this purpose.
For that, you can use the fact that the activation argument can be a callable object.
lrelu = lambda x: tf.keras.activations.relu(x, alpha=0.1)
model.add(Conv2D(..., activation=lrelu, ...)
Since a Layer is also a callable object, you could also simply use
model.add(Conv2D(..., activation=tf.keras.layers.LeakyReLU(alpha=0.1), ...)
which now works in TF2. This is a better solution as this avoids the need to use a custom_object during loading as #ChristophorusReyhan mentionned.
you can import the function to make the code cleaner and then use it like any other activation.
if you choose not to define alpha, don't forget to add brackets "LeakyReLU()"
from tensorflow.keras.layers import LeakyReLU
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(512, activation=LeakyReLU()))
model.add(tf.keras.layers.Dense(512, activation=LeakyReLU(alpha=0.1)))
I am working in python and tensor flow but I miss 'units' argument and I do not know how to solve it, It looks like your post is mostly code; please add some more details.It looks like your post is mostly code; please add some more details.
here the code
def createModel():
model = Sequential()
# first set of CONV => RELU => MAX POOL layers
model.add(Conv2D(32, (3, 3), padding='same', activation='relu', input_shape=inputShape))
model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(output_dim=NUM_CLASSES, activation='softmax'))
# returns our fully constructed deep learning + Keras image classifier
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
# use binary_crossentropy if there are two classes
model.compile(loss="categorical_crossentropy", optimizer=opt, metrics=["accuracy"])
return model
print("Reshaping trainX at..."+ str(datetime.now()))
#print(trainX.sample())
print(type(trainX)) # <class 'pandas.core.series.Series'>
print(trainX.shape) # (750,)
from numpy import zeros
Xtrain = np.zeros([trainX.shape[0],HEIGHT, WIDTH, DEPTH])
for i in range(trainX.shape[0]): # 0 to traindf Size -1
Xtrain[i] = trainX[i]
print(Xtrain.shape) # (750,128,128,3)
print("Reshaped trainX at..."+ str(datetime.now()))
print("Reshaping valX at..."+ str(datetime.now()))
print(type(valX)) # <class 'pandas.core.series.Series'>
print(valX.shape) # (250,)
from numpy import zeros
Xval = np.zeros([valX.shape[0],HEIGHT, WIDTH, DEPTH])
for i in range(valX.shape[0]): # 0 to traindf Size -1
Xval[i] = valX[i]
print(Xval.shape) # (250,128,128,3)
print("Reshaped valX at..."+ str(datetime.now()))
# initialize the model
print("compiling model...")
sys.stdout.flush()
model = createModel()
# print the summary of model
from keras.utils import print_summary
print_summary(model, line_length=None, positions=None, print_fn=None)
# add some visualization
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot
SVG(model_to_dot(model).create(prog='dot', format='svg'))
Try changing this line:
model.add(Dense(output_dim=NUM_CLASSES, activation='softmax'))
to
model.add(Dense(NUM_CLASSES, activation='softmax'))
I'm not experience in keras but I could not find a parameter called output_dim on the documentation page for Dense. I think you meant to provide units but labelled it as output_dim
The Keras Dense layer documentation is as follows:
keras.layers.Dense(units, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)
Using the following :
classifier.add(Dense(6, activation='relu', kernel_initializer='glorot_uniform',input_dim=11))
Will work as here the units means the output_dim saying that we need 6 neurons in the hidden layer. The weights are initialized with the uniform function and the input layer has 11 independent variables of the dataset (input_dim) to feed the above-hidden layer.
I think it's a version issue. In updated version of keras for Dense there is no "output_dim" argument.
You can see this documentation link for Dense arguments.
https://keras.io/api/layers/core_layers/dense/
tf.keras.layers.Dense(
units,
activation=None,
use_bias=True,
kernel_initializer="glorot_uniform",
bias_initializer="zeros",
kernel_regularizer=None,
bias_regularizer=None,
activity_regularizer=None,
kernel_constraint=None,
bias_constraint=None,
**kwargs
)
So the first argument is "units", Which is mandatory.
instead of this line:
model.add(Dense(output_dim=NUM_CLASSES, activation='softmax'))
use this:
model.add(Dense(units=NUM_CLASSES, activation='softmax'))
or
model.add(Dense(NUM_CLASSES, activation='softmax'))
I'm trying to combine LSTM with CNN but I got stuck because of an error.
Here is the model I'm trying to implement:
model=Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(28, 28,3), activation='relu'))
model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(32, activation='relu'))
model.add(LSTM(128, return_sequences=True,input_shape=(1,32), activation='relu'))
model.add(LSTM(256))
model.add(Dropout(0.25))
model.add(Dense(37))
model.compile(loss='categorical_crossentropy', optimizer='adam')
and error happens in the first LSTM layer:
ERROR: Input 0 is incompatible with layer lstm_12: expected ndim=3, found ndim=2
The input of LSTM layer should be a 3D array which represents a sequence or a timeseries (this is what the error is trying to say: expected ndim=3). However, in your model the input of LSTM layer, which is actually the output of the Dense layer before it, is a 2D array (i.e. found ndim=2). To make it into a 3D array of shape (n_samples, n_timesteps, n_features), one solution is to use a RepeatVector layer to repeat it as much as the number of timesteps (which you need to specify in your code):
model.add(Dense(32, activation='relu'))
model.add(RepeatVector(n_timesteps))
model.add(LSTM(128, return_sequences=True, input_shape=(n_timesteps,32), activation='relu'))
I'm using a custom loss function:
def ratio_loss(y, y0):
return (K.mean(K.abs(y - y0) / y))
and get negative predicted values - which in my case doesn't makes scene (I use CNN and regression as last layer to get a length of an object).
I used division in order to penalize more where the true value is relative small to the predicted).
how can i prevent the negative predictions ?
this is the mode (for now..):
def create_model():
model = Sequential()
model.add(Conv2D(128, kernel_size=(3, 3), activation='relu', padding='same', input_shape=(128, 128, 1)))
model.add(Dropout(0.5))
model.add(Conv2D(128, kernel_size=(3, 3), activation='relu', padding='same'))
model.add(Dropout(0.25))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, kernel_size=(3, 3), activation='relu', padding='same'))
model.add(Dropout(0.25))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
#
#
model.add(Conv2D(64, kernel_size=(3, 3), activation='relu', padding='same'))
model.add(Dropout(0.25))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.15))
model.add(Dense(1))
#model.compile(loss=keras.losses.mean_squared_error, optimizer=keras.optimizers.Adadelta(), metrics=[sacc])
model.compile(loss=ratio_loss, optimizer=keras.optimizers.Adadelta(), metrics=[sacc])
return model
Thanks,
Amir
You could continue training your neural network, and hopefully it will learn not to make any prediction below 0 (assuming all of the training data has output below 0). You could then add a post-prediction step where you turn an And if it makes any predictions below 0, then you can just convert it to 0.
You could add an activation function as Daniel Möller answered.
That would involve changing
model.add(Dense(1))
to
model.add(Dense(1, activation='softplus'))
since you mentioned you wanted the output to be from 0 to ~200 in a comment.
This would guarantee there's not output below 0.
def ratio_loss(y, y0):
return (K.mean(K.abs(y - y0 / y)))
But what is the range of your expected output?
You should probably be using some activation function at the end such as:
activation ='sigmoid' - from 0 to 1
activation = 'tanh' - from -1 to +1
activation = 'softmax' - if it's a classification problem with only one correct class
actication = 'softplus' - from 0 to +inf.
etc.
Usage in the last layer:
model.add(Dense(1,activation='sigmoid')) #from 0 to 1
#optional, from 0 to 200 after using the sigmoid above
model.add(Lambda(lambda x: 200*x))
Hint: if you're a starter, avoid using too much "relu", it often gets stuck in 0 and must be used with carefully selected learning rates.