I know we can access the hidden layer from each layer by the following code. However, each hidden output we obtains is the result mixed from 12 head by a fully connected layers. Is there a way to obtain the outputs before they enter the fully connected layers?
from transformers import BertModel, BertConfig
config = BertConfig.from_pretrained("xxx", output_hidden_states=True)
model = BertModel.from_pretrained("xxx", config=config)
outputs = model(inputs)
print(len(outputs)) # 3
hidden_states = outputs[2]
Thank you!
Related
I want to use the Segmentation_Models UNet (with ResNet34 Backbone) for uncertainty estimation, so i want to add some Dropout Layers into the upsampling part. The Model is not Sequential, so i think i have to reconnect some outputs to the new Dropout Layers and the following layer inputs to the output of Dropout.
I'm not sure, whats the right way to do this. I'm currently trying this:
# create model
model = sm.Unet('resnet34', classes=1, activation='sigmoid', encoder_weights='imagenet')
# define optimizer, loss and metrics
optim = tf.keras.optimizers.Adam(0.001)
total_loss = sm.losses.binary_focal_dice_loss # or sm.losses.categorical_focal_dice_loss
metrics = ['accuracy', sm.metrics.IOUScore(threshold=0.5), sm.metrics.FScore(threshold=0.5)]
# get input layer
updated_model_layers = model.layers[0]
# iterate over old model and add Dropout after given Convolutions
for layer in model.layers[1:]:
# take old layer and add to new Model
updated_model_layers = layer(updated_model_layers.output)
# after some convolutions, add Dropout
if layer.name in ['decoder_stage0b_conv', 'decoder_stage0a_conv', 'decoder_stage1a_conv', 'decoder_stage1b_conv', 'decoder_stage2a_conv',
'decoder_stage2b_conv', 'decoder_stage3a_conv', 'decoder_stage3b_conv', 'decoder_stage4a_conv']:
if (uncertain):
# activate dropout in predictions
next_layer = Dropout(0.1) (updated_model_layers, training=True)
else:
# add dropout layer
next_layer = Dropout(0.1) (updated_model_layers)
# add reconnected Droput Layer
updated_model_layers = next_layer
model = Model(model.layers[0], updated_model_layers)
This throws the following Error: AttributeError: 'KerasTensor' object has no attribute 'output'
But I think I'm doing something wrong. Does anybody have a Solution for this?
There is a problem with the Resnet model you are using. It is complex and has Add and Concatenate layers (residual layers, I guess), which take as input a list of tensors from several "subnetworks". In other words, the network is not linear, so you can't walk through the model with a simple loop.
Regarding your error, in the loop of your code: layer is a layer and updated_model_layers is a tensor (functional API). Therefore, updated_model_layers.output does not exist. You confuse the two a bit
I have trained a model using sklearn.neural_network.MLPClassifier and I want to know how many layers are in my clssifier. The result shows :
>>from sklearn.neural_network import MLPClassifier
>>clf = MLPClassifier()
>>clf = clf.fit(train_matrix,train_label)
>>clf.n_layers_
>>3
The document shows attribute n_layers_ means :
Number of layers
Dose it mean there is a hidden layer or there are three hidden layers?
n_layers_ denotes all the layers in the neural network which include
Input layer = 1
All hidden layers = len(hidden_layer_sizes)
Output layer = 1
So if you initialized the classifier as
clf = MLPClassifier()
The default hidden_layer_sizes param = (100,), so number of hidden layers = 1.
So total layers = 1+1+1 = 3 as you are getting.
If instead you initialized it as:
clf = MLPClassifier(hidden_layer_sizes=(100,100,))
Now the number of hidden layers = 2, so total layers = 4
I want to modify the resnet_v2.resnet_v2_50 model so that I concatenate a number to the pool5 layer.
After I import the network I can see the layers in the end_points variable.
with slim.arg_scope(resnet_v2.resnet_arg_scope()):
net, end_points = resnet_v2.resnet_v2_50(self.imageIn, num_classes = numClasses)
So I have access to the different layers
curr_conv1 = end_points['resnet_v2_50/conv1']
curr_pred = end_points['resnet_v2_50/predictions']
curr_block4 = end_points['mainQN/resnet_v2_50/block4/unit_3/bottleneck_v2']
But I don't have access to the last part of the network after the pooling layer.
curr_pool5 = end_points['resnet_v2_50/pool5']
But I can see in Tensorboard and in the code of resnet_v2_50 that there is some kind of pool5 layer. How can I get access to it, so I can modify it and concatenate a number to it?
end_points['global_pool']
is what you want. That immediately follows the global pooling layer called pool5 in the code.
https://github.com/tensorflow/models/blob/master/research/slim/nets/resnet_v2.py#L214
I have an ANN model and i am trying to get the activation values of all the hidden layers. I have trained the network with a 90dim matrix and i have 1 hidden layers which is 150dim. My model structure is one 90dim input layer, one hidden layer of 150dim and one output of 90dim. I have trained and tested the data. After that i m using the .predict() function to predict output using my test dataset. I am feeding the predicted output as the next input and so on. Now i want to get the activation value of the hidden layers of the predict function. I am using the following code to achieve it but its not working:
write_predict_data = pd.ExcelWriter("/home/workstation/ANN/prediction_data_2.xlsx",engine="xlsxwriter")
write_activations_data = pd.ExcelWriter("/home/rianzaman/Downloads/activition_of_hidden_node_2.xlsx",engine="xlsxwriter")
for i in range(0, 200):
print("Predicting ...",)
next_prediction = my_model.predict(X_test, 1,)
output_file_data = pd.DataFrame(next_prediction)
output_file_data.to_excel(write_predict_data, sheet_name='Sheet1')
#To get activation
get_activations = theano.function([my_model.layers[0].input], my_model.layers[1].get_output(train=False),
allow_input_downcast=True)
activations = get_activations(next_prediction)
output_file_data_activation = pd.DataFrame(activations)
output_file_data_activation.to_excel(write_activations_data, sheet_name='Sheet1')
X_test = next_prediction
write_predict_data.save()
When i m running the code i m getting a 90dim output which basically i think is the output layers dataset.
Can anybody tell me what's wrong with the code?
get_activations(next_prediction) should be get_activations(X_test) - you want to pass inputs to get_activations, not labels.
I have a Neural Network with two hidden layers. I want to add a bias unit only to the second hidden layer. How do I do that?
The code for my network is as follows:
nn = FeedForwardNetwork()
inLayer = LinearLayer(numFeatures)
hiddenLayer1 = LinearLayer(numFeatures+1)
hiddenLayer2 = SigmoidLayer(numFeatures+1)
outLayer = LinearLayer(1)
nn.addInputModule(inLayer)
nn.addModule(hiddenLayer1)
nn.addModule(hiddenLayer2)
nn.addOutputModule(outLayer)
in_to_hidden1 = FullConnection(inLayer, hiddenLayer1)
hidden1_to_hidden2 = FullConnection(hiddenLayer1, hiddenLayer2)
hidden2_to_out = FullConnection(hiddenLayer2, outLayer)
nn.addConnection(in_to_hidden1)
nn.addConnection(hidden1_to_hidden2)
nn.addConnection(hidden2_to_out)
nn.sortModules()
This is fairly simple task. First you have to create Bias module:
bias = BiasUnit()
Then add it to your NeuralNetwork, so:
nn = FeedForwadNetwork()
nn.addModule(bias)
Then, assuming you already added other layers you have to connect bias to hidden layer of your choice:
bias_to_hiden = FullConnection(bias, hiden_layer)
And then add it to neural network:
nn.addConnection(bias_to_hiden)
Apart from that you do everything same as before.
For the reference check code of the buildNetwork function from pybrain.tools.shortcuts module. Here is some code that connects bias unit to other layers (lines 75-79):
if opt['bias']:
# add bias module and connection to out module, if desired
n.addModule(BiasUnit(name='bias'))
if opt['outputbias']:
n.addConnection(FullConnection(n['bias'], n['out']))
Hope that helped.