I have a model in keras and I want to use it as a feature extractor in an autoencoder configuration. The first test I want to do is to mirror the model and use the original version as the encoder, and the second one as the decoder.
e.g. if the blocks on the original model goes
input->A->B->C->output
I want to get
input->A->B->C->(latent space)->C->B->A->reconstructed_input
However, the original model is not small and I am quite new to keras. It has a number of blocks defined with which the total model is built. What I am confused about is the order I have to follow when I reverse things. There are several convolutional, reshaping, and dense layers, as well as skip connections, and I am not sure about the strategy I should follow with each individual case. I know for a fact that I should replace Conv2d layers with Conv2DTranspose to upscale the feature maps back to the original image, but that's about it. I am baffled by the rest.
Does anyone have any recommendations or resources I could follow to make my life easier?
Cheers!
Related
If a big model consists of end-to-end individual models, can I (after training) preserve only one model and freeze/discard other models during inference?
An example: this struct2depth (see below) have three models training in an unsupervised fashion. However, what I really need is the object motion, namely 3D Object Motion Estimation part. So I wonder if this is feasible to
train on the original networks, but
inference with only Object Motion Estimator, i.e. other following layers frozen/discarded?
I saw that in tensorflow one can obtain tensor-output of a specified layer, but to save unnecessary computation I'd like to simply freeze all other parts... don't know if it's possible.
Looking forward to some insights. Thanks in advance!
You can ignore weights by setting them to 0. For this, you can directly get a weight W and do W.assign(tf.mul(W,0)). I know that you care about speeding up inference but unless you rewrite your code to use sparse representations, you will probably not be speeding up inference since weights can't be removed fully.
What you can alternatively do, is look at existing solutions for pruning in custom layers:
class MyDenseLayer(tf.keras.layers.Dense, tfmot.sparsity.keras.PrunableLayer):
def get_prunable_weights(self):
# Prune bias also, though that usually harms model accuracy too much.
return [self.kernel, self.bias]
# Use `prune_low_magnitude` to make the `MyDenseLayer` layer train with pruning.
model_for_pruning = tf.keras.Sequential([
tfmot.sparsity.keras.prune_low_magnitude(MyDenseLayer(20, input_shape=input_shape)),
tf.keras.layers.Flatten()
])
You can e.g. use ConstantSparsity (see here) and set the parameters such that your layers are fully pruned.
Another alternative is to construct a second, smaller model that you only use for inference. You can then save the required weights separately (instead of saving the entire model) after training and load them in the second model.
This is my first post, so I'm sorry if I'm doing something wrong. I'm a beginner when it comes to machine learning. I want to create an accurate model with the cifar10 dataset. I've learned that a Conv2D layer looks for a specific thing in an image classification model. But how do I know what the layer is looking for? I'm not sure if I make sense here, or if anything I've learned so far is true could someone help me out in understanding how layers work in tensorfow?
Convolutional layers extract features from images. They apply a filter to images and output a feature map. Then, one may add dense layers afterwards to obtain labels from these features.
Conv2D is just a 2d convolution used with 2d images - this means that the kernel is 2d also.
Here are a few helpful links: https://machinelearningmastery.com/convolutional-layers-for-deep-learning-neural-networks/
https://towardsdatascience.com/simple-introduction-to-convolutional-neural-networks-cdf8d3077bac
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I want to train a model with CT Grayscale images such as this one:
For certain classes of diseases my training set is limited , e.g 2,000 positives with 98,000 negatives.
I am thinking to use transfer learning to avoid overfitting and boost the effectiveness of my model but I also realize that I should fine tune the model since the kind of images I am feeding the model are very different from the kind of images with which the Inception model I will use has been trained.
My problem then is that I am not sure how many layers I should keep fixed and how many trainable.
I am thinking then to use skip connections to apply a stochastic depth letting the network to learn how many layers trully needs.
So I am thinking to implement the following architecture.
I.e. I will add skip connections between the layers of the pretrained inception model that comes with Keras (Tensorflow 2.0).
I would welcome advice on how to implement these ideas. In particular how to split the network into three parts, leave the first part untouched (untrainable) and train the second part after adding the skip connections. The implementation should be in Keras.
Transfer learning is indeed the right approach. This allows you to make use of the trained weights to take care of the 'generic' tasks of DL image processing, such as shape recognition, edge detection, etc, and, in a manner of speaking, conserve your data (input and labels) for retooling the existing the neural network for your specific task.
As a rule of thumb, the closer the weights are to the input, the more generic their function and the less you want to retrain them. Conversely, the closer the weights are to the output, the more task specific is their function and the more retraining they require.
I suggest training the endpoint classifier before retraining existing weights. 1-2 fully connected layers (read: Dense) with your favorite activation function + 1 fully connected layer with the softmax activation (as we want the output to be the predicted probability of each disease) should probably do the trick. When training this (the endpoint classifier) be sure to freeze all other layers (or use bottleneck features - see how in the link below).
Only then you should retrain existing weights - this is called fine-tuning. I suggest unfreezing the first inception module*, then allowing it (and the already trained endpoint classifier from the last step!) to retrain. then maybe unfreeze the next inception module, and allow it to train as well (again, while also allowing the first inception module and the new endpoint classifier to retrain as well. When retraining a segment, always allow the weights downstream to retrain as well).
Be advised that fine-tuning should use a slow training rate.
To the best of my understanding skip connections DO NOT "let the network to learn how many layers truly needs.". They mostly allow the circumvention of the diminishing gradient problem. The "skipped" layers do not become "optional", they will participate in generating the output. And, since the weights used do not account for inputs that will be added via skip connections, I believe it will render the training of the weights irrelevant, requiring the retraining of the entire network and thus preventing transfer learning.
It would be an interesting experiment, but try it at your own peril.
If you really want skip connections, I suggest you make use of a model that already has them (and whose weights are therefore adjusted to it), such as ResNet.
Please view this link for more ideas regarding the utilization of transfer learning, bottle neck features, and fine tuning. And augmentation, while your at it.
https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
*By that I mean first one from the prescriptive of the output, that is, the one highest in your image.
Now, I want feature of image to compute their similarity. We can get feature using pre-trained VGG19 model in tensorflow easily. But VGG19 model has many layers, and I don't know which layer should I use to get feature. Which layer's output is appropriate for this problem?
# I think this how is correct to extract feature
model = tf.keras.application.VGG19(include_top=True,
weight='imagenet')
input = model.input
output = model.layers[-2].output
extract_model = tf.keras.Model(input, output)
It's my infer that the more closer to last output, the more the model output powerful feature. But some tutorials says 'use include_top=False to extract feature' (e.g Image Captioning with Attention TensorFlow)
So, I don't know which layer should I use. Please try to help me here in this thread.
The include_top=False may be used because the last 3 layers (for that specific model) are fully connected layers which are not typically good feature vectors. If the model directly outputs a feature vector, then you don't need it.
Most people use the last layer for transfer learning, but it may depend on your application. For example, Gatys et. al. show that the first few layers of VGG are sensitive to the style of the image and later layers are sensitive to the content.
I would probably try all of them in a hyperparameter search and see which gives the best performance. If by image similarity you mean the similarity of objects contained inside, I would probably start with the last layer.
I am searching for a way to forward lower layer output to a higher layer with a loaded VGG16 model using CNTK.
The background of my problem is:
I reimplemented some parts of Fully Convolutional Networks for Semantic Segmentation but then I ran into some problems: Starting with this example I first replaced the fully connected layers with fully convolutional and slit the sequence in the model definition part into chunks where I could simply access pool3 and pool4 for the later usage in eg. Convolution2D((1,1), num_classes, name='score_pool4')(pool4). This works fine but after building the model I noticed, that I need to implement an own way to read batches because the build-in reader does not support 2D labels right now. Now I simply read the images using OpenCV and replaced the training_session(...).train() with a for loop and trainer.train_minibatch({model['features']: my_loaded_features, model['labels']: my_2D_labels}) this works well but because of the removed training_session part I don't know where I could apply the existing VGG16 weights.
My problem is:
I searched for transfer learning examples where those guys load models using C.load_model(...) and then clone the needed layers but now I am wondering how could I access cloned_layers->pool4 (in the middle of the loaded model) if I also want to use it in deeper layers.
I tried Convolution2D((1,1), num_classes, name='score_pool4')(cloned_layers.find_by_name('pool4'))but I ended up with some error messages while learner initialization because of "unknown shape information" in used weight variables.
So how can I access those layers within the loaded model for later (deeper) usage?
Thanks for reading (and maybe helping)!
If you are looking to read custom data. There are two tutorials on building your own readers. https://cntk.ai/pythondocs/manuals.html
Regarding cloning parts of a network - here is a link to another post on StackOverflow that has exemplar code