We Keep Coding

Check and modify the weights of a layer of an onnx model at runtime

I'm using Onnx Models from ONNX Models ZOO with ONNXRuntime as a engine for inference.
I want to check and modify the weights of a convolutional layer (or any layer) of an ONNX model at runtime. This is to do it inside 'OnnxRuntime.InferenceSession.run()'. Is that possible? Did anyone do something similar?
Could be done by adding a layer before the layer I want to check and modify its weights? This new layer should do this procedure before the convolutional layer, but it would need access to the weights of the next layer.
Thanks.
Izan.

Will freezing layers in tensorflow save model update time?

I'm trying to train a model using transfer learning method, which involves loading and freezing a very large embedding layer. As training speed is also what I concern in this task, I expect freezing layers to also boost training speed. However, I haven't observed any speed improvement at all.
I freeze the embedding layer using
self.embedding_layer.trainable = False
And did observered non-trainable parameters increased in model summary
As the frozen layers may occur in the middle of a model, and the gradients need to be passed down to the first several trainable layers, I reckon tensorflow still calculate the gradients of frozen layers but just skip them during updating stage.
If my guess is correct, is there any way to remove the frozen layer from gradient calculation?

"Fading in" new layers in Keras

I'm trying to implement this paper in Keras, with a tensorflow backend.
In my understanding, they progressively grow a GAN, fading in additional blocks of layers as the model is trained. New layers are faded in linearly over iterations.
I'm not sure how to introduce the "fading in" they used.
To do this Keras, I'm thinking I'll probably need a Lambda layer -- but that's about as much as I know.
Any suggestions?
Thanks!

I think you should use Keras Functional API. That way you can rearrange the inputs to the layer as well as the outputs and interconnect them the way you want to while the layers keep the weights they learned. You will have to have a few nested if-statements, but I think it should work.
Or you could have all the models prepared (functions that return you the model architecture and can set layer weights) and just populate layers in the new model with weights from corresponding layers from the old model.

Modify pretrained model in tensorflow

I want to know how to make changes to a graph loaded from tensorflow's meta and checkpoint files like:
saver = tf.train.import_meta_graph('***.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))
For example, there are old_layer1 -> old_layer2 in existing graph with pretrained weights. I want to insert one then it becomes old_layer1 -> new_layer -> old_layer2, and new_layer are randomly initialized since there are no pretrained parameter for it. Answer here said its impossible, since tf's graph only allow append, is this true?
So I wonder if this can be worked around by loading the pretrained layers as individual variables, and assigning pre-trained weights as initial values and connect them by myself, so that I can add new layers between old ones. But I don't know how to do this in code.

Doing this with raw tensorflow can be complicated since the tf graph does not encode directly the structure of the layers. If your model was built with tf.keras, however, this is fairly straightforward as loading a keras model also loads its layer structure.

Keras BatchNormalization population parameters update while training in tensorflow

I am using Keras 2.0.8 with Tensorflow 1.3.0 in Ubuntu 16.04 with Cuda 8.0 and cuDNN 6.
I am using two BatchNormalization layers( keras layers ) in my model and training using tensorflow pipeline.
I am facing two problems here -
BatchNorm layer population parameters( mean and variance ) are not being updated while training even after setting K.learning_phase to True. As a result, inference is failing completely. I need some advice on how to update these parameters between training steps manually.
Secondly, after saving the trained model using tensorflow saver op, when I try to load it, the results cannot be reproduced. It seems the weights are changing. Is there a way to keep the weights same in save-load operation?

I ran into the same problem a few weeks ago. Internally, keras layers can add additional update operations to a model (e.g. batchnorm). So you need to run these additional ops explicitly. For the batchnorm these updates seem to be just some assign_ops which swap the current mean/variance with the new values. If you do not create a keras model this might work; assuming x is a tensor you like to normalize
bn = keras.layers.BatchNormalization()
x = bn(x)
....
sess.run([minimizer_op,bn.updates],K.learning_phase(): 1)
In my workflow, I am creating a keras model (w/o compiling it) and then run the following
model = keras.Model(inputs=inputs, outputs=prediction)
sess.run([minimizer_op,model.updates],K.learning_phase(): 1)
where inputs can be something like
inputs = [keras.layers.Input(tensor=input_variables)]
and outputs is a list of tensorflow tensors. The model seems to aggregate all additional updates operations between inputs and outputs automatically.