We Keep Coding

How to smooth LSTM output

I'm trying to predict some time-series signal using a number of other time-series signals. For that purpose i'm using a LSTM network. The input signals are normalized and the same are the output signals. I'm using MSE loss, and implementing using tensorflow
The network give farely good prediction, but it is very noisy. I want to make it more smooth, as if a LPF filter was used on the LSTM output.
The optimal solution for me is some parameter that i can change that will filter more/less frequencies from the LSTM output.
How can i do that? i was thinking about trying to constrain the loss function somehow?
Thanks
I tried adding a fully connected layers after the LSTM, batch normalization and single and multi-layer LSTM networks

"Fading in" new layers in Keras

I'm trying to implement this paper in Keras, with a tensorflow backend.
In my understanding, they progressively grow a GAN, fading in additional blocks of layers as the model is trained. New layers are faded in linearly over iterations.
I'm not sure how to introduce the "fading in" they used.
To do this Keras, I'm thinking I'll probably need a Lambda layer -- but that's about as much as I know.
Any suggestions?
Thanks!

I think you should use Keras Functional API. That way you can rearrange the inputs to the layer as well as the outputs and interconnect them the way you want to while the layers keep the weights they learned. You will have to have a few nested if-statements, but I think it should work.
Or you could have all the models prepared (functions that return you the model architecture and can set layer weights) and just populate layers in the new model with weights from corresponding layers from the old model.

Dropout for LSTM recurrent weights in tensorflow

Tensorflow's DropoutWrapper allows to apply dropout to either the cell's inputs, outputs or states. However, I haven't seen an option to do the same thing for the recurrent weights of the cell (4 out of the 8 different matrices used in the original LSTM formulation). I just wanted to check that this is the case before implementing a Wrapper of my own.
EDIT:
Apparently this functionality has been added in newer versions (my original comment referred to v1.4): https://github.com/tensorflow/tensorflow/issues/13103

It's because original LSTM model only applies dropout on the input and output layers (only to the non-recurrent layers.) This paper is considered as a "textbook" that describes the LSTM with dropout: https://arxiv.org/pdf/1409.2329.pdf
Recently some people tried applying dropout in recurrent layers as well. If you want to look at the implementation and the math behind it, search for "A Theoretically Grounded Application of Dropout in Recurrent Neural Networks" by Yarin Gal. I'm not sure Tensorflow or Keras already implemented this approach though.

Deep learning using Caffe - Python

I am studying deep learning and trying to implement it using CAFFE- Python. can anybody tell that how we can assign the weights to each node in input layer instead of using weight filler in caffe?

There is a fundamental difference between weights and input data: the training data is used to learn the weights (aka "trainable parameters") during training. Once the net is trained, the training data is no longer needed while the weights are kept as part of the model to be used for testing/deployment.
Make sure this difference is clear to you before you precede.
Layers with trainable parameters has a filler to set the weights initially.
On the other hand, an input data layer does not have trainable parameters, but it should supply the net with input data. Thus, input layers has no filler.
Based on the type of input layer you use, you will need to prepare your training data.

Unset trainable attributes for parameters in lasagne / nolearn neural networks

I'm implementing a convolutional neural network using lasagne nolearn.
I'd like to fix some parameters that prelearned.
How can I set some layers untrainable?
Actually, though I removed 'trainable' attribute of some layers,
the number shown in the layer information before fitting, namely, such as
Neural Network with *** learnable parameters never change.
Besides, I'm afraid that the greeting function
in 'handers.py'
def _get_greeting(nn):
shapes = [param.get_value().shape for param in
nn.get_all_params() if param]
should be
nn.get_all_params(trainable=True) if param]
but I'm not sure how it affect on training.