I want to build a Deep Believe Network with scikit-learn. As I know one should train many Restricted Boltzmann Machines (RBM) individually. Then one should create a Multilayer Perceptron (MLP) that has the same number of layers as the number of (RBMs), and the weights of the MLP should be initialized with the weights of the RBMs. However I'm unable to find a way to get the weights of the RBMs from scikit-learn's BernoulliRBM. Also it doesn't seem to be a way also to initialize the weights of a MLP in scikit-learn.
Is there a way to do what I described?
scikit-learn does not currently have an MLP implemented which you can initialize via an RBM, but you can still access the weights which are stored in the components_ attribute and the bias which is stored in the intercept_hidden_ attribute.
If you're interested in using modern MLPs, torch7, pylearn2, and deepnet are all modern libraries and most of them contain pretraining routines like you describe.
Related
In my previous project, I need to frame an image classification task as a regression problem. I implement the regression model using Tensorflow, with standard Sequential model with a 1 node Dense layer with no activation function as the last layer. In order to measure the performance, I need to use standard classification metrics, such as accuracy and cohen kappa.
However, I can't directly use those metrics because my model is a regression model, so I need to clip and round the output before feeding them to the metrics. I use a workaround by defining my own metric, however that workaround is not practical. Therefore, I'm thinking about contributing to Tensorflow by implementing a custom transformation_function to transform y_pred by a Tensor lambda function before storing them in the __update_state method. After reading the source code, I get doubts regarding this idea. So, I'm asking out to you, fellow Tensorflow user/contributors, what is the best practice of transforming y_pred before feeding it to a metric? Is this functionality already implemented in the newest version?
Thank you!
R's package 'forecast' has a function nnetar, which uses feed-forward neural networks with a single hidden layer to predict in time series.
Now I am using Python to do the similar analysis. I want to use neural network which does not need to be as complex as deep learning. Maybe 2 layers and a couple of nodes are good enough for my case.
So, does Python have a model of simple neural networks which can be used in time series lik nnetar? If not, how to deal with this problem?
Any NN model that uses 1 or more hidden layers is a multi-layer perceptron model, and for that case it is trivial to make it extendable to N layers. So any library that you pick will support it. My guess for you not picking a complex library like pytorch/Tensorflow is its size.
Tensorflow does have TF-Lite which can work for smaller IOT devices.
Sklearn does have MLPRegressor that can train NNs if that is more to your liking.
You can always write your model. There are plenty of examples for this that use numpy and are plenty fast for cpu computation.( Single Hidden layer NN I am guessing will be more memory bound than computation bound)
Use another ML algorithm. Single Hidden layer NNs will not perform nearly as well as other other simpler algorithms.
If there are other reasons for not using a standard library like tensorflow/pytorch then you should mention them.
I am using scikit-learn's multilayer perceptron classifier and I want to evaluate some pruning techniques for neural networks, such as Optimal Brain Damage. This method requires, iteratively, to remove weights from the network, i.e. manually setting them to 0 and retraining, and repeat the process until some criteria is satisfied.
So I would like to know if there is a simple way of setting one or more weights to zero and keeping them like that all along the training of the net. I want to point out that while it is possible to easily access the weights of the MLP (it is an attribute of the object) once it has already been trained, I don't know how to preset it before training.
PD: if you know another more automatic way of evaluating pruning methods in sklearn, that would also be helpful.
I am trying to understand the Large-scale Linear Models with TensorFlow documentation. The docs motivate these models as follows:
Linear model can be interpreted and debugged more easily than neural
nets. You can examine the weights assigned to each feature to figure
out what's having the biggest impact on a prediction.
So I ran the extended code example from the accompanying TensorFlow Linear Model Tutorial. In particular, I ran the example code from GitHub with the model-type flag set to wide. This correctly ran and produced accuracy: 0.833733, similar to the accuracy: 0.83557522 on the Tensorflow web page.
The example uses a tf.estimator.LinearClassifier to train the weights. However, in contrast to the quoted motivation of being able to examine the weights, I can't find any function to actually extract the trained weights in the LinearClassifier documentation.
Question: how do I access the trained weights for the various feature columns in a tf.estimator.LinearClassifier? I'd prefer to be able to extract all the weights in a NumPy array.
Note: I am coming from an R environment where linear regression / classification models have a coefs method to extract learned weights. I want to be able to compare linear models in both R and TensorFlow on the same datasets.
After training the model with Estimator, you could use the tf.train.load_variable to retrieve the weights from checkpoint. You can use tf.train.list_variables to find the names for model weights.
There are plans to add this support in Estimator directly also.
I built an Autoencoder using Python and Tensorflow. To build the Autoencoder I used the Tensorflow tutorial on how to build an Autoencoder to read the MNIST Data set on handwritten digits. I used it to find features of CGRA compositions.
So far I restructured the Code for the Autoencoder in a way that I can use it on my own data. I found features and already managed to reconstruct the Input,
up to a certain error. NOW, I am trying to use the Autoencoders weights, to initialize a Neural Network with parameters similiar to the encoder part of my Autoencoder. Then, add one extra Layer with a single neuron and a linear activation function to perform Regression Analysis(or basically supervised learning).
So my question is: How do I initialize a neural network with specific weights (not random) using tensorflow?
I'd be grateful for any kind of help. Links to Tutorials or other links to other Threads.
Tahnks in Advance!
When you build tf.Variable, the first argument is an initial_value.
https://www.tensorflow.org/api_docs/python/state_ops/variables#Variable.init
You can provide any Tensor you like to initialize the variables, not just, say, random initialization.
Another option is you can assign values to the variables after construction, if you find that easier.
Hope that helps!