I've built some neural networks with TensorFlow, like basic MLPs and convolutional neural networks. Now I want to move on to recurrent neural networks. However, I'm not experienced in natural language processing. Therefore the TensorFlow NLP tutorials for RNNs are not easy to read for me (and not really interesting, too).
Basically I want to start off with something simple, not a LSTM.
How would one build a simple recurrent neural network, like an Elman network, in TensorFlow?
I were only able to find GRU- or LSTM RNN examples for TensorFlow, mostly for NLP. Does anyone know of some simple recurrent neural network tutorials or examples for TensorFlow?
This figure shows a basic Elman network, which is often simply called SRN (simple recurrent network):
One option is to use the built-in RNNCell located in tensorflow/python/ops/rnn_cell.py.
If you don't want to do that you can make your own RNN. The RNN will train using back-propagation through time. Try unrolling the network a fixed number of steps, e.g. consider input sequences of length ten. Then you can write a loop in python to do all of the matrix multiplications for each step of the network. Each time you can take the output from the previous step and concatenate it with the input to that step. It will not be too many lines of code to get this working.
Related
I'm building a Sequential NN model in Keras for binary classification. The training data has about 600,000 rows and 2,000 features, so every epoch and every layer is very time consuming. I believe many of the features are not relevant to the model, and can be dropped altogether, to make the model thinner, so it it would be faster to work with.
I run a simple model with one hidden-layer of 200 neurons. How can I tell which of the features (which are actually the nodes in the input layer) are meaningless, so I could drop them from the data set and re run the model without them?
There is a very big topic in machine learning called feature selection. Though, neural networks are considered to automatically choose the best features for the problem, to an extent, by using their weights, to either consider more or less some of them. Neural networks also need a lot of experience to be tuned correctly. I would definitely suggest you to increase the layers of the network, because you have a lot of data and features and use l1 regularisation, in order to get sparse weights and exclude most of the features. Also, these information are indicative, since I do not know anything about your dataset and your network architecture. At last, I would suggest you to study more about the basics of machine learning and then continue learning about neural networks, before practicing with real data.
R's package 'forecast' has a function nnetar, which uses feed-forward neural networks with a single hidden layer to predict in time series.
Now I am using Python to do the similar analysis. I want to use neural network which does not need to be as complex as deep learning. Maybe 2 layers and a couple of nodes are good enough for my case.
So, does Python have a model of simple neural networks which can be used in time series lik nnetar? If not, how to deal with this problem?
Any NN model that uses 1 or more hidden layers is a multi-layer perceptron model, and for that case it is trivial to make it extendable to N layers. So any library that you pick will support it. My guess for you not picking a complex library like pytorch/Tensorflow is its size.
Tensorflow does have TF-Lite which can work for smaller IOT devices.
Sklearn does have MLPRegressor that can train NNs if that is more to your liking.
You can always write your model. There are plenty of examples for this that use numpy and are plenty fast for cpu computation.( Single Hidden layer NN I am guessing will be more memory bound than computation bound)
Use another ML algorithm. Single Hidden layer NNs will not perform nearly as well as other other simpler algorithms.
If there are other reasons for not using a standard library like tensorflow/pytorch then you should mention them.
I am trying to set up a Multiple Artificial Neural Network as you can see here on image (a):
(source)
I want that each of the networks work independently on its own domain. The single networks must be built and trained for their specific task. The final decision will be make on the results of the individual networks, often called expert networks or agents.
Because of privacy, I could not share my data.
I try to set up this with Tensorflow in Python. Do you have an idea of how I would do it if that is achievable? At the moment I have not found any examples of this.
The way to go about this is to just take the outputs of the two networks and concatenate the resulting output tensors (and reshape them if needed) and then pass them into the final network. Take a look at here for the concatenation documentation and here for an example of taking the output from one network and feeding it into another. This should give you a place to start from.
As for (a), it is simple, just train the networks before hand and load them when you are training the final network. Then do the concatenation on the outputs.
Hope this helps
I built an Autoencoder using Python and Tensorflow. To build the Autoencoder I used the Tensorflow tutorial on how to build an Autoencoder to read the MNIST Data set on handwritten digits. I used it to find features of CGRA compositions.
So far I restructured the Code for the Autoencoder in a way that I can use it on my own data. I found features and already managed to reconstruct the Input,
up to a certain error. NOW, I am trying to use the Autoencoders weights, to initialize a Neural Network with parameters similiar to the encoder part of my Autoencoder. Then, add one extra Layer with a single neuron and a linear activation function to perform Regression Analysis(or basically supervised learning).
So my question is: How do I initialize a neural network with specific weights (not random) using tensorflow?
I'd be grateful for any kind of help. Links to Tutorials or other links to other Threads.
Tahnks in Advance!
When you build tf.Variable, the first argument is an initial_value.
https://www.tensorflow.org/api_docs/python/state_ops/variables#Variable.init
You can provide any Tensor you like to initialize the variables, not just, say, random initialization.
Another option is you can assign values to the variables after construction, if you find that easier.
Hope that helps!
Is there a way to learn unsupervised features from set of images. Similar to word2vec or doc2vec, where neural network is learnt and given new document we get its features.
Expecting similar to this example shows that it can load learnt nn-model and predict features for new images.
Is there any simple example how to implement cnn over images and get their features back will help !!
Suppose in this example
If I want to get cnn features for all X_train and X_test ... is there any way?
Also, if we can get weights per layer per image, we can stack them and use as features. In that case is there a way to get the same.
Using those features for unsupervised task would be easier, if we consider them as vectors.
If I correctly understood your question, this task is quite common in a deep learning field. In case of images what I consider the best is a convolutional autoencoder. You may read about this architecture e.g. here
http://people.idsia.ch/~ciresan/data/icann2011.pdf
Previous version of Keras supported this architecture as one of core layers, though from version 1.0 I noticed that it disappeared from documentation. But - it's still quite easy to build it from a scratch :)
In noimage cases there are also another approaches like e.g. Restricted Boltzmann Machines.
UPDATE :
When it comes to what sort of activations are the best for obtaining new features from neural network activations - from my personal experience - it depends on the size of the net which you use. If you use a network which last layer is wide (has a lot of nodes) it might be useful to get only last layer (due to number of parameters if you want to consider also previous layers - it may harm the performance of learning). But - if (like in case of some MNIST networks) your last layer is not sufficient for this task - you may try using also previous layers activation or even all net activity. To be honest - I'm not expecting much of improvement in this case - but you may try. I think that you should use both approaches - starting from taking only last layer activations - and then trying to check the behaviour of your new classifier when you add activations from previous layers.
What I will strongly advise to you is also getting some insights from what sort of features network is learning - using T-SNE embeddings of it activations. In many cases I found it useful - e.g. checking if the size of a layer is sufficient. Using T-SNE you may check if the features obtained from last layer are good discriminators of your classes. It may also give you good insights about your data and what neural networks are really learning (alongside with amazing visualizations :) )