I want to use the Glove vectors in Language Modeling. But the problem is if I use Embedding layer in the model, I can't predict the output vector and match the word. What I mean here is, I want to give the glove vector representation for my sentences as the Input. and get them out from the Lstm layer and get the vectors and match it with Glove vectors I want to use the glove vectors without embedding layer. can someone propose a method to do it? I am using the keras and python3
What I want is to use the embedding layer as one model1 and return the output vector and give it to another LSTM model2 as the input. where it gives the index of the word vector.
Related
I am solving a multi-class classification problem using Keras. But I am assuming the accuracy is bad due to poor word embedding of my data (domain-specific data).
Keras has its own Embedding layer, which is a supervised learning method.
So I have 2 questions regarding this :
Can I use word2vec embedding in Embedding layer of Keras, because word2vec is a form of unsupervised learning/self-supervised?
If yes, then can I use transfer learning on word2vec pre-train model to put extra knowledge of my domain specific features.
You can initialize the embeddings layer with word2vec or any other pre-trained embeddings (maybe FastText?) in such a way that you manually construct the embedding matrix, i.e., just load all the numbers form the word2vec files and make an np.array of it. Then you create a constant initializer and pass it as an argument to your embeddings layer constructor.
If you don't want the embeddings to get updated during training, just set trainable to False on the layer object.
I'm using an RNN on sequences of word embeddings to classify sentences. At first I was feeding pre-trained word embeddings and everything worked fine. I made the embeddings matrix a tf.placeholder with dimension (Vocab size, Embedding size) and fed some pre-trained embeddings from GloVe. I also use tf.nn.embedding_lookup to translate my inputs (which are sequences of word IDs) into sequences of embeddings.
Then I wanted to allow the model to train the embeddings as well, so I made the embedding matrix a tf.Variable instead of a placeholder. Now TensorFlow gives me this error -- apparently the AdamOptimizer can't handle the embedding lookup. Any idea what's up or how to fix this?
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input 0 of node
Adam/update_embeddings/AssignSub was passed float from _recv_embeddings_0:0
incompatible with expected float_ref.
You cannot feed a value to a variable and optimize it at the same time. Instead, you must first run a tf.assign on that variable to initialize it to the fed value, and then run the optimier. Or, more easily, you can just pass the glove vectors as the initializer of the variable and run tf.global_variables_initializer.
We are only using the RNN decoder (without encoder) for text generation, how is RNN decoder different from pure RNN operation?
RNN Decoder in TensorFlow: https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/dynamic_rnn_decoder
Pure RNN in TensorFlow: https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn
Thanks for your time
RNN Decoder is sequence to sequence mapping model. In combination with Encoder takes a sequence as input and generates another sequence as output. In many to many sequence learning setups, the input is a sequence of vectors and the output is another sequence of vectors.
Eg: Speech Wave to Text, Language Translation etc
In General We use Normal RNN for one to one sequence learning setup.
Eg: Next Character Prediction.
My goal is to ask if it is possible to use pre-trained GloVe vectors in a Tensorflow word-rnn LSTM generative model, and if so any guidance on how to achieve this?
I am referencing this from here
and I understand(I think) that I am supposed to put the vectors in the embeddings in line 35-37 of the model.py. From the code, I see that he is not using any pre-trained vectors but the words from the input text.
I have seen other answers like this but as I am new to Tensorflow and Python I do not fully understand how to apply this into the code.
GloVe generates two files, namely:
vocabulary file, with the count of all word occurrences
vector file. e.g the word [also -0.5432 -0.3210 0.1234...n_dimensions..]
Also, do I have to generate the GloVe vectors and train the LSTM model on the same corpus or can they be separate? eg. GloVe(100k words), text_to_train(50k words)
Thank you for the assistance!
Embeddings are word encoding, you load a pre-trained Glove encoding "dictionary" with 400 000 entries, where each token or entry is encoded in a 1D-vector of dim 50 for Glove 50, 100 for Glove 100 etc.
Your input dataset of dim N, M will go through encoding, each entry in the input dataset is encoded in the Glove encoding and stored in a row of the embedding matrix, of dim N, 50 or N, 100 etc.
You build a Keras embedding layer from this embedding matrix, which output is fed into the LSTM.
https://keras.io/examples/nlp/pretrained_word_embeddings/
I am building a seq2seq model using functions in tensorflow's seq2seq.py, where they have a function like this:
embedding_rnn_seq2seq(encoder_inputs, decoder_inputs, cell,
num_encoder_symbols, num_decoder_symbols,
embedding_size, output_projection=None,
feed_previous=False, dtype=dtypes.float32,
scope=None)
however, it seems that this function does not take pre-trained embeddings as input, are there any ways that I can take pre-trained word embeddings as input in this function?
There is no parameter you just hand over. Read in your embeddings (make sure vocabulary IDs match). Then, once you initialized all variables, find the embedding tensor (iterate through tf.all_variables to find the name). Then use tf.assign to overwrite the randomly initialized embeddings there with your embeddings.