I created my model using Word2Vec.
But the results were not good.
So I want to add a word.
The code I created the first time
Creation is possible, but can not be added.
Please tell me how can add.
createModel.py
token = loadCsv("test_data")
embeddingmodel = []
for i in range(len(token)):
temp_embeddingmodel = []
for k in range(len(token[i][0])):
temp_embeddingmodel.append(token[i][0][k])
embeddingmodel.append(temp_embeddingmodel)
embedding = Word2Vec(embeddingmodel, size=300, window=5, min_count=3, iter=100, sg=1,workers=4, max_vocab_size = 360000000)
embedding.save('post.embedding')
loadWord2Vec.py
tokens = W2V.tokenize(sentence)
embedding = Convert2Vec('Data/post.embedding', tokens)
zero_pad = W2V.Zero_padding(embedding, Batch_size, Maxseq_length, Vector_size)
Tell me how to add or merge the results of Word2Vec
There's no easy way to merge two Word2Vec models.
Only word-vectors that were trained together are "in the same space" and thus comparable.
The best policy would be to combine the two training corpuses of texts, and train a new model on the combined data, thus obtaining word-vectors for all words from the same training session.
Related
I'm evaluating Doc2Vec for a recommender API. I wasn't able to find a decent pre-trained model, so I trained a model on the corpus, which is about 8,000 small documents.
model = Doc2Vec(vector_size=25,
alpha=0.025,
min_alpha=0.00025,
min_count=1,
dm=1)
I then looped through the corpus to find similar documents for each document. Results were not very good (in comparison to TF-IDF). Note this is after testing different epochs and vector sizes.
inferred_vector = model.infer_vector(row['cleaned'].split())
sims = model.docvecs.most_similar([inferred_vector], topn=4)
I also tried extracting the trained vectors and using cosine_similarty, but results were oddly even worse.
cosine_similarities = cosine_similarity(model.docvecs.vectors_docs, model.docvecs.vectors_docs)
Am I doing something wrong or is the smallish corpus the problem?
Edit:
Prep and training code
def unesc(s):
for idx, row in s.iteritems():
s[idx] = html.unescape(row)
return s
custom_pipeline = [
preprocessing.lowercase,
unesc,
preprocessing.remove_urls,
preprocessing.remove_html_tags,
preprocessing.remove_diacritics,
preprocessing.remove_digits,
preprocessing.remove_punctuation,
lambda s: hero.remove_stopwords(s, stopwords=custom_stopwords),
preprocessing.remove_whitespace,
preprocessing.tokenize
]
ds['cleaned'] = ds['body'].pipe(hero.clean, pipeline=custom_pipeline)
w2v_total_data = list(ds['cleaned'])
tag_data = [TaggedDocument(words=doc, tags=[str(i)]) for i, doc in enumerate(w2v_total_data)]
model = Doc2Vec(vector_size=25,
alpha=0.025,
min_alpha=0.00025,
min_count=1,
epochs=20,
dm=1)
model.build_vocab(tag_data)
model.train(tag_data, total_examples=model.corpus_count, epochs=model.epochs)
model.save("lmdocs_d2v.model")
Without seeing your training code, there could easily be errors in text prep & training. Many online code examples are bonkers wrong in their Doc2Vec training technique!
Note that min_count=1 is essentially always a bad idea with this sort of algorithm: any example suggesting that was likely from a misguided author.
Is a mere .split() also the only tokenization applied for training? (The inference list-of-tokens should be prepped the same as the training lists-of-tokens.)
How was "not very good" and "oddly even worse" evaluated? For example, did the results seem arbitrary, or in-the-right-direction-but-just-weak?
"8,000 small documents" is a bit on the thin side for a training corpus, but it somewhat depends on "how small" – a few words, a sentence, a few sentences? Moving to smaller vectors, or more training epochs, can sometimes make the best of a smallish training set - but this sort of algorithm works best with lots of data, such that dense 100d-or-more vectors can be trained.
I have a dataset of synonyms and non-synonyms. These are stored in a list of python dictionaries like {"sentence1": <string>, "sentence2": <string>, "label": <1.0 or 0.0> }. Note that this words (or sentences) do not have to be a single token in the tokenizer.
I want to fine-tune a BERT-based model to take both sentences like: [[CLS], <sentence1_token1>, ...,<sentence1_tokenN>, [SEP], <sentence2_token1>, ..., <sentence2_tokenM>, [SEP]]. I want to take the embedding for the [CLS] token (or the pooled_ouput available in some models) and run it through one or more perceptron layers (MLP).
Once I have this new model with the additional layers I want to train it using my data. I have found some examples and I have been able to create the desired pipeline (using PyTorch's torch.nn for the perceptron layers, although I am open to hear recommendations on what is best).
model = AutoModel.from_pretrained(modelname)
tokenizer = AutoTokenizer.from_pretrained(modelname)
# input_sentences1 is a list of the first sentence of every pair
# input_sentences2 is a list of the second sentence of every pair
input = tokenizer( input_sentences1,
input_sentences2,
add_special_tokens = True,
padding=True,
return_tensors="pt" )
bert_output = model(**input)
# Extract embedding that will go through additional layers
pooled_output = bert_output.pooler_output
pooled_ouput_CLS_embedding = pooled_output[:]
## OR
# sequence_output = bert_output.last_hidden_state
# sequence_ouput_CLS_embedding = sequence_output[:,0,:]
# First layer
linear1 = nn.Linear(768, 256)
linear1_output = linear1(pooled_ouput_CLS_embedding)
# Second layer
linear2 = nn.Linear(256, 1)
linear2_output = linear2(linear1_output)
linear2_output # Random results becuase the layers have not been trained
How do I encapsulate this to facilitate training and how do I perform the fine tuning?
I have a movie review data set which has two columns Review(Sentences) and Sentiment(1 or 0).
I want to create a classification model using word2vec for the embedding and a CNN for the classification.
I've looked for tutorials on youtube but all they do is create vectors for every words and show me the similar words. Like this-
model= gensim.models.Word2Vec(cleaned_dataset, min_count = 2, size = 100, window = 5)
words= model.wv.vocab
simalar= model.wv.most_similar("bad")
I already have my dependent variable(y) which is my 'Sentiment' column all I need is the independent variable(X) which I can pass on to my CNN model.
Before using word2vec I used the Bag Of Words(BOW) model which generated a sparse matrix which was my independent(X) variable. How can I achieve something similar using word2vec?
Kindly correct me if I'm doing something wrong.
To get the word vector, you have to do this:
model['word_that_you_want']
You may also want to handle the KeyError that could arise if you don't find that given word in your model. You also might want to read about what an embedding layer is, which is usually used as the first layer of the neural network (for NLP generally) and is basically a lookup mapping of a word to its corresponding word vector.
To get the word vectors for an entire sentence, you need to first initialize a numpy array of zeros to the dimensions you want.
You might need other variables such as the length of the longest sentence so that you can pad all sentences to that length. The documentation of the pad_sequences method for Keras is here.
A simple example of getting a sentence of word vectors is:
import numpy as np
embedding_matrix = np.zeros((vocab_len, size_of_your_word_vector))
Then iterate over the index of embedding_matrix and add to it, if you find a word vector in your model.
I use this resource which has a lot of examples and I have referenced some of the code there (which I have also used myself sometimes):
embedding_matrix = np.zeros((vocab_length, 100))
for word, index in word_tokenizer.word_index.items():
embedding_vector = model[word] # using your w2v model, KeyError possible
if embedding_vector is not None:
embedding_matrix[index] = embedding_vector
And in your model (I'm assuming Tensorflow with Keras)
embedding_layer = Embedding(vocab_length, 100, weights=[embedding_matrix], input_length=length_long_sentence, trainable=False)
I hope this helps.
Word2Vec doesn't inherently create vectors for a text (set of words) – just individual words.
But, sometimes a not-so-bad vector for a multi-word text is the average of all its word-vectors.
If list_of_words is a list of the words in your text, and all the words are in the Word2Vec model, a simple way to get the average of those words' vectors is:
avg_vector_of_words = model.wv[list_of_words].mean(axis=0)
(If some words aren't present, you'd need to filter them before attempting this to avoid KeyErrors. If you wanted to leave out some words, or use unit-normed word-vectors, or unit-normalize the final vector, you'd need more code.)
Then avg_vector_of_words is a small, dense/'embedded' feature vector for the list_of-words text.
You could pass these vectors, one per text, to another downstream classifier, like your CNN, exactly analogously to how you were previously using sparse BOW vectors.
When doing training, I initialize my embedding matrix, using the pretrained embeddings picked for words in training set vocabulary.
import torchtext as tt
contexts = tt.data.Field(lower=True, sequential=True, tokenize=tokenizer, use_vocab=True)
contexts.build_vocab(data, vectors="fasttext.en.300d",
vectors_cache=config["vectors_cache"])
In my model I pass contexts.vocab as parameter and initialize embeddings:
embedding_dim = vocab.vectors.shape[1]
self.embeddings = nn.Embedding(len(vocab), embedding_dim)
self.embeddings.weight.data.copy_(vocab.vectors)
self.embeddings.weight.requires_grad=False
I train my model and during training I save its 'best' state via torch.save(model, f).
Then I want to test/create demo for model in separate file for evaluation. I load the model via torch.load. How do I extend the embedding matrix to contain test vocabulary? I tried to replace embedding matrix
# data is TabularDataset with test data
contexts.build_vocab(data, vectors="fasttext.en.300d",
vectors_cache=config["vectors_cache"])
model.embeddings = torch.nn.Embedding(len(contexts.vocab), contexts.vocab.vectors.shape[1])
model.embeddings.weight.data.copy_(contexts.vocab.vectors)
model.embeddings.weight.requires_grad = False
But the results are terrible (almost 0 accuracy). Model was doing good during training. What is the 'correct way' of doing this?
I am training my own word2vec model on Gensim in python, on a relatively small dataset. The data consist of about 3000 short-text entries from different people, most of which are two or three sentences. I know this is small for a word2vec dataset, but I've seen similar ones work in the past.
For some reason, when I train my model all of the features are impractically close to one another. For instance:
model.most_similar('jesus/NN')
[(u'person/NN', 0.9999418258666992),
(u'used/VB', 0.9998890161514282),
(u'so/RB', 0.9998359680175781),
(u'question/NN', 0.9997845888137817),
(u'just/RB', 0.9996646642684937),
(u'other/NN', 0.9995589256286621),
(u'allow/VB', 0.9995476603507996),
(u'feel/VB', 0.9995381236076355),
(u'attend/VB', 0.9995047450065613),
(u'make/VB', 0.9994802474975586)]
The parts of speech are included because I lemmatize the data.
Here is my training code:
cleanedResponses = []
#For every response
for rawValue in df['QB17_W6'].values:
lemValue = lemmatize(rawValue)
cleanedResponses.append(lemValue)
df['cleaned_responses'] = cleanedResponses
bigram_transformer = Phrases(df.cleaned_responses.values)
model = Word2Vec(bigram_transformer[df.cleaned_responses.values], size=5)
This also happens when I train without the bigram transformer. Does anybody have an idea as to why the distances are so close?