We Keep Coding

Transformers (Attention is all you need) with Word2Vec or GloVe?

I am new to NLP and i am confused about the embedding.
Is it possible, if i already have trained GloVe embeddings / or Word2Vec embeddings and send these into Transformer? Or does the Transformer needs raw data and do its own embedding?
(Language: python, keras)

If you train a new transformer, you can do whatever you want with the bottom layer.
Most likely you are asking about pretrained transformers, though. Pretrained transformers such as Bert will have their own embeddings of the word pieces. In that case, you will probably get sufficient results just by using the results of the transformer.

Per https://en.wikipedia.org/wiki/BERT_(language_model)
BERT models are pre-trained from unlabeled data extracted from the
BooksCorpus with 800M words and English Wikipedia with 2,500M
words.
Whether to train your model depends on your data.
For simple english text, the out-of-the-box model should work well.
If your data concentrates on certain domain e.g. job requisitions and job applications, then you can extend the model by training it on your corpus (aka transfer learning).
https://huggingface.co/docs/transformers/training

Training a new dataset on bert

I am new to BERT
I have a amazon review dataset, where I want to predict the star rating based on the review
I know I can use a pretrained bert model as shown here
But I want to train the bert model on my own dataset. Is that whats being done here? And can I apply this type of 'fine tuning' on a pretrained model with any dataset to get more accurate results or do I have to do something else to train the model from scratch
And if I do want to train a model from scratch, where would I start

First of all what is pretraining? The procedure helps the model to learn syntactic <==> semantic (this is a spectrum) features of the language using an enormous amount of raw text (40GB) and processing power. objective function: casual language model and mask language model
What about fine-tuning a pre-trained model? suppose there is a model which has knowledge about the general aspect of the English language (POS, dependency tree, subj ... a little of everything). fine-tuning help us to direct the focus of the model on the most important features in our dataset, let's say in your dataset some syntactic feature is the game-changer, and the model should be careful about it!
objective function: based on downstream task
Training from scratch isn't feasible for most of us, but there is an approach to continue the pre-training phase using your own corpus/corpora (task-specific) without damaging model pieces of knowledge (hopefully)!
objective function: casual language model and mask language model
Here is an article about this approach and its effectiveness and you can be inspired by Scibert and COVIDbert. As you expect the use pre-trained bert as a starting point and continue pre-training using domain-specified corpus!

Fine-tune a BERT model for context specific embeddigns

I'm trying to find information on how to train a BERT model, possibly from the Huggingface Transformers library, so that the embedding it outputs are more closely related to the context o the text I'm using.
However, all the examples that I'm able to find, are about fine-tuning the model for another task, such as classification.
Would anyone happen to have an example of a BERT fine-tuning model for masked tokens or next sentence prediction, that outputs another raw BERT model that is fine-tuned to the context?
Thanks!

Here is an example from the Transformers library on Fine tuning a language model for masked token prediction.
The model that is used is one of the BERTForLM familly. The idea is to create a dataset using the TextDataset that tokenizes and breaks the text into chunks. Then use a DataCollatorForLanguageModeling to randomly mask tokens in the chunks when traing, and pass the model, the data and the collator to the Trainer to train and evaluate the results.

pre-trained spacy model or spacy.blank,for custom NER which is the right way?

I want train a spacy custom NER model,which is the best option?
the train data is ready (doccano)
option 1. use an existing pre-trained spacy model and update it with custom NER?.
option 2. create an empty model using spacy.blank() with custom NER?
I just want to identify my custom entity in a text, the other types of entities are not necessary...currently

You want to leverage transfer learning as much as possible: this means you most likely want to use a pre-trained model (e.g. on Wikipedia data) and fine-tune it for your use case. This is because training a spacy.blank model from scratch will require lots of data, whereas fine tuning a pretrained model might require as few as a couple hundreds labels.
However, pay attention to catastrophic forgetting which is the fact that when fine-tuning on some of your new labels, the model might 'forget' some of the old labels because they are no longer present in the training set.
For example, let's say you are trying to label the entity DOCTOR on top of a pre-trained NER model that labels LOC, PERSON and ORG. You label 200 DOCTOR records and fine tune your model with them. You might find that the model now predicts every PERSON as a DOCTOR.
That's all one can say without knowing more about your data. Please check out the spacy docs on training ner for more details.

How to predict the given value is anomaly or not with trained pyod.LOCI and Machine learning models?

I used the pyod.loci model to detect outliers. Trained the model with 100 records data.
I see most of the outliers detector models find the outliers with in the data only. I want to use the same trained model to predict the unseen data point whether it is anomaly or not?.
Can some one help me with an idea or solution? Model should work for individual or multiple column data as well.