I have big chucks of text in English (avg length 800 words) which I would like to evaluate with a good and reliable sentiment analysis API.
Some threads seem to suggest APIs like Alchemy but I would like an evaluation of the sentiment along multiple dimensions and not just a single score. Example of such dimensions could be Positivity and Emotionality etc.
Do you know any APIs that would provide such more elaborate results?
The terms used in the natural language processing literature for positivity and emotionality are "valence" (or sometimes "polarity") and "arousal", respectively, so searching for APIs using those terms might be more useful to you. A quick search on those terms + sentiment + API revealed the following:
http://talc2.loria.fr/empathic/ can give positivity (valence) as well as the specific type of emotion (e.g. "sadness" vs. "disgust")
SentiStrength gives a positivity score as well as a negativity score. You can sum the scores to get positivity, or sum the absolute values of the score to get emotionality. For example a high-magnitude positivity score (+5) and a high-magnitude negativity score (-5) corresponds to high emotionality, but neutral positivity.
Mashape's Repustate ( https://www.mashape.com/repustate/repustate-sentiment-and-text-analytics ) can give positivity towards different aspects of a service (e.g. pos/neg sentiment towards price, food, staff, location, atmosphere, events). Some of their other APIs on this list may also be of interest: http://blog.mashape.com/list-of-20-sentiment-analysis-apis/ . Apparently they used to have sentiment detection APIs specific to the dimensions of anger and excitement, but these seem to have been phased out.
We recently compared 15 Sentiment Analysis APIs. Here are some relevant points:
sentiment score and positivity is essentially the same thing. Some APIs return the sentiment score, others - sentiment polarity labels (negative, positive etc) together with a confidence for each label. They could be mapped into each other (and we do that in our uniform API). The only difference is that the latter approach allows for expressing a mixed sentiment, while with the sentiment score it requires adding sentiment agreement (like Meaning Cloud does).
aspect-based sentiment is when the subject can be evaluated along different dimensions or aspects. An example is a restaurant review, which may combine sentiment towards service, meals, and prices in one sentence. We have found aspect-based sentiment in Aylien, Meaning Cloud and Repustate, with different domain models available at each of the services.
entity-based sentiment. another way to get more details is to perform entity extraction and then to analyze sentiment towards each of the entities mentioned in the sentence. This is supported by Google Cloud Natural Language.
Additionally, Aylien and Meaning Cloud provide sentiment subjectivity score, measuring how subjective is the writer opinion.
Surprisingly, only Meaning Cloud provides explicit irony detection. It is not clear if it is used in other models implicitly.
Here's the picture:
Take a look at this API: http://sentic.net/
They're doing sentiment analysis for a wide variety of different emotional dimensions at concept level and so much more...
Related
I have a manually labelled set of ~120K tweets. If I use VADER's compound score it only matches the manual labelling for ~24% of the records, textblob matches ~35% of the manually labelled record. If I take Vaders compound score and textblobs score and add then together and divide by 2 the resulting sentiment result matches the manual labelling ~70% of the time. Is there any reason for why its more accurate or is it just coincidence?
I think you're stumbling upon the idea behind ensemble learning. More often than not, putting multiple models together and combining their predictions leads to better results. Your implementation could be thought of as an equally weighted soft-voting ensemble. For more examples and additional implementations, the scikit-learn Voting Classifier docs are great.
The idea
I am collecting tweets talking about the three major candidates for the US presidency in November. After collecting the tweets from different states, I will score these tweets, and then analyze each candidate's followers/supporters on various aspects.
The problem
I am sure what method I should use to classify the tweets in order to produce reasonable outcomes. More precisely, I don't know how to tell if a tweet is supporting or opposing a specific candidate.
What I tried
I tried to use a library called textblob. Given a tweet, it returns a tuple of the form Sentiment(polarity, subjectivity). Polarity is a float which lies in the range of [-1,1] where 1 means positive statement and -1 means a negative statement. This method does not return reasonable results at all when applied. For example, given a tweet like "Donald Trump is horrible! I still support him tho.", it returns a polarity of -1.0 (negative), which does not make any sense at all.
Further Research
I looked for more examples and found this. In that example, the author uses a mood vocabulary (from the internet) and later assigns a mood to each tweet. I am planning to take a close look at that article and apply the method used there.
My questions
What is a proper way to categorize those tweets? Should I consider the one I mentioned in the further research section?
What if a tweet contains both names? something like "Trump will beat Biden". How something like this is going to be scored using a specific method?
Sentiment Analysis is a topic of NLP (Natural language processing). What you are wondering is about NLP which is one of many interesting branches of machine learning.
Your approach to weigh out the polarity of a tweet is correct. Since you have 3 different candidates aka. labels, this is a more interesting problem.
I would store a sentiment array of length 3 for each tweet in order to calculate weights for each candidate. For example; a tweet like "A is bad. B will do much better! But also C might too?" might produce [-1, 0.8, 0.4].
To do that, you need a corpus. The corpus aka your dataset should contain tweets and labels for each tweet so that your machine learning model can learn from the tweets.
There are many ways to build a machine learning model and train it with your dataset. That is a topic of data science. Data scientists try to improve some performance indicators in order to improve their model.
The simpliest would be something like, parse out all words out of tweet, increment their values in a hash map with labels and normalize. Now you have a hash map containing the sentiment value for each word.
But in reality, this would not work out well as outliers and lack of dataset would affect your result. Therefore, you need to look at your data, problem and choose the right machine learning model. Look at this article to learn more about building a sentiment classifier.
If you have a corpus of text, how can you identify all the categories (from a list of pre-defined categories) and the associated sentiment (positive/negative writing) with it?
I will be doing this in Python but at this stage I am not necessarily looking for a language specific solution.
Let's look at this question with an example to try and clarify what I am asking.
If I have a whole corpus of reviews for products e.g.:
Microsoft's Xbox One offers impressive graphics and a solid list of exclusive 2015 titles. The Microsoft console currently edges ahead of the PS4 with a better selection of media apps. The console's fall-2015 dashboard update is a noticeable improvement. The console has backward compatibility with around 100 Xbox 360 titles, and that list is poised to grow. The Xbox One's new interface is still more convoluted than the PS4's. In general, the PS4 delivers slightly better installation times, graphics and performance on cross-platform games. The Xbox One also lags behind the PS4 in its selection of indie games. The Kinect's legacy is still a blemish. While the PS4 remains our overall preferred choice in the game console race, the Xbox One's significant course corrections and solid exclusives make it a compelling alternative.
And I have a list of pre-defined categories e.g. :
Graphics
Game Play
Game Selection
Apps
Performance
Irrelevant/Other
I could take my big corpus of reviews and break them down by sentence. For each sentence in my training data I can hand tag them with the appropriate categories. The problem is that there could be various categories in 1 sentence.
If it was 1 category per sentence then any classification algorithm from scikit-learn would do the trick. When working with multi-classes I could use something like multi-label classification.
Adding in the sentiment is the trickier part. Identifying sentiment in a sentence is a fairly simple task but if there is a mix of sentiment on different labels that becomes different.
The example sentence "The Xbox One has a good selection of games but the performance is worse than the PS4". We can identify two of our pre-defined categories (game selection, performance) but we have positive sentiment towards game selection and a negative sentiment towards performance.
What would be a way to identify all categories in text (from our pre-defined list) with their associated sentiment?
One simple method is to break your training set into minimal sentences using a parser and use that as the input for labelling and sentiment classification.
Your example sentence:
The Xbox One has a good selection of games but the performance is worse than the PS4
Using the Stanford Parser, take S tags that don't have child S tags (and thus are minimal sentences) and put the tokens back together. For the above sentence that would give you these:
The Xbox One has a good selection of games
the performance is worse than the PS4
Sentiment within an S tag should be consistent most of the time. If sentences like The XBox has good games and terrible graphics are common in your dataset you may need to break it down to NP tags but that seems unlikely.
Regarding labelling, as you mentioned any multi-label classification method should work.
For more sophisticated methods, there's a lot of research on join topic-sentiment models - a search for "topic sentiment model" turns up a lot of papers and code. Here's sample training data from a paper introducing a Hidden Topic Sentiment Model that looks right up your alley. Note how in the first sentence with labels there are two topics.
Hope that helps!
The only approach I could think of would consists of a set of steps.
1) Use some library to extract entities from text and their relationships. For example, check this article:
http://www.nltk.org/book/ch07.html
By parsing each text you may figure out which entities you have in each text and which chunks of text are related to the entity.
2) Use NLTKs sentiment extraction to analyze chunks specifically related to each entity and obtain their sentiment. That gives you sentiment of each entity.
3) After that you need to come of with a way to map entities which you may face in text to what you call 'topics'. Unfortunately, I don't see a way to automate it since you clearly not define topics conventionally, through word frequency (like in topic modelling algorithms - LDA, NMF etc).
I want to classify the tweets into predefined categories (like: sports, health, and 10 more). If I had labeled data, I would be able to do the classification by training Naive Bayes or SVM. As described in http://cucis.ece.northwestern.edu/publications/pdf/LeePal11.pdf
But I cannot figure out a way with unlabeled data. One possibility could be using Expectation-Maximization and generating clusters and label those clusters. But as said earlier I have predefined set of classes, so clustering won't be as good.
Can anyone guide me on what techniques I should follow. Appreciate any help.
Alright by what i can understand i think there are multiple ways to attend to this case.
there will be trade offs and the accuracy rate may vary. because of the well know fact and observation
Each single tweet is distinct!
(unless you are extracting data from twitter stream api based on tags and other keywords). Please define the source of data and how are you extracting it. i am assuming you're just getting general tweets which can be about anything
The thing you can do is to generate a set of dictionary for each class you have
(i.e Music => pop , jazz , rap , instruments ...)
which will contain relevant words to that class. You can use NLTK for python or Stanford NLP for other languages.
You can start with extracting
Synonyms
Hyponyms
Hypernyms
Meronyms
Holonyms
Go see these NLP Lexical semantics slides. it will surely clear some of the concepts.
Once you have dictionaries for each classes. cross compare them with the tweets you have got. the tweet which has the most similarity (you can rank them according to the occurrences of words from the these dictionaries) you can label it to that class. This will make your tweets labeled like others.
Now the question is the accuracy! But it depends on the data and versatility of your classes. This may be an "Over kill" But it may come close to what you want.
Furthermore you can label some set of tweets this way and use Cosine Similarity to cross identify other tweets. This will help with the optimization part. But then again its up-to you. As you know what Trade offs you can bear
The real struggle will be the machine learning part and how you manage that.
Actually this seems as a typical use case of semi-supervised learning. There are plenty methods of use here, including clustering with constraints (where you force model to cluster samples from the same class together), transductive learning (where you try to extrapolate model from labeled samples onto distribution of unlabeled ones).
You could also simply cluster data as #Shoaib suggested, but then you will have to come up the the heuristic approach how to deal with clusters with mixed labeling. Futhermore - obviously solving optimziation problem not related to the task (labeling) will not be as good as actually using this knowledge.
You can use clustering for that task. For that you have to label some examples for each class first. Then using these labeled examples, you can identify the class of each cluster easily.
I downloaded WN-Affect. I am however not sure how to use it to detect the mood of a sentence. For example if I have a string "I hate football." I want to be able to detect whether the mood is bad and the emotion is fear. WN-Affect has no tutorial on how to do it, and I am kind of new to python. Any help would be great!
In short: Use SentiWordNet instead and look at https://github.com/kevincobain2000/sentiment_classifier
In Long:
Affectedness vs Sentiment
The line between affect and sentiment is very fine. One should looking into Affectedness in linguistics studies, e.g. http://compling.hss.ntu.edu.sg/events/2014-ws-affectedness/ and Sentiment Analysis in computational researches. For now, let's call both the task of identifying affect and sentiment, sentiment analysis.
Also note that WN-Affect is a rather old resource compared to SentiWordNet, http://sentiwordnet.isti.cnr.it/.
Here's a good resource for using SentiWordNet for sentiment analysis: https://github.com/kevincobain2000/sentiment_classifier.
Often sentiment analysis has only two classes, positive or negative sentiment. Whereas the WN-affect uses 11 types of affectedness labels:
emotion
mood
trait
cognitive state
physical state
hedonic signal
emotion-eliciting
emotional response
behaviour
attitude
sensation
For each type, there are multiple classes, see https://github.com/larsmans/wordnet-domains-sentiwords/blob/master/wn-domains/wn-affect-1.1/a-hierarchy.xml
To answer the question of how one can use the WN-Affect, there're several things you need to do:
First map WN1.6 to WN3.0 (it's not an easy task, you have to do several mappings, especially the mapping between 2.0-2.1)
Now using the WN-Affect with WN3.0, you can apply
the same classification technique as he SentiWordNet sentiment classifier or
try to maximize the classes within text and then use some heuristics to choose 'positive' / 'negative'
WordNet-Affect uses WordNet 1.6 offsets.
However, WordNet 1.6 is still available for download. You can use the nltk.corpus.WordNetCorpusReader class to load it. I wrote all the code to do it here.