We Keep Coding

Split data frame of comments into multiple rows

I have a data frame with long comments and I want to split them into indiviual sentences using spacy sentencizer.
Comments = pd.read_excel('Comments.xlsx', sheet_name = 'Sheet1')
Comments
>>>
reviews
0 One of the rare films where every discussion leaving the theater is about how much you
just had, instead of an analysis of its quotients.
1 Gorgeous cinematography, insane flying action sequences, thrilling, emotionally moving,
and a sequel that absolutely surpasses its predecessor. Well-paced, executed & has that
re-watchability factor.
I loaded the model like this
import spacy
nlp = spacy.load("en_core_news_sm")
And using sentencizer
from spacy.lang.en import English
nlp = English()
nlp.add_pipe('sentencizer')
Data = Comments.reviews.apply(lambda x : list( nlp(x).sents))
But when I check the sentence is in just one row like this
[One of the rare films where every discussion leaving the theater is about how much you just had.,
Instead of an analysis of its quotients.]
Thanks a lot for any help. I'm new using NLP tools in Data Frame.

Currently, Data is a Series whose rows are lists of sentences, or actually, lists of Spacy's Span objects. You probably want to obtain the text of these sentences and to put each sentence on a different row.
comments = [{'reviews': 'This is the first sentence of the first review. And this is the second.'},
{'reviews': 'This is the first sentence of the second review. And this is the second.'}]
comments = pd.DataFrame(comments) # building your input DataFrame
+----+--------------------------------------------------------------------------+
| | reviews |
|----+--------------------------------------------------------------------------|
| 0 | This is the first sentence of the first review. And this is the second. |
| 1 | This is the first sentence of the second review. And this is the second. |
+----+--------------------------------------------------------------------------+
Now let's define a function which, given a string, returns the list of its sentences as texts (strings).
def obtain_sentences(s):
doc = nlp(s)
sents = [sent.text for sent in doc.sents]
return sents
The function can be applied to the comments DataFrame to produce a new DataFrame containing sentences.
data = comments.copy()
data['reviews'] = comments.apply(lambda x: obtain_sentences(x['reviews']), axis=1)
data = data.explode('reviews').reset_index(drop=True)
data
I used explode to transform the elements of the lists of sentences into rows.
And this is the obtained output!
+----+--------------------------------------------------+
| | reviews |
|----+--------------------------------------------------|
| 0 | This is the first sentence of the first review. |
| 1 | And this is the second. |
| 2 | This is the first sentence of the second review. |
| 3 | And this is the second. |
+----+--------------------------------------------------+

How to construct a dataframe with LDA in Python

Based on 37,000 article texts, I implemented LDA mallet topic modeling. Each article was properly categorized and the dominant topic of each was determined.
Now I want to create a dataframe that shows each topic's percentages for each article, in Python.
I want the data frame to look like this:
no | Text | Topic_Num_1 | Topic_Num_2 | .... | Topic_Num_25
01 | article text1 | 0.7529 | 0.0034 | .... | 0.0011
02 | article text2 | 0.3529 | 0.0124 | .... | 0.0001
....
(37000 x 27 row)
How would I do this?
+
All the code I've been doing is based on the following site.
http://machinelearningplus.com/nlp/topic-modeling-gensim-python
How can I see the all probability list of the topics of every single article?

Here's a useful link for anyone that has just discovered this question.
I'm also pasting some example code, assuming that you have built a LDA-model and that you want to concatenate the topic-scores to a dataframe df.
import gensim
import numpy as np
lda_model = gensim.models.LdaMulticore(corpus, id2word, num_topics)
lda_scores = lda_model[corpus]
all_topics_csr = gensim.matutils.corpus2csc(lda_scores)
all_topics_numpy = all_topics_csr.T.toarray()
all_topics_pandas = pd.DataFrame(all_topics_numpy).reindex(df1.index).fillna(0)
df = pd.concat([df, all_topics_pandas.reindex(df.index)], axis=1, join="inner")

extracting the text after certain value in pandas

I am trying to extract the values in a column which has text data as below:
create date:1953/01/01 | first author:REAGAN RL
How can I extract the author name from the columns and store in a new column.
I tried the following ways:
df.str.extract("first author:(.*?)")
and
authorname=df['EntrezUID'].apply(lambda x:x.split("first author:")). The second one worked.
How can I use the regualr expressions achieve the similar thing

You can do:
## sample data
df = pd.DataFrame({'dd':['create date:1953/01/01 | first author:REAGAN RL','create date:1953/01/01 | first author:MEGAN RL']})
## output
df['names'] = df['dd'].str.extract(r'author\:(.*)')
print(df)
dd names
0 create date:1953/01/01 | first author:REAGAN RL REAGAN RL
1 create date:1953/01/01 | first author:MEGAN RL MEGAN RL

How do I test the Naive Bayes classifier with totally new data after I train/test?

I train/tested a binary classifier to give me output as 0 or 1 based on class. It is just like a spam classifier. Now, I have some extra data and I just want to test them and get an output array like:
[0 1 0 0 0... 1 0]
Here is what I did:
I used the pandas library to create a data frame
def dataFromDirectory(path):
rows = []
index = []
for filename, message in readFiles(path):
rows.append({'resume': message})
index.append(filename)
return DataFrame(rows, index=index)
test= DataFrame({'resume':[]})
test=test.append(dataFromDirectory(r'<folder path>'))
This worked and I successfully created a data frame, with 10 sample .txt files.
So,
test.head()
will give a 5x2 array of file path and content of the txt file in the two columns respectively. Something like this:
| data |
<path1> | <text> |
<path2> | <text> |
<path3> | <text> |
.
.
.
But, when I do Tf-Idf transformation:
testtf=tf.transform(test) #tf is the Tf-Idf vectorizer
pred1=mnb.predict(testtf) #MultinomialNaiveBayes is mnb
I get output as
[0]
What am I doing wrong? Please note that I am using Python 3.

Use the result from Cross tab (spark dataframe) for chi-square test in SparkMlib

I've generated a dataframe data from crosstab in Spark DataFrame and want to perform the chi-squared test.
It seems that Statistics.chiSqTest can only be applied to a matrix. My DataFrame looks like as below and I want to see whether the level distribution is the same across three groups:
true
false
and Undefined.
from pyspark.mllib.stat import Statistics
+-----------------------------+-------+--------+----------+
|levels | true| false|Undefined |
+-----------------------------+-------+--------+----------+
| 1 |32783 |634460 |2732340 |
| 2 | 2139 | 41248 |54855 |
| 3 |28837 |573746 |5632147 |
| 4 |16473 |320529 |8852552 |
+-----------------------------+-------+--------+----------+
Is there any easy way to transform this in order to be used for chi-squared test?

One way to handle this without using mllib.Statistics:
import scipy.stats
crosstab = ...
scipy.stats.chi2_contingency(
crosstab.drop(crosstab.columns[0]).toPandas().as_matrix()
)
If you really want Spark statistics:
from itertools import chain
Statistics.chiSqTest(DenseMatrix(
numRows=crosstab.count(), numCols=len(crosstab.columns) - 1,
values=list(chain(*zip(*crosstab.drop(crosstab.columns[0]).collect())))
))