We Keep Coding

How does the ISRI Stemmer give better stem words than Lancaster or Snowball Stemmer

I have this sample text which i want to tokenize and subsequently find the stem words
sample_text = "'I am a student from the University of Alabama. \
I was born in Ontario, Canada and I am a huge fan of the United States. \
I am going to get a degree in Philosophy to improve\
my chances of becoming a Philosophy professor. \
I have been working towards this goal for 4 years. \
I am currently enrolled in a PhD program. \
It is very difficult, but I am confident that it will be a good decision'"
Using Lancaster Stemmer I am getting the following result -
sentences = sent_tokenize(sample_text)
from nltk.stem import LancasterStemmer
lancaster = LancasterStemmer()
for i in range(len(sentences)):
sentences[i] = re.sub('[^A-Za-z0-9]', ' ', sentences[i])
sentences[i] = word_tokenize(sentences[i])
stopwds = [word.lower() for word in stopwords.words('english')]
sentences[i] = [word.lower() for word in sentences[i] if word.lower() not in stopwds]
sentences[i] = [lancaster.stem(word) for word in sentences[i]]
print(sentences[i])
Output of Lancaster Stemmer:
['stud', 'univers', 'alabam']
['born', 'ontario', 'canad', 'hug', 'fan', 'unit', 'stat']
['going', 'get', 'degr', 'philosoph', 'improvemy', 'chant', 'becom', 'philosoph', 'profess']
['work', 'toward', 'goal', '4', 'year']
['cur', 'enrol', 'phd', 'program']
['difficult', 'confid', 'good', 'decid']
Output with Snowball stemmer -
['student', 'univers', 'alabama']
['born', 'ontario', 'canada', 'huge', 'fan', 'unit', 'state']
['go', 'get', 'degre', 'philosophi', 'improvemi', 'chanc', 'becom', 'philosophi', 'professor']
['work', 'toward', 'goal', '4', 'year']
['current', 'enrol', 'phd', 'program']
['difficult', 'confid', 'good', 'decis']
Output of Porter Stemmer
['student', 'univers', 'alabama']
['born', 'ontario', 'canada', 'huge', 'fan', 'unit', 'state']
['go', 'get', 'degre', 'philosophi', 'improvemi', 'chanc', 'becom', 'philosophi', 'professor']
['work', 'toward', 'goal', '4', 'year']
['current', 'enrol', 'phd', 'program']
['difficult', 'confid', 'good', 'decis']
Whereas ISRI Stemmer almost gives me same results as if the words had been lemmatized
sentences = sent_tokenize(sample_text)
from nltk.stem import ISRIStemmer
isri = ISRIStemmer()
for i in range(len(sentences)):
sentences[i] = re.sub('[^A-Za-z0-9]', ' ', sentences[i])
sentences[i] = word_tokenize(sentences[i])
stopwds = [word.lower() for word in stopwords.words()]
sentences[i] = [word.lower() for word in sentences[i] if word.lower() not in stopwds]
sentences[i] = [ isri.stem(word) for word in sentences[i]]
print(sentences[i])
Output :
['student', 'university', 'alabama']
['born', 'ontario', 'canada', 'huge', 'fan', 'united', 'states']
['going', 'get', 'degree', 'philosophy', 'improvemy', 'chances', 'becoming', 'philosophy', 'professor']
['working', 'towards', 'goal', '4', 'years']
['currently', 'enrolled', 'phd', 'program']
['difficult', 'confident', 'good', 'decision']
Can someone explain how ISRI Stemmer gives almost Lemmatized words

This stemmer was developed specifically for Arabic, that is why you are getting such results when using it on English.
Here is a link to the nltk doc https://www.nltk.org/_modules/nltk/stem/isri.html

how to for each item in a list create class objects by passing parameters from functions

defined few functions (not induced body of the functions to simplify question here)
def policyname(i):
retrun policyname
def policytype(i):
retrun policytype
def active(i):
retrun active
def backupselection(i):
retrun backupselection
defined a list -
clients = ['winwebint16', 'winwebtpie03', 'winwebtpie04', 'winwtsdt08', 'winwtsmwg03', 'winwtsqnr03', 'winwtswrl37', 'winwtswrl60', 'winwtswrl62', 'winwtswrl63', 'winwtswrl75', 'winwtszsim03', 'winwww0016','winsbk0100', 'winsbk0100a0', 'winsbk0100a1', 'winsbk0101', 'iinf065', 'iinf130', 'iinf185', 'iinf2126', 'inbf005', 'inis001', 'ipdataisbic01', 'ipdataisbic02', 'ipdataispre01', 'ipdataispre02', 'iproip02', 'isis002', 'isyn002', 'isyn006', 'isyn011', 'isyn012','isyn014', 'isyn038', 'isyn039', 'isyn040', 'mu2ssql1001', 'mu2ssql1003', 'macrsz0001', 'macrsz0005']
defined a class -
class client():
def __init__(self,policyname,policytype,active,backupselection):
self.policyname = policyname
self.policytype = policytype
self.active = active
self.backupselection = backupselection
For each item in clients list create class objects by passing parameters from functions.
Is below code is correct ?
for i in clients:
i = client(policyname(i),policytype(i),active(i),backupselection(i))
with above code, will i be able to access specific class objects like ?
print(winwebint16.policyname)
print(winwebint16.policytype)
print(winwebint16.active)
print(winwebint16.backupselection)

Classnames should be capital letter but that's just convention. Further you have a problem with the for loop
for i in clients:
i = client(policyname(i),policytype(i),active(i),backupselection(i))
The created client is not visible outside of the for loop scope, so you might wanna add them to some kind of list or dict
client_list: dict = {}
for i in clients:
client_list[i] = client(policyname(i),policytype(i),active(i),backupselection(i))
you should then be able to print like
print(client_list['winwebint16'].policyname)

This really worked for me :
clients = ['669165933', '963881480', '341417157', '514321792', '115456712', '547995746', '135425221', '871543967', '770463311', '616607081', '814711606', '939825713']
policynames = ['Tuvalu', 'Grenada', 'Russia', 'Sao Tome and Principe', 'Rwanda', 'Solomon Islands', 'Angola', 'Burkina Faso', 'Republic of the Congo', 'Senegal', 'Kyrgyzstan', 'Cape Verde']
policytypes = ['Offline', 'Online', 'Offline', 'Online', 'Offline', 'Online', 'Offline', 'Online', 'Offline', 'Online', 'Online', 'Offline']
actives = ['Baby Food', 'Cereal', 'Office Supplies', 'Fruits', 'Office Supplies', 'Baby Food', 'Household', 'Vegetables', 'Personal Care', 'Cereal', 'Vegetables', 'Clothes']
backupselections = ['H', 'C', 'L', 'C', 'L', 'C', 'M', 'H', 'M', 'H', 'H', 'H']
def policyname(i):
return policynames[i]
def policytype(i):
return policytypes[i]
def active(i):
return actives[i]
def backupselection(i):
return backupselections[i]
class client():
def __init__(self,policyname,policytype,active,backupselection):
self.policyname = policyname
self.policytype = policytype
self.active = active
self.backupselection = backupselection
for i in range(0,len(clients)):
i = client(policyname(i),policytype(i),active(i),backupselection(i))
print(i.policyname,i.policytype,i.active,i.backupselection)
Tuvalu Offline Baby Food H
Grenada Online Cereal C
Russia Offline Office Supplies L
Sao Tome and Principe Online Fruits C
Rwanda Offline Office Supplies L
Solomon Islands Online Baby Food C
Angola Offline Household M
Burkina Faso Online Vegetables H
Republic of the Congo Offline Personal Care M
Senegal Online Cereal H
Kyrgyzstan Online Vegetables H
Cape Verde Offline Clothes H
But still I can't print
print(669165933.policyname)
File "<ipython-input-20-72da397c032b>", line 1
print(669165933.policyname)
^
SyntaxError: invalid syntax

Get continent name from country using pycountry

How to convert continent name from country name using pycountry. I have a list of country like this
country = ['India', 'Australia', ....]
And I want to get continent name from it like.
continent = ['Asia', 'Australia', ....]

All the tools you need are provided in pycountry-convert.
Here is an example of how you can create your own function to make a direct conversion from country to continent name, using pycountry's tools:
import pycountry_convert as pc
def country_to_continent(country_name):
country_alpha2 = pc.country_name_to_country_alpha2(country_name)
country_continent_code = pc.country_alpha2_to_continent_code(country_alpha2)
country_continent_name = pc.convert_continent_code_to_continent_name(country_continent_code)
return country_continent_name
# Example
country_name = 'Germany'
print(country_to_continent(country_name))
Out[1]: Europe
Hope it helps!
Remember you can access function descriptions using '?? function_name'
?? pc.country_name_to_country_alpha2

Looking at the documentation, would something like this do the trick?:
country_alpha2_to_continent_code()
It converts country code(eg: NO, SE, ES) to continent name.
If first you need to acquire the country code you could use:
country_name_to_country_alpha2(cn_name, cn_name_format="default")
to get the country code from country name.
Full example:
import pycountry_convert as pc
country_code = pc.country_name_to_country_alpha2("China", cn_name_format="default")
print(country_code)
continent_name = pc.country_alpha2_to_continent_code(country_code)
print(continent_name)

from pycountry_convert import country_alpha2_to_continent_code, country_name_to_country_alpha2
continents = {
'NA': 'North America',
'SA': 'South America',
'AS': 'Asia',
'OC': 'Australia',
'AF': 'Africa',
'EU': 'Europe'
}
countries = ['India', 'Australia']
[continents[country_alpha2_to_continent_code(country_name_to_country_alpha2(country))] for country in countries]
Don't know what to do with Antarctica continent ¯_(ツ)_/¯

More efficient way to clean dataframe than loc

My code looks like:
import pandas as pd
df = pd.read_excel("Energy Indicators.xls", header=None, footer=None)
c_df = df.copy()
c_df = c_df.iloc[18:245, 2:]
c_df = c_df.rename(columns={2: 'Country', 3: 'Energy Supply', 4:'Energy Supply per Capita', 5:'% Renewable'})
c_df['Energy Supply'] = c_df['Energy Supply'].apply(lambda x: x*1000000)
c_df.loc[c_df['Country'] == 'Korea, Rep.'] = 'South Korea'
c_df.loc[c_df['Country'] == 'United States of America20'] = 'United States'
c_df.loc[c_df['Country'] == 'United Kingdom of Great Britain and Northern Ireland'] = 'United Kingdom'
c_df.loc[c_df['Country'] == 'China, Hong Kong Special Administrative Region'] = 'Hong Kong'
c_df.loc[c_df['Country'] == 'Venezuela (Bolivarian Republic of)'] = 'Venezuela'
c_df.loc[c_df['Country'] == 'Bolivia (Plurinational State of)'] = 'Bolivia'
c_df.loc[c_df['Country'] == 'Switzerland17'] = 'Switzerland'
c_df.loc[c_df['Country'] == 'Australia1'] = 'Australia'
c_df.loc[c_df['Country'] == 'China2'] = 'China'
c_df.loc[c_df['Country'] == 'Falkland Islands (Malvinas)'] = 'Bolivia'
c_df.loc[c_df['Country'] == 'Greenland7'] = 'Greenland'
c_df.loc[c_df['Country'] == 'Iran (Islamic Republic of'] = 'Iran'
c_df.loc[c_df['Country'] == 'Italy9'] = 'Italy'
c_df.loc[c_df['Country'] == 'Japan10'] = 'Japan'
c_df.loc[c_df['Country'] == 'Kuwait11'] = 'Kuwait'
c_df.loc[c_df['Country'] == 'Micronesia (Federal States of)'] = 'Micronesia'
c_df.loc[c_df['Country'] == 'Netherlands12'] = 'Netherlands'
c_df.loc[c_df['Country'] == 'Portugal13'] = 'Portugal'
c_df.loc[c_df['Country'] == 'Saudi Arabia14'] = 'Saudi Arabia'
c_df.loc[c_df['Country'] == 'Serbia15'] = 'Serbia'
c_df.loc[c_df['Country'] == 'Sint Maarteen (Dutch part)'] = 'Sint Marteen'
c_df.loc[c_df['Country'] == 'Spain16'] = 'Spain'
c_df.loc[c_df['Country'] == 'Ukraine18'] = 'Ukraine'
c_df.loc[c_df['Country'] == 'Denmark5'] = 'Denmark'
c_df.loc[c_df['Country'] == 'France6'] = 'France'
c_df.loc[c_df['Country'] == 'Indonesia8'] = 'Indonesia'
I feel like there must be an easier way to change the values of the countries with parentheses and numbers in their names. What pandas method can I use to look within the column for names with numbers of parentheses? isin?

You can start by getting rid of numbers and text in parentheses. After that, for everything else that requires non-trivial replacement, declare a map and apply it using pd.Series.replace.
mapper = {'Korea, Rep' : 'South Korea', 'Falkland Islands' : 'Bolivia', ...}
df['Country'] = (
df['Country'].str.replace(r'\d+|\s*\(.*\)', '').str.strip().replace(mapper)
)
Simple enough, done.
Details
\d+ # one or more digits
| # regex OR pipe
\s* # zero or more whitespace characters
\( # literal parentheses (opening brace)
.* # match anything
\) # closing brace

Using a dictionary and then df.replace:
dict_to_replace = {'Korea, Rep.':'South Korea',
'United States of America20':'United States',
'United Kingdom of Great Britain and Northern Ireland': 'United Kingdom'
...}
df['c_df'] = df['c_df'].replace(dict_to_replace)

Failing to append to dictionary. Python

I am experiencing a strange faulty behaviour, where a dictionary is only appended once and I can not add more key value pairs to it.
My code reads in a multi-line string and extracts substrings via split(), to be added to a dictionary. I make use of conditional statements. Strangely only the key:value pairs under the first conditional statement are added.
Therefore I can not complete the dictionary.
How can I solve this issue?
Minimal code:
#I hope the '\n' is sufficient or use '\r\n'
example = "Name: Bugs Bunny\nDOB: 01/04/1900\nAddress: 111 Jokes Drive, Hollywood Hills, CA 11111, United States"
def format(data):
dic = {}
for line in data.splitlines():
#print('Line:', line)
if ':' in line:
info = line.split(': ', 1)[1].rstrip() #does not work with files
#print('Info: ', info)
if ' Name:' in info: #middle name problems! /maiden name
dic['F_NAME'] = info.split(' ', 1)[0].rstrip()
dic['L_NAME'] = info.split(' ', 1)[1].rstrip()
elif 'DOB' in info: #overhang
dic['DD'] = info.split('/', 2)[0].rstrip()
dic['MM'] = info.split('/', 2)[1].rstrip()
dic['YY'] = info.split('/', 2)[2].rstrip()
elif 'Address' in info:
dic['STREET'] = info.split(', ', 2)[0].rstrip()
dic['CITY'] = info.split(', ', 2)[1].rstrip()
dic['ZIP'] = info.split(', ', 2)[2].rstrip()
return dic
if __name__ == '__main__':
x = format(example)
for v, k in x.iteritems():
print v, k

Your code doesn't work, at all. You split off the name before the colon and discard it, looking only at the value after the colon, stored in info. That value never contains the names you are looking for; Name, DOB and Address all are part of the line before the :.
Python lets you assign to multiple names at once; make use of this when splitting:
def format(data):
dic = {}
for line in data.splitlines():
if ':' not in line:
continue
name, _, value = line.partition(':')
name = name.strip()
if name == 'Name':
dic['F_NAME'], dic['L_NAME'] = value.split(None, 1) # strips whitespace for us
elif name == 'DOB':
dic['DD'], dic['MM'], dic['YY'] = (v.strip() for v in value.split('/', 2))
elif name == 'Address':
dic['STREET'], dic['CITY'], dic['ZIP'] = (v.strip() for v in value.split(', ', 2))
return dic
I used str.partition() here rather than limit str.split() to just one split; it is slightly faster that way.
For your sample input this produces:
>>> format(example)
{'CITY': 'Hollywood Hills', 'ZIP': 'CA 11111, United States', 'L_NAME': 'Bunny', 'F_NAME': 'Bugs', 'YY': '1900', 'MM': '04', 'STREET': '111 Jokes Drive', 'DD': '01'}
>>> from pprint import pprint
>>> pprint(format(example))
{'CITY': 'Hollywood Hills',
'DD': '01',
'F_NAME': 'Bugs',
'L_NAME': 'Bunny',
'MM': '04',
'STREET': '111 Jokes Drive',
'YY': '1900',
'ZIP': 'CA 11111, United States'}