I have a column that has a lot of doctor specialties. I want to clean it up and created a function below:
def specialty(x):
if x.str.contains('Urolog'):
return 'Urology'
elif x.str.contains('Nurse'):
return 'Nurse Practioner'
elif x.str.contains('Oncology'):
return 'Oncology'
elif x.str.contains('Physician'):
return 'Physician Assistant'
elif x.str.contains('Family Medicine'):
return 'Family Medicine'
elif x.str.contains('Anesthes'):
return 'Anesthesiology'
else:
return 'Other'
df['desc_clean'] = df['desc'].apply(specialty)
However I get an error TypeError: 'function' object is not subscriptable
There are too many values to use a manual mapping so i wanted to use str.contains. Is there a way to do this better?
EDIT: Sample DF
{'person_id': {39063: 33081476009,
50538: 33033519093,
56075: 33170508793,
36593: 33061707789,
51656: 33047685345,
95512: 33022026049,
40286: 33038034707,
3887: 33076466195,
40161: 33052807819,
52905: 33190526939,
35418: 33008425164,
35934: 33015737122,
3389: 33055125864,
136: 33139641318,
105460: 33113871389,
52568: 33075745388,
24725: 33052090907,
34838: 33205449839,
31908: 33183672635,
36115: 33006692696},
'final_desc': {39063: 'None',
50538: 'Urology',
56075: 'Anesthesiology',
36593: 'None',
51656: 'Urology',
95512: 'None',
40286: 'Anesthesiology',
3887: 'Specialist',
40161: 'None',
52905: 'Anesthesiology',
35418: 'Urology',
35934: 'None',
3389: 'Ophthalmology',
136: 'Rheumatology',
105460: 'None',
52568: 'Urology',
24725: 'Family Medicine',
34838: 'None',
31908: 'Nurse Practitioner',
36115: 'None'}}
To do this, we can define a mapping between matches, then iterate through them and set the column's value, keeping track of columns we've changed. At the end, any columns we never matched get set to 'Other'.
mapping = {'Urolog': 'Urology',
'Nurse': 'Nurse Practioner',
'Oncology': 'Oncology',
'Physician': 'Physician Assistant',
'Family Medicine': 'Family Medicine',
'Anesthes': 'Anesthesiology'}
def specialty(column):
column = column.copy()
matches = pd.Series(False, index=column.index)
for k,v in mapping.items():
match = column.str.contains(k)
column[match] = v
matches[match] = True
column[~matches] = 'Other'
return column
specialty(df['final_desc'])
39063 Other
50538 Urology
56075 Anesthesiology
36593 Other
51656 Urology
95512 Other
40286 Anesthesiology
3887 Other
40161 Other
52905 Anesthesiology
35418 Urology
35934 Other
3389 Other
136 Other
105460 Other
52568 Urology
24725 Family Medicine
34838 Other
31908 Nurse Practioner
36115 Other
Name: final_desc, dtype: object
x received by specialty function is string itself. So no x.str and since it is string you can use 'in' to check as below. Modified some data to see the result
Tip: You should use a dictionary or list rather than using the elif chain.
Code:
import pandas as pd
import numpy as np
def specialty(x):
print(x)
if x in 'Urolog':
return 'Urology'
elif x in 'Nurse':
return 'Nurse Practioner'
elif x in 'Oncology':
return 'Oncology'
elif x in 'Physician':
return 'Physician Assistant'
elif x in 'Family Medicine':
return 'Family Medicine'
elif x in 'Anesthes':
return 'Anesthesiology'
else:
return 'Other'
df = pd.DataFrame(data={'person_id': {39063: 33081476009, 50538: 33033519093, 56075: 33170508793, 36593: 33061707789, 51656: 33047685345, 95512: 33022026049, 40286: 33038034707, 3887: 33076466195, 40161: 33052807819, 52905: 33190526939, 35418: 33008425164, 35934: 33015737122, 3389: 33055125864, 136: 33139641318, 105460: 33113871389, 52568: 33075745388, 24725: 33052090907, 34838: 33205449839, 31908: 33183672635, 36115: 33006692696},
'final_desc': {39063: 'None', 50538: 'Urolog', 56075: 'Anesthes', 36593: 'None', 51656: 'Urology', 95512: 'None', 40286: 'Anesthes', 3887: 'Specialist', 40161: 'None', 52905: 'Anesthesiology', 35418: 'Urology', 35934: 'None', 3389: 'Ophthalmology', 136: 'Rheumatology', 105460: 'None', 52568: 'Urology', 24725: 'Family Medicine', 34838: 'None', 31908: 'Nurse', 36115: 'None'}})
df['desc_clean'] = df['final_desc'].apply(specialty)
print(df)
Output:
person_id final_desc desc_clean
39063 33081476009 None Other
50538 33033519093 Urolog Urology
56075 33170508793 Anesthes Anesthesiology
36593 33061707789 None Other
51656 33047685345 Urology Other
95512 33022026049 None Other
40286 33038034707 Anesthes Anesthesiology
3887 33076466195 Specialist Other
40161 33052807819 None Other
52905 33190526939 Anesthesiology Other
35418 33008425164 Urology Other
35934 33015737122 None Other
3389 33055125864 Ophthalmology Other
136 33139641318 Rheumatology Other
105460 33113871389 None Other
52568 33075745388 Urology Other
24725 33052090907 Family Medicine Family Medicine
34838 33205449839 None Other
31908 33183672635 Nurse Nurse Practioner
36115 33006692696 None Other
You can use a library like fuzzywuzzy for fuzzy string matching. The benefit of this approach is it is more flexible than some rule set, as demonstrated below.
This solution generates the max score of substrings and candidate categories, returning the one that matches best. If it's below the threshold it will return the default value ("None"):
from fuzzywuzzy import fuzz
CATEGORIES = [
'Urology',
'Nurse Practioner',
'Oncology',
'Physician Assistant',
'Family Medicine',
'Anesthesiology',
'Specialist',
]
def best_match(
text,
categories=CATEGORIES,
default="None",
threshold=65
):
matches = {fuzz.partial_ratio(cat, text): cat for cat in categories}
best_score = max(matches)
best_match = matches[best_score]
if best_score >= threshold:
return best_match
else:
return default
df["final_desc"] = df.desc.apply(best_match)
result:
person_id final_desc desc
52568 33075745388 Urology urologist
36593 33061707789 Nurse Practioner nruse practition
136 33139641318 Specialist oncology specialist
50538 33033519093 Physician Assistant physicians assistant
3389 33055125864 Family Medicine fam. medicine
51656 33047685345 Anesthesiology anesthesiology
35418 33008425164 Anesthesiology anesthesiologist
52905 33190526939 Nurse Practioner Nurses practitioner
36115 33006692696 Specialist Occupational specialist
31908 33183672635 Oncology Oncologist
You can iterate directly using the index :
ix = df[df.desc.str.contains('Urolog')].index
df.loc[ix, 'desc_clean'] = "Urology"
So iterating would be something like :
dict_specialties = {"Urolog":"Urology",}
for key, val in dict_specialties.items():
ix = df[df.desc.str.contains(key)].index
df.loc[ix, 'desc_clean'] = val
Related
I have this one string, which is actually price, this price value comes with any currency symbol (currency_list), I am trying to remove these currency symbols from price and return only price.\
Till now I am able to do it for prefix and suffix currency symbol using below code , everything works till here.
I just want to add one validation where if the symbol is not prefix or suffix like "200$434" in btw, then it should return not valid format. which I am not able to understand how should be implemented.
currency_list = ['USD', 'UNITED STATES DOLLAR', '$', 'EUR', 'EURO', '€', 'GBP','BRITISH POUND', '£']
Normally input string can be
"$1212212"
"1212212EURO"
"1212212"
"1212212 BRITISH POUND"
need help to validate values like "1212$343" or "1212212EURO323.23"
Code:
for symb in currency_list:
if symb in amount:
data = amount.replace(symb, '')
After going through multiple blog post, I found this answer which gets the job done.
def validateCurrency(amount):
new_amount=None
for cur in currency_list:
if amount.startswith(cur) or amount.endswith(cur):
new_amount = amount.replace(cur, "", 1)
if new_amount == None:
return "Currency is not valid a string."
return f"Price after removeing symbol is {new_amount}"
// print(validateCurrency('$1212212'))
You can use regex to achieve your purpose.
import re
currency_list = ['USD', 'UNITED STATES DOLLAR', '$', 'EUR', 'EURO', '€', 'GBP', 'BRITISH POUND', '£']
p = re.compile(r'([\D]*)([\d]+\.?[\d]+)(.*)')
def verify_or_get_amount(amount):
first, mid, last = [i.strip() for i in p.search(amount).groups()]
if (first and first not in currency_list) or (last and last not in currency_list):
print('invalid:', amount)
else:
amount = mid
print('amount:', amount)
return mid
for i in ['EURO123', 'EURO 123', 'EURO 123.', 'EURO .12', 'EURO 12.12', '$1212212', '1212212EURO', '1212212', '1212212 BRITISH POUND', '1212$343']:
verify_or_get_amount(i)
using regex:
import re
currency_list = ['USD', 'UNITED STATES DOLLAR', '\$', 'EUR', 'EURO', '€', 'GBP', 'BRITISH POUND', '£']
currencies = '|'.join(currency_list)
c = re.compile(rf'^({currencies})? *(\d+(\.\d+)?) *({currencies})?$')
for i in ['$1212212', '1212212EURO', '1212212', '1212212 BRITISH POUND', '1212$343']:
match_obj = c.match(i)
if match_obj:
print(match_obj.group(2))
else:
print('not found')
output :
1212212
1212212
1212212
1212212
not found
Explanation :
to see actual pattern : print(c.pattern) which gives :
^(USD|UNITED STATES DOLLAR|\$|EUR|EURO|€|GBP|BRITISH POUND|£)?(\d+(\.\d+)?) *(USD|UNITED STATES DOLLAR|\$|EUR|EURO|€|GBP|BRITISH POUND|£)?$
I've escaped $ in the currency_list.
currencies = '|'.join(currency_list) for building possible prefixes or suffixes.
(\d+(\.\d+)?) is for matching price which accept float as well. (you can omit the (\.\d+) part)
the * that you see in regex, is for for example BRITISH POUND which have a space after the number.
I am assuming you want a currency validation function
def validateCurrency(input):
input_length = len(input)
if input.isdigit():return False
split = [re.findall(r'(\D+?)(\d+)|(\d+?)(\D+)', input)[0] ]
total_length = 0
for i in split[0]:
if i in currency_list:
total_length+=len(i)
if str(i).isdigit():
total_length+=len(i)
if total_length == input_length:
return True
else:
return False
Is there a smart way to shorten very long if-elif-elif-elif... statements?
Let's say I have a function like this:
def very_long_func():
something = 'Audi'
car = ['VW', 'Audi', 'BMW']
drinks = ['Cola', 'Fanta', 'Pepsi']
countries = ['France', 'Germany', 'Italy']
if something in car:
return {'type':'car brand'}
elif something in drinks:
return {'type':'lemonade brand'}
elif something in countries:
return {'type':'country'}
else:
return {'type':'nothing found'}
very_long_func()
>>>> {'type': 'car brand'}
The actual function is much longer than the example. What would be the best way to write this function (not in terms of speed but in readability)
I was reading this, but I have trouble to apply it to my problem.
You can't hash lists as dictionary values. So go other way round. Create a mapping of type -> list. And initialize your output with the default type. This allows you to keep on adding new types to your mapping without changing any code.
def very_long_func():
something = 'Audi'
car = ['VW', 'Audi', 'BMW']
drinks = ['Cola', 'Fanta', 'Pepsi']
countries = ['France', 'Germany', 'Italy']
out = {'type': 'nothing found'} # If nothing matches
mapping = {
'car brand': car,
'lemonade brand': drinks,
'country': countries
}
for k,v in mapping.items() :
if something in v:
out['type'] = k # update if match found
break
return out # returns matched or default value
you can create dictionary like this and then use map_dict.
from functools import reduce
car = ['VW', 'Audi', 'BMW']
drinks = ['Cola', 'Fanta', 'Pepsi']
countries = ['France', 'Germany', 'Italy']
li = [car, drinks, countries]
types = ['car brand', 'lemonade brand', 'country', 'nothing found']
dl = [dict(zip(l, [types[idx]]*len(l))) for idx, l in enumerate(li)]
map_dict = reduce(lambda a, b: dict(a, **b), dl)
Try this:
def create_dct(lst, flag):
return {k:flag for k in lst}
car = ['VW', 'Audi', 'BMW']
drinks = ['Cola', 'Fanta', 'Pepsi']
countries = ['France', 'Germany', 'Italy']
merge_dcts = {}
merge_dcts.update(create_dct(car, 'car brand'))
merge_dcts.update(create_dct(drinks, 'lemonade brand'))
merge_dcts.update(create_dct(countries, 'country'))
something = 'Audi'
try:
print("type: ", merge_dcts[something])
except:
print("type: nothing found")
You can simulate a switch statement with a helper function like this:
def switch(v): yield lambda *c: v in c
The your code could be written like this:
something = 'Audi'
for case in switch(something):
if case('VW', 'Audi', 'BMW'): name = 'car brand' ; break
if case('Cola', 'Fanta', 'Pepsi'): name = 'lemonade brand' ; break
if case('France', 'Germany', 'Italy'): name = 'country' ; break
else: name = 'nothing found'
return {'type':name}
If you don't have specific code to do for each value, then a simple mapping dictionary would probably suffice. For ease of maintenance, you can start with a category-list:type-name mapping and expand it before use:
mapping = { ('VW', 'Audi', 'BMW'):'car brand',
('Cola', 'Fanta', 'Pepsi'):'lemonade brand',
('France', 'Germany', 'Italy'):'country' }
mapping = { categ:name for categs,name in mapping.items() for categ in categs }
Then your code will look like this:
something = 'Audi'
return {'type':mapping.get(something,'nothing found')}
using a defaultdict would make this even simpler to use by providing the 'nothing found' value automatically so you could write: return {'type':mapping[something]}
defined few functions (not induced body of the functions to simplify question here)
def policyname(i):
retrun policyname
def policytype(i):
retrun policytype
def active(i):
retrun active
def backupselection(i):
retrun backupselection
defined a list -
clients = ['winwebint16', 'winwebtpie03', 'winwebtpie04', 'winwtsdt08', 'winwtsmwg03', 'winwtsqnr03', 'winwtswrl37', 'winwtswrl60', 'winwtswrl62', 'winwtswrl63', 'winwtswrl75', 'winwtszsim03', 'winwww0016','winsbk0100', 'winsbk0100a0', 'winsbk0100a1', 'winsbk0101', 'iinf065', 'iinf130', 'iinf185', 'iinf2126', 'inbf005', 'inis001', 'ipdataisbic01', 'ipdataisbic02', 'ipdataispre01', 'ipdataispre02', 'iproip02', 'isis002', 'isyn002', 'isyn006', 'isyn011', 'isyn012','isyn014', 'isyn038', 'isyn039', 'isyn040', 'mu2ssql1001', 'mu2ssql1003', 'macrsz0001', 'macrsz0005']
defined a class -
class client():
def __init__(self,policyname,policytype,active,backupselection):
self.policyname = policyname
self.policytype = policytype
self.active = active
self.backupselection = backupselection
For each item in clients list create class objects by passing parameters from functions.
Is below code is correct ?
for i in clients:
i = client(policyname(i),policytype(i),active(i),backupselection(i))
with above code, will i be able to access specific class objects like ?
print(winwebint16.policyname)
print(winwebint16.policytype)
print(winwebint16.active)
print(winwebint16.backupselection)
Classnames should be capital letter but that's just convention. Further you have a problem with the for loop
for i in clients:
i = client(policyname(i),policytype(i),active(i),backupselection(i))
The created client is not visible outside of the for loop scope, so you might wanna add them to some kind of list or dict
client_list: dict = {}
for i in clients:
client_list[i] = client(policyname(i),policytype(i),active(i),backupselection(i))
you should then be able to print like
print(client_list['winwebint16'].policyname)
This really worked for me :
clients = ['669165933', '963881480', '341417157', '514321792', '115456712', '547995746', '135425221', '871543967', '770463311', '616607081', '814711606', '939825713']
policynames = ['Tuvalu', 'Grenada', 'Russia', 'Sao Tome and Principe', 'Rwanda', 'Solomon Islands', 'Angola', 'Burkina Faso', 'Republic of the Congo', 'Senegal', 'Kyrgyzstan', 'Cape Verde']
policytypes = ['Offline', 'Online', 'Offline', 'Online', 'Offline', 'Online', 'Offline', 'Online', 'Offline', 'Online', 'Online', 'Offline']
actives = ['Baby Food', 'Cereal', 'Office Supplies', 'Fruits', 'Office Supplies', 'Baby Food', 'Household', 'Vegetables', 'Personal Care', 'Cereal', 'Vegetables', 'Clothes']
backupselections = ['H', 'C', 'L', 'C', 'L', 'C', 'M', 'H', 'M', 'H', 'H', 'H']
def policyname(i):
return policynames[i]
def policytype(i):
return policytypes[i]
def active(i):
return actives[i]
def backupselection(i):
return backupselections[i]
class client():
def __init__(self,policyname,policytype,active,backupselection):
self.policyname = policyname
self.policytype = policytype
self.active = active
self.backupselection = backupselection
for i in range(0,len(clients)):
i = client(policyname(i),policytype(i),active(i),backupselection(i))
print(i.policyname,i.policytype,i.active,i.backupselection)
Tuvalu Offline Baby Food H
Grenada Online Cereal C
Russia Offline Office Supplies L
Sao Tome and Principe Online Fruits C
Rwanda Offline Office Supplies L
Solomon Islands Online Baby Food C
Angola Offline Household M
Burkina Faso Online Vegetables H
Republic of the Congo Offline Personal Care M
Senegal Online Cereal H
Kyrgyzstan Online Vegetables H
Cape Verde Offline Clothes H
But still I can't print
print(669165933.policyname)
File "<ipython-input-20-72da397c032b>", line 1
print(669165933.policyname)
^
SyntaxError: invalid syntax
My code looks like:
import pandas as pd
df = pd.read_excel("Energy Indicators.xls", header=None, footer=None)
c_df = df.copy()
c_df = c_df.iloc[18:245, 2:]
c_df = c_df.rename(columns={2: 'Country', 3: 'Energy Supply', 4:'Energy Supply per Capita', 5:'% Renewable'})
c_df['Energy Supply'] = c_df['Energy Supply'].apply(lambda x: x*1000000)
c_df.loc[c_df['Country'] == 'Korea, Rep.'] = 'South Korea'
c_df.loc[c_df['Country'] == 'United States of America20'] = 'United States'
c_df.loc[c_df['Country'] == 'United Kingdom of Great Britain and Northern Ireland'] = 'United Kingdom'
c_df.loc[c_df['Country'] == 'China, Hong Kong Special Administrative Region'] = 'Hong Kong'
c_df.loc[c_df['Country'] == 'Venezuela (Bolivarian Republic of)'] = 'Venezuela'
c_df.loc[c_df['Country'] == 'Bolivia (Plurinational State of)'] = 'Bolivia'
c_df.loc[c_df['Country'] == 'Switzerland17'] = 'Switzerland'
c_df.loc[c_df['Country'] == 'Australia1'] = 'Australia'
c_df.loc[c_df['Country'] == 'China2'] = 'China'
c_df.loc[c_df['Country'] == 'Falkland Islands (Malvinas)'] = 'Bolivia'
c_df.loc[c_df['Country'] == 'Greenland7'] = 'Greenland'
c_df.loc[c_df['Country'] == 'Iran (Islamic Republic of'] = 'Iran'
c_df.loc[c_df['Country'] == 'Italy9'] = 'Italy'
c_df.loc[c_df['Country'] == 'Japan10'] = 'Japan'
c_df.loc[c_df['Country'] == 'Kuwait11'] = 'Kuwait'
c_df.loc[c_df['Country'] == 'Micronesia (Federal States of)'] = 'Micronesia'
c_df.loc[c_df['Country'] == 'Netherlands12'] = 'Netherlands'
c_df.loc[c_df['Country'] == 'Portugal13'] = 'Portugal'
c_df.loc[c_df['Country'] == 'Saudi Arabia14'] = 'Saudi Arabia'
c_df.loc[c_df['Country'] == 'Serbia15'] = 'Serbia'
c_df.loc[c_df['Country'] == 'Sint Maarteen (Dutch part)'] = 'Sint Marteen'
c_df.loc[c_df['Country'] == 'Spain16'] = 'Spain'
c_df.loc[c_df['Country'] == 'Ukraine18'] = 'Ukraine'
c_df.loc[c_df['Country'] == 'Denmark5'] = 'Denmark'
c_df.loc[c_df['Country'] == 'France6'] = 'France'
c_df.loc[c_df['Country'] == 'Indonesia8'] = 'Indonesia'
I feel like there must be an easier way to change the values of the countries with parentheses and numbers in their names. What pandas method can I use to look within the column for names with numbers of parentheses? isin?
You can start by getting rid of numbers and text in parentheses. After that, for everything else that requires non-trivial replacement, declare a map and apply it using pd.Series.replace.
mapper = {'Korea, Rep' : 'South Korea', 'Falkland Islands' : 'Bolivia', ...}
df['Country'] = (
df['Country'].str.replace(r'\d+|\s*\(.*\)', '').str.strip().replace(mapper)
)
Simple enough, done.
Details
\d+ # one or more digits
| # regex OR pipe
\s* # zero or more whitespace characters
\( # literal parentheses (opening brace)
.* # match anything
\) # closing brace
Using a dictionary and then df.replace:
dict_to_replace = {'Korea, Rep.':'South Korea',
'United States of America20':'United States',
'United Kingdom of Great Britain and Northern Ireland': 'United Kingdom'
...}
df['c_df'] = df['c_df'].replace(dict_to_replace)
I am experiencing a strange faulty behaviour, where a dictionary is only appended once and I can not add more key value pairs to it.
My code reads in a multi-line string and extracts substrings via split(), to be added to a dictionary. I make use of conditional statements. Strangely only the key:value pairs under the first conditional statement are added.
Therefore I can not complete the dictionary.
How can I solve this issue?
Minimal code:
#I hope the '\n' is sufficient or use '\r\n'
example = "Name: Bugs Bunny\nDOB: 01/04/1900\nAddress: 111 Jokes Drive, Hollywood Hills, CA 11111, United States"
def format(data):
dic = {}
for line in data.splitlines():
#print('Line:', line)
if ':' in line:
info = line.split(': ', 1)[1].rstrip() #does not work with files
#print('Info: ', info)
if ' Name:' in info: #middle name problems! /maiden name
dic['F_NAME'] = info.split(' ', 1)[0].rstrip()
dic['L_NAME'] = info.split(' ', 1)[1].rstrip()
elif 'DOB' in info: #overhang
dic['DD'] = info.split('/', 2)[0].rstrip()
dic['MM'] = info.split('/', 2)[1].rstrip()
dic['YY'] = info.split('/', 2)[2].rstrip()
elif 'Address' in info:
dic['STREET'] = info.split(', ', 2)[0].rstrip()
dic['CITY'] = info.split(', ', 2)[1].rstrip()
dic['ZIP'] = info.split(', ', 2)[2].rstrip()
return dic
if __name__ == '__main__':
x = format(example)
for v, k in x.iteritems():
print v, k
Your code doesn't work, at all. You split off the name before the colon and discard it, looking only at the value after the colon, stored in info. That value never contains the names you are looking for; Name, DOB and Address all are part of the line before the :.
Python lets you assign to multiple names at once; make use of this when splitting:
def format(data):
dic = {}
for line in data.splitlines():
if ':' not in line:
continue
name, _, value = line.partition(':')
name = name.strip()
if name == 'Name':
dic['F_NAME'], dic['L_NAME'] = value.split(None, 1) # strips whitespace for us
elif name == 'DOB':
dic['DD'], dic['MM'], dic['YY'] = (v.strip() for v in value.split('/', 2))
elif name == 'Address':
dic['STREET'], dic['CITY'], dic['ZIP'] = (v.strip() for v in value.split(', ', 2))
return dic
I used str.partition() here rather than limit str.split() to just one split; it is slightly faster that way.
For your sample input this produces:
>>> format(example)
{'CITY': 'Hollywood Hills', 'ZIP': 'CA 11111, United States', 'L_NAME': 'Bunny', 'F_NAME': 'Bugs', 'YY': '1900', 'MM': '04', 'STREET': '111 Jokes Drive', 'DD': '01'}
>>> from pprint import pprint
>>> pprint(format(example))
{'CITY': 'Hollywood Hills',
'DD': '01',
'F_NAME': 'Bugs',
'L_NAME': 'Bunny',
'MM': '04',
'STREET': '111 Jokes Drive',
'YY': '1900',
'ZIP': 'CA 11111, United States'}