Related
I have the following 122 countries which I couldn't look up their corresponding alpha 3 code. I tried search_fuzzy but nothing is found.
By looking at some of the countries names, I can "manually" assign the alpha 3 based on common knowledge (such like creating a dic for rename). However, I wonder if there is a better way to look up the alpha 3 in an automated way, such like by using other function from pycoutry or even re?
Any suggestions and advice are greatly appreciated.
import pandas as pd
import numpy as np
import regex as re
import pycountry
missing = ['Americas', 'Asia', 'Australia and New Zealand', 'Bolivia (Plurinational State of)', 'Caribbean', 'Central America', 'Central and Southern Asia', 'Central Asia', 'China, Hong Kong Special Administrative Region', 'China, Macao Special Administrative Region', 'Democratic Republic of the Congo', 'Eastern Africa', 'Eastern and South-Eastern Asia', 'Eastern Asia', 'Eastern Europe', 'Europe', 'Europe and Northern America', 'Iran (Islamic Republic of)', 'Landlocked developing countries (LLDCs)', 'Latin America and the Caribbean', 'Least Developed Countries (LDCs)', 'Melanesia', 'Micronesia (Federated States of)', 'Middle Africa', 'Northern Africa', 'Northern Africa and Western Asia', 'Northern America', 'Northern Europe', 'Oceania', 'Oceania (exc. Australia and New Zealand)', 'Small island developing States (SIDS)', 'South America', 'South-Eastern Asia', 'Southern Africa', 'Southern Asia', 'Southern Europe', 'Sub-Saharan Africa', 'Türkiye', 'Venezuela (Bolivarian Republic of)', 'Western Africa', 'Western Asia', 'Western Europe', 'World', 'European Union (27)', 'Chinese Taipei', 'UAE', 'Belgium-Luxembourg', 'Channel Islands', 'China, Hong Kong SAR', 'China, Macao SAR', 'China, mainland', 'China, Taiwan Province of', 'Czechoslovakia', 'Ethiopia PDR', 'French Guyana', 'Netherlands Antilles (former)', 'Pacific Islands Trust Territory', 'Serbia and Montenegro', 'Sudan (former)', 'Svalbard and Jan Mayen Islands', 'United States Virgin Islands', 'USSR', 'Wallis and Futuna Islands', 'Yugoslav SFR', 'Global average', 'Cocos Islands', 'Macquarie Island', 'Northern Mariana Islands and Guam', 'Comoro Islands', 'Glorioso Islands', 'Juan de Nova Island', 'Bassas da India', 'Ile Europa', 'Ile Tromelin', 'Azores', 'Cape Verde', 'Canary Islands', 'Prince Edward Islands', 'Crozet Islands', 'Amsterdam Island and Saint Paul Island', 'Kerguelen Islands', 'Heard and McDonald Islands', 'Republique du Congo', 'Clipperton Island', 'Puerto Rico and Virgin Islands of the United States', 'Guadeloupe and Martinique', 'Faeroe Islands', 'Line Islands (Kiribati)', 'Phoenix Islands (Kiribati)', 'Howland Island and Baker Island', 'Guinea Bissau', 'Ivory Coast', 'Gilbert Islands (Kiribati)', 'Northern Saint-Martin', 'East Timor', 'Oecussi Ambeno', 'Laos', 'Republic of Congo', 'Dem. Rep. Congo', 'ASEAN', 'BRIICS', 'DRC', 'EA19', 'EECCA', 'EU27_2020', 'European Union', 'G20', 'G7M', 'Lao PDR', 'OECD', 'OECDAM', 'OECDAO', 'OECDE', 'Grenade', 'Korea, Rep.', 'Egypt, Arab Rep.', 'Iran, Islamic Rep.', 'Korea (Rep.)', 'Hong Kong, China', 'Iran (Islamic Republic)', 'Cote dIvoire', 'Congo (Democratic Republic)']
not_found = []
for country in missing:
try:
print(pycountry.countries.search_fuzzy(country))
print(country)
except:
print('not found')
not_found.append(country)
print(len(missing)) #122
print(len(not_found)) #122
I am trying to get just some of the fields of a record because I do not want to delete the not wanted columns afterwards but can't figure out how to do it. My real JSON has a lot more fields in the "countries" path, this is just an example.
Example JSON
data = [{'state': 'Florida',
'shortname': 'FL',
'info': {
'governor': 'Rick Scott'
},
'counties': [{'name': 'Dade', 'population': 12345},
{'name': 'Broward', 'population': 40000},
{'name': 'Palm Beach', 'population': 60000}]},
{'state': 'Ohio',
'shortname': 'OH',
'info': {
'governor': 'John Kasich'
},
'counties': [{'name': 'Summit', 'population': 1234},
{'name': 'Cuyahoga', 'population': 1337}]}]
json_normalize
result = pd.json_normalize(
data=data,
record_path='counties',
meta=['state', 'shortname',
['info', 'governor']])
output
name population state shortname info.governor
0 Dade 12345 Florida FL Rick Scott
1 Broward 40000 Florida FL Rick Scott
2 Palm Beach 60000 Florida FL Rick Scott
3 Summit 1234 Ohio OH John Kasich
4 Cuyahoga 1337 Ohio OH John Kasich
but I do not want the "population" in this example, I just want the name of the counties
I tried all kind of combinations in the meta attribute.
I am trying to match capital cities that contain three consecutive consonants.
This is my code:
result = [i for i in capitals if re.match("\w*[^aeiouAEIOU\W]{3}\w*", i)]
print(*result)
result = [i for i in capitals if re.match(r"\b(?=[a-z]*[aeiou]{3})[a-z]+\b", i)]
print(*result)
This is the source:
capitals = ('Kabul', 'Tirana (Tirane)', 'Algiers', 'Andorra la Vella', 'Luanda', "Saint John's", 'Buenos Aires', 'Yerevan', 'Canberra', 'Vienna', 'Baku', 'Nassau', 'Manama', 'Dhaka', 'Bridgetown', 'Minsk', 'Brussels', 'Belmopan', 'Porto Novo', 'Thimphu', 'Sucre', 'Sarajevo', 'Gaborone', 'Brasilia', 'Bandar Seri Begawan', 'Sofia', 'Ouagadougou', 'Gitega', 'Phnom Penh', 'Yaounde', 'Ottawa', 'Praia', 'Bangui', "N'Djamena", 'Santiago', 'Beijing', 'Bogota', 'Moroni', 'Kinshasa', 'Brazzaville', 'San Jose', 'Yamoussoukro', 'Zagreb', 'Havana', 'Nicosia', 'Prague', 'Copenhagen', 'Djibouti', 'Roseau', 'Santo Domingo', 'Dili', 'Quito', 'Cairo', 'San Salvador', 'London', 'Malabo', 'Asmara', 'Tallinn', 'Mbabana', 'Addis Ababa', 'Palikir', 'Suva', 'Helsinki', 'Paris', 'Libreville', 'Banjul', 'Tbilisi', 'Berlin', 'Accra', 'Athens', "Saint George's", 'Guatemala City', 'Conakry', 'Bissau', 'Georgetown', 'Port au Prince', 'Tegucigalpa', 'Budapest', 'Reykjavik', 'New Delhi', 'Jakarta', 'Tehran', 'Baghdad', 'Dublin', 'Jerusalem', 'Rome', 'Kingston', 'Tokyo', 'Amman', 'Nur-Sultan', 'Nairobi', 'Tarawa Atoll', 'Pristina', 'Kuwait City', 'Bishkek', 'Vientiane', 'Riga', 'Beirut', 'Maseru', 'Monrovia', 'Tripoli', 'Vaduz', 'Vilnius', 'Luxembourg', 'Antananarivo', 'Lilongwe', 'Kuala Lumpur', 'Male', 'Bamako', 'Valletta', 'Majuro', 'Nouakchott', 'Port Louis', 'Mexico City', 'Chisinau', 'Monaco', 'Ulaanbaatar', 'Podgorica', 'Rabat', 'Maputo', 'Nay Pyi Taw', 'Windhoek', 'No official capital', 'Kathmandu', 'Amsterdam', 'Wellington', 'Managua', 'Niamey', 'Abuja', 'Pyongyang', 'Skopje', 'Belfast', 'Oslo', 'Muscat', 'Islamabad', 'Melekeok', 'Panama City', 'Port Moresby', 'Asuncion', 'Lima', 'Manila', 'Warsaw', 'Lisbon', 'Doha', 'Bucharest', 'Moscow', 'Kigali', 'Basseterre', 'Castries', 'Kingstown', 'Apia', 'San Marino', 'Sao Tome', 'Riyadh', 'Edinburgh', 'Dakar', 'Belgrade', 'Victoria', 'Freetown', 'Singapore', 'Bratislava', 'Ljubljana', 'Honiara', 'Mogadishu', 'Pretoria, Bloemfontein, Cape Town', 'Seoul', 'Juba', 'Madrid', 'Colombo', 'Khartoum', 'Paramaribo', 'Stockholm', 'Bern', 'Damascus', 'Taipei', 'Dushanbe', 'Dodoma', 'Bangkok', 'Lome', "Nuku'alofa", 'Port of Spain', 'Tunis', 'Ankara', 'Ashgabat', 'Funafuti', 'Kampala', 'Kiev', 'Abu Dhabi', 'London', 'Washington D.C.', 'Montevideo', 'Tashkent', 'Port Vila', 'Vatican City', 'Caracas', 'Hanoi', 'Cardiff', "Sana'a", 'Lusaka', 'Harare')
This is the output:
It is missing one city this one "Port Moresby"
Minsk Thimphu Phnom Penh Kinshasa Accra Conakry Reykjavik Baghdad Kingston Bishkek Lilongwe Nouakchott Windhoek Kathmandu Amsterdam Wellington Pyongyang Castries Kingstown Edinburgh Belgrade Ljubljana Stockholm Bangkok Ashgabat Washington D.C. Tashkent
This is my expected output:
Including "Port Moresby"
Minsk Thimphu Phnom Penh Kinshasa Accra Conakry Reykjavik Baghdad Kingston Bishkek Lilongwe Nouakchott Windhoek Kathmandu Amsterdam Wellington Pyongyang Port Moresby Castries Kingstown Edinburgh Belgrade Ljubljana Stockholm Bangkok Ashgabat Washington D.C. Tashkent
Try give this a whirl:
result = [i for i in capitals if re.search("[^aeiou\W]{3}", i.lower())]
print(*result)
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I'm a beginner in Python.
Now I have 2 columns on my Excel file. One is country column and the other one is city column.
For the country column, most of the values are shown in country code and some of them are shown in country full name, while some values are U.S.A states code and less than 1% of them are blank.
For the city column, it clearly shows the full city name (not city code), while nearly 20% of them are blank.
How can I use Python to create a new column to show the full country name based on the country code and remain the same name if it shows the full country name in the country column, and show the U.S.A states code as the United States in the new column?
The tricky part is, in the country column, take CO as the example, Co can stand for Columbia and Colorado, I cannot be sure whether it's a country or a state at the beginning, but when I check the corresponding city name I can know it's a country or a state (ex: Longmont for Colorado, Bogota for Columbia). How can I avoid this issue in the new column and infer the full country name in the new column based on the corresponding city name?
I appreciate your help!
Explanation
Coded the task using following logic.
Process simple abbreviations such as U.S.
A country length greater than 3
Have Country and City
Find closest Country City Pair in Cities
Country Only
Find closest country match in list of countries in two letter country codes
Country length equals 3
Find country with 3 letter country codes
Country length equals 2 (could be country or state code)
Code does not exist in list of states
Must be country code, so look up country in two letter country codes
Code does not exist in list of countries
Must be state code for USA, so country is United States
Could be country or state code
Check if city with this as a state code
Check if city with this as a country code
Must be best match of these two possibilities
Note: String matching uses fuzzy matching to allow for flexibility in spelling of names
rapidfuzz library was used over fuzzywuzzy since its an order of magnitude faster
Code
import pandas as pd
from rapidfuzz import fuzz
def find_closest_country(country):
' Country with the closest name in list of countries in country code '
ratios = [fuzz.partial_ratio(country, x) for x in alpha2.values()]
rated_countries = [(info, r) for info, r in zip(alpha2.values(), ratios)]
# Best match with shortest name
return sorted(rated_countries, key = lambda x: (x[1], -len(x[0])), reverse = True)[0]
def check_city_country(city, country):
' City, Country pair closest in list of cities '
ratios = [fuzz.partial_ratio(city, x['name']) * fuzz.partial_ratio(country, x['country']) for x in cities]
rated_cities = [(info, r) for info, r in zip(cities, ratios)]
# Best match with shortest name
return sorted(rated_cities, key = lambda x: (x[1], -len(x[0])), reverse = True)[0]
def check_city_subregion(city, subregion):
' City, subresion pair closest in list of cities '
ratios = [fuzz.partial_ratio(city, x['name']) * fuzz.partial_ratio(subregion, x['subcountry']) for x in cities]
rated_cities = [(info, r) for info, r in zip(cities, ratios)]
# Best match with shortest name
return sorted(rated_cities, key = lambda x: (x[1], -len(x[0])), reverse = True)[0]
def lookup(country, city):
'''
Finds country based upon country and city
country - country name or country code
city - name of city
'''
if country.lower() == 'u.s.':
# Picks up common US acronym
country = "US"
if len(country) > 3:
# Must be country since too long for abbreviation
if city:
# Find closest city country pair in list of cities
city_info = check_city_country(city, country)
if city_info:
return city_info[0]['country']
# No city, so find closest country in list of countries (2 code abbreviations reverse lookup)
countries = find_closest_country(country)
if countries:
return countries[0]
return None
elif len(country) == 3:
# 3 letter abbreviation
country = country.upper()
return alpha3.get(country, None)
elif len(country) == 2:
# Two letter country abbreviation
country = country.upper()
if not country in states:
# Not a state code, so lookup contry from code
return alpha2.get(country, None)
if not country in alpha2:
# Not a country code, so must be state code for US
return "United States of America"
# Could be country of state code
if city:
# Have 2 digit code (could be country or state)
pos_country = alpha2[country] # possible country
pos_state = states[country] # possible state
# check closest country with this city
pos_countries = check_city_country(city, pos_country)
# If state code, country would be United States
pos_us = check_city_country(city, "United States")
if pos_countries[1] > pos_us[1]:
# Provided better match as country code
return pos_countries[0]['country']
else:
# Provided better match as state code (i.e. "United States")
return pos_us[0]['country']
else:
return alpha2[country]
else:
return None
Data
# State Codes
# https://gist.github.com/rugbyprof/76575b470b6772ce8fa0c49e23931d97
states = {"AL":"Alabama","AK":"Alaska","AZ":"Arizona","AR":"Arkansas","CA":"California","CO":"Colorado","CT":"Connecticut","DE":"Delaware","FL":"Florida","GA":"Georgia","HI":"Hawaii","ID":"Idaho","IL":"Illinois","IN":"Indiana","IA":"Iowa","KS":"Kansas","KY":"Kentucky","LA":"Louisiana","ME":"Maine","MD":"Maryland","MA":"Massachusetts","MI":"Michigan","MN":"Minnesota","MS":"Mississippi","MO":"Missouri","MT":"Montana","NE":"Nebraska","NV":"Nevada","NH":"New Hampshire","NJ":"New Jersey","NM":"New Mexico","NY":"New York","NC":"North Carolina","ND":"North Dakota","OH":"Ohio","OK":"Oklahoma","OR":"Oregon","PA":"Pennsylvania","RI":"Rhode Island","SC":"South Carolina","SD":"South Dakota","TN":"Tennessee","TX":"Texas","UT":"Utah","VT":"Vermont","VA":"Virginia","WA":"Washington","WV":"West Virginia","WI":"Wisconsin","WY":"Wyoming"}
# two letter country codes
# https://gist.github.com/carlopires/1261951/d13ca7320a6abcd4b0aa800d351a31b54cefdff4
alpha2 = {
'AD': 'Andorra',
'AE': 'United Arab Emirates',
'AF': 'Afghanistan',
'AG': 'Antigua & Barbuda',
'AI': 'Anguilla',
'AL': 'Albania',
'AM': 'Armenia',
'AN': 'Netherlands Antilles',
'AO': 'Angola',
'AQ': 'Antarctica',
'AR': 'Argentina',
'AS': 'American Samoa',
'AT': 'Austria',
'AU': 'Australia',
'AW': 'Aruba',
'AZ': 'Azerbaijan',
'BA': 'Bosnia and Herzegovina',
'BB': 'Barbados',
'BD': 'Bangladesh',
'BE': 'Belgium',
'BF': 'Burkina Faso',
'BG': 'Bulgaria',
'BH': 'Bahrain',
'BI': 'Burundi',
'BJ': 'Benin',
'BM': 'Bermuda',
'BN': 'Brunei Darussalam',
'BO': 'Bolivia',
'BR': 'Brazil',
'BS': 'Bahama',
'BT': 'Bhutan',
'BU': 'Burma (no longer exists)',
'BV': 'Bouvet Island',
'BW': 'Botswana',
'BY': 'Belarus',
'BZ': 'Belize',
'CA': 'Canada',
'CC': 'Cocos (Keeling) Islands',
'CF': 'Central African Republic',
'CG': 'Congo',
'CH': 'Switzerland',
'CI': 'Côte D\'ivoire (Ivory Coast)',
'CK': 'Cook Iislands',
'CL': 'Chile',
'CM': 'Cameroon',
'CN': 'China',
'CO': 'Colombia',
'CR': 'Costa Rica',
'CS': 'Czechoslovakia (no longer exists)',
'CU': 'Cuba',
'CV': 'Cape Verde',
'CX': 'Christmas Island',
'CY': 'Cyprus',
'CZ': 'Czech Republic',
'DD': 'German Democratic Republic (no longer exists)',
'DE': 'Germany',
'DJ': 'Djibouti',
'DK': 'Denmark',
'DM': 'Dominica',
'DO': 'Dominican Republic',
'DZ': 'Algeria',
'EC': 'Ecuador',
'EE': 'Estonia',
'EG': 'Egypt',
'EH': 'Western Sahara',
'ER': 'Eritrea',
'ES': 'Spain',
'ET': 'Ethiopia',
'FI': 'Finland',
'FJ': 'Fiji',
'FK': 'Falkland Islands (Malvinas)',
'FM': 'Micronesia',
'FO': 'Faroe Islands',
'FR': 'France',
'FX': 'France, Metropolitan',
'GA': 'Gabon',
'GB': 'United Kingdom (Great Britain)',
'GD': 'Grenada',
'GE': 'Georgia',
'GF': 'French Guiana',
'GH': 'Ghana',
'GI': 'Gibraltar',
'GL': 'Greenland',
'GM': 'Gambia',
'GN': 'Guinea',
'GP': 'Guadeloupe',
'GQ': 'Equatorial Guinea',
'GR': 'Greece',
'GS': 'South Georgia and the South Sandwich Islands',
'GT': 'Guatemala',
'GU': 'Guam',
'GW': 'Guinea-Bissau',
'GY': 'Guyana',
'HK': 'Hong Kong',
'HM': 'Heard & McDonald Islands',
'HN': 'Honduras',
'HR': 'Croatia',
'HT': 'Haiti',
'HU': 'Hungary',
'ID': 'Indonesia',
'IE': 'Ireland',
'IL': 'Israel',
'IN': 'India',
'IO': 'British Indian Ocean Territory',
'IQ': 'Iraq',
'IR': 'Islamic Republic of Iran',
'IS': 'Iceland',
'IT': 'Italy',
'JM': 'Jamaica',
'JO': 'Jordan',
'JP': 'Japan',
'KE': 'Kenya',
'KG': 'Kyrgyzstan',
'KH': 'Cambodia',
'KI': 'Kiribati',
'KM': 'Comoros',
'KN': 'St. Kitts and Nevis',
'KP': 'Korea, Democratic People\'s Republic of',
'KR': 'Korea, Republic of',
'KW': 'Kuwait',
'KY': 'Cayman Islands',
'KZ': 'Kazakhstan',
'LA': 'Lao People\'s Democratic Republic',
'LB': 'Lebanon',
'LC': 'Saint Lucia',
'LI': 'Liechtenstein',
'LK': 'Sri Lanka',
'LR': 'Liberia',
'LS': 'Lesotho',
'LT': 'Lithuania',
'LU': 'Luxembourg',
'LV': 'Latvia',
'LY': 'Libyan Arab Jamahiriya',
'MA': 'Morocco',
'MC': 'Monaco',
'MD': 'Moldova, Republic of',
'MG': 'Madagascar',
'MH': 'Marshall Islands',
'ML': 'Mali',
'MN': 'Mongolia',
'MM': 'Myanmar',
'MO': 'Macau',
'MP': 'Northern Mariana Islands',
'MQ': 'Martinique',
'MR': 'Mauritania',
'MS': 'Monserrat',
'MT': 'Malta',
'MU': 'Mauritius',
'MV': 'Maldives',
'MW': 'Malawi',
'MX': 'Mexico',
'MY': 'Malaysia',
'MZ': 'Mozambique',
'NA': 'Namibia',
'NC': 'New Caledonia',
'NE': 'Niger',
'NF': 'Norfolk Island',
'NG': 'Nigeria',
'NI': 'Nicaragua',
'NL': 'Netherlands',
'NO': 'Norway',
'NP': 'Nepal',
'NR': 'Nauru',
'NT': 'Neutral Zone (no longer exists)',
'NU': 'Niue',
'NZ': 'New Zealand',
'OM': 'Oman',
'PA': 'Panama',
'PE': 'Peru',
'PF': 'French Polynesia',
'PG': 'Papua New Guinea',
'PH': 'Philippines',
'PK': 'Pakistan',
'PL': 'Poland',
'PM': 'St. Pierre & Miquelon',
'PN': 'Pitcairn',
'PR': 'Puerto Rico',
'PT': 'Portugal',
'PW': 'Palau',
'PY': 'Paraguay',
'QA': 'Qatar',
'RE': 'Réunion',
'RO': 'Romania',
'RU': 'Russian Federation',
'RW': 'Rwanda',
'SA': 'Saudi Arabia',
'SB': 'Solomon Islands',
'SC': 'Seychelles',
'SD': 'Sudan',
'SE': 'Sweden',
'SG': 'Singapore',
'SH': 'St. Helena',
'SI': 'Slovenia',
'SJ': 'Svalbard & Jan Mayen Islands',
'SK': 'Slovakia',
'SL': 'Sierra Leone',
'SM': 'San Marino',
'SN': 'Senegal',
'SO': 'Somalia',
'SR': 'Suriname',
'ST': 'Sao Tome & Principe',
'SU': 'Union of Soviet Socialist Republics (no longer exists)',
'SV': 'El Salvador',
'SY': 'Syrian Arab Republic',
'SZ': 'Swaziland',
'TC': 'Turks & Caicos Islands',
'TD': 'Chad',
'TF': 'French Southern Territories',
'TG': 'Togo',
'TH': 'Thailand',
'TJ': 'Tajikistan',
'TK': 'Tokelau',
'TM': 'Turkmenistan',
'TN': 'Tunisia',
'TO': 'Tonga',
'TP': 'East Timor',
'TR': 'Turkey',
'TT': 'Trinidad & Tobago',
'TV': 'Tuvalu',
'TW': 'Taiwan, Province of China',
'TZ': 'Tanzania, United Republic of',
'UA': 'Ukraine',
'UG': 'Uganda',
'UM': 'United States Minor Outlying Islands',
'US': 'United States of America',
'UY': 'Uruguay',
'UZ': 'Uzbekistan',
'VA': 'Vatican City State (Holy See)',
'VC': 'St. Vincent & the Grenadines',
'VE': 'Venezuela',
'VG': 'British Virgin Islands',
'VI': 'United States Virgin Islands',
'VN': 'Viet Nam',
'VU': 'Vanuatu',
'WF': 'Wallis & Futuna Islands',
'WS': 'Samoa',
'YD': 'Democratic Yemen (no longer exists)',
'YE': 'Yemen',
'YT': 'Mayotte',
'YU': 'Yugoslavia',
'ZA': 'South Africa',
'ZM': 'Zambia',
'ZR': 'Zaire',
'ZW': 'Zimbabwe',
'ZZ': 'Unknown or unspecified country',
}
# Three letter codes
#https://en.wikipedia.org/wiki/ISO_3166-1_alpha-3#Uses_and_applications
alpha3 = """ABW Aruba
AFG Afghanistan
AGO Angola
AIA Anguilla
ALA Åland Islands
ALB Albania
AND Andorra
ARE United Arab Emirates
ARG Argentina
ARM Armenia
ASM American Samoa
ATA Antarctica
ATF French Southern Territories
ATG Antigua and Barbuda
AUS Australia
AUT Austria
AZE Azerbaijan
BDI Burundi
BEL Belgium
BEN Benin
BES Bonaire, Sint Eustatius and Saba
BFA Burkina Faso
BGD Bangladesh
BGR Bulgaria
BHR Bahrain
BHS Bahamas
BIH Bosnia and Herzegovina
BLM Saint Barthélemy
BLR Belarus
BLZ Belize
BMU Bermuda
BOL Bolivia (Plurinational State of)
BRA Brazil
BRB Barbados
BRN Brunei Darussalam
BTN Bhutan
BVT Bouvet Island
BWA Botswana
CAF Central African Republic
CAN Canada
CCK Cocos (Keeling) Islands
CHE Switzerland
CHL Chile
CHN China
CIV Côte d'Ivoire
CMR Cameroon
COD Congo, Democratic Republic of the
COG Congo
COK Cook Islands
COL Colombia
COM Comoros
CPV Cabo Verde
CRI Costa Rica
CUB Cuba
CUW Curaçao
CXR Christmas Island
CYM Cayman Islands
CYP Cyprus
CZE Czechia
DEU Germany
DJI Djibouti
DMA Dominica
DNK Denmark
DOM Dominican Republic
DZA Algeria
ECU Ecuador
EGY Egypt
ERI Eritrea
ESH Western Sahara
ESP Spain
EST Estonia
ETH Ethiopia
FIN Finland
FJI Fiji
FLK Falkland Islands (Malvinas)
FRA France
FRO Faroe Islands
FSM Micronesia (Federated States of)
GAB Gabon
GBR United Kingdom of Great Britain and Northern Ireland
GEO Georgia
GGY Guernsey
GHA Ghana
GIB Gibraltar
GIN Guinea
GLP Guadeloupe
GMB Gambia
GNB Guinea-Bissau
GNQ Equatorial Guinea
GRC Greece
GRD Grenada
GRL Greenland
GTM Guatemala
GUF French Guiana
GUM Guam
GUY Guyana
HKG Hong Kong
HMD Heard Island and McDonald Islands
HND Honduras
HRV Croatia
HTI Haiti
HUN Hungary
IDN Indonesia
IMN Isle of Man
IND India
IOT British Indian Ocean Territory
IRL Ireland
IRN Iran (Islamic Republic of)
IRQ Iraq
ISL Iceland
ISR Israel
ITA Italy
JAM Jamaica
JEY Jersey
JOR Jordan
JPN Japan
KAZ Kazakhstan
KEN Kenya
KGZ Kyrgyzstan
KHM Cambodia
KIR Kiribati
KNA Saint Kitts and Nevis
KOR Korea, Republic of
KWT Kuwait
LAO Lao People's Democratic Republic
LBN Lebanon
LBR Liberia
LBY Libya
LCA Saint Lucia
LIE Liechtenstein
LKA Sri Lanka
LSO Lesotho
LTU Lithuania
LUX Luxembourg
LVA Latvia
MAC Macao
MAF Saint Martin (French part)
MAR Morocco
MCO Monaco
MDA Moldova, Republic of
MDG Madagascar
MDV Maldives
MEX Mexico
MHL Marshall Islands
MKD North Macedonia
MLI Mali
MLT Malta
MMR Myanmar
MNE Montenegro
MNG Mongolia
MNP Northern Mariana Islands
MOZ Mozambique
MRT Mauritania
MSR Montserrat
MTQ Martinique
MUS Mauritius
MWI Malawi
MYS Malaysia
MYT Mayotte
NAM Namibia
NCL New Caledonia
NER Niger
NFK Norfolk Island
NGA Nigeria
NIC Nicaragua
NIU Niue
NLD Netherlands
NOR Norway
NPL Nepal
NRU Nauru
NZL New Zealand
OMN Oman
PAK Pakistan
PAN Panama
PCN Pitcairn
PER Peru
PHL Philippines
PLW Palau
PNG Papua New Guinea
POL Poland
PRI Puerto Rico
PRK Korea (Democratic People's Republic of)
PRT Portugal
PRY Paraguay
PSE Palestine, State of
PYF French Polynesia
QAT Qatar
REU Réunion
ROU Romania
RUS Russian Federation
RWA Rwanda
SAU Saudi Arabia
SDN Sudan
SEN Senegal
SGP Singapore
SGS South Georgia and the South Sandwich Islands
SHN Saint Helena, Ascension and Tristan da Cunha
SJM Svalbard and Jan Mayen
SLB Solomon Islands
SLE Sierra Leone
SLV El Salvador
SMR San Marino
SOM Somalia
SPM Saint Pierre and Miquelon
SRB Serbia
SSD South Sudan
STP Sao Tome and Principe
SUR Suriname
SVK Slovakia
SVN Slovenia
SWE Sweden
SWZ Eswatini
SXM Sint Maarten (Dutch part)
SYC Seychelles
SYR Syrian Arab Republic
TCA Turks and Caicos Islands
TCD Chad
TGO Togo
THA Thailand
TJK Tajikistan
TKL Tokelau
TKM Turkmenistan
TLS Timor-Leste
TON Tonga
TTO Trinidad and Tobago
TUN Tunisia
TUR Turkey
TUV Tuvalu
TWN Taiwan, Province of China
TZA Tanzania, United Republic of
UGA Uganda
UKR Ukraine
UMI United States Minor Outlying Islands
URY Uruguay
USA United States of America
UZB Uzbekistan
VAT Holy See
VCT Saint Vincent and the Grenadines
VEN Venezuela (Bolivarian Republic of)
VGB Virgin Islands (British)
VIR Virgin Islands (U.S.)
VNM Viet Nam
VUT Vanuatu
WLF Wallis and Futuna
WSM Samoa
YEM Yemen
ZAF South Africa
ZMB Zambia
ZWE Zimbabwe"""
# Convert to dictionary
alpha3 = dict(tuple(re.split(r" {2,}", s)) for s in alpha3.split('\n'))
# List of World Cities & Country
# cities https://pkgstore.datahub.io/core/world-cities/world-cities_csv/data/6cc66692f0e82b18216a48443b6b95da/world-cities_csv.csv
# Online CSV File
import csv
import urllib.request
import io
def csv_import(url):
url_open = urllib.request.urlopen(url)
csvfile = csv.DictReader(io.StringIO(url_open.read().decode('utf-8')), delimiter=',')
return csvfile
url = 'https://pkgstore.datahub.io/core/world-cities/world-cities_csv/data/6cc66692f0e82b18216a48443b6b95da/world-cities_csv.csv'
cities = csv_import(url)
Test
Excel File (Input)
country city
u.s.
DZ
AS
co Longmont
co Bogota
AL
AL Huntsville
usa
AFG
BLR Minsk
AUS
united states
Korea seoul
Korea Pyongyang
Test Code
df = pd.read_excel('country_test.xlsx') # Load Excel File
df.fillna('', inplace=True)
# Get name of country based upon country and city
df['country_'] = df.apply(lambda row: lookup(row['country'], row['city']), axis = 1)
Resulting Dataframe
country city country_
0 u.s. United States of America
1 DZ Algeria
2 AS American Samoa
3 co Longmont United States
4 co Bogota Colombia
5 AL Albania
6 AL Huntsville United States
7 usa United States of America
8 AFG Afghanistan
9 BLR Minsk Belarus
10 AUS Australia
11 united states United States of America
12 Korea seoul South Korea
13 Korea Pyongyang North Korea
Well, You can have a {key (state) : Values (cities belonging to states)} json and use python to read the file and arrange the list to the corresponding city, state.
An advice for this approach is to create dictionaries(i.e. dic = {'CO':'Colombia',...} and dic_state = {'CO':'Colorado',...}). Then, probably have an if statement to check if the country is USA. If USA, then use dic_state. Finally, you can create a new column by using the appropriate command (this depends on the package/module that you are using)
Good luck!
I have a data set which is in data format and looks like this:
[{'session_id': ['X061RFWB06K9V'],
'unix_timestamp': [1442503708],
'cities': ['New York NY, Newark NJ'],
'user': [[{'user_id': 2024,
'joining_date': '2015-03-22',
'country': 'UK'}]]},
{'session_id': ['5AZ2X2A9BHH5U'],
'unix_timestamp': [1441353991],
'cities': ['New York NY, Jersey City NJ, Philadelphia PA'],
'user': [[{'user_id': 2853,
'joining_date': '2015-03-28',
'country': 'DE'}]]},
{'session_id': ['SHTB4IYAX4PX6'],
'unix_timestamp': [1440843490],
'cities': ['San Antonio TX'],
'user': [[{'user_id': 10958,
'joining_date': '2015-03-06',
'country': 'UK'}]]}
I am using pandas and processing it and when i use read_json, I get the following:
cities session_id unix_timestamp user
0 [New York NY, Newark NJ] [X061RFWB06K9V] [1442503708] [[{'user_id': 2024, 'joining_date': '2015-03-2...
1 [New York NY, Jersey City NJ, Philadelphia PA] [5AZ2X2A9BHH5U] [1441353991] [[{'user_id': 2853, 'joining_date': '2015-03-2...
2 [San Antonio TX] [SHTB4IYAX4PX6] [1440843490] [[{'user_id': 10958, 'joining_date': '2015-03-...
How do I process this data so that its in a better format?
Here is the data definition:
Columns:
session_id: session id.
unix_timestamp: unix timestamp of session start time
cities: the unique cities which were searched within the same session
user:
user_id: the id of the user
joining_date: when the user created the account
country: where the user is based
I tried using json_normalize but keep getting error:
AttributeError: 'int' object has no attribute 'values'
and also different types of error. Kindly help
You could use a function that completely flattens it out, then reconstruct your dataframe:
import re
import pandas as pd
import numpy as np
jsonData = [{'session_id': ['X061RFWB06K9V'],
'unix_timestamp': [1442503708],
'cities': ['New York NY, Newark NJ'],
'user': [[{'user_id': 2024,
'joining_date': '2015-03-22',
'country': 'UK'}]]},
{'session_id': ['5AZ2X2A9BHH5U'],
'unix_timestamp': [1441353991],
'cities': ['New York NY, Jersey City NJ, Philadelphia PA'],
'user': [[{'user_id': 2853,
'joining_date': '2015-03-28',
'country': 'DE'}]]},
{'session_id': ['SHTB4IYAX4PX6'],
'unix_timestamp': [1440843490],
'cities': ['San Antonio TX'],
'user': [[{'user_id': 10958,
'joining_date': '2015-03-06',
'country': 'UK'}]]} ]
def flatten_json(y):
out = {}
def flatten(x, name=''):
if type(x) is dict:
for a in x:
flatten(x[a], name + a + '_')
elif type(x) is list:
i = 0
for a in x:
flatten(a, name + str(i) + '_')
i += 1
else:
out[name[:-1]] = x
flatten(y)
return out
flat = flatten_json(jsonData)
results = pd.DataFrame()
columns_list = list(flat.keys())
for item in columns_list:
row_idx = re.findall(r'(\d+)\_', item )[0]
column = item.replace(row_idx+'_', '',1)
column = column.replace('_0', '')
row_idx = int(row_idx)
value = flat[item]
results.loc[row_idx, column] = value
# If you don't want to expand/split the `cities` column, remove line below
results = results.join(results['cities'].str.split(',', expand=True).add_prefix('cities_').fillna(np.nan))
print (results)
Output:
print (results.to_string())
session_id unix_timestamp cities user_user_id user_joining_date user_country cities_0 cities_1 cities_2
0 X061RFWB06K9V 1.442504e+09 New York NY, Newark NJ 2024.0 2015-03-22 UK New York NY Newark NJ NaN
1 5AZ2X2A9BHH5U 1.441354e+09 New York NY, Jersey City NJ, Philadelphia PA 2853.0 2015-03-28 DE New York NY Jersey City NJ Philadelphia PA
2 SHTB4IYAX4PX6 1.440843e+09 San Antonio TX 10958.0 2015-03-06 UK San Antonio TX NaN NaN