ValueError: array must not contain infs or NaNs - python

I have a csv file with data that is formatted for example, as follows(my data set is much much larger):
Image Id,URL,Latitude,Longitude,Address
10758202333,https://farm8.staticflickr.com/7408/10758202333_b6c29d93b1_q.jpg,51.482826,-0.167112,Cadogan Pier Chelsea Embankment Chelsea Royal Borough of Kensington and Chelsea London
23204019400,https://farm6.staticflickr.com/5688/23204019400_fb6879abe3_q.jpg,51.483106,-3.171207,Greggs Station Terrace Plasnewydd Cardiff Wales CF United Kingdom
11243511074,https://farm3.staticflickr.com/2818/11243511074_e1e2f1b99c_q.jpg,51.483297,-0.166534,Albert Bridge Chelsea Embankment Chelsea Royal Borough of Kensington and Chelsea London Greater London England SW3 5SY United Kingdom
22186903335,https://farm6.staticflickr.com/5697/22186903335_de53168305_q.jpg,51.483394,-3.176926,Greyfriars House Greyfriars Road Plasnewydd Cardiff Wales CF United Kingdom
22197179851,https://farm6.staticflickr.com/5786/22197179851_a818b17fae_q.jpg,51.483394,-3.176926,Greyfriars House Greyfriars Road Plasnewydd Cardiff Wales CF United Kingdom
22174235522,https://farm1.staticflickr.com/589/22174235522_3ffd1de2bb_q.jpg,51.483394,-3.176926,Greyfriars House Greyfriars Road Plasnewydd Cardiff Wales CF United Kingdom
22160755536,https://farm1.staticflickr.com/761/22160755536_8e23e9ed32_q.jpg,51.483394,-3.176926,Greyfriars House Greyfriars Road Plasnewydd Cardiff Wales CF United Kingdom
7667114130,https://farm8.staticflickr.com/7269/7667114130_117849250a_q.jpg,51.484563,-3.178181,Oybike Gorsedd Gardens Road Cathays Cardiff Wales CF United Kingdom
17136775881,https://farm9.staticflickr.com/8780/17136775881_363c2379ef_q.jpg,51.484608,-3.178845,Oybike Gorsedd Gardens Road Cathays Cardiff Wales CF United Kingdom
7110881411,https://farm9.staticflickr.com/8162/7110881411_f0fe3d7214_q.jpg,51.484644,-3.178099,Oybike Gorsedd Gardens Road Cathays Cardiff Wales CF United Kingdom
11718453936,https://farm4.staticflickr.com/3700/11718453936_148af12df6_q.jpg,51.484661,-3.179117,King Edward VII Avenue Cathays Cardiff Wales CF United Kingdom
20218915752,https://farm1.staticflickr.com/352/20218915752_4282c1f9b8_q.jpg,51.484683,-3.179147,King Edward VII Avenue Cathays Cardiff Wales CF United Kingdom
My code is as follows, I know it is not much but I simply want to be able to see a cluster plot figure showing up for now with centroids. However I am getting an error "ValueError: array must not contain infs or NaNs"
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.vq import kmeans, kmeans2, whiten
df = pd.read_csv('dataset_import.csv')
df.head()
coordinates = df.as_matrix(columns=['latitude', 'longitude'])
N = len(coordinates)
k = 100
i = 50
w = whiten(coordinates)
cluster_centroids, closest_centroids = kmeans2(w, k, iter=i, minit='points')
plt.figure(figsize=(10, 6), dpi=100)
plt.scatter(cluster_centroids[:,0], cluster_centroids[:,1], c='r', alpha=.7, s=150)
plt.scatter(w[:,0], w[:,1], c='k', alpha=.3, s=10)
plt.show()
Can anyone shed some light as to why this is happening, perhaps some of the fugures in my code are wrong etc. Thanks!

I have met the same problem with you, and I solved by wipe out the NaNs and infs.
def clean(serie):
output = serie[(np.isnan(serie) == False) & (np.isinf(serie) == False)]
return output
When I draw a plot, I use this function to clean my data in a temporary way, and it works now.
fig = plt.figure()
clean(data[col]).plot(kind='kde')
plt.show()
Or like this:
sns.kdeplot(clean(data[col]), bw=0.1, shade=True, legend=False)

Related

Draw a Map of cities in python

I have a ranking of countries across the world in a variable called rank_2000 that looks like this:
Seoul
Tokyo
Paris
New_York_Greater
Shizuoka
Chicago
Minneapolis
Boston
Austin
Munich
Salt_Lake
Greater_Sydney
Houston
Dallas
London
San_Francisco_Greater
Berlin
Seattle
Toronto
Stockholm
Atlanta
Indianapolis
Fukuoka
San_Diego
Phoenix
Frankfurt_am_Main
Stuttgart
Grenoble
Albany
Singapore
Washington_Greater
Helsinki
Nuremberg
Detroit_Greater
TelAviv
Zurich
Hamburg
Pittsburgh
Philadelphia_Greater
Taipei
Los_Angeles_Greater
Miami_Greater
MannheimLudwigshafen
Brussels
Milan
Montreal
Dublin
Sacramento
Ottawa
Vancouver
Malmo
Karlsruhe
Columbus
Dusseldorf
Shenzen
Copenhagen
Milwaukee
Marseille
Greater_Melbourne
Toulouse
Beijing
Dresden
Manchester
Lyon
Vienna
Shanghai
Guangzhou
San_Antonio
Utrecht
New_Delhi
Basel
Oslo
Rome
Barcelona
Madrid
Geneva
Hong_Kong
Valencia
Edinburgh
Amsterdam
Taichung
The_Hague
Bucharest
Muenster
Greater_Adelaide
Chengdu
Greater_Brisbane
Budapest
Manila
Bologna
Quebec
Dubai
Monterrey
Wellington
Shenyang
Tunis
Johannesburg
Auckland
Hangzhou
Athens
Wuhan
Bangalore
Chennai
Istanbul
Cape_Town
Lima
Xian
Bangkok
Penang
Luxembourg
Buenos_Aires
Warsaw
Greater_Perth
Kuala_Lumpur
Santiago
Lisbon
Dalian
Zhengzhou
Prague
Changsha
Chongqing
Ankara
Fuzhou
Jinan
Xiamen
Sao_Paulo
Kunming
Jakarta
Cairo
Curitiba
Riyadh
Rio_de_Janeiro
Mexico_City
Hefei
Almaty
Beirut
Belgrade
Belo_Horizonte
Bogota_DC
Bratislava
Dhaka
Durban
Hanoi
Ho_Chi_Minh_City
Kampala
Karachi
Kuwait_City
Manama
Montevideo
Panama_City
Quito
San_Juan
What I would like to do is a map of the world where those cities are colored according to their position on the ranking above. I am opened to further solutions for the representation (such as bubbles of increasing dimension according to the position of the cities in the rank or, if necessary, representing only a sample of countries taken from the top rank, the middle and the bottom).
Thank you,
Federico
Your question has two parts; finding the location of each city and then drawing them on the map. Assuming you have the latitude and longitude of each city, here's how you'd tackle the latter part.
I like Folium (https://pypi.org/project/folium/) for drawing maps. Here's an example of how you might draw a circle for each city, with it's position in the list is used to determine the size of that circle.
import folium
cities = [
{'name':'Seoul', 'coodrs':[37.5639715, 126.9040468]},
{'name':'Tokyo', 'coodrs':[35.5090627, 139.2094007]},
{'name':'Paris', 'coodrs':[48.8588787,2.2035149]},
{'name':'New York', 'coodrs':[40.6976637,-74.1197631]},
# etc. etc.
]
m = folium.Map(zoom_start=15)
for counter, city in enumerate(cities):
circle_size = 5 + counter
folium.CircleMarker(
location=city['coodrs'],
radius=circle_size,
popup=city['name'],
color="crimson",
fill=True,
fill_color="crimson",
).add_to(m)
m.save('map.html')
Output:
You may need to adjust the circle_size calculation a little to work with the number of cities you want to include.

finding id name of 5 most frequent value in a column in pandas

We have a data which has column name "birth_country"
i executed following code;
import pandas as pd
df=pd.read_csv("data.csv")
df['birth_country'].value_counts()[:5]
output:
United States of America 259
United Kingdom 85
Germany 61
France 51
Sweden 29
I want my output to be look like;
United States of America
United Kingdom
Germany
France
Sweden
How to do it?
Like;
df['birth_country'].value_counts().idxmax()
gives output:
United States of America
For series by index values use:
pd.Series(df['birth_country'].value_counts()[:5].index)

Assign values from a dictionary to a new column based on condition

This my data frame
City
sales
San Diego
500
Texas
400
Nebraska
300
Macau
200
Rome
100
London
50
Manchester
70
I want to add the country at the end which will look like this
City
sales
Country
San Diego
500
US
Texas
400
US
Nebraska
300
US
Macau
200
Hong Kong
Rome
100
Italy
London
50
England
Manchester
200
England
The countries are stored in below dictionary
country={'US':['San Diego','Texas','Nebraska'], 'Hong Kong':'Macau', 'England':['London','Manchester'],'Italy':'Rome'}
It's a little complicated because you have lists and strings as the values and strings are technically iterable, so distinguishing is more annoying. But here's a function that can flatten your dict:
def flatten_dict(d):
nd = {}
for k,v in d.items():
# Check if it's a list, if so then iterate through
if ((hasattr(v, '__iter__') and not isinstance(v, str))):
for item in v:
nd[item] = k
else:
nd[v] = k
return nd
d = flatten_dict(country)
#{'San Diego': 'US',
# 'Texas': 'US',
# 'Nebraska': 'US',
# 'Macau': 'Hong Kong',
# 'London': 'England',
# 'Manchester': 'England',
# 'Rome': 'Italy'}
df['Country'] = df['City'].map(d)
You can implement this using geopy
You can install geopy by pip install geopy
Here is the documentation : https://pypi.org/project/geopy/
# import libraries
from geopy.geocoders import Nominatim
# you need to mention a name for the app
geolocator = Nominatim(user_agent="some_random_app_name")
# get country name
df['Country'] = df['City'].apply(lambda x : geolocator.geocode(x).address.split(', ')[-1])
print(df)
City sales Country
0 San Diego 500 United States
1 Texas 400 United States
2 Nebraska 300 United States
3 Macau 200 中国
4 Rome 100 Italia
5 London 50 United Kingdom
6 Manchester 70 United Kingdom
# to get country name in english
df['Country'] = df['City'].apply(lambda x : geolocator.reverse(geolocator.geocode(x).point, language='en').address.split(', ')[-1])
print(df)
City sales Country
0 San Diego 500 United States
1 Texas 400 United States
2 Nebraska 300 United States
3 Macau 200 China
4 Rome 100 Italy
5 London 50 United Kingdom
6 Manchester 70 United Kingdom

Pandas Groupby results coming up based on the value_counts and ascending values

highest_medals_countries = olympics_merged.groupby(['Sport'])['Team'].value_counts()
highest_medals_countries.sort_values(ascending = False)[:10]
Output:
Sport Team
Athletics United States 3202
Great Britain 2240
Gymnastics United States 1939
Swimming United States 1622
Gymnastics France 1576
Athletics France 1494
Gymnastics Italy 1345
Swimming Great Britain 1291
Athletics Germany 1254
Gymnastics Hungary 1242
In the above output, I am stacking the teams with the most number of medals based on sport together but when I look at the output the sports are coming up based on the value counts. How can I get rid of this and put countries together for athletics , Gymnastics, Swimming, etc?
Expected output is:
Sport Team
Athletics United States 3202
Great Britain 2240
France 1494
Gymnastics United States 1939
France 1576
Italy 1345
Hungary 1242
Swimming United States 1622
Great Britain 1291
Athletics Germany 1254
By running sort_values on your stacked dataframe you force it to sort the entire dataframe by value whereas the values were already sorted within the categories in the first place. So don't run highest_medals_countries.sort_values(ascending = False)[:10] and you're fine.

Rounding and sorting dataframe with pandas

https://github.com/haosmark/jupyter_notebooks/blob/master/Coursera%20week%203%20assignment.ipynb
All the way at the bottom of the code, with question 3, I'm trying to average, round, and sort the data, however for some reason rounding and sorting isn't working at all
i = df.columns.get_loc('2006')
avgGDP = df[df.columns[i:]].copy()
avgGDP = avgGDP.mean(axis=1).round(2).sort_values(ascending=False)
avgGDP
what am I doing wrong here?
This is what df looks like before I apply average, round, and sort.
Your series is actually sorted, the first line being 1.5e+13 and the last one 4.4e+11:
Country
United States 1.536434e+13
China 6.348609e+12
Japan 5.542208e+12
Germany 3.493025e+12
France 2.681725e+12
United Kingdom 2.487907e+12
Brazil 2.189794e+12
Italy 2.120175e+12
India 1.769297e+12
Canada 1.660648e+12
Russian Federation 1.565460e+12
Spain 1.418078e+12
Australia 1.164043e+12
South Korea 1.106714e+12
Iran 4.441558e+11
Rounding doesn't do anything visible here because the smallest value is 4e+11, and rounding it to 2 decimal places doesn't show on this scale. If you want to keep only 2 decimal places in the scientific notation, you can use .map('{:0.2e}'.format), see my note below.
Note: just for fun, you could also calculate the same with a one-liner:
df.filter(regex='^2').mean(1).sort_values()[::-1].map('{:0.2e}'.format)
Output:
Country
United States 1.54e+13
China 6.35e+12
Japan 5.54e+12
Germany 3.49e+12
France 2.68e+12
United Kingdom 2.49e+12
Brazil 2.19e+12
Italy 2.12e+12
India 1.77e+12
Canada 1.66e+12
Russian Federation 1.57e+12
Spain 1.42e+12
Australia 1.16e+12
South Korea 1.11e+12
Iran 4.44e+11

Categories