I have a DataFrame with latitude and longitude of places (restaurants) and a DataFrame with latitude and longitude of neighborhoods (area).
I would like, for each neighborhood, to count the number of restaurants in a 3km area (numberR).
I have written the following code, and it works:
df=pd.DataFrame()
numberR=[]
radius=3
for element in range(0,area['lon'].count()): #for every neighborhood
df=pd.DataFrame()
df['destLat']=restaurants['lat']
df['originLat']=areas['lat'][element]
df['destLon']= restaurants['lng']
df['originLon']=area['lon'][element]
for i, row in df.iterrows():
#for every restaurant I compute the distance from my neighborhood in km
l=[haversine(df.originLon[i],df.originLat[i],df.destLon[i],df.destLat[i]) for i, row in df.iterrows()]
numberR.append(sum(x<radius for x in l))
However, I would like to make the code quicker as it is very slow.
Do you have any idea on how could I do to reach the same result in less time?
Thanks in advance.
P.S. haversine is the well known function for getting distance in km starting from lat and lng.
I would recommend you to use functions from scipy.spacial.distance.
from scipy.spatial.distance import cdist
distances = cdist(areas, restaurants, metric=haversine) # metric accepts a callable
sum(distances > 3) # sums columns
The cdist function computes distances between each pair of rows of the two DataFrames.
Also, you should modify the haversine function as to be able to accept DataFrame rows.
Related
I have a csv file with a table that has the columns Longitude, Latitude, and Wind Speed. I have a code that takes a csv file and deletes values outside of a specified bound. I would like to retain values whose longitude/latitude is within a 0.5 lon/lat radius of a point located at -71.5 longitude and 40.5 latitude.
My example code below deletes any values whose longitude and latitude isn't between -71 to -72 and 40 to 41 respectively. Of course, this retains values within a square bound ±0.5 lon/lat around my point of interest. But I am interested in finding values within a circular bound with radius 0.5 lon/lat of my point of interest. How should I modify my code?
import pandas as pd
import numpy
df = pd.read_csv(r"C:\\Users\\xil15102\\Documents\\results\\EasternLongIsland50.csv") #file path
indexNames=df[(df['Longitude'] <= -72)|(df['Longitude']>=-71)|(df['Latitude']<=40)|(df['Latitude']>=41)].index
df.drop(indexNames,inplace=True)
df.to_csv(r"C:\\Users\\xil15102\\Documents\\results\\EasternLongIsland50.csv")
Basically you need to check if a value is a certain distance from a central point (-71.5 and 40.5); to do this use the pythagorean theorem/distance formula:
d = sqrt(dx^2+dy^2).
So programmatically, I would do this like:
from math import sqrt
drop_indices = []
for row in range(len(df)):
if (sqrt(abs(-71.5 - df[row]['Longitude'])*abs(-71.5 - df[row]['Longitude']) + abs(40.5-df[row]['Latitude'])*abs(40.5-df[row]['Latitude']))) > 0.5:
drop_indices.append(row)
df.drop(drop_indices)
Sorry that is a sort for disgusting way to get rid of the rows and your way looks much better, but the code should work.
You should write a function to calculate the distance from your point of interest and drop those. Some help here. Pretty sure the example below should work if you implement is_not_in_area as a function to calculate the distance and check if dist < 0.5.
df = df.drop(df[is_not_in_area(df.lat, df.lon)].index)
(This code lifted from here)
Edit: drop the ones that aren't in area, not the ones that are haha.
In case this has been answered in the past I want to apologize, I was not sure how to phrase the question.
I have a dataframe with 3d coordinates and rows with a scalar value (magnetic field in this case) for each point in space. I calculated the radius as the distance from the line at (x,y)=(0,0) for each point. The unique radius and z values are transferred into a new dataframe. Now I want to calculate the scalar values for every point (Z,R) in the volume by averaging over all points in the 3d system with equal radius.
Currently I am iterating over all unique Z and R values. It works but is awfully slow.
df is the original dataframe, dfn is the new one which - in the beginning - only contains the unique combinations of R and Z values.
for r in dfn.R.unique():
for z in df.Z.unique():
dfn.loc[(df["R"]==r)&(df["Z"]==z), "B"] = df["B"][(df["R"]==r)&(df["Z"]==z)].mean()
Is there any way to speed this up by writing a single line of code, in which pandas is given the command to grab all rows from the original dataframe, where Z and R have the values according to each row in the new dataframe?
Thank you in advance for your help.
Try groupby!!!
It looks like you can achieve with something like:
df[['R', 'Z', 'B']].groupby(['R', 'Z']).mean()
I have over 1 million rows of Latitude Longitude positions. My goal is to check each of these rows against a data set of about 43000 ZipCodes that have a central Latitude Longitude.
I want to calculate the haversine distance between each row with the large ZipCodes list. I then want to take the closest lat/long and return that or the corresponding zip code to the left most frame (in essence, giving the closest ZipCode to the latitude/longitudes in the large frame.
I have tried several things including vectorized haversine functions and looping through each row, calculating and moving to next but I can't quite get them to work. Given the large size of my data I know that simply looping through each row and calculating won't work. I need a new solution. I think it might involve vectorization.
Here are some sample frames of my data. df is the large frame I am trying to calculate the smallest distance from the zip_list and return the corresponding zip code to the large frame.
df = pd.DataFrame(np.array([[42.801104,-76.827879],[38.187102,-83.433917],
[35.973115,-83.955932]]), columns = ['Lat', 'Long'])
zip_list = pd.DataFrame(np.array([[49544, 42.999561,-85.75371],[49648,
45.000254,-85.3651],[49654, 45.023384,-85.75697],[50265,
41.570916,-93.73568]]), columns = ['ZipCode', 'Latitude', 'Longitude'])
I would like to return the minimum distance zip code to the corresponding row in the df frame.
Any ideas would be great. I am a beginner with vectorization and numpy/pandas.
I have 4 Dataframes (ticket_data.csv, providers.csv, stations.csv and cities.csv)
In stations.csv I have 2 colls called o_city (origin city) and d_city (destination city) those two colls gives me the id of the city i need to look for in cities.csv
In cities.csv I have the lat and long of each city.
How can i calculate the distance between o_city and d_city for each ticket ? I tried to use pyproj but I didn't find a way to make it work with each ticket..
Screenshot of csv files :
ticket_data.csv
cities.csv
Welcome to StackOverflow! In your cities dataframe, assuming here it is called city_df; for each row you can use something called the haversine distance formula from Euclidean geometry to calculate the distance between two coordinate pairs on Earth's surface. Here is an example of some dummy Python3 code of roughly how you may go about this (just using two pairs of coordinates for ease of communication):
from haversine import haversine
distance = haversine((city_df[origin_lat][0], city_df[origin_lon][0]), (city_df[destination_lat][0], city_df[destination_lon][0]))
The coordinates must be in decimal degree notation as in 43.9202 instead of 43* 38" 67' notation. Given this, the output value of distance will be in km units.
Hope this helps you get closer to solving your problem!
P. S. - you may need to install haversine, as it is not in the standard libary
I am looking to find a center of mass for N-dimensional space in Python.I have a dataframe with K columns (some contain text and some contain numbers)
{X1...Xk}
...
{Z1..Zk}
k > 10000
I need to calculate center of mass for all numerical values in the dataframe.
What is the best way to do it?
The center of mass is simply the mean of the values on each dimension, and you just want to calculate it on non-object columns, so:
df.ix[:,df.dtypes != 'O'].mean()
EDIT: although the OP only mentioned "text" and "numbers", the following alternative is indeed more general (thanks MaxU):
df.select_dtypes(include=['number']).mean()