We Keep Coding

Simple API Rest python script takes a lot of time (or doesn't end)

For a business process I need to calculate driving distance between 1 origin and 30 k destinations.
I get both origins and destinations coordinates from a Google Sheet. Destinations is a matrix (approx 100 x 30).
I'm using HERE api to calculate the distance.
The result should be the same destinations matrix but with the distance (in the same order as the destinations coordinates).
This is the part of the script that calculates the distance and, I think, the one which lasts a lot:
distance= []
distance= pd.DataFrame(distance)
for row in destinations.itertuples():
a= row[1:]
distance1 = []
for column in a:
try:
args = {'waypoint0': 'geo!'+origins, 'waypoint1': 'geo!'+column, 'mode': 'fastest;truck'}
qstr = urlencode(args)
url = "https://route.ls.hereapi.com/routing/7.2/calculateroute.json?apiKey=xxxx" + qstr
response = urllib.request.urlopen(url)
dist = json.loads(response.read())['response']['route'][0]['leg'][0]['length']/1000
except Exception:
dist = 10000
distance1.append(dist)
distance2 = pd.DataFrame(distance1)
distance2 = distance2.T
distance = distance.append(distance2)
Does anyone think of a better way to make the script to actually finish?
Thanks!!

The logic looks pretty much accurate. If you need to limit the loop count, please check Large-Scale Matrix Routing API if it aligns with the use case.
The Large-Scale Matrix Routing service is a HTTP JSON API for calculating routing matrices with a large number of start and destinations points (e.g. 10,000 x 10,000).
For more details, please refer the following doc :
https://developer.here.com/documentation/large-matrix/api-reference-swagger.html
Note : please remove appKey from the shared code snippet

What's the most efficient way to fill a longitude / latitude map with circular points of specified radius?

So I'm trying to get around restrictions on a certain company's API that provides data on places of interest. The API doesn't allow me to collect results for an entire state. Instead, I must specify Lat/Lon coordinates and gather the nearby places in a circular 1 to 50000m radius. This API also only returns 60 results at a time, regardless of how many reside in the location specified.
I'm aware I will encounter more than 60 places at specified locations, but I'm planning to recursively bisect and process each of these cases.
I'd like to use New York State as my first test case.
I'm using a shapefile from
https://www.arcgis.com/home/item.html?id=b07a9393ecbd430795a6f6218443dccc
to get the shape of NY, and shapely to determine whether my point is inside the border.
import geopandas as gpd
from shapely.geometry import Point, Polygon
usa = gpd.read_file('states_21basic/states.shp')
polygon = usa[usa.STATE_ABBR == 'NY']['geometry'].values[0]
point = Point(-74.005974,40.712776) # create point
print(polygon.contains(point)) # check if polygon contains point
It's been suggested I try a flood fill algorithm, but I'm not exactly sure how to get a list of spaced out coordinates from it (to minimize API calls) and how to ensure that every part is covered, even in weird shapes like NY.
My main goal is to collect all the places whilst minimizing API calls. I'm not really expecting any code, just an idea of how to tackle this.
**Unfortunately, I have been removed from the company and won't be able to mark answers as accepted

Your problem might have a good solution not a perfect solution. I worked with maps for a long time and mostly all of them are complex themselves. You have another level of complexity which is an external dependency you cannot control.
I had to do something similar at a lower level querying Google Maps to obtain sites on a small area (no more than 20 meters) to try to match the closest place to where I was in terms of geolocation.
First, you need to make an assumption and try it. The assumption could be to only use 100 meters radius which more or less cover one block.
The second problem you have is density. You can explore a map with zero places of interest for a while (New York has a lot of open areas that might not have nearly anything around).
Let's suppose you can enclose your solution in a shape. Suggestion is you start from a predefined point in a dense area (Manhattan?) to maximize the algorithm findings at the beginning. Longer it runs, closer to detect less it will get.
The flood fill algorithm is good in your case but it might not be the best. I would probably go with something more complex that follows streets for example but a first approach using flood fill will work.
"Flood filling" is like using paint, you walk in a direction; in your case I suggest to use 8, (N, S, W, E, NW, NE, SW, SE), detect if you have "painted it" (you will need to store somewhere you've been in that position already to avoid duplicate calls, and call the API if not.
Walking will open a tree of different executions, start from point A, walk N, S, E, W, NW, NE, SW, SE recursively. This recursion might be extreme in an area like New York, 100 meters each in that area will lead to several thousands level of recursion. You will require to optimize it.
The third thing to keep into consideration is checking if the point is outside of the polygon. This is pretty straightforward using PNPoly. Once you are outside the polygon, walking in that direction must stop.
I share with you the PNPoly implementation in C# I coded for a project:
public bool IsCoordinateWithinPolygon(double latitude, double longitude, Polygon polygon)
{
if(polygon == null || polygon.Coordinates.Count() == 0)
{
return false;
}
List<double> coordinatesY = new List<double>();
List<double> coordinatesX = new List<double>();
var minLatitude = double.MaxValue;
var maxLatitude = double.MinValue;
var minLongitude = double.MaxValue;
var maxLongitude = double.MinValue;
// Quick-check, determine if the coordinate is outside of the bounding rectangle
foreach(var linearRing in polygon.Coordinates)
{
foreach (var coordinate in linearRing.Coordinates)
{
coordinatesY.Add(coordinate.Latitude);
coordinatesX.Add(coordinate.Longitude);
if (coordinate.Latitude < minLatitude)
{
minLatitude = coordinate.Latitude;
}
if(coordinate.Latitude > maxLatitude)
{
maxLatitude = coordinate.Latitude;
}
if(coordinate.Longitude < minLongitude)
{
minLongitude = coordinate.Longitude;
}
if(coordinate.Longitude > maxLongitude)
{
maxLongitude = coordinate.Longitude;
}
}
}
// Determine if the coordinate is outside the bounding box
if( (latitude < minLatitude || latitude > maxLatitude) &&
(longitude < minLongitude || longitude < maxLongitude))
{
// Out of the box
return false;
}
// PNPoly Algorithm - Point Inclusion in Polygon Test
bool inclusion = false;
var verty = coordinatesY.ToArray();
var vertx = coordinatesX.ToArray();
var nvert = vertx.Length;
var testy = latitude;
var testx = longitude;
for (int i = 0, j = nvert - 1; i < nvert; j = i++)
{
if (((verty[i] > testy) != (verty[j] > testy)) &&
(testx < (vertx[j] - vertx[i]) * (testy - verty[i]) / (verty[j] - verty[i]) + vertx[i]))
inclusion = !inclusion;
}
return inclusion;
}

Decompress Bing Maps GeoData (borders) with Python

I have been trying to decompress the Bing Maps location/border shape algorithm using Python. My end goal is to have custom regions/borders created from combining multiple counties and cities, and save the location data to our database for faster and more accurate location based analysis.
My strategy is as follows, but I'm a little stuck on #2, since I can't seem to accurately decompress the code:
Retrieve County/City borders from Bing Maps GeoData API - They refer to it as "shape"
Decompress their "shape" data to get the latitude and longitude of the border points
Remove the points that have the same lat/lng as other shapes (The goal is to make one large shape of multiple counties, as opposed to 5-6 separate shapes)
Compress the end result and save in the database
The function I am using seems to work for the example of 'vx1vilihnM6hR7mEl2Q' provided in the Point Compression Algorithm documentation. However, when I insert something a little more complex, like Cook County, the formula seems to be off (tested by inserting several of the points into different polygon mapping/drawing applications that also use Bing Maps). It basically creates a line at the south side of Chicago that vigorously goes East and West into Indiana, without much North-South movement. Without knowing what the actual coordinates of any counties are supposed to be, I'm not sure how to figure out where I'm going wrong.
Any help is greatly appreciated, even if it is a suggestion of a different strategy.
Here is the python code (sorry for the overuse of the decimal format - my poor attempt to ensure the error wasn't a result of inadvertently losing precision):
safeCharacters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-'
def decodeBingBorder(compressedData):
latLng = []
pointsArray = []
point = []
lastLat = Decimal(0)
lastLng = Decimal(0)
# Assigns the number of of each character based on the respective index of 'safeCharacters'
# numbers under 32 indicate it is the last number of the combination of the point, and a new point is begun
for char in compressedData:
num = Decimal(safeCharacters.index(char))
if num < 32:
point.append(num)
pointsArray.append(point)
point = []
else:
num -= Decimal(32)
point.append(num)
# Loops through each point to determine the lat/lng of each point
for pnt in pointsArray:
result = Decimal(0)
# This revereses step 7 of the Point Compression Algorithm https://msdn.microsoft.com/en-us/library/jj158958.aspx
for num in reversed(pnt):
if result == 0:
result = num
else:
result = result * Decimal(32) + num
# This was pretty much taken from the Decompression Algorithm (not in Python format) at https://msdn.microsoft.com/en-us/library/dn306801.aspx
# Determine which diaganal it's on
diag = Decimal(int(round((math.sqrt(8 * result + 5) - 1) / 2)))
# submtract the total number of points from lower diagonals, and get the X and Y from what's left over
latY = Decimal(result - Decimal(diag * (diag + 1) / 2))
lngX = Decimal(diag - latY)
# undo the sign encoding
if latY % 2 == 1:
latY = (latY + Decimal(1)) * Decimal(-1)
if lngX % 2 == 1:
lngX = (lngX + Decimal(1)) * Decimal(-1)
latY /= 2
lngX /= 2
# undo the delta encoding
lat = latY + lastLat
lng = lngX + lastLng
lastLat = lat
lastLng = lng
# position the decimal point
lat /= Decimal(100000)
lng /= Decimal(100000)
# append the point to the latLng list in a string format, as opposed to the decimal format
latLng.append([str(lat), str(lng)])
return latLng
The compressed algorithm:
1440iqu9vJ957r8pB_825syB6rh_gXh1-ntqB56sk2B2nq07Mwvq5f64r0m0Fni11ooE4kkvxEy4wzMuotr_DvsiqvFozvt-Lw9znxH-r5oxLv9yxCwhh7wKnk4sB8o0Rvv56D8snW5n1jBg50K4kplClkpqBpgl9F4h4X_sjMs85Ls6qQi6qvqBr188mBqk-pqIxxsx5EpsjosI-8hgIoygDigU94l_4C
This is the result:
[['41.46986', '-87.79031'], ['41.47033', '-87.52569'], ['41.469145',
'-87.23372'], ['41.469395', '-87.03741'], ['41.41014', '-86.7114'],
['41.397545', '-86.64553'], ['41.3691', '-86.47018'], ['41.359585',
'-86.41984'], ['41.353585', '-86.9637'], ['41.355725', '-87.43971'],
['41.35561', '-87.52716'], ['41.3555', '-87.55277'], ['41.354625',
'-87.63504'], ['41.355635', '-87.54018'], ['41.360745', '-87.40351'],
['41.362315', '-87.29262'], ['41.36214', '-87.43194'], ['41.360915',
'-87.44473'], ['41.35598', '-87.58256'], ['41.3551', '-87.59025'],
['41.35245', '-87.59828'], ['41.34782', '-87.60784'], ['41.34506',
'-87.61664'], ['41.34267', '-87.6219'], ['41.34232', '-87.62643'],
['41.33809', '-87.63286'], ['41.33646', '-87.63956'], ['41.32985',
'-87.65056'], ['41.33069', '-87.65596'], ['41.32965', '-87.65938'],
['41.33063', '-87.6628'], ['41.32924', '-87.66659'], ['41.32851',
'-87.71306'], ['41.327105', '-87.75963'], ['41.329515', '-87.64388'],
['41.32698', '-87.73614'], ['41.32876', '-87.61933'], ['41.328275',
'-87.6403'], ['41.328765', '-87.63857'], ['41.32866', '-87.63969'],
['41.32862', '-87.70802']]

As mentioned by rbrundritt, storing the data from Big Maps is against the terms of use. However, there are other sources of this same data available, such as http://nationalmap.gov/boundaries.html
In the interest of solving the problem, and to store this and other coordinate data more efficiently, I solved the problem by removing the 'round' function when calculating 'diag'. This should be what replaces it:
diag = int((math.sqrt(8 * result + 5) - 1) / 2)
All of the 'Decimal' crap I added is not necessary, so you can remove it if you wish.

You can also do
diag=int(round((sqrt(8 * number + 1)/ 2)-1/2.))
Don't forget to subtract longitude*2 from latitude to get N/E coordinates!

Maybe it will be usefull, i found bug in code.
invert pair function should be
diag = math.floor((math.sqrt(8 * result + 1) - 1) / 2)
after fixing this, your implementation work correct

You can't store the boundary data from the Bing Maps GeoData API or any data derived from it in a database. This is against the terms of use of the platform.

GeoDjango + PostGIS calculates wrong Distances

I just installed PostGIS with GeoDjango. Everything worked fine, but now I have a problem and cant find out the reason for this.
I have model like this:
from django.contrib.gis.db import models
class Shop(models.Model):
name = models.CharField(max_length=80)
point = models.PointField(null=True, blank=True)
objects = models.GeoManager()
And I set its point to this position (49.794254,9.927489). Then i create a point like this:
pnt = fromstr('POINT(50.084068 8.238381)')
The distance between this points should be about ~ 125 km, but when i do this:
results = Shop.objects.distance(pnt)
print results[0].distance.km
I'm getting always about 60 km too much in my result, so it returns 190 km! My SRIDs of both points are 4326... probably something wrong with that?
And maybe another interesting fact, when i do this:
pnt.distance(shop.point)
it returns 1.713790... as a result.
What am I doing wrong? Any alternatives for me to use with python + django? If there is a better solution I would not need to use PostGIS.
Hope you can help me!
Chris

I just ran this query in postgis :
select round(CAST(ST_Distance_Sphere(ST_GeomFromText('POINT(49.794254 9.927489)',4326), ST_GeomFromText('POINT(50.084068 8.238381)',4326)) As numeric)/1000.0,2) as distance_km;
distance_km
-------------
190.50
the result is in fact 190.50, so it seems there's nothing wrong with your 190 km result
same result with this awesome page, there is a brief explanation of how to calculate this distances.
the 1.713790... result seems to be in the same units of the srid, or in other words that number is not in meters.
EDIT
Ooohh I just saw your problem you misplaced the lat and lon, in the WKT format, Longitude comes first so the real query should be:
select round(CAST(ST_Distance_Sphere(ST_GeomFromText('POINT(9.927489 49.794254)',4326), ST_GeomFromText('POINT(8.238381 50.084068)',4326)) As numeric)/1000.0,2) as distance_km;
distance_km
-------------
125.10
so the points should be created like this
POINT(9.927489 49.794254)
POINT(8.238381 50.084068)

Store 3 nearest coordinates

I have an XML file that contains a number of points with their longitude and latitude.
My python code at the moment gets the nearest point by simply looping through the XML file, finding the nearest, in miles or whatever, then comparing it with the previous closest point. If its nearer then I assign the variable the value of this new point. So everything is working in that regard.
Now, what I want to do is actually store the closest 2 or 3 points.
How do I go about doing this? The XML file isn't ordered by closest, and besides, the users location will change each time a request is made. Can I do this with an XML file or will I perhaps have to look into storing the data is SQL Server or MySQL?
Thanks for the help.
PS, the sample code is available here if anyone is interested. This is part of a college project.

You should store in a list of tuples (for example) all the point pairs and their distances as you parse de xml file.
mypoints = [(distance12, x1, x2),...,(distancenm, xn, xm)]
mypoints.sort()
three_closer = mypoints[:3]
Adapting this to your code:
..............
mypoints = []
for row in rows:
# Get coords for current record
curr_coords = row.getAttribute("lat") + ',' + row.getAttribute("lng")
# Get distance
tempDistance = distance.distance(user_coords, curr_coords).miles
mypoints.append((tempDistance, row))
mypoints.sort()
#the three closest points:
mythree_shorter = mypoints[0:3]
for distance, row in mythree_shorter:
shortestStation = json.dumps(
{'number': row.getAttribute("number"),
'address': row.getAttribute("address"),
'lat': row.getAttribute("lat"),
'lng': row.getAttribute("lng"),
'open': row.getAttribute("open")},
sort_keys=True,
indent=4)
save_in_some_way(shortestStation) #maybe writing to a file?
..................

Here's a solution that will work for any number of points:
closest = points[:NUM_CLOSEST]
closest.sort()
for point in points[NUM_CLOSEST:]:
if point.distance < closest[-1].distance:
closest[-1] = point
closest.sort()
Obviously, a bit pseudo-cody. The sort() calls will probably need an argument so they are sorted in a useful way, and you'll probably want a function to calculate the distance to replace the distance member.