Unfortunately my projection from Irish Transverse Mercator (ITM) to WGS84 latitude-longitude seems to have gone wrong as the plotted coordinates don't line up with a map of Dublin sourced from the CSO (see below).
My transformed coordinates plotted on a map of Dublin
The transformed data was sourced from the Irish Valuation Office and the ITM X & Y coordinates were fed into a function adapted from a previous stackoverflow discussion which uses geopandas' built-in points_from_xy method to transform coordinates between Coordinate Reference Systems:
def create_geodf_from_GPS (df, latitude, longitude, crs):
locations = gpd.points_from_xy(longitude, latitude)
geo_df = gpd.GeoDataFrame(df, geometry=locations)
geo_df.crs = crs
return geo_df
VO_geo = create_geodf_from_GPS(VO, VO[" X ITM"], VO[" Y ITM"], crs = 'epsg:2157')
VO_geo = VO_geo.to_crs('epsg:4326')
Does anyone have any idea what may have gone wrong here?
Very simple fix thanks to #joris
Altered function using x & y as arguments for gpd.points_from_xy instead of the previously mixed up longitude & latitude:
def create_geodf_from_GPS (df, x, y, crs):
locations = gpd.points_from_xy(x, y)
geo_df = gpd.GeoDataFrame(df, geometry=locations)
geo_df.crs = crs
return geo_df
Now plotting the data in WGS84 latitude-longitude works as expected:
VO_geo = create_geodf_from_GPS(VO, x=VO[" X ITM"], y=VO[" Y ITM"], crs = 'epsg:2157')
VO_geo.to_crs('epsg:4326').plot()
Note: the data had to be cleaned to remove obvious outliers by filtering out non-Dublin data using geopandas' (gpd) spatial join function
VO_geo_clean = gpd.sjoin(VO_geo.to_crs('epsg:4326'), map_of_Dublin)
Result:
VO data plotted over a map of Dublin
Related
I have an xarray dataset that contains a data variable 'tmpc_obs' that was created using the interpolate_to_grid function within MetPy. I would like to be able to interpolate specific point values from lat lon pairs using xarray.interp, but I need assign a crs and then assign x and y values. MetPy has options to do this, but I am not having much luck getting it to work.
I created map_crs using cartopy and is what I will transform my lat lon points to, and then create the interpolated field. latlon_crs is the crs for lat lon points.
map_crs = ccrs.AlbersEqualArea(central_longitude=-79.0,
central_latitude=35.5,
standard_parallels=(34.33,36.16),
)
latlon_crs = ccrs.PlateCarree()
Transform the points to the map_crs and then interpolate the point values to a grid.
xp, yp, _ = map_crs.transform_points(latlon_crs, obs['lon'], obs['lat']).T
obsx, obsy, tmpc = interpolate_to_grid(xp, yp, obs['tmpc'].values,
interp_type='rbf',
rbf_func='thin_plate',
hres=5000)
This turns the 3 arrays generated into an xarray dataset. x_p and y_p are x and y coordinates that are transformed to the map crs.
obs_ds = xr.Dataset(
data_vars=dict(
tmpc_obs=(["x", "y"], tmpc),
),
coords=dict(
x_p=(["x", "y"], obsx),
y_p=(["x", "y"], obsy),
),
)
It's not possible however to interpolate point values at this time using the following code block as it will only give you nan's.
vals = []
for x, y in zip(stat['lon'], stat['lat']):
x_t, y_t = map_crs.transform_point(x, y, src_crs=latlon_crs)
tmp_c = obs_ds["tmpc_obs"].interp(x=x_t, y=y_t, method='cubic').values.item()
tmp_c = float("{:.2f}".format(tmp_c))
vals.append(tmp_c)
I believe you need to add y and x to the coordinates of the dataset that reference the dimensions. I am having issues doing this. I think I should use the same crs info from the mapcrs, but I don't know what to use for semi major and semi minor axis. Whether I use the axes or not, I get an error, and it doesnt attach the x and y to the coordinates, it only attaches the metpy_crs
# Assign the grid mapping to be able to index specific lat/lon pairs
obs_ds = obs_ds.metpy.assign_crs({
'semi_major_axis': ???,
'semi_minor_axis': ???,
'grid_mapping_name': 'albers_conical_equal_area',
"standard_parallel": [34.33,36.16],
"latitude_of_projection_origin": 35.5,
"longitude_of_central_meridian": -79.0,
}).metpy.assign_y_x()
Error:
UserWarning: No y and x coordinates assigned since horizontal coordinates were not found
warnings.warn('No y and x coordinates assigned since horizontal coordinates '
<xarray.Dataset>
Dimensions: (x: 57, y: 75)
Coordinates:
x_p (x, y) float64 -1.963e+05 -1.913e+05 ... 1.692e+05 1.742e+05
y_p (x, y) float64 -1.663e+05 -1.663e+05 ... 1.137e+05 1.137e+05
metpy_crs object Projection: albers_conical_equal_area
Dimensions without coordinates: x, y
Data variables:
tmpc (x, y) float64 0.0618 0.07065 0.07945 ... 0.08191 0.09538 0.1064
I'm using this dataset:
https://sedac.ciesin.columbia.edu/data/set/gpw-v4-population-density-rev11/data-download
(Gridded population density of the world)
With this map:
https://data.humdata.org/dataset/uganda-administrative-boundaries-as-of-17-08-2018
(Uganda administrative boundaries shapefile)
I have clipped the uganda map to the region I need, like so:
shape_records = uganda.shapeRecords()
desired_shapes = []
for s in shape_records:
for x in s.record:
if 'FORT PORTAL' in str(x):
desired_shapes.append(s)
Loaded them into a single geopandas dataframe:
forgpd=[]
for x in desired_shapes:
forgpd.append(x.__geo_interface__)
gdf = gpd.GeoDataFrame.from_features(forgpd, crs=4326)
Then I'm reading the .tif world population file with rasterio.
gpw = rio.open('UgandaData/gpw_v4_population_density_rev11_2020_30_sec.tif')
gpw_region = gpw.read(1, window=gpw.window(*box))
And I'd like to crop it, using this:
from rasterio import mask as msk
region_mask, region_mask_tf = msk.mask(dataset=gpw, shapes=gdf.geometry, all_touched=True, filled=True, crop=True) #error here
region_mask = np.where(region_mask < 0, 0, region_mask).squeeze()
I get the following errors:
WindowError: windows do not intersect
ValueError: Input shapes do not overlap raster.
This is my crs:
Gridded population of world: CRS.from_epsg(4326)
Uganda(Fort Portal) :
<Geographic 2D CRS: EPSG:4326>
Name: WGS 84
Axis Info [ellipsoidal]:
- Lat[north]: Geodetic latitude (degree)
- Lon[east]: Geodetic longitude (degree)
Area of Use:
- name: World
- bounds: (-180.0, -90.0, 180.0, 90.0)
Datum: World Geodetic System 1984
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich
Could the difference be that I have not specified WGS 84 for the gridded world population? If so, how is this specified?
The problem is the shapefile is in UTM coordinates and the raster is a world coordinate system (lat/long). Even though you assign the epsg:4326 crs to gdf it's coordinates are still in UTM. You can convert these manually doing something like this.
Otherwise, you can re-projected the world raster into EPSG:21096 (estimation based off UTM zone from the uganda shapefile) using QGIS or you can use gdalwarp.
After changing the projection on the raster the rest of your code worked.
I have plotted a heatmap with the following data.
I have thousands of rows. Its just a sample. I also wanted to see the google map view of that coordinate. So I did something like this.
import folium
from folium.plugins import HeatMap
from folium.plugins import FastMarkerCluster
default_location=[11.1657, 45.4515]
m = folium.Map(location=default_location, zoom_start=13)
heat_data = [[row['lat'],row['lon']] for index, row in test.iterrows()]
# Plot it on the map
HeatMap(heat_data).add_to(m)
callback = ('function (row) {'
'var marker = L.marker(new L.LatLng(row[0], row[1]), {color: "red"});'
'var icon = L.AwesomeMarkers.icon({'
"icon: 'info-sign',"
"iconColor: 'white',"
"markerColor: 'green',"
"prefix: 'glyphicon',"
"extraClasses: 'fa-rotate-0'"
'});'
'marker.setIcon(icon);'
"var popup = L.popup({maxWidth: '300'});"
"const display_text = {text1: row[0], text2: row[1]};"
"var mytext = $(`<div id='mytext' class='display_text' style='width: 100.0%; height: 100.0%;'>\
<a href=https://https://www.google.com/maps?ll=${display_text.text1},${display_text.text2} target='_blank'>Open Google Maps</a></div>`)[0];"
"popup.setContent(mytext);"
"marker.bindPopup(popup);"
'return marker};')
m.add_child(FastMarkerCluster(heat_data, callback=callback))
# Display the map
m
Now for every gps coordinate I want to plot a small arrow or few small arrows in the angle of heading_direction and if possible show the distance_of_item in that angle from the gps coordinate. The expected outcome may be something like this.
In the above image, the location pointer is the gps coordinate, the direction and angle would be according to heading direction angle and there is a little star plotted which is the object. The object should be placed at a distance(in meters) mentioned in the dataset. I am not sure how to achieve that. Any lead or suggestions are most welcome. Thanks!
given your sample data is an image, have used alternate GPS data (UK hospitals) then added distance and direction columns as random values
given requirement is to plot a marker at location defined by distance and direction, first step is to calculate GPS co-ordinates of this.
use UTM CRS so that distance is meaningful
use high school maths to calculate x and y in UTM CRS
convert CRS back to WSG 84 so that have GPS co-ordinates
you have tagged question as plotly so I have used mapbox line and scatter traces to demonstrate building a tiled map
sample data is 1200+ hospitals, performance is decent
geopandas data frame could also be used to build folium tiles / markers. Key step is calculating the GPS co-ordinates
import geopandas as gpd
import pandas as pd
import numpy as np
import shapely
import math
import plotly.express as px
import plotly.graph_objects as go
import io, requests
# get some public addressess - hospitals. data that has GPS lat / lon
dfhos = pd.read_csv(io.StringIO(requests.get("http://media.nhschoices.nhs.uk/data/foi/Hospital.csv").text),
sep="¬",engine="python",).loc[:, ["OrganisationName", "Latitude", "Longitude"]]
# debug with fewer records
# df = dfhos.loc[0:500]
df = dfhos
# to use CRS transformations use geopandas, initial data is WSG 84, transform to UTM geometry
# directions and distances are random
gdf = gpd.GeoDataFrame(
data=df.assign(
heading_direction=lambda d: np.random.randint(0, 360, len(d)),
distance_of_item=lambda d: np.random.randint(10 ** 3, 10 ** 4, len(d)),
),
geometry=df.loc[:, ["Longitude", "Latitude"]].apply(
lambda r: shapely.geometry.Point(r["Longitude"], r["Latitude"]), axis=1
),
crs="EPSG:4326",
).pipe(lambda d: d.to_crs(d.estimate_utm_crs()))
# standard high school geometry...
def new_point(point, d, alpha):
alpha = math.radians(alpha)
return shapely.geometry.Point(
point.x + (d * math.cos(alpha)),
point.y + (d * math.sin(alpha)),
)
# calculate points based on direction and distance in UTM CRS. Then convert back to WSG 84 CRS
gdf["geometry2"] = gpd.GeoSeries(
gdf.apply(
lambda r: new_point(
r["geometry"], r["distance_of_item"], r["heading_direction"]
),
axis=1,
),
crs=gdf.geometry.crs,
).to_crs("EPSG:4326")
gdf = gdf.to_crs("EPSG:4326")
# plot lines to show start point and direct. plot markers of destinations for text of distance, etc
fig = px.line_mapbox(
lon=np.stack(
[gdf.geometry.x.values, gdf.geometry2.x.values, np.full(len(gdf), np.nan)],
axis=1,
).reshape([1, len(gdf) * 3])[0],
lat=np.stack(
[gdf.geometry.y.values, gdf.geometry2.y.values, np.full(len(gdf), np.nan)],
axis=1,
).reshape([1, len(gdf) * 3])[0],
).add_traces(
px.scatter_mapbox(
gdf,
lat=gdf.geometry2.y,
lon=gdf.geometry2.x,
hover_data=["distance_of_item", "OrganisationName"],
).data
)
# c = gdf.loc[]
fig.update_layout(mapbox={"style": "open-street-map", "zoom": 8, 'center': {'lat': 52.2316838387109, 'lon': -1.4577750831062155}}, margin={"l":0,"r":0,"t":0,"r":0})
I have a satellite image file. Loaded into dask array. I want to get pixel value (nearest) of a latitude, longitude of interest.
Satellite image is in GEOS projection. I have longitude and latitude information as 2D numpy arrays.
Satellite Image file
I have loaded it into a dask data array
from satpy import Scene
import matplotlib as plt
import os
cwd = os.getcwd()
fn = os.path.join(cwd, 'EUMETSAT_data/1Jan21/MSG1-SEVI-MSG15-0100-NA-20210101185741.815000000Z-20210101185757-1479430.nat')
files = [fn]
scn = Scene(filenames=files, reader='seviri_l1b_native')
scn.load(["VIS006"])
da = scn['VIS006']
This is what the dask array looks like:
I read lon lats from the area attribute with the help of satpy:
lon, lat = scn['VIS006'].attrs['area'].get_lonlats()
print(lon.shape)
print(lat.shape)
(1179, 808)
(1179, 808)
I get a 2d numpy array each, for longitude and latitude that are coordinates but I can not use them for slicing or selecting.
What is the best practice/method to get nearest lat long, pixel information?
How do I project the data onto lat long coordinates that I can then use for indexing to arrive at the pixel value.
At the end, I want to get pixel value (nearest) of lat long of interest.
Thanks in advance!!!
The AreaDefinition object you are using (.attrs['area']) has a few methods for getting different coordinate information.
area = scn['VIS006'].attrs['area']
col_idx, row_idx = area.get_xy_from_lonlat(lons, lats)
scn['VIS006'].values[row_idx, col_idx]
Note that row and column are flipped. The get_xy_from_lonlat method should work for arrays or scalars.
There are other methods for getting X/Y coordinates of each pixel if that is what you're interesting in.
You can find the location with following:
import numpy as np
px,py = (23.0,55.0) # some location to take out values:
dist = np.sqrt(np.cos(lat*np.pi/180.0)*(lon-px)**2+(lat-py)**2); # this is the distance matrix from point (px,py)
kkout = np.squeeze(np.where(np.abs(dist)==np.nanmin(dist))); # find location where distance is minimum
print(kkout) # you will see the row and column, where to take out data
#serge ballesta - thanks for the direction
Answering my own question.
Project the latitude and longitude (platecaree projection) onto the GEOS projection CRS. Find x and y. Use this x and y and nearest select method of xarray to get pixel value from dask array.
import cartopy.crs as ccrs
data_crs = ccrs.Geostationary(central_longitude=41.5, satellite_height=35785831, false_easting=0, false_northing=0, globe=None, sweep_axis='y')
lon = 77.541677 # longitude of interest
lat = 8.079148 # latitude of interst
# lon lat system in
x, y = data_crs.transform_point(lon, lat, src_crs=ccrs.PlateCarree())
dn = ds.sel(x=x,y=y, method='nearest')
I have a pandas dataframe containing MULTIPOLYGON coordinates in (LON, LAT) format. I need to use this coordinates to add a polygon to an ipyleaflet map but I need to change the order of the coordinates to (LAT, LON)
df['Footprint'][0]
''MULTIPOLYGON (((-3.870231 39.827106,-3.49322 41.329609,-6.624273 41.739006,-6.931492 40.237854,-3.870231 39.827106)))''
# Here in locations, I have manually changed the order
polygon = Polygon(
locations=[(39.827106,-3.870231),(41.329609,-3.49322),(41.739006,-6.624273),(40.237854,-6.931492),(39.827106,-3.870231)],
color="green",
fill_color="green"
)
m = Map(center=(39.5531, -3.6914), zoom=6)
m.add_layer(polygon);
m
Any idea on how to do the trick?