I would like to know the file formats of the following files data files in Matplotlib Basemap toolkit
countries*.dat
countriesmeta*.dat
gshhs*.dat
rivers*.dat
riversmeta*.dat
states*.dat
statesmeta*.dat
Also I would like to know if there are tools available to manipulate these files.
I have just experimented a bit:
"gshhs_c.dat" is a binary file containing a long list of lon,lat points of all coasts as single precision 32b floating point numbers:
lon1,lat1, lon2,lat2, ..., lonn,latn.
the file "gshhsmeta_c.dat" contains the connectivity information of these points:
1, area, numpoints, limit_south, limit_north, startbyte, numbytes, id-(E/W crosses dateline east or west)
In my case the first entry (eurasia) is:
1 50654050.7558 1004 1.26950 77.71625 0 8032 0-E
We can read and plot it with:
import numpy as np
import matplotlib.pyplot as plt
binfile = open('gshhs_c.dat','rb')
data = np.fromfile(binfile,'<f4')
data = data.reshape(len(data)/2,2)
plt.plot(data[:1004,0],data[:1004,1])
plt.show()
The other files should have more or less the same format because they are read in by the same function.
EDIT:
some basemap versions don't have the dateline crossing. The file format is essentially the same
Related
How to convert the NetCDF to TIFF, when the coordinates are stored in another NetCDF file (and are a irregular grid, since this covers the Arctic region)?
An example of the NetCDF file can be downloaded here: https://drive.google.com/uc?export=download&id=1i4OGCQhKlZ056H1YHq4hTb0EbEkl-pYd
The NetCDF file with the coordinates can be donwnloaded here: https://drive.google.com/uc?export=download&id=1WVzZ--NnHSPkJmBqlGwXAN7abXM5_uNh
(Just additional information files only provide the following in what regards coordinates):
NC_GLOBAL#geospatial_bounds_crs=EPSG:4326
NC_GLOBAL#geospatial_lat_max=90
NC_GLOBAL#geospatial_lat_min=57.8
NC_GLOBAL#geospatial_lon_max=180
NC_GLOBAL#geospatial_lon_min=-180
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 512.0)
Upper Right ( 512.0, 0.0)
Lower Right ( 512.0, 512.0)
Center ( 256.0, 256.0)
I know how to a do a conversion with Gdaltranslate, but the problem is if I apply it, my generated netCDF file will not be georeferenced, as lat/lon are not as coordinates, but stored as variables on another netCDF file. So below I have my so far progress trying to do this
with a GDAL-Python. It results on the a rotated image, still seems not georeferenced.
Also: it seems I managed to insert the coordinates but their name do not change to y and x, and keep as c and r, despite having changed them (see pics below).
EDIT - -
This is what I tried so far, and the output is a tiff (wrongly rotated) and no coordinates on the axis:
import xarray as xr
import numpy as np
import matplotlib.pyplot as plt
import rioxarray as rio
xds = xr.open_dataset(r'path_to_netdfc')
xdc = xr.open_dataset(r"path_to_netcdf_with_coordinates")
# Adds coordinates to x and y
xds.coords["c"] = xdc.mp_lon[1,:]
xds.coords["r"] = xdc.mp_lat[:,1]
xds
# Reorganize the netCDF file into standard names/locations
xds = xds.squeeze().rename_dims({"c": "x", "r": "y"}).transpose('y', 'x')
xds.rio.write_crs('epsg:4326', inplace=True)
#Take the variable that I'm interested in
df = xds['daily_fraction']
#It was giving me error later on, so i needed to set_spatial_dims
df = df.rio.set_spatial_dims(x_dim='x', y_dim='y')
#Save the GeoTIFF file:
df.rio.to_raster(r"C:\PHD\name_of_output.tiff")
Your files do not follow any standard that I know of. Each dimension is in its separate dataset.
If you are sure that the longitude/latitude is linear - which it might not be given that your dataset covers the polar regions - you can simply use gdal_translate to convert to TIFF and then gdal_edit.py -a_ulurll ulx uly urx ury llx lly to set a geotransform with your coordinates. But this will work only if the longitude/latitude are linear relative to your pixels.
I have a series of rgb files in png format, as well as the corresponding depth file in txt format, which can be loaded with np.loadtxt. How could I merge these two files to point cloud using open3d?
I followed the procedure as obtain point cloud from depth numpy array using open3d - python, but the result is not readable for human.
The examples is listed here:
the source png:
the pcd result:
You can get the source file from this link ![google drive] to reproduce my result.
By the way, the depth and rgb are not registerd.
Thanks.
I had to play a bit with the settings and data and used mainly the answer of your SO link.
import cv2
import numpy as np
import open3d as o3d
color = o3d.io.read_image("a542c.png")
depth = np.loadtxt("a542d.txt")
vertices = []
for x in range(depth.shape[0]):
for y in range(depth.shape[1]):
vertices.append((float(x), float(y), depth[x][y]))
pcd = o3d.geometry.PointCloud()
point_cloud = np.asarray(np.array(vertices))
pcd.points = o3d.utility.Vector3dVector(point_cloud)
pcd.estimate_normals()
pcd = pcd.normalize_normals()
o3d.visualization.draw_geometries([pcd])
However, if you keep the code as provided, the whole scene looks very weird and unfamiliar. That is because your depth file contains data between 0 and almost 2.5 m.
I introduced a cut-off at 500 or 1000 mm plus removed all 0s as suggested in the other answer. Additionally I flipped the x-axis (float(-x) instead of float(x)) to resemble your photo.
# ...
vertices = []
for x in range(depth.shape[0]):
for y in range(depth.shape[1]):
if 0< depth[x][y]<500:
vertices.append((float(-x), float(y), depth[x][y]))
For a good perspective I had to rotate the images manually. Probably open3d provides methods to do it automatically (I quickly tried pcd.transform() from your SO link above, it can help you if needed).
Results
500 mm cut-off: and 1000 mm cut-off: .
I used laspy instead of open3d because wanted to give some colors to your image:
import imageio
import numpy as np
# first reading the image for RGB values
image = imageio.imread(".../a542c.png")
loading the depth file
depth = np.loadtxt("/home/shaig93/Documents/internship_FWF/a542d.txt")
# creating fake x, y coordinates with meshgrid
xv, yv = np.meshgrid(np.arange(400), np.arange(640), indexing='ij')
# save_las is a function based on laspy that was provided to me by my supervisor
save_las("fn.laz", image[:400, :, 0].flatten(), np.c_[yv.flatten(), xv.flatten(), depth.flatten()], cmap = plt.cm.magma_r)
and the result is this. As you can see objects are visible from front.
However from side they are not easy to distinguish.
This means to me to think that your depth file is not that good.
Another idea would be also getting rid off 0 values from your depth file so that you can get point cloud without a wall kind of structure in the front. But still does not solve depth issue of course.
ps. I know this is not a proper answer but I hope it was helpful on identifying the problem.
I am new to python and am currently in the process of attempting to plot the retrograde motion of mars. I have a txt file that has R.A. and Declination in addition 12 other rows of data (like apparent magnitude etc). However, from that file I am trying to convert only the R.A and Dec. to decimal degrees in order to create a scatter plot with dec on the x axis and R.A. on the y axis. After researching I discovered that atrophy/skycoord may be the best tool to use. The problem I am having is how to code the conversion for the two specific rows of data I'm needing. Any help is greatly appreciated![][1]
I am currently in the process of attempting to plot the retrograde motion of mars. I have a txt file that has R.A. and Declination in addition 12 other rows of data (like apparent magnitude etc). However, from that file I am trying to convert only the R.A and Dec. to decimal degrees in order to create a scatter plot with dec on the x axis and R.A. on the y axis. After researching I discovered that atrophy/skycoord may be the best tool to use. The problem I am having is how to code the conversion for the two specific rows of data I'm needing. Any help is greatly appreciated!
import numpy as np
import pandas as pd
import csv
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
f = open("Mars2.txt", "r")
print(f.read())
df = pd.read_csv('Mars2.txt', sep=";", names=['Date(0 UT)','Apparent R.A.','Apparent Declination','Distance to Earth','Distance to Sun','App. Mag.','Ang. Diam.','Phase Illum','Phase Angle','S.E Long','S.E Lat','P.A Axis','Ls','Solar Elong'])
print (df)
df.plot(x ='Apparent Declination', y='Apparent R.A.', kind = 'scatter')
from astropy import units as u
from astropy.coordinates import SkyCoord
from astropy.io import ascii
c = SkyCoord(ra=10.625*u.degree, dec=41.2*u.degree, frame='icrs')
I am new to Python, so I apologize for the rudimentary programming skills, I am aware I am using a bit too much "loop for" (coming from Matlab it is dragging me down).
I have millions of points (timestep, long, lat, pointID) and hundreds of irregular non-overlapping polygons (vertex_long,vertex_lat,polygonID).points and polygons format sample
I want to know what polygon contains each point.
I was able to do it this way:
from matplotlib import path
def inpolygon(lon_point, lat_point, lon_poly, lat_poly):
shape = lon_point.shape
lon_point = lon_point.reshape(-1)
lat_point = lat_point.reshape(-1)
lon_poly = lon_poly.values.reshape(-1)
lat_poly = lat_poly.values.reshape(-1)
points = [(lon_point[i], lat_point[i]) for i in range(lon_point.shape[0])]
polys = path.Path([(lon_poly[i], lat_poly[i]) for i in range(lon_poly.shape[0])])
return polys.contains_points(points).reshape(shape)
And then
import numpy as np
import pandas as pd
Areas_Lon = Areas.iloc[:,0]
Areas_Lat = Areas.iloc[:,1]
Areas_ID = Areas.iloc[:,2]
Unique_Areas = np.unique(Areas_ID)
Areas_true=np.zeros((Areas_ID.shape[0],Unique_Areas.shape[0]))
for i in range(Areas_ID.shape[0]):
for ii in range(Unique_Areas.shape[0]):
Areas_true[i,ii]=(Areas_ID[i]==Unique_Areas[ii])
Areas_Lon_Vertex=np.zeros(Unique_Areas.shape[0],dtype=object)
Areas_Lat_Vertex=np.zeros(Unique_Areas.shape[0],dtype=object)
for i in range(Unique_Areas.shape[0]):
Areas_Lon_Vertex[i]=(Areas_Lon[(Areas_true[:,i]==1)])
Areas_Lat_Vertex[i]=(Areas_Lat[(Areas_true[:,i]==1)])
import f_inpolygon as inpolygon
Areas_in=np.zeros((Unique_Areas.shape[0],Points.shape[0]))
for i in range (Unique_Areas.shape[0]):
for ii in range (PT.shape[0]):
Areas_in[i,ii]=(inpolygon.inpolygon(Points[ii,2], Points[ii,3], Areas_Lon_Vertex[i], Areas_Lat_Vertex[i]))
This way the final outcome Areas_in Areas_in format contains as many rows as polygons and as many columns as points, where every column is true=1 at the row where the point is relative to polygon index (1st given polygon ID --> 1st row, and so).
The code works but very slowly for what it is supossed to do. When locating points in a regular grid or within a point radius I have succesfully tried implement a KDtree, what increases dramatically the speed, but I can`t do the same or whatever faster to irregular non-overlapping polygons.
I have seen some related questions but rather than asking for what polygons a point is were about whether a point is inside a polygon or not.
Any idea please?
Have you tried Geopandas Spatial join?
install the Package using pip
pip install geopandas
or conda
conda install -c conda-forge geopandas
then you should able to read the data as GeoDataframe
import geopandas
df = geopandas.read_file("file_name1.csv") # you can read shp files too.
right_df = geopandas.read_file("file_name2.csv") # you can read shp files too.
# Convert into geometry column
geometry = [Point(xy) for xy in zip(df['longitude'], df['latitude'])] # Coordinate reference system : WGS84
crs = {'init': 'epsg:4326'}
# Creating a Geographic data frame
left_df = geopandas.GeoDataFrame(df, crs=crs, geometry=geometry)
Then you can apply the sjoin
jdf = geopandas.sjoin(left_df, right_df, how='inner', op='intersects', lsuffix='left', rsuffix='right')
the option in op are:
intersects
contains
within
All should do the same in your case when you joining two geometry columns of type Polygon and Point
I am trying to use model output on flows in a tidal basin. The model uses a curvilinear grid. My first task is to just plot one component of the velocity of the highest water layer. I wrote a little bit of code based on the question under the name: Matplotlib Streamplot for Unevenly (curvilinear) Grid.
Now as far as I can see, I didn't change anything essential except for the numbers in comparison to the earlier metioned question, but the figures remain empty. I put the code and some numbers below.
import numpy as np
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
Lat = np.array([[ 30.40098833, 30.40103752, 30.40108727, 30.40113704],
[ 30.40140046, 30.40145021, 30.40149997, 30.40154973],
[ 30.40186559, 30.40191478, 30.40196453, 30.4020143 ],
[ 30.40239781, 30.402447, 30.40249676, 30.40254652]])
Lon = np.array([[-86.51729818, -86.51794126, -86.5185871, -86.51923603],
[-86.51725858, -86.51790149, -86.51854717, -86.51919595],
[-86.51721383, -86.51785659, -86.51850228, -86.51915089],
[-86.51716242, -86.51780518, -86.51845087, -86.51909948]])
Xvel = np.array([[ 0.0325774, -0.02811189, -0.04972513, -0.07736091],
[ 0.00592685, -0.00043959, -0.00735147, -0.05015078],
[-0.03365543, -0.03183309, -0.03701356, -0.07232581],
[-0.09578606, -0.10139448, -0.11220678, -0.13221299]])
plt.ion()
fig,(ax1) = plt.subplots(1,1)
m = Basemap(llcrnrlon=Lon.min(),llcrnrlat=Lat.min(),
urcrnrlon=Lon.max(), urcrnrlat=Lat.max(),
projection='merc',resolution='i',ax=ax1)
m.contourf(Lat,Lon,Xvel,latlon=True)
m.drawcoastlines()
m.drawrivers()
m.plot(Lat,Lon,'-k',alpha=0.3,latlon=True)
m.plot(Lat.T,Lon.T,'-k',alpha=0.3,latlon=True)
Could someone tell me what it is that causes the plots to remain empty?
I have another question regarding the use of Basemap: My datasheet also contains a lot of NaN's (gridpoints with no information). I was wondering how I can let Basemap know that I just don't have any information on these positions and that I don't want any plotting there. In the current code it causes an 'Points of LinearRing do not form a closed linestring' error.
Regarding the second part of your question (since Ajean appears to have solved the first half), the standard way to tell Matplotlib (and hence Basemap) to not plot data is to create a masked array. Lets say your Xvel contained NaNs, then to plot it you would do
import numpy.ma as ma
m.contourf(Lon, Lat, ma.masked_invalid(Xvel), latlon=True)
the function ma.masked_invalid, as its name implies, masks all invalid (i.e., NaN) values, so that they're not plotted.