We Keep Coding

Plotting a heatmap with interpolation in Python using excel file

I need to plot a HEATMAP in python using x, y, z data from the excel file.
All the values of z are 1 except at (x=5,y=5). The plot should be red at point (5,5) and blue elsewhere. But I am getting false alarms which need to be removed. The COLORMAP I have used is 'jet'
X=[0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3,3,4,4,4,4,4,4,4,4,4,4,5,5,5,5,5,5,5,5,5,5,6,6,6,6,6,6,6,6,6,6,7,7,7,7,7,7,7,7,7,7,8,8,8,8,8,8,8,8,8,8,9,9,9,9,9,9,9,9,9,9]
Y=[0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9]
Z=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,9,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]
Code I have used is:
import matplotlib.pyplot as plt
import numpy as np
from numpy import ravel
from scipy.interpolate import interp2d
import pandas as pd
import matplotlib as mpl
excel_data_df = pd.read_excel('test.xlsx')
X= excel_data_df['x'].tolist()
Y= excel_data_df['y'].tolist()
Z= excel_data_df['z'].tolist()
x_list = np.array(X)
y_list = np.array(Y)
z_list = np.array(Z)
# f will be a function with two arguments (x and y coordinates),
# but those can be array_like structures too, in which case the
# result will be a matrix representing the values in the grid
# specified by those arguments
f = interp2d(x_list,y_list,z_list,kind="linear")
x_coords = np.arange(min(x_list),max(x_list))
y_coords = np.arange(min(y_list),max(y_list))
z= f(x_coords,y_coords)
fig = plt.imshow(z,
extent=[min(x_list),max(x_list),min(y_list),max(y_list)],
origin="lower", interpolation='bicubic', cmap= 'jet', aspect='auto')
# Show the positions of the sample points, just to have some reference
fig.axes.set_autoscale_on(False)
#plt.scatter(x_list,y_list,400, facecolors='none')
plt.xlabel('X Values', fontsize = 15, va="center")
plt.ylabel('Y Values', fontsize = 15,va="center")
plt.title('Heatmap', fontsize = 20)
plt.tight_layout()
plt.show()
For your ease you can also use the X, Y, Z arrays instead of reading excel file.
The result that I am getting is:
Here you can see dark blue regions at (5,0) and (0,5). These are the FALSE ALARMS I am getting and I need to REMOVE these.
I am probably doing some beginner's mistake. Grateful to anyone who points it out. Regards

There are at least three problems in your example:
x_coords and y_coords are not properly resampled;
the interpolation z does to fill in the whole grid leading to incorrect output;
the output is then forced to be plotted on the original grid (extent) that add to the confusion.
Leading to the following interpolated results:
On what you have applied an extra smoothing with imshow.
Let's create your artificial input:
import matplotlib.pyplot as plt
import numpy as np
x = np.arange(0, 11)
y = np.arange(0, 11)
X, Y = np.meshgrid(x, y)
Z = np.ones(X.shape)
Z[5,5] = 9
Depending on how you want to proceed, you can simply let imshow smooth your signal by interpolation:
fig, axe = plt.subplots()
axe.imshow(Z, origin="lower", cmap="jet", interpolation='bicubic')
And you are done, simple and efficient!
If you aim to do it by yourself, then choose the interpolant that suits you best and resample on a grid with a higher resolution:
interpolant = interpolate.interp2d(x, y, Z.ravel(), kind="linear")
xlin = np.linspace(0, 10, 101)
ylin = np.linspace(0, 10, 101)
zhat = interpolant(xlin, ylin)
fig, axe = plt.subplots()
axe.imshow(zhat, origin="lower", cmap="jet")
Have a deeper look on scipy.interpolate module to pick up the best interpolant regarding your needs. Notice that all methods does not expose the same interface for imputing parameters. You may need to reshape your data to use another objects.
MCVE
Here is a complete example using the trial data generated above. Just bind it to your excel columns:
# Flatten trial data to meet your requirement:
x = X.ravel()
y = Y.ravel()
z = Z.ravel()
# Resampling on as square grid with given resolution:
resolution = 11
xlin = np.linspace(x.min(), x.max(), resolution)
ylin = np.linspace(y.min(), y.max(), resolution)
Xlin, Ylin = np.meshgrid(xlin, ylin)
# Linear multi-dimensional interpolation:
interpolant = interpolate.NearestNDInterpolator([r for r in zip(x, y)], z)
Zhat = interpolant(Xlin.ravel(), Ylin.ravel()).reshape(Xlin.shape)
# Render and interpolate again if necessary:
fig, axe = plt.subplots()
axe.imshow(Zhat, origin="lower", cmap="jet", interpolation='bicubic')
Which renders as expected:

Cartopy projection scale not consistent

I am using cartopy to display a KDE overlayed on a world map. Initially, I was using the ccrs.PlateCarree projection with no issues, but the moment I tried to use another projection it seemed to explode the scale of the projection. For reference, I have included an example that you can test on your own machine below (just comment out the two projec lines to switch between projections)
from scipy.stats import gaussian_kde
import numpy as np
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.feature as cfeature
projec = ccrs.PlateCarree()
#projec = ccrs.InterruptedGoodeHomolosine()
fig = plt.figure(figsize=(12, 12))
ax = fig.add_subplot(projection=projec)
np.random.seed(1)
discrete_points = np.random.randint(0,10,size=(2,400))
kde = gaussian_kde(discrete_points)
x, y = discrete_points
# https://www.oreilly.com/library/view/python-data-science/9781491912126/ch04.html
resolution = 1
x_step = int((max(x)-min(x))/resolution)
y_step = int((max(y)-min(y))/resolution)
xgrid = np.linspace(min(x), max(x), x_step+1)
ygrid = np.linspace(min(y), max(y), y_step+1)
Xgrid, Ygrid = np.meshgrid(xgrid, ygrid)
Z = kde.evaluate(np.vstack([Xgrid.ravel(), Ygrid.ravel()]))
Zgrid = Z.reshape(Xgrid.shape)
ext = [min(x)*5, max(x)*5, min(y)*5, max(y)*5]
earth = plt.cm.gist_earth_r
ax.add_feature(cfeature.NaturalEarthFeature('physical', 'land', '50m',
edgecolor='black', facecolor="none"))
ax.imshow(Zgrid,
origin='lower', aspect='auto',
extent=ext,
alpha=0.8,
cmap=earth, transform=projec)
ax.axis('on')
ax.get_xaxis().set_visible(True)
ax.get_yaxis().set_visible(True)
ax.set_xlim(-30, 90)
ax.set_ylim(-60, 60)
plt.show()
You'll notice that when using the ccrs.PlateCarree() projection, the KDE is nicely placed over Africa, however when using the ccrs.InterruptedGoodeHomolosine() projection, you can't see the world map at all. This is because the world map is on an enormous scale. Below is an image of both examples:
Plate Carree projection:
Interrupted Goode Homolosine projection (standard zoom):
Interrupted Goode Homolosine projection (zoomed out):
If anyone could explain why this is occurring, and how to fix it so I can plot the same data on different projections, that would be greatly appreciated.
EDIT:
I would also like to specify that I tried adding transform=projec to line 37 in the example I included, namely:
ax.add_feature(cfeature.NaturalEarthFeature('physical', 'land', '50m',
edgecolor='black', facecolor="none", transform=projec))
However this did not help. In fact, it seemed upon adding this the world map no longer appeared at all.
EDIT:
In response to JohanC's answer, this is the plot I get when using that code:
And zoomed out:

Comments on your plots:
Plot1: (the reference map)
projection: PlateCarree projection
(Zgrid) image extents cover (approx) square area, about 40 degrees on each side
image's lower-left corner is at lat/long: (0,0)
Plot2
Q: Why the topo features are not shown on the map?
A: The plot covers very small area that does not include any of them.
projection: InterruptedGoodeHomolosine
the image data, Zgrid is declared to fit within grid (mapprojection) coordinates (unit: meters)
the map is plotted within a small extents of a few meters in both x and y, and aspect ratio is not equal.
Plot3
Q: Why the Zgrid image are not seen on the map?
A: The plot covers very large area that the image become too small to plot.
projection: InterruptedGoodeHomolosine projection
the (Zgrid) image extent is very small, not visible at this scale
the map is plotted within a large extents, and aspect ratio is not equal.
The remedies (for Plot2 and 3)
Zgrid need proper transformation from lat/long to the axes' projection coordinates
map's extents also need to be transformed and set appropriately
the aspect ratio must be set 'equal', to prevent unequal stretches in x and y
About 'gridlines' plots
useful for location reference
latitude/parallels: OK with InterruptedGoodeHomolosine in this case
longitude/meridians: is problematic (dont know how to fix !!)
Here is the modified code that runs and produces the required map.
# proposed code
from scipy.stats import gaussian_kde
import numpy as np
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.feature as cfeature
fig = plt.figure(figsize=(7, 12))
ax = plt.axes(projection=ccrs.InterruptedGoodeHomolosine())
np.random.seed(1)
discrete_points = np.random.randint(0,10,size=(2,400))
kde = gaussian_kde(discrete_points)
x, y = discrete_points
# https://www.oreilly.com/library/view/python-data-science/9781491912126/ch04.html
resolution = 1
x_step = int((max(x)-min(x))/resolution)
y_step = int((max(y)-min(y))/resolution)
xgrid = np.linspace(min(x), max(x), x_step+1)
ygrid = np.linspace(min(y), max(y), y_step+1)
Xgrid, Ygrid = np.meshgrid(xgrid, ygrid)
Z = kde.evaluate(np.vstack([Xgrid.ravel(), Ygrid.ravel()]))
Zgrid = Z.reshape(Xgrid.shape)
ext = [min(x)*5, max(x)*5, min(y)*5, max(y)*5]
earth = plt.cm.gist_earth_r
ocean110 = cfeature.NaturalEarthFeature('physical', 'ocean', \
scale='110m', edgecolor='none', facecolor=cfeature.COLORS['water'])
ax.add_feature(ocean110, zorder=-5)
land110 = cfeature.NaturalEarthFeature('physical', 'land', '110m', \
edgecolor='black', facecolor="silver")
ax.add_feature(land110, zorder=5)
# extents used by both Zgrid and axes
ext = [min(x)*5, max(x)*5, min(y)*5, max(y)*5]
# plot the image's data array
# note the options: `extent` and `transform`
ax.imshow(Zgrid,
origin='lower', aspect='auto',
extent=ext, #set image's extent
alpha=0.75,
cmap=earth, transform=ccrs.PlateCarree(),
zorder=10)
# set the plot's extent with proper coord transformation
ax.set_extent(ext, ccrs.PlateCarree())
ax.coastlines()
#ax.add_feature(cfeature.BORDERS) #uncomment if you need
ax.gridlines(linestyle=':', linewidth=1, draw_labels=True, dms=True, zorder=30, color='k')
ax.set_aspect('equal') #make sure the aspect ratio is 1
plt.show()
The output map:

Matplotlib contour plot with nonuniform, matrix grids

I would like to contour plot a function, f(x,y), against x and x-y. The spacing in the y grid is not the same as the x grid, so x-y is 2 dimensional, whereas x is one-dimensional.
I do not know how to set up the grids. The function, tricontourf, can handle non-uniform grids, but only it seems, if both the axes are one-dimensional. contour can handle matrices, but only for f(x,y), whereas I need one of the axes to be a matrix.
Pseudocode would look like the following:
import matplotlib.pyplot as plt
def twoDfunction(x,y):
return x + y # my function is more complicated than this
xaxis = np.linspace(0,10,100)
yaxis = np.linspace(0,10,22)
xminusyaxis = np.subtract(xaxis,yaxis)
functionsurfacevalues = twoDfunction(xaxis,yaxis)
fig =plt.figure(figsize=(10,10),dpi=300,facecolor='w')
ax1 = plt.subplot(111)
ax1.tricontourf(xaxis, xminusyaxis, functionsurfacevalues)
I would like the pseudocode to plot functionsurfacevalues versus x and xminusy.

What you need to do is create your grid using np.meshgrid() and then plot a contour or contourf plot.np.meshgrid will make irregular grids based on whatever you give it. You do not need a surface plot because your data isn't really a surface.
The main problem you are having is that because your x and y axis are different lengths, you can't subtract them. Otherwise the solution is easy and you can follow the following code.
import matplotlib.pyplot as plt
def twoDfunction(x,y):
return (x + y) # my function is more complicated than this
xaxis = np.linspace(0,10,100)
yaxis = np.linspace(0,5,100)
xminusyaxis = np.subtract(xaxis,yaxis)
xx,yy = np.meshgrid(xaxis,xminusyaxis)
fig =plt.figure(figsize=(10,10),dpi=300,facecolor='w')
ax1 = plt.subplot(111)
ax1.contourf(xx, yy, twoDfunction(xx,yy))
plt.show()

Is there a way to plot a polar heatmap incrementally?

I am trying to have a polar heatmap appear incrementally. I want the plot to grow by adding a deltasector to the existing plot. The same maximal radius is always used.
For now I replot the old data as well, but that is only because I do not know how to add to the existing plot.
How do I add z values for the new angle to an existing heatmap?
The accepted answer here gives shows how to plot a polar heatmap:
Polar heatmaps in python
In the code below the z values are calculated as a function of the r and th. My situation is however that I read the values from a file instead.
How would I add them to the heatmap?
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import numpy as np
import time
fig = plt.figure()
ax = Axes3D(fig)
angle = 0
rad = np.linspace(0, 5, 100)
d_angle = np.pi/100
while angle < np.pi:
azm = np.linspace(0, angle, 100)
r, th = np.meshgrid(rad, azm)
z = r/5
plt.subplot(projection="polar")
plt.pcolormesh(th, r, z)
plt.plot(azm, r, color='k', ls='none')
plt.grid()
plt.ion()
plt.show()
plt.pause(0.0001)
plt.clf()
angle += d_angle
I do not know where to start. Any pointers to docs? Or other advices?

You can retrieve the data from a plot by looking into ax.lines. Add a "gid" to your curve like so plt.plot(azm, r, color='k', ls='none', gid="a custom name") then we have a little work to do:
def append_data_to_curve(ax, gid):
for line in ax.lines: # Check every curve.
if line.get_gid() == "a custom name": # If the one you seek is found:
X = line.get_xdata() # Get its X and Y data.
Y = line.get_ydata()
X.append(x) # Add the new point (x,y) you want.
Y.append(y)
line.set_xdata(X) # Put back the modified list as curve data.
line.set_ydata(Y)
You can call this function for every iteration of a loop and add a single new point by giving it its (x,y) coordinates.

Contour plot from xy data in python

I want to plot contour lines for 2D data (xy) representing 68%, 95%, 99.7% of the data. My question is almost similar to this question but I do not have Z-axis values. Is there any possibility to do it in python ?.
EDIT: From the reference question, I have applied the same code for my problem. (This is half code)
from scipy.stats import gaussian_kde
import matplotlib.pyplot as plt
import numpy as np
x = X_data
y = Y_data
k = gaussian_kde(np.vstack([X_data, Y_data]))
xi, yi = np.mgrid[x.min():x.max():x.size**0.5*1j,y.min():y.max():y.size**0.5*1j]
zi = k(np.vstack([xi.flatten(), yi.flatten()]))
#set zi to 0-1 scale
zi = (zi-zi.min())/(zi.max() - zi.min())
zi =zi.reshape(xi.shape)
plt.plot(X_data, Y_data, 'bo', ms =1)
#set up plot
origin = 'lower'
levels = [0,0.1,0.25,0.5,0.68, 0.95, 0.975,1]
CS = plt.contour(xi, yi, zi,levels = levels,
colors=('k',),
linewidths=(3,),
origin=origin)
plt.clabel(CS, fmt='%.3f', colors='b', fontsize=8)
plt.xlabel(r'$X_{data} [V]$', fontsize=fontall)
plt.ylabel(r'$Y_{data} [A]$', fontsize=fontall)
plt.ticklabel_format(axis='y', style='sci', scilimits=(0,0))
plt.show()
and I got this figure:
I want to understand is this the correct result ?. I want to see the contour levels that represent 68%, 95% and 99.7% of my data.