Related
I need to plot a HEATMAP in python using x, y, z data from the excel file.
All the values of z are 1 except at (x=5,y=5). The plot should be red at point (5,5) and blue elsewhere. But I am getting false alarms which need to be removed. The COLORMAP I have used is 'jet'
X=[0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3,3,4,4,4,4,4,4,4,4,4,4,5,5,5,5,5,5,5,5,5,5,6,6,6,6,6,6,6,6,6,6,7,7,7,7,7,7,7,7,7,7,8,8,8,8,8,8,8,8,8,8,9,9,9,9,9,9,9,9,9,9]
Y=[0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9]
Z=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,9,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]
Code I have used is:
import matplotlib.pyplot as plt
import numpy as np
from numpy import ravel
from scipy.interpolate import interp2d
import pandas as pd
import matplotlib as mpl
excel_data_df = pd.read_excel('test.xlsx')
X= excel_data_df['x'].tolist()
Y= excel_data_df['y'].tolist()
Z= excel_data_df['z'].tolist()
x_list = np.array(X)
y_list = np.array(Y)
z_list = np.array(Z)
# f will be a function with two arguments (x and y coordinates),
# but those can be array_like structures too, in which case the
# result will be a matrix representing the values in the grid
# specified by those arguments
f = interp2d(x_list,y_list,z_list,kind="linear")
x_coords = np.arange(min(x_list),max(x_list))
y_coords = np.arange(min(y_list),max(y_list))
z= f(x_coords,y_coords)
fig = plt.imshow(z,
extent=[min(x_list),max(x_list),min(y_list),max(y_list)],
origin="lower", interpolation='bicubic', cmap= 'jet', aspect='auto')
# Show the positions of the sample points, just to have some reference
fig.axes.set_autoscale_on(False)
#plt.scatter(x_list,y_list,400, facecolors='none')
plt.xlabel('X Values', fontsize = 15, va="center")
plt.ylabel('Y Values', fontsize = 15,va="center")
plt.title('Heatmap', fontsize = 20)
plt.tight_layout()
plt.show()
For your ease you can also use the X, Y, Z arrays instead of reading excel file.
The result that I am getting is:
Here you can see dark blue regions at (5,0) and (0,5). These are the FALSE ALARMS I am getting and I need to REMOVE these.
I am probably doing some beginner's mistake. Grateful to anyone who points it out. Regards
There are at least three problems in your example:
x_coords and y_coords are not properly resampled;
the interpolation z does to fill in the whole grid leading to incorrect output;
the output is then forced to be plotted on the original grid (extent) that add to the confusion.
Leading to the following interpolated results:
On what you have applied an extra smoothing with imshow.
Let's create your artificial input:
import matplotlib.pyplot as plt
import numpy as np
x = np.arange(0, 11)
y = np.arange(0, 11)
X, Y = np.meshgrid(x, y)
Z = np.ones(X.shape)
Z[5,5] = 9
Depending on how you want to proceed, you can simply let imshow smooth your signal by interpolation:
fig, axe = plt.subplots()
axe.imshow(Z, origin="lower", cmap="jet", interpolation='bicubic')
And you are done, simple and efficient!
If you aim to do it by yourself, then choose the interpolant that suits you best and resample on a grid with a higher resolution:
interpolant = interpolate.interp2d(x, y, Z.ravel(), kind="linear")
xlin = np.linspace(0, 10, 101)
ylin = np.linspace(0, 10, 101)
zhat = interpolant(xlin, ylin)
fig, axe = plt.subplots()
axe.imshow(zhat, origin="lower", cmap="jet")
Have a deeper look on scipy.interpolate module to pick up the best interpolant regarding your needs. Notice that all methods does not expose the same interface for imputing parameters. You may need to reshape your data to use another objects.
MCVE
Here is a complete example using the trial data generated above. Just bind it to your excel columns:
# Flatten trial data to meet your requirement:
x = X.ravel()
y = Y.ravel()
z = Z.ravel()
# Resampling on as square grid with given resolution:
resolution = 11
xlin = np.linspace(x.min(), x.max(), resolution)
ylin = np.linspace(y.min(), y.max(), resolution)
Xlin, Ylin = np.meshgrid(xlin, ylin)
# Linear multi-dimensional interpolation:
interpolant = interpolate.NearestNDInterpolator([r for r in zip(x, y)], z)
Zhat = interpolant(Xlin.ravel(), Ylin.ravel()).reshape(Xlin.shape)
# Render and interpolate again if necessary:
fig, axe = plt.subplots()
axe.imshow(Zhat, origin="lower", cmap="jet", interpolation='bicubic')
Which renders as expected:
With matplotlib I am trying to plot 3D data as a 2D colormap. Each point has a x and a y coordinate, and a 'height' z. This height should determine the color a certain x/y region is colored in.
Here is the code I have been trying:
import random
import numpy as np
import matplotlib.pyplot as plt
x = []
y = []
z = []
for index in range(100):
a = random.random()
b = random.random()
c = np.exp(-a*a - b*b)
x.append(a)
y.append(b)
z.append(c)
cmap = plt.get_cmap('PiYG')
fig, ax = plt.subplots()
ax.pcolormesh(x, y, z, cmap=cmap)
But it gives an error
ValueError: not enough values to unpack (expected 2, got 1)
Maybe I am trying the wrong thing?
Remark: The three lists x,y,z and calculated for the example above, but in reality I have just three lists with "random" numbers in it I want to vizualize. I cannot calculate z given x and y.
I could also use imshow to create the plot I want, but I have to convert my original data into a matrix first. Maybe there is a function I can use?
pcolormesh might not be the choice for this kind of problem. pcolormesh expects ordered cell edges as data rather than random data points. You could do this if you know your grid before hand e.g.
import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(0, 1, 51)
# meshgrid makes a 2D grid of points
xx, yy = np.meshgrid(x, x)
z = np.exp(-xx**2 - yy*2)
fig, ax = plt.subplots()
ax.pcolormesh(xx, yy, z, cmap="PiYG")
which will give you
Alternatively, you could use one of the tri functions such as tripcolor with your existing setup
import random
import numpy as np
import matplotlib.pyplot as plt
x = []
y = []
z = []
for index in range(100):
a = random.random()
b = random.random()
c = np.exp(-a*a - b*b)
x.append(a)
y.append(b)
z.append(c)
fig, ax = plt.subplots()
ax.tripcolor(x, y, z, cmap="PiYG")
which will give
Note it would be simpler to use np.random to generate your data
x, y = np.random.random(size=(2, 100))
z = np.exp(-x**2 - y**2)
fig, ax = plt.subplots()
ax.tripcolor(x, y, z, cmap="PiYG")
There is an issue with x, y and z shapes: they have to be 2D arrays (matrices) but they are 1-dimensional.
In order to generate x and y axis, you could use:
x = []
y = []
for index in range(100):
x.append(random.random())
y.append(random.random())
Then you have to create a meshgrid:
X, Y = np.meshgrid(x, y)
Finally you can compute Z over the meshgrid:
Z = np.exp(-X**2 - Y**2)
In this way, your code:
cmap = plt.get_cmap('PiYG')
fig, ax = plt.subplots()
ax.pcolormesh(X, Y, Z, cmap=cmap)
gives:
If you you cannot compute Z on the meshgrid, then you should not use pcolormesh.
Some alternative could be:
3D scatterplot:
import random
import numpy as np
import matplotlib.pyplot as plt
x = []
y = []
z = []
for index in range(100):
a = random.random()
b = random.random()
c = np.exp(-a*a - b*b)
x.append(a)
y.append(b)
z.append(c)
cmap = plt.get_cmap('PiYG')
fig = plt.figure()
ax = fig.add_subplot(projection = '3d')
ax.scatter(x, y, z, cmap=cmap)
plt.show()
2D colored scatterplot:
import random
import numpy as np
import matplotlib.pyplot as plt
x = []
y = []
z = []
for index in range(100):
a = random.random()
b = random.random()
c = np.exp(-a*a - b*b)
x.append(a)
y.append(b)
z.append(c)
cmap = plt.get_cmap('PiYG')
plt.style.use('seaborn-darkgrid')
fig, ax = plt.subplots()
ax.scatter(x, y, c = z, cmap=cmap)
plt.show()
I am trying to plot hatches over contours lines that
statisfy certian criteria folliwng the example found here. Yet, I got regular contours (the yellow lines) instead of the hatches. Any ideas how to resolve that. Thanks
import matplotlib.pyplot as plt
import numpy as np
# invent some numbers, turning the x and y arrays into simple
# 2d arrays, which make combining them together easier.
x = np.linspace(-3, 5, 150).reshape(1, -1)
y = np.linspace(-3, 5, 120).reshape(-1, 1)
z = np.cos(x) + np.sin(y)
# we no longer need x and y to be 2 dimensional, so flatten them.
x, y = x.flatten(), y.flatten()
fig2, ax2 = plt.subplots()
n_levels = 6
a=ax2.contourf(x, y, z, n_levels)
fig2.colorbar(a)
[m,n]=np.where(z > 0.5)
z1=np.zeros(z.shape)
z1[m,n]=99
cs = ax2.contour(x, y, z1,2,hatches=['','.'])
plt.show()enter code here
Use contourf() with proper parameters to get useful plot with hatching. See important comment within the working code below:
import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(-3, 5, 150).reshape(1, -1)
y = np.linspace(-3, 5, 120).reshape(-1, 1)
z = np.cos(x) + np.sin(y)
x, y = x.flatten(), y.flatten()
fig2, ax2 = plt.subplots()
n_levels = 6
a = ax2.contourf(x, y, z, n_levels)
fig2.colorbar(a)
[m,n] = np.where(z > 0.5)
z1=np.zeros(z.shape)
z1[m, n] = 99
# use contourf() with proper hatch pattern and alpha value
cs = ax2.contourf(x, y, z1 ,3 , hatches=['', '..'], alpha=0.25)
plt.show()
The output plot:
I have data of a plot on two arrays that are stored in unsorted way, so the plot jumps from one place to another discontinuously:
I have tried one example of finding the closest point in a 2D array:
import numpy as np
def distance(pt_1, pt_2):
pt_1 = np.array((pt_1[0], pt_1[1]))
pt_2 = np.array((pt_2[0], pt_2[1]))
return np.linalg.norm(pt_1-pt_2)
def closest_node(node, nodes):
nodes = np.asarray(nodes)
dist_2 = np.sum((nodes - node)**2, axis=1)
return np.argmin(dist_2)
a = []
for x in range(50000):
a.append((np.random.randint(0,1000),np.random.randint(0,1000)))
some_pt = (1, 2)
closest_node(some_pt, a)
Can I use it somehow to "clean" my data? (in the above code, a can be my data)
Exemplary data from my calculations is:
array([[ 2.08937872e+001, 1.99020033e+001, 2.28260611e+001,
6.27711094e+000, 3.30392288e+000, 1.30312878e+001,
8.80768833e+000, 1.31238275e+001, 1.57400130e+001,
5.00278061e+000, 1.70752624e+001, 1.79131456e+001,
1.50746185e+001, 2.50095731e+001, 2.15895974e+001,
1.23237801e+001, 1.14860312e+001, 1.44268222e+001,
6.37680265e+000, 7.81485403e+000],
[ -1.19702178e-001, -1.14050879e-001, -1.29711421e-001,
8.32977493e-001, 7.27437322e-001, 8.94389885e-001,
8.65931116e-001, -6.08199292e-002, -8.51922900e-002,
1.12333841e-001, -9.88131292e-324, 4.94065646e-324,
-9.88131292e-324, 4.94065646e-324, 4.94065646e-324,
0.00000000e+000, 0.00000000e+000, 0.00000000e+000,
-4.94065646e-324, 0.00000000e+000]])
After using radial_sort_line (of Joe Kington) I have received the following plot:
This is actually a problem that's tougher than you might think in general.
In your exact case, you might be able to get away with sorting by the y-values. It's hard to tell for sure from the plot.
Therefore, a better approach for somewhat circular shapes like this is to do a radial sort.
For example, let's generate some data somewhat similar to yours:
import numpy as np
import matplotlib.pyplot as plt
t = np.linspace(.2, 1.6 * np.pi)
x, y = np.cos(t), np.sin(t)
# Shuffle the points...
i = np.arange(t.size)
np.random.shuffle(i)
x, y = x[i], y[i]
fig, ax = plt.subplots()
ax.plot(x, y, color='lightblue')
ax.margins(0.05)
plt.show()
Okay, now let's try to undo that shuffle by using a radial sort. We'll use the centroid of the points as the center and calculate the angle to each point, then sort by that angle:
x0, y0 = x.mean(), y.mean()
angle = np.arctan2(y - y0, x - x0)
idx = angle.argsort()
x, y = x[idx], y[idx]
fig, ax = plt.subplots()
ax.plot(x, y, color='lightblue')
ax.margins(0.05)
plt.show()
Okay, pretty close! If we were working with a closed polygon, we'd be done.
However, we have one problem -- This closes the wrong gap. We'd rather have the angle start at the position of the largest gap in the line.
Therefore, we'll need to calculate the gap to each adjacent point on our new line and re-do the sort based on a new starting angle:
dx = np.diff(np.append(x, x[-1]))
dy = np.diff(np.append(y, y[-1]))
max_gap = np.abs(np.hypot(dx, dy)).argmax() + 1
x = np.append(x[max_gap:], x[:max_gap])
y = np.append(y[max_gap:], y[:max_gap])
Which results in:
As a complete, stand-alone example:
import numpy as np
import matplotlib.pyplot as plt
def main():
x, y = generate_data()
plot(x, y).set(title='Original data')
x, y = radial_sort_line(x, y)
plot(x, y).set(title='Sorted data')
plt.show()
def generate_data(num=50):
t = np.linspace(.2, 1.6 * np.pi, num)
x, y = np.cos(t), np.sin(t)
# Shuffle the points...
i = np.arange(t.size)
np.random.shuffle(i)
x, y = x[i], y[i]
return x, y
def radial_sort_line(x, y):
"""Sort unordered verts of an unclosed line by angle from their center."""
# Radial sort
x0, y0 = x.mean(), y.mean()
angle = np.arctan2(y - y0, x - x0)
idx = angle.argsort()
x, y = x[idx], y[idx]
# Split at opening in line
dx = np.diff(np.append(x, x[-1]))
dy = np.diff(np.append(y, y[-1]))
max_gap = np.abs(np.hypot(dx, dy)).argmax() + 1
x = np.append(x[max_gap:], x[:max_gap])
y = np.append(y[max_gap:], y[:max_gap])
return x, y
def plot(x, y):
fig, ax = plt.subplots()
ax.plot(x, y, color='lightblue')
ax.margins(0.05)
return ax
main()
Sorting the data base on their angle relative to the center as in #JoeKington 's solution might have problems with some parts of the data:
In [1]:
import scipy.spatial as ss
import matplotlib.pyplot as plt
import numpy as np
import re
%matplotlib inline
In [2]:
data=np.array([[ 2.08937872e+001, 1.99020033e+001, 2.28260611e+001,
6.27711094e+000, 3.30392288e+000, 1.30312878e+001,
8.80768833e+000, 1.31238275e+001, 1.57400130e+001,
5.00278061e+000, 1.70752624e+001, 1.79131456e+001,
1.50746185e+001, 2.50095731e+001, 2.15895974e+001,
1.23237801e+001, 1.14860312e+001, 1.44268222e+001,
6.37680265e+000, 7.81485403e+000],
[ -1.19702178e-001, -1.14050879e-001, -1.29711421e-001,
8.32977493e-001, 7.27437322e-001, 8.94389885e-001,
8.65931116e-001, -6.08199292e-002, -8.51922900e-002,
1.12333841e-001, -9.88131292e-324, 4.94065646e-324,
-9.88131292e-324, 4.94065646e-324, 4.94065646e-324,
0.00000000e+000, 0.00000000e+000, 0.00000000e+000,
-4.94065646e-324, 0.00000000e+000]])
In [3]:
plt.plot(data[0], data[1])
plt.title('Unsorted Data')
Out[3]:
<matplotlib.text.Text at 0x10a5c0550>
See x values between 15 and 20 are not sorted correctly.
In [10]:
#Calculate the angle in degrees of [0, 360]
sort_index = np.angle(np.dot((data.T-data.mean(1)), np.array([1.0, 1.0j])))
sort_index = np.where(sort_index>0, sort_index, sort_index+360)
#sorted the data by angle and plot them
sort_index = sort_index.argsort()
plt.plot(data[0][sort_index], data[1][sort_index])
plt.title('Data Sorted by angle relatively to the centroid')
plt.plot(data[0], data[1], 'r+')
Out[10]:
[<matplotlib.lines.Line2D at 0x10b009e10>]
We can sort the data based on a nearest neighbor approach, but since the x and y are of very different scale, the choice of distance metrics becomes an important issue. We will just try all the distance metrics available in scipy to get an idea:
In [7]:
def sort_dots(metrics, ax, start):
dist_m = ss.distance.squareform(ss.distance.pdist(data.T, metrics))
total_points = data.shape[1]
points_index = set(range(total_points))
sorted_index = []
target = start
ax.plot(data[0, target], data[1, target], 'o', markersize=16)
points_index.discard(target)
while len(points_index)>0:
candidate = list(points_index)
nneigbour = candidate[dist_m[target, candidate].argmin()]
points_index.discard(nneigbour)
points_index.discard(target)
#print points_index, target, nneigbour
sorted_index.append(target)
target = nneigbour
sorted_index.append(target)
ax.plot(data[0][sorted_index], data[1][sorted_index])
ax.set_title(metrics)
In [6]:
dmetrics = re.findall('pdist\(X\,\s+\'(.*)\'', ss.distance.pdist.__doc__)
In [8]:
f, axes = plt.subplots(4, 6, figsize=(16,10), sharex=True, sharey=True)
axes = axes.ravel()
for metrics, ax in zip(dmetrics, axes):
try:
sort_dots(metrics, ax, 5)
except:
ax.set_title(metrics + '(unsuitable)')
It looks like standardized euclidean and mahanalobis metrics give the best result. Note that we choose a starting point of the 6th data (index 5), it is the data point this the largest y value (use argmax to get the index, of course).
In [9]:
f, axes = plt.subplots(4, 6, figsize=(16,10), sharex=True, sharey=True)
axes = axes.ravel()
for metrics, ax in zip(dmetrics, axes):
try:
sort_dots(metrics, ax, 13)
except:
ax.set_title(metrics + '(unsuitable)')
This is what happens if you choose the starting point of max. x value (index 13). It appears that mahanalobis metrics is better than standardized euclidean as it is not affected by the starting point we choose.
If we do the assumption that the data are 2D and the x axis should be in an increasing fashion, then you could:
sort the x axis data, e.g. x_old and store the result in a different variable, e.g. x_new
for each element in the x_new find its index in the x_old array
re-order the elements in the y_axis array according to the indices that you got from previous step
I would do it with python list instead of numpy array due to list.index method been more easily manipulated than the numpy.where method.
E.g. (and assume that x_old and y_old are your previous numpy variables for x and y axis respectively)
import numpy as np
x_new_tmp = x_old.tolist()
y_new_tmp = y_old.tolist()
x_new = sorted(x_new_tmp)
y_new = [y_new_tmp[x_new_tmp.index(i)] for i in x_new]
Then you can plot x_new and y_new
I am trying to make a contour plot like:
Using a table of data like 3 columns in a txt file, with a long number of lines.
Using this code:
import numpy as np
import matplotlib.pyplot as plt
import scipy.interpolate
data = np.loadtxt(r'dataa.txt')
a = [data[:,0]]
b = [data[:,1]]
n = [data[:,2]]
x = np.asarray(a)
y = np.asarray(b)
z = np.asarray(n)
print "x = ", x
print "y = ", y
print "z = ", z
fig=plt.figure()
CF = contour(x,y,z,colors = 'k')
plt.xlabel("X")
plt.ylabel("Y")
plt.colorbar()
plt.show()
I don't know why, it is not working. Python gives me the right axes for the values that I am expecting to see, but in the graph is just a blank and I know that it is importing the data in right way because it shows me my values before the plot.
Example of table: (the diference is because my table has 90000 lines)
Using this code:
import numpy as np
import matplotlib.pyplot as plt
import scipy.interpolate
N = 1000 #number of points for plotting/interpolation
x, y, z = np.genfromtxt(r'dataa.txt', unpack=True)
xi = np.linspace(x.min(), x.max(), N)
yi = np.linspace(y.min(), y.max(), N)
zi = scipy.interpolate.griddata((x, y), z, (xi[None,:], yi[:,None]), method='cubic')
fig = plt.figure()
plt.contour(xi, yi, zi)
plt.xlabel("X")
plt.ylabel("Y")
plt.show()
Ive got this result:
I think I've got the advices wrongly.
Followup from my comment... first, I would replace all these lines:
data = np.loadtxt(r'dataa.txt')
a = [data[:,0]]
b = [data[:,1]]
n = [data[:,2]]
x = np.asarray(a)
y = np.asarray(b)
z = np.asarray(n)
With:
x, y, z = np.genfromtxt(r'dataa.txt', unpack=True)
Your original code is adding an extra axis at the front, since [data[:,0]] is a list of arrays with one element. The result is that x.shape will be (1, N) instead if (N,). All of this can be done automatically using the last line above, or you could just use the same data loading and say:
x = data[:,0]
y = data[:,1]
z = data[:,2]
since those slices will give you an array back.
However, you're not quite done, because plt.contour expects you to give it a 2d array for z, not a 1d array of values. Right now, you seem to have z values at given x, y points, but contour expects you to give it a 2d array, like an image.
Before I can answer that, I need to know how x and y are spaced. If regularly, you can just populate an array pretty easily. If not regularly, you basically have to interpolate before you can make a contour plot.
To do the interpolation, use
import numpy as np
import matplotlib.pyplot as plt
import scipy.interpolate
N = 1000 #number of points for plotting/interpolation
x, y, z = np.genfromtxt(r'dataa.txt', unpack=True)
xi = np.linspace(x.min(), x.max(), N)
yi = np.linspace(y.min(), y.max(), N)
zi = scipy.interpolate.griddata((x, y), z, (xi[None,:], yi[:,None]), method='cubic')
fig = plt.figure()
plt.contour(xi, yi, zi)
plt.xlabel("X")
plt.ylabel("Y")
plt.show()
The code below worked for me:
import scipy.interpolate
import numpy as np
N = 500 #number of points for plotting/interpolation
x, y, z = np.genfromtxt(r'data.dat', unpack=True)
xll = x.min(); xul = x.max(); yll = y.min(); yul = y.max()
xi = np.linspace(xll, xul, N)
yi = np.linspace(yll, yul, N)
zi = scipy.interpolate.griddata((x, y), z, (xi[None,:], yi[:,None]), method='cubic')
contours = plt.contour(xi, yi, zi, 6, colors='black')
plt.clabel(contours, inline=True, fontsize=7)
plt.imshow(zi, extent=[xll, xul, yll, yul], origin='lower', cmap=plt.cm.jet, alpha=0.9)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.clim(0, 1)
plt.colorbar()
plt.show()