Related
I have created a simple visualizing function in python using pyplot. It takes a dataframe, an upper and lower limit and the start/end points to visualize. Here is the full code:
def visualize(DATASET, DATASET_LIMITS, DATASET_START, DATASET_END):
# DATASET = df_I
# DATASET_LIMITS = df_I_limits
# DATASET_START = 0
# DATASET_END = len(df_I)
plt.figure(figsize=(20,10))
values = []
values_above = []
for data in DATASET['Temp'].iloc[DATASET_START:DATASET_END]:
if data < DATASET_LIMITS[0] or data > DATASET_LIMITS[1]:
values.append(math.nan)
values_above.append(data)
else:
values.append(data)
values_above.append(math.nan)
plt.plot(range(0, DATASET_END - DATASET_START), values, 'b-')
plt.plot(range(0, DATASET_END - DATASET_START), values_above, 'r-')
plt.hlines(DATASET_LIMITS[0], 0, DATASET_END - DATASET_START, colors='g', linestyles='dashed')
plt.hlines(DATASET_LIMITS[1], 0, DATASET_END - DATASET_START, colors='g', linestyles='dashed')
plt.show()
Here is what a generated graph looks like:
You can already see some gaps where the dotted-greed limit line bisects the graph, but here's a zoomed in version to show the problem more clearly. Here is just the largest spike in the data, in the ~205000 range:
You can clearly see that the red and blue segments of the graph are not connected. I believe this is likely due to the method that I use to visualize the data, that being two arrays - values and values_above. Is there a better way of accomplishing this graphing behaviour? Or perhaps a way to get connected lines using this approach?
Since you didn't provide dataset, I'm going to post a simple function I usually use on this occasions.
You take the x and y coordinates from your dataframe and pass them to this function: modify_coords(x, y, y_lim) where y_lim is the y coordinates of the your horizontal line. This function will insert new points where the intersection happens. Then, you can proceed with your usual code.
def modify_coords(x, y, y_lim):
"""If a line segment defined by `(x1, y1) -> (x2, y2)` intercepts
a limiting y-value, divide this segment by inserting a new point
such that y_newpoint = y_lim.
"""
xv, yv = [x[0]], [y[0]]
for i in range(len(x) - 1):
xc, xn = x[i:i+2]
yc, yn = y[i:i+2]
if ((yc < y_lim) and (yn > y_lim)) or ((yc > y_lim) and (yn < y_lim)):
xv.append(((y_lim - yc) / ((yn - yc) / (xn - xc))) + xc)
yv.append(y_lim)
xv.append(xn)
yv.append(yn)
return np.array(xv), np.array(yv)
Note:
it uses linear interpolation to add the new points.
you can see this function in action at this question/answer, where I used a much smoother function as a test. You should give it a try on your data and eventually adapt it to achieve your goal.
I'm working on a Python-based data analysis. I have some x-y data points, and some ellipses, and I want to determine whether points are inside any of the ellipses. The way that I've been doing this works, but it's kludgy. As I think about distributing my software to other people, I find myself wanting a cleaner way.
Right now, I'm using matplotlib.patches.Ellipse objects. Matplotlib Ellipses have a useful method called contains_point(). You can work in data coordinates on a Matplotlib Axes object by calling Axes.transData.transform().
The catch is that I have to create a Figure and an Axes object to hold the Ellipses. And when my program runs, an annoying Matplotlib Figure object will get rendered, showing the Ellipses, which I don't actually need to see. I have tried several methods to suppress this output. I have succeeded in deleting the Ellipses from the Axes, using Axes.clear(), resulting in an empty graph. But I can't get Matplolib's pyplot.close(fig_number) to delete the Figure itself before calling pyplot.show().
Any advice is appreciated, thanks!
Inspired by how a carpenter draws an ellipse using two nails and a piece of string, here is a numpy-friendly implementation to test whether points lie inside given ellipses.
One of the definitions of an ellipse, is that the sum of the distances to the two foci is constant, equal to the width (or height if it would be larger) of the ellipse. The distance between the center and the foci is sqrt(a*a - b*b), where a and b are half of the width and height. Using that distance and rotation by the desired angle finds the locations of the foci. numpy.linalg.norm can be used to calculate the distances using numpy's efficient array operations.
After the calculations, a plot is generated to visually check whether everything went correct.
import numpy as np
from numpy.linalg import norm # calculate the length of a vector
x = np.random.uniform(0, 40, 20000)
y = np.random.uniform(0, 20, 20000)
xy = np.dstack((x, y))
el_cent = np.array([20, 10])
el_width = 28
el_height = 17
el_angle = 20
# distance between the center and the foci
foc_dist = np.sqrt(np.abs(el_height * el_height - el_width * el_width) / 4)
# vector from center to one of the foci
foc_vect = np.array([foc_dist * np.cos(el_angle * np.pi / 180), foc_dist * np.sin(el_angle * np.pi / 180)])
# the two foci
el_foc1 = el_cent + foc_vect
el_foc2 = el_cent - foc_vect
# for each x,y: calculate z as the sum of the distances to the foci;
# np.ravel is needed to change the array of arrays (of 1 element) into a single array
z = np.ravel(norm(xy - el_foc1, axis=-1) + norm(xy - el_foc2, axis=-1) )
# points are exactly on the ellipse when the sum of distances is equal to the width
# z = np.where(z <= max(el_width, el_height), 1, 0)
# now create a plot to check whether everything makes sense
from matplotlib import pyplot as plt
from matplotlib import patches as mpatches
fig, ax = plt.subplots()
# show the foci as red dots
plt.plot(*el_foc1, 'ro')
plt.plot(*el_foc2, 'ro')
# create a filter to separate the points inside the ellipse
filter = z <= max(el_width, el_height)
# draw all the points inside the ellipse with the plasma colormap
ax.scatter(x[filter], y[filter], s=5, c=z[filter], cmap='plasma')
# draw all the points outside with the cool colormap
ax.scatter(x[~filter], y[~filter], s=5, c=z[~filter], cmap='cool')
# add the original ellipse to verify that the boundaries match
ellipse = mpatches.Ellipse(xy=el_cent, width=el_width, height=el_height, angle=el_angle,
facecolor='None', edgecolor='black', linewidth=2,
transform=ax.transData)
ax.add_patch(ellipse)
ax.set_aspect('equal', 'box')
ax.autoscale(enable=True, axis='both', tight=True)
plt.show()
The simplest solution here is to use shapely.
If you have an array of shape Nx2 containing a set of vertices (xy) then it is trivial to construct the appropriate shapely.geometry.polygon object and check if an arbitrary point or set of points (points) is contained within -
import shapely.geometry as geom
ellipse = geom.Polygon(xy)
for p in points:
if ellipse.contains(geom.Point(p)):
# ...
Alternatively, if the ellipses are defined by their parameters (i.e. rotation angle, semimajor and semiminor axis) then the array containing the vertices must be constructed and then the same process applied. I would recommend using the polar form relative to center as this is the most compatible with how shapely constructs the polygons.
import shapely.geometry as geom
from shapely import affinity
n = 360
a = 2
b = 1
angle = 45
theta = np.linspace(0, np.pi*2, n)
r = a * b / np.sqrt((b * np.cos(theta))**2 + (a * np.sin(theta))**2)
xy = np.stack([r * np.cos(theta), r * np.sin(theta)], 1)
ellipse = affinity.rotate(geom.Polygon(xy), angle, 'center')
for p in points:
if ellipse.contains(geom.Point(p)):
# ...
This method is advantageous because it supports any properly defined polygons - not just ellipses, it doesn't rely on matplotlib methods to perform the containment checking, and it produces a very readable code (which is often important when "distributing [one's] software to other people").
Here is a complete example (with added plotting to show it working)
import shapely.geometry as geom
from shapely import affinity
import matplotlib.pyplot as plt
import numpy as np
n = 360
theta = np.linspace(0, np.pi*2, n)
a = 2
b = 1
angle = 45.0
r = a * b / np.sqrt((b * np.cos(theta))**2 + (a * np.sin(theta))**2)
xy = np.stack([r * np.cos(theta), r * np.sin(theta)], 1)
ellipse = affinity.rotate(geom.Polygon(xy), angle, 'center')
x, y = ellipse.exterior.xy
# Create a Nx2 array of points at grid coordinates throughout
# the ellipse extent
rnd = np.array([[i,j] for i in np.linspace(min(x),max(x),50)
for j in np.linspace(min(y),max(y),50)])
# Filter for points which are contained in the ellipse
res = np.array([p for p in rnd if ellipse.contains(geom.Point(p))])
plt.plot(x, y, lw = 1, color='k')
plt.scatter(rnd[:,0], rnd[:,1], s = 50, color=(0.68, 0.78, 0.91)
plt.scatter(res[:,0], res[:,1], s = 15, color=(0.12, 0.67, 0.71))
plt.show()
I am trying to calculate the FWHM of spectra using python. The spectral description (I'm talking in terms of the physics) for me it's bit complicated and I can't fit the data using some simple Gaussian or Lorentizian profile.
So far I managed to manage interpolation of the data and draw a straight line parallel to the X axis through the half maxima.
How can I find the coordinates of the intersection of the two lines on both sides of the peak?
I know if I take the cursor in those points it will give me the coordinates but I want to automate this process so that it becomes much more user friendly. How can I do that?
from matplotlib import pyplot as mp
import numpy as np
def peak(x, c):
return np.exp(-np.power(x - c, 2) / 16.0)
def lin_interp(x, y, i, half):
return x[i] + (x[i+1] - x[i]) * ((half - y[i]) / (y[i+1] - y[i]))
def half_max_x(x, y):
half = max(y)/2.0
signs = np.sign(np.add(y, -half))
zero_crossings = (signs[0:-2] != signs[1:-1])
zero_crossings_i = np.where(zero_crossings)[0]
return [lin_interp(x, y, zero_crossings_i[0], half),
lin_interp(x, y, zero_crossings_i[1], half)]
# make some fake data
x=np.linspace(0,20,21)
y=peak(x,10)
# find the two crossing points
hmx = half_max_x(x,y)
# print the answer
fwhm = hmx[1] - hmx[0]
print("FWHM:{:.3f}".format(fwhm))
# a convincing plot
half = max(y)/2.0
mp.plot(x,y)
mp.plot(hmx, [half, half])
mp.show()
The (x, y) coordinates of the two points are (hmx[0], half) and (hmx[1], half).
In addition to the previos answer, in case of baseline is not 0, then ((max-min)/2) + min. That's what I did to solve my problem. Tks.
I am trying to find pairs of (x,y) points within a maximum distance of each other. I thought the simplest thing to do would be to generate a DataFrame and go through each point, one by one, calculating if there are points with coordinates (x,y) within distance r of the given point (x_0, y_0). Then, divide the total number of discovered pairs by 2.
%pylab inline
import pandas as pd
def find_nbrs(low, high, num, max_d):
x = random.uniform(low, high, num)
y = random.uniform(low, high, num)
points = pd.DataFrame({'x':x, 'y':y})
tot_nbrs = 0
for i in arange(len(points)):
x_0 = points.x[i]
y_0 = points.y[i]
pt_nbrz = points[((x_0 - points.x)**2 + (y_0 - points.y)**2) < max_d**2]
tot_nbrs += len(pt_nbrz)
plot (pt_nbrz.x, pt_nbrz.y, 'r-')
plot (points.x, points.y, 'b.')
return tot_nbrs
print find_nbrs(0, 1, 50, 0.1)
First of all, it's not always finding the right pairs (I see points that are within the stated distance that are not labeled).
If I write plot(..., 'or'), it highlights all the points. Which means that pt_nbrz = points[((x_0 - points.x)**2 + (y_0 - points.y)**2) < max_d**2] returns at least one (x,y). Why? Shouldn't it return an empty array if the comparison is False?
How do I do all of the above more elegantly in Pandas? For example, without having to loop through each element.
The functionality you're looking for is included in scipy's spatial distance module.
Here's an example of how you could use it. The real magic is in squareform(pdist(points)).
from scipy.spatial.distance import pdist, squareform
import numpy as np
import matplotlib.pyplot as plt
points = np.random.uniform(-.5, .5, (1000,2))
# Compute the distance between each different pair of points in X with pdist.
# Then, just for ease of working, convert to a typical symmetric distance matrix
# with squareform.
dists = squareform(pdist(points))
poi = points[4] # point of interest
dist_min = .1
close_points = dists[4] < dist_min
print("There are {} other points within a distance of {} from the point "
"({:.3f}, {:.3f})".format(close_points.sum() - 1, dist_min, *poi))
There are 27 other points within a distance of 0.1 from the point (0.194, 0.160)
For visualization purposes:
f,ax = plt.subplots(subplot_kw=
dict(aspect='equal', xlim=(-.5, .5), ylim=(-.5, .5)))
ax.plot(points[:,0], points[:,1], 'b+ ')
ax.plot(poi[0], poi[1], ms=15, marker='s', mfc='none', mec='g')
ax.plot(points[close_points,0], points[close_points,1],
marker='o', mfc='none', mec='r', ls='') # draw all points within distance
t = np.linspace(0, 2*np.pi, 512)
circle = dist_min*np.vstack([np.cos(t), np.sin(t)]).T
ax.plot((circle+poi)[:,0], (circle+poi)[:,1], 'k:') # Add a visual check for that distance
plt.show()
I have a list of x and y values for two curves, both having weird shapes, and I don't have a function for any of them. I need to do two things:
Plot it and shade the area between the curves like the image below.
Find the total area of this shaded region between the curves.
I'm able to plot and shade the area between those curves with fill_between and fill_betweenx in matplotlib, but I have no idea on how to calculate the exact area between them, specially because I don't have a function for any of those curves.
Any ideas?
I looked everywhere and can't find a simple solution for this. I'm quite desperate, so any help is much appreciated.
Thank you very much!
EDIT: For future reference (in case anyone runs into the same problem), here is how I've solved this: connected the first and last node/point of each curve together, resulting in a big weird-shaped polygon, then used shapely to calculate the polygon's area automatically, which is the exact area between the curves, no matter which way they go or how nonlinear they are. Works like a charm! :)
Here is my code:
from shapely.geometry import Polygon
x_y_curve1 = [(0.121,0.232),(2.898,4.554),(7.865,9.987)] #these are your points for curve 1 (I just put some random numbers)
x_y_curve2 = [(1.221,1.232),(3.898,5.554),(8.865,7.987)] #these are your points for curve 2 (I just put some random numbers)
polygon_points = [] #creates a empty list where we will append the points to create the polygon
for xyvalue in x_y_curve1:
polygon_points.append([xyvalue[0],xyvalue[1]]) #append all xy points for curve 1
for xyvalue in x_y_curve2[::-1]:
polygon_points.append([xyvalue[0],xyvalue[1]]) #append all xy points for curve 2 in the reverse order (from last point to first point)
for xyvalue in x_y_curve1[0:1]:
polygon_points.append([xyvalue[0],xyvalue[1]]) #append the first point in curve 1 again, to it "closes" the polygon
polygon = Polygon(polygon_points)
area = polygon.area
print(area)
EDIT 2: Thank you for the answers. Like Kyle explained, this only works for positive values. If your curves go below 0 (which is not my case, as showed in the example chart), then you would have to work with absolute numbers.
The area calculation is straightforward in blocks where the two curves don't intersect: thats the trapezium as has been pointed out above. If they intersect, then you create two triangles between x[i] and x[i+1], and you should add the area of the two. If you want to do it directly, you should handle the two cases separately. Here's a basic working example to solve your problem. First, I will start with some fake data:
#!/usr/bin/python
import numpy as np
# let us generate fake test data
x = np.arange(10)
y1 = np.random.rand(10) * 20
y2 = np.random.rand(10) * 20
Now, the main code. Based on your plot, looks like you have y1 and y2 defined at the same X points. Then we define,
z = y1-y2
dx = x[1:] - x[:-1]
cross_test = np.sign(z[:-1] * z[1:])
cross_test will be negative whenever the two graphs cross. At these points, we want to calculate the x coordinate of the crossover. For simplicity, I will calculate x coordinates of the intersection of all segments of y. For places where the two curves don't intersect, they will be useless values, and we won't use them anywhere. This just keeps the code easier to understand.
Suppose you have z1 and z2 at x1 and x2, then we are solving for x0 such that z = 0:
# (z2 - z1)/(x2 - x1) = (z0 - z1) / (x0 - x1) = -z1/(x0 - x1)
# x0 = x1 - (x2 - x1) / (z2 - z1) * z1
x_intersect = x[:-1] - dx / (z[1:] - z[:-1]) * z[:-1]
dx_intersect = - dx / (z[1:] - z[:-1]) * z[:-1]
Where the curves don't intersect, area is simply given by:
areas_pos = abs(z[:-1] + z[1:]) * 0.5 * dx # signs of both z are same
Where they intersect, we add areas of both triangles:
areas_neg = 0.5 * dx_intersect * abs(z[:-1]) + 0.5 * (dx - dx_intersect) * abs(z[1:])
Now, the area in each block x[i] to x[i+1] is to be selected, for which I use np.where:
areas = np.where(cross_test < 0, areas_neg, areas_pos)
total_area = np.sum(areas)
That is your desired answer. As has been pointed out above, this will get more complicated if the both the y graphs were defined at different x points. If you want to test this, you can simply plot it (in my test case, y range will be -20 to 20)
negatives = np.where(cross_test < 0)
positives = np.where(cross_test >= 0)
plot(x, y1)
plot(x, y2)
plot(x, z)
plt.vlines(x_intersect[negatives], -20, 20)
Define your two curves as functions f and g that are linear by segment, e.g. between x1 and x2, f(x) = f(x1) + ((x-x1)/(x2-x1))*(f(x2)-f(x1)).
Define h(x)=abs(g(x)-f(x)). Then use scipy.integrate.quad to integrate h.
That way you don't need to bother about the intersections. It will do the "trapeze summing" suggested by ch41rmn automatically.
Your set of data is quite "nice" in the sense that the two sets of data share the same set of x-coordinates. You can therefore calculate the area using a series of trapezoids.
e.g. define the two functions as f(x) and g(x), then, between any two consecutive points in x, you have four points of data:
(x1, f(x1))-->(x2, f(x2))
(x1, g(x1))-->(x2, g(x2))
Then, the area of the trapezoid is
A(x1-->x2) = ( f(x1)-g(x1) + f(x2)-g(x2) ) * (x2-x1)/2 (1)
A complication arises that equation (1) only works for simply-connected regions, i.e. there must not be a cross-over within this region:
|\ |\/|
|_| vs |/\|
The area of the two sides of the intersection must be evaluated separately. You will need to go through your data to find all points of intersections, then insert their coordinates into your list of coordinates. The correct order of x must be maintained. Then, you can loop through your list of simply connected regions and obtain a sum of the area of trapezoids.
EDIT:
For curiosity's sake, if the x-coordinates for the two lists are different, you can instead construct triangles. e.g.
.____.
| / \
| / \
| / \
|/ \
._________.
Overlap between triangles must be avoided, so you will again need to find points of intersections and insert them into your ordered list. The lengths of each side of the triangle can be calculated using Pythagoras' formula, and the area of the triangles can be calculated using Heron's formula.
The area_between_two_curves function in pypi library similaritymeasures (released in 2018) might give you what you need. I tried a trivial example on my side, comparing the area between a function and a constant value and got pretty close tie-back to Excel (within 2%). Not sure why it doesn't give me 100% tie-back, maybe I am doing something wrong. Worth considering though.
I had the same problem.The answer below is based on an attempt by the question author. However, shapely will not directly give the area of the polygon in purple. You need to edit the code to break it up into its component polygons and then get the area of each. After-which you simply add them up.
Area Between two lines
Consider the lines below:
Sample Two lines
If you run the code below you will get zero for area because it takes the clockwise and subtracts the anti clockwise area:
from shapely.geometry import Polygon
x_y_curve1 = [(1,1),(2,1),(3,3),(4,3)] #these are your points for curve 1
x_y_curve2 = [(1,3),(2,3),(3,1),(4,1)] #these are your points for curve 2
polygon_points = [] #creates a empty list where we will append the points to create the polygon
for xyvalue in x_y_curve1:
polygon_points.append([xyvalue[0],xyvalue[1]]) #append all xy points for curve 1
for xyvalue in x_y_curve2[::-1]:
polygon_points.append([xyvalue[0],xyvalue[1]]) #append all xy points for curve 2 in the reverse order (from last point to first point)
for xyvalue in x_y_curve1[0:1]:
polygon_points.append([xyvalue[0],xyvalue[1]]) #append the first point in curve 1 again, to it "closes" the polygon
polygon = Polygon(polygon_points)
area = polygon.area
print(area)
The solution is therefore to split the polygon into smaller pieces based on where the lines intersect. Then use a for loop to add these up:
from shapely.geometry import Polygon
x_y_curve1 = [(1,1),(2,1),(3,3),(4,3)] #these are your points for curve 1
x_y_curve2 = [(1,3),(2,3),(3,1),(4,1)] #these are your points for curve 2
polygon_points = [] #creates a empty list where we will append the points to create the polygon
for xyvalue in x_y_curve1:
polygon_points.append([xyvalue[0],xyvalue[1]]) #append all xy points for curve 1
for xyvalue in x_y_curve2[::-1]:
polygon_points.append([xyvalue[0],xyvalue[1]]) #append all xy points for curve 2 in the reverse order (from last point to first point)
for xyvalue in x_y_curve1[0:1]:
polygon_points.append([xyvalue[0],xyvalue[1]]) #append the first point in curve 1 again, to it "closes" the polygon
polygon = Polygon(polygon_points)
area = polygon.area
x,y = polygon.exterior.xy
# original data
ls = LineString(np.c_[x, y])
# closed, non-simple
lr = LineString(ls.coords[:] + ls.coords[0:1])
lr.is_simple # False
mls = unary_union(lr)
mls.geom_type # MultiLineString'
Area_cal =[]
for polygon in polygonize(mls):
Area_cal.append(polygon.area)
Area_poly = (np.asarray(Area_cal).sum())
print(Area_poly)
A straightforward application of the area of a general polygon (see Shoelace formula) makes for a super-simple and fast, vectorized calculation:
def area(p):
# for p: 2D vertices of a polygon:
# area = 1/2 abs(sum(p0 ^ p1 + p1 ^ p2 + ... + pn-1 ^ p0))
# where ^ is the cross product
return np.abs(np.cross(p, np.roll(p, 1, axis=0)).sum()) / 2
Application to area between two curves. In this example, we don't even have matching x coordinates!
np.random.seed(0)
n0 = 10
n1 = 15
xy0 = np.c_[np.linspace(0, 10, n0), np.random.uniform(0, 10, n0)]
xy1 = np.c_[np.linspace(0, 10, n1), np.random.uniform(0, 10, n1)]
p = np.r_[xy0, xy1[::-1]]
>>> area(p)
4.9786...
Plot:
plt.plot(*xy0.T, 'b-')
plt.plot(*xy1.T, 'r-')
p = np.r_[xy0, xy1[::-1]]
plt.fill(*p.T, alpha=.2)
Speed
For both curves having 1 million points:
n = 1_000_000
xy0 = np.c_[np.linspace(0, 10, n), np.random.uniform(0, 10, n)]
xy1 = np.c_[np.linspace(0, 10, n), np.random.uniform(0, 10, n)]
%timeit area(np.r_[xy0, xy1[::-1]])
# 42.9 ms ± 140 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Simple viz of polygon area calculation
# say:
p = np.array([[0, 3], [1, 0], [3, 3], [1, 3], [1, 2]])
p_closed = np.r_[p, p[:1]]
fig, axes = plt.subplots(ncols=2, figsize=(10, 5), subplot_kw=dict(box_aspect=1), sharex=True)
ax = axes[0]
ax.set_aspect('equal')
ax.plot(*p_closed.T, '.-')
ax.fill(*p_closed.T, alpha=0.6)
center = p.mean(0)
txtkwargs = dict(ha='center', va='center')
ax.text(*center, f'{area(p):.2f}', **txtkwargs)
ax = axes[1]
ax.set_aspect('equal')
for a, b in zip(p_closed, p_closed[1:]):
ar = 1/2 * np.cross(a, b)
pos = ar >= 0
tri = np.c_[(0,0), a, b, (0,0)].T
# shrink a bit to make individual triangles easier to visually identify
center = tri.mean(0)
tri = (tri - center)*0.95 + center
c = 'b' if pos else 'r'
ax.plot(*tri.T, 'k')
ax.fill(*tri.T, c, alpha=0.2, zorder=2 - pos)
t = ax.text(*center, f'{ar:.1f}', color=c, fontsize=8, **txtkwargs)
t.set_bbox(dict(facecolor='white', alpha=0.8, edgecolor='none'))
plt.tight_layout()