Contour Plot of the Gradient Descent Algorithm in Python - python

I'm trying to apply gradient descent to a simple linear regression model, when plotting a 2D graph I get the intended result but when I switch into a contour plot I don't the intended plot, I would like to know where my mistake is.
Here is the code:
def J(b_0, b_1, x, y):
return (1/len(y))*(y - b_0 - b_1*x)**2
def dJ_1(b_0, b_1, x, y):
return (2/len(y))*np.sum(x*(b_1*x + b_0 - y))
def dJ_0(b_0, b_1, x, y):
return (2/100)*np.sum((b_1*x + b_0 - y))
x = np.linspace(-1, 1, 100)
y = np.linspace(-2, 2, 100)
b_0 = 5
b_1 = 5
parameters_0 = [b_0]
parameters_1 = [b_1]
for i in range(99):
b_1 -= 0.1*dJ_1(b_0, b_1, x, y)
b_0 -= 0.1*dJ_0(b_0, b_1, x, y)
parameters_0.append(b_0)
parameters_1.append(b_1)
plt.figure(figsize=(4, 8))
plt.plot(np.linspace(-2, 7, 100), J(np.linspace(-2, 7, 100), parameters_1[-1], -1, -2))
plt.plot(np.array(parameters_0), J(np.array(parameters_0), parameters_1[-1], -1, -2), color="C1")
plt.plot(np.array(parameters_0), J(np.array(parameters_0), parameters_1[-1], -1, -2), '-o', color="C1")
plt.xlabel(r"$\beta_0$")
plt.ylabel(r"$J(\beta_0)$")
plt.show()
The first plot:
plt.figure(figsize=(4, 8))
plt.plot(np.linspace(-4, 7, 100), J(parameters_0[-1], np.linspace(-4, 7, 100), -1, -2))
plt.plot(np.array(parameters_1), J(parameters_0[-1], np.array(parameters_1), -1, -2), color="C1")
plt.plot(np.array(parameters_1), J(parameters_0[-1], np.array(parameters_1), -1, -2), '-o', color="C1")
plt.xlabel(r"$\beta_1$")
plt.ylabel(r"$J(\beta_1)$")
plt.show()
The second plot:
b_0 = np.linspace(-10, 10, 100)
b_1 = np.linspace(-10, 10, 100)
X, Y = np.meshgrid(b_0, b_1)
Z = J(X, Y, x=-1, y=-2)
fig,ax=plt.subplots(1,1)
cp = ax.contourf(X, Y, Z)
fig.colorbar(cp)
ax.set_xlabel(r"$\beta_0$")
ax.set_ylabel(r"$\beta_1$")
plt.show()
The contour plot is:
Why do I get the above plot rather than a plot similar to this below one for example when the global minima of the cost function is at (0, 2)? Thanks in advance.


Well I think there's no mistake there, you can see from the 2d plot that your gradient descent plot is a quadratic function, thus the way you see it from the contour is as if you see it from the sky to the valley. As to why it doesn't look like a circle, well it's because it's just a 3d quadratic function. I also once made something similar, and the gradient descent plot is just as what you plot. Check it out here at the end of the page

Related

Dealing with infinite singularities in matplotlib

I have been graphing the graph for 1/r or 1/sqrt(x^2 + y^2) in Python recently. When (x,y) -> (0,0), 1/r goes to inf. To deal with this, I used np.nan_to_num to deal with the infinities but it just gives an error. I just wanna make my graph look nice and not have these diamond things like in this picture:
Here is my code:
min, max, ds = -10, 10, 0.5
x = np.arange(min, max, ds)
y = np.arange(min, max, ds)
X, Y = np.meshgrid(x, y)
def potential(X, Y, X0, Y0):
return 1/np.sqrt((X-X0)**2 + (Y-Y0)**2)
plt.figure(figsize = (9.5,8))
plt.contourf(X, Y, potential(X, Y, 0, 0), 100)
plt.colorbar()
plt.title("Electric potential", fontsize = 20)
plt.show()

How to use curve_fit with barplot?

I'm new with the function curve_fit() from scipy.optimize, but I can't get it to work.
I've a barplot, really simple, and I would like to create a curve that "fit" it.
My code :
import matplotlib.pyplot as plt
import numpy as np
from scipy.optimize import curve_fit
x = [i for i in range(15)]
y = [1,3,4,6,8,4,2,1,5,8,6,5,5,8,5]
plt.bar(x,y,color='yellow')
plt.show()
# that is working
curve_fit(x,y) # I want the curve to fit the barplot
# but it returns an error...
plt.show()
result : error because of curve_fit.
If you could help me, that would be really great.
That is bonus, don t waste too much time, but would you know how to do the curve and do some forecast ? For instance, the result could be:

curve_fit
You need to pass a fitting function to curve_fit. Note that the line you've drawn is quite overfit for such a small sample and would require a high-order polynomial (even a cubic fit won't look like that).
Here is an example of using a quartic fitting function f_curve4:
# curve_fit requires x and y to be arrays (not lists)
x = np.arange(15)
y = np.array([1, 3, 4, 6, 8, 4, 2, 1, 5, 8, 6, 5, 5, 8, 5])
plt.bar(x, y, color='cyan')
# fit
f_curve4 = lambda x, a, b, c, d, e: a*x**4 + b*x**3 + c*x**2 + d*x + e
popt, pcov = curve_fit(f_curve4, x, y)
plt.plot(x, f_curve4(x, *popt), '--', label='fit')
# forecast
x_new = np.arange(max(x), max(x) + 2)
plt.plot(x_new, f_curve4(x_new, *popt), 'r:', label='forecast')
polyfit
Alternatively use polyfit and just pass deg=N without manually defining an Nth-order fitting function:
plt.bar(x, y, color='cyan')
# fit
f_poly4 = np.polyfit(x, y, deg=4)
x_fit = np.linspace(min(x), max(x), 100)
y_fit = np.polyval(f_poly4, x_fit)
plt.plot(x_fit, y_fit, '--', label='fit')
# forecast
x_new = np.linspace(max(x), max(x) + 1, 10)
y_new = np.polyval(f_poly4, x_new)
plt.plot(x_new, y_new, 'r:', lw=2, label='forecast')
interp1d
Depending on your use case, consider interpolating with interp1d instead of fitting a polynomial. Here is an example of using cubic interpolation function f_interp:
plt.bar(x, y, color='cyan')
f_interp = interpolate.interp1d(x, y, kind='cubic')
x2 = np.linspace(min(x), max(x), 100)
plt.plot(x2, f_interp(x2), '--')

How to plot this 2D sinusoidal parametric function

I have a 2D sinusoidal function and I want to plot its boundary threshold of t=0. Could someone give me a hint on how to do it?
f (x, y) = sin(10x) + cos(4y) − cos(3xy)
x ∈ [0, 1], y ∈ [0, 2], with a boundary threshold of t = 0
The expected plot should look like this: Plot A dashed lines
Actually the function I am referring to is a toy one from paper "Active Learning For Identifying Function Threshold Boundaries"(https://papers.nips.cc/paper/2005/file/8e930496927757aac0dbd2438cb3f4f6-Paper.pdf)
Page 4 of that paper
Update: I tried the following code but apparently it does not give what I want. The top view is a straight line from (0,0) to (1,2), instead of some curves...
ax = plt.axes(projection='3d')
# Data for a three-dimensional line
xline = np.linspace(0, 1, 1000)
yline = np.linspace(0, 2, 1000)
zline = np.sin(10*xline)+np.cos(4*yline)-np.cos(3*xline*yline)
ax.plot3D(xline, yline, zline, 'gray')

Welcome to stackoverflow. Your math is wrong. Your function f is a function of two variables, f(x, y). Hence you need to evaluate it on a grid (all combinations of valid x and y values), if you want to find the solutions for f = 0 computationally. Your code is currently evaluating f only on the y = 2x axis (hence the "straight line from (0,0) to (1, 2) in top-down view").
import numpy as np
import matplotlib.pyplot as plt
def f(x, y):
return np.sin(10*x)+np.cos(4*y)-np.cos(3*x*y)
x = np.arange(0, 1, 1e-3)
y = np.arange(0, 2, 1e-3)
XX, YY = np.meshgrid(x, y)
ZZ = f(XX, YY)
plt.contour(XX, YY, ZZ, levels=[0.])
plt.show()

How can I get this quadratic fit to plateau?

I have two variables, x and y, that are random variables. I want to fit a curve to them that plateaus. I've been able to do this using an exponential fit but I'd like to do so with a quadratic fit as well.
How can I get the fit to flatten out at the top? FWIW, the y data were generated such that no value goes above: 4300. So probably in the new curve it should have this requirement.
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
x = np.asarray([70,37,39,42,35,35,44,40,42,51,65,32,56,51,33,47,33,42,33,44,46,38,53,38,54,54,51,46,50,51,48,48,50,32,54,60,41,40,50,49,58,35,53,66,41,48,43,54,51])
y = np.asarray([3781,3036,3270,3366,2919,2966,3326,2812,3053,3496,3875,1823,3510,3615,2987,3589,2791,2819,1885,3570,3431,3095,3678,2297,3636,3569,3547,3553,3463,3422,3516,3538,3671,1888,3680,3775,2720,3450,3563,3345,3731,2145,3364,3928,2720,3621,3425,3687,3630])
def polyfit(x, y, degree):
results = {}
coeffs = np.polyfit(x, y, degree)
# Polynomial Coefficients
results['polynomial'] = coeffs.tolist()
# r-squared, fit values, and average
p = np.poly1d(coeffs)
yhat = p(x)
ybar = np.sum(y)/len(y)
ssreg = np.sum((yhat-ybar)**2)
sstot = np.sum((y - ybar)**2)
results['determination'] = ssreg / sstot
return results, yhat, ybar
def plot_polyfit(x=None, y=None, degree=None):
# degree = degree of the fitting polynomial
xmin = min(x)
xmax = max(x)
fig, ax = plt.subplots(figsize=(5,4))
p = np.poly1d(np.polyfit(x, y, degree))
t = np.linspace(xmin, xmax, len(x))
ax.plot(x, y, 'ok', t, p(t), '-', markersize=3, alpha=0.6, linewidth=2.5)
results, yhat, ybar = polyfit(x,y,degree)
R_squared = results['determination']
textstr = r'$r^2=%.2f$' % (R_squared, )
props = dict(boxstyle='square', facecolor='lightgray', alpha=0.5)
fig.text(0.05, 0.95, textstr, transform=ax.transAxes, fontsize=12,
verticalalignment='top', bbox=props)
results['polynomial'][0]
plot_polyfit(x=x, y=y, degree=2)
In contrast, I can use the same functions and get the curve to plateau better when the data are so:
x2 = np.asarray([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12])
y2 = np.asarray([2, 4, 8, 12, 14, 18, 20, 21, 22, 23, 24, 24])
plot_polyfit(x=x2, y=y2, degree=2)
Edits suggested by #tstanisl:
def plot_newfit(xdat, ydat):
x,y = xdat, ydat
xmax = 4300
def new_fit(A,x,B):
return A*(x - xmax)**2+B # testing this out
fig, axs = plt.subplots(figsize=(5,4))
# Find best fit.
popt, pcov = curve_fit(new_fit, x, y)
# Top plot
# Plot data and best fit curve.
axs.plot(x, y,'ok', alpha=0.6)
axs.plot(np.sort(x), new_fit(np.sort(x), *popt),'-')
#r2
residuals = y - new_fit(x, *popt)
ss_res = np.sum(residuals**2)
ss_tot = np.sum((y-np.mean(y))**2)
r_squared = 1 - (ss_res / ss_tot)
r_squared
# Add text
textstr = r'$r^2=%.2f$' % (r_squared, )
props = dict(boxstyle='square', facecolor='lightgray', alpha=0.5)
fig.text(0.05, 0.95, textstr, transform=axs.transAxes, fontsize=12,
verticalalignment='top', bbox=props)
plot_newfit(x,y)

You just need to slightly modify new_fit() to fit A, B rather x and B.
Set xmax to the desired location of the peek. Using x.max() will guarantee that the fit curve will flatten at the last sample.
def new_fit(x, A, B):
xmax = x.max() # or 4300
return A*(x - xmax)**2+B # testing this out
Result:

I'm not too familiar with scipy.optimise but, if you find the Euclidian distance between the point that contains x-max and the one that contains your y-max, divide it in half and do some trig, you could use that coord to either force your quadratic through it, or use it in your array. (again not too familiar with scipy.optimise so I'm not sure if that first option is possible, but the second should lessen the downwards curve)
I can provide the proof if you don't understand.

Curve fitting with boundary conditions

I'm trying to fit a curve using LSQUnivariateSpline.
from scipy.interpolate import LSQUnivariateSpline, UnivariateSpline
import matplotlib.pyplot as plt
x = np.linspace(-3, 3, 50)
y = np.exp(-x**2) + 0.1 * np.random.randn(50)
t = [-2, -1, 0, 1, 2]
spl = LSQUnivariateSpline(x, y, t, k=4 )
xs = np.linspace(-3, 3, 1000)
plt.plot(x, y, 'ro', ms=5)
plt.plot(xs, spl(xs), 'g-', lw=3)
plt.show()
This code produce below graph.
I want to declare boundary conditions like dy/dx = 0 or d2x/dy = 0. Is there a way or another function to fit a curve with boundary conditions?

Categories