It is some days that I am trying to visualize the so-called kernel trick resulting from a RBF kernel transformation in a SVC model. Basically, I am trying to map a 2D space to a 3D space in order to let the viewer see how the kernel trick adds a dimension in order to linearly separate the space between two classes.
Following sklearn examples, I managed to plot a 2D example of the trick. However, I feel it is not enough to really grasp what it is happening behind the scences.
Below is what I managed to plot:
I would like to plot the same data on a three dimensional space, representing also the plane that splits the space between the two classes.
I am not asking for the actual code here. Rather, I would like to understand what goes on the axis for the third dimension. Indeed, I think that such axis should be equal to exp(-gamma||x-y||^2). However, due to my poor vector algebra skills, I do not know how to compute it.
Any help would be much appreciated.
Cheers!
UPDATE
The following allowed me to build a new matrix for a 3D plot:
def feature_map_2(X):
return np.asarray((X[:,0], X[:,1], np.exp( -gam*(( X[:,0]**2 + X[:,1]**2 -2*X[:,0]*X[:,1]))))).T
Z = feature_map_2(X)
Where gam = 1/n_features
Then, I computed the boundary as follows:
#SVM
clf = svm.NuSVC(kernel = 'linear', nu=0.5)
clf.fit(Z, y)
w = clf.coef_.flatten()
b = clf.intercept_.flatten()
# create x,y
xx, yy = np.meshgrid(np.linspace(-6,6), np.linspace(-2,2))
# calculate corresponding z
boundary = lambda xx, yy: (-w[0] * xx - w[1] * yy - b) * 1. /w[2]
However, results differ from what one might have expected looking at the 2D plot.
Do you mean SVM model?
https://jgreitemann.github.io/svm-demo
Visualizing 3D may be difficult because you will need to project to the screen bringing again to a 2D image.
To find the plane in the 3D space you simply apply your kernel to make your classes linearly separable and then apply a
Linear SVM
The equation w' x - b = 0 expressed in terms of scalars as w[0] * x[0] + w[1] * x[1] + w[2] * x[2] - b = 0, can be made parametric by choosing element of x (with non-zero coefficient). For instance if w[2] != 0 you can write the plane as.
(U, V, (b - w[0] * U - w[1] * V) / w[2])
And this may be used in common surface plot functions, for instance in python it would be like this
U, V = meshgrid(np.linspace(-1, 1, 100), np.linspace(-1, 1, 100))
plt.pcolormesh(U, V, (b - w[0] * U - w[1] * V) / w[2]);
Related
I have a collection of curved lines, representing the third degree polynomial line of best fit for some datasets.
I want to differentiate relatively flat lines, filtering these plots, for further analyses.
For example I want to filter subplots 20935, 21004, 21010, 18761, 21037.
How can I do this, with a list of floats as input for these lines?
(using Python 3.8, Numpy, Math, mathplotlib in an anaconda env)
If you have got a list of xs and their respective ys, you can compute the slope for each point and check if the slope is always a constant value.
threshold = 0.001 # add your precision here. zero indicates a perfect straight line
is_straight_line = True
slope = (y[1]-y[0]) / (x[1] - x[0])
for i, (xval, yval) in enumerate(zip(x[2:], y[2:])):
s = (yval - y[i-1]) / (xval - x[i-1])
if abs(s - slope) > threshold:
is_straight_line = False
break
print(is_straight_line)
if you need the computation to be efficient, you should consider using numpy instead.
Knowledge of first-year calculus is assumed. There's a geometric property called "curvature" that basically determines how much a shape bends at a certain point (really the inverse of the radius of the osculating circle at that point).
We can use this link to develop a formula for a cubic function with coefficients [a, b, c, d] at x = x.
def cubic_curvature(a, b, c, d, x):
k = abs(6*a*x + 2*b) / (1 + (3*a*x**2 + 2*b*x + c)**2) ** 1.5
return k
More general algorithms can be created for any polynomial, possibly with assistance from the sympy library depending on your needs.
With this in mind, you can set some threshold for curvature that determines whether the cubic is "straight" enough given its coefficients (I believe scipy or similar should be able to give you these from a list of points) and the x-value to be evaluated at (try the median independent variable).
I am trying to implement a gradient descent algorithm from scratch in python, which should be fairly easy. however, I have been scratching my head for quite while with my code now, unable to make it work.
I generate data as follow:
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('darkgrid')
#Defining the x array.
x=np.array(range(1,100))
#Defining the y array.
y=10+2*x.ravel()
y=y+np.random.normal(loc=0, scale=70, size=99)
Then define the parameters:
alpha = 0.01 # Which will be the learning rate
NbrIter = 100 # Representing the number of iteration
m = len(y)
theta = np.random.randn(2,1)
and my GD is as follow:
for iter in range(NbrIter):
theta = theta - (1/m) * alpha * ( X.T # ((X # theta) - y) )
What I get is a huge matrix, meaning that I have some problem with the linear algebra. However, I really fail to see where the issue is.
(Playing around with the matrices to try to get them to match I reached a theta having the correct form (2x1) with:
theta = theta - (1/m) * alpha * ( X.T # ((X # theta).T - y).T )
But it does look wrong and the actual value are way off (array([[-8.92647663e+148],
[-5.92079000e+150]]))
)
I guess you were hit by broadcasting. Variable y's shape is (100,). When y is subtracted from result of X.T#X#theta. Theta is column vector so I guess the result is a column vector. Variable y is broadcasted to a row vector of shape (1,100). The result of subtraction is (100,100). To fix this reshape y as column vector with y.reshape(-1,1)
Now, a few optimizations:
X.T # ((X # theta) - y[:,None])
can be rewritten as:
(X.T#X) # theta - (X.T*y[:,None])
The most costly computation can be taken out of the loop:
XtX = X.T#X
Xty = X.T*y[:,None]
for iter in range(NbrIter):
theta = theta - (1/m) * alpha * (XtX # theta - Xty)
Now you operate on 2x2 matrix rather that 100x2.
Let's take a look on convergence.
Assuming that X is constructed like: X=np.column_stack((x, np.ones_like(x)) it is possible to check matrix condition:
np.linalg.cond(XtX)
Which produced:
13475.851490419038
It means that the ratio between minimal and maximal eigenvector is about 13k. Therefore using alpha larger then 1/13k will likely result in bad convergence.
If you use alpha=1e-5 the algorithm will converge.
Good luck!
This question already has answers here:
Generating 3D Gaussian distribution in Python
(2 answers)
Closed 2 years ago.
I'm trying to generate a 3D distribution, where x, y represents the surface plane, and z is the magnitude of some value, distributed over a range.
I'm looking at numpy's multivariate_normal, but it only lets me get a number of samples. I'd like the ability to specify some x, y coordinate, and get back what the z value should be; so I'd be able to query gp(x, y) and get back a z value that adheres to some mean and covariance.
Perhaps a more illustrative (toy) example: assume I have some temperature distribution that can be modeled as a gaussian process. So I might have a mean temperature of 20 at (0, 0), and some covariance [[1, 0], [0, 1]]. I'd like to be able to create a model that I can then query at different x, y locations to get the temperature at that position (so, at (5, 5) I might get back something like 7 degrees).
How to best accomplish this?
I assume that your data can be copied to a single np.array, which I will refer to as X in my code, with shape X.shape = (n,2), where n is the number of data points you have and you can have n = 1, if you wish to test a single point at a time. 2, of course, refers to the 2D space spanned by your coordinates (x and y) base. Then:
def estimate_gaussian(X):
return X.mean(axis=0), np.cov(X.T)
def mva_gaussian( X, mu, sigma2 ):
k = len(mu)
# check if sigma2 is a vector and, if yes, use as the diagonal of the covariance matrix
if sigma2.ndim == 1 :
sigma2 = np.diag(sigma2)
X = X - mu
return (2 * np.pi)**(-k/2) * np.linalg.det(sigma2)**(-0.5) * \
np.exp( -0.5 * np.sum( np.multiply( X.dot( np.linalg.inv(sigma2) ), X ), axis=1 ) ).reshape( ( X.shape[0], 1 ) )
will do what you want - that is, given data points you will get the value of the gaussian function at those points (or a single point). This is actually a generalized version of what you need, as this function can describe a multivariate gaussian. You seem to be interested in the k = 2 case and a diagonal covariance matrix sigma2.
Moreover, this is also a probability distribution - which you say you don't want. We don't have enough info to know what exactly it is you're trying to fit to (i.e. what you expect the three parameters of the gaussian function to be. Usually, people are interested in a normal distribution). Nevertheless, you can simply change the parameters in the return statement of the mva_gaussian function according to your needs and ignore the estimate gaussian function if you don't want a normalized distribution (although a normalized function would still give you what you seek - a real valued temperature - as long as you know the normalization process - which you do :-) ).
You can create a multivariate normal using scipy.stats.multivariate_normal.
>>> import scipy.stats
>>> dist = scipy.stats.multivariate_normal(mean=[2,3], cov=[[1,0],
[0,1]])
Then to find p(x,y) you can use pdf
>>> dist.pdf([2,3])
0.15915494309189535
>>> dist.pdf([1,1])
0.013064233284684921
Which represents the probability (which you called z) given any [x,y]
I have a set of 3D points defining a 3D contour.
What I want to do is to obtain the minimal surface representation corresponding to this contour (see Minimal Surfaces in Wikipedia). Basically, this requires to solve a nonlinear partial differential equation.
In Matlab, this is almost straightforward using the pdenonlinfunction (see Matlab's documentation). An example of its usage for solving a minimal surface problem can be found here: Minimal Surface Problem on the Unit Disk.
I need to make such an implementation in Python, but up to know I haven't found any web resources on how to do this.
Can anyone point me any resources/examples of such implementation?
Thanks,
Miguel.
UPDATE
The 3D surface (ideally a triangular mesh representation) I want to find is bounded by this set of 3D points (as seen in this figure, the points lie in the best-fit plane):
Ok, so doing some research I found that this minimal surface problem is related with the solution of the Biharmonic Equation, and I also found that the Thin-plate spline is the fundamental solution to this equation.
So I think the approach would be to try to fit this sparse representation of the surface (given by the 3D contour of points) using thin-plate splines. I found this example in scipy.interpolate where scattered data (x,y,z format) is interpolated using thin-plate splines to obtain the ZI coordinates on a uniform grid (XI,YI).
Two questions arise:
Would thin-plate spline interpolation be the correct approach for the problem of computing the surface from the set of 3D contour points?
If so, how to perform thin-plate interpolation on scipy with a NON-UNIFORM grid?
Thanks again!
Miguel
UPDATE: IMPLEMENTATION IN MATLAB (BUT IT DOESN'T WORK ON SCIPY PYTHON)
I followed this example using Matlab's tpaps function and obtained the minimal surface fitted to my contour on a uniform grid. This is the result in Matlab (looks great!):
However I need to implement this in Python, so I'm using the package scipy.interpolate.Rbf and the thin-plate function. Here's the code in python (XYZ contains the 3D coordinates of each point in the contour):
GRID_POINTS = 25
x_min = XYZ[:,0].min()
x_max = XYZ[:,0].max()
y_min = XYZ[:,1].min()
y_max = XYZ[:,1].max()
xi = np.linspace(x_min, x_max, GRID_POINTS)
yi = np.linspace(y_min, y_max, GRID_POINTS)
XI, YI = np.meshgrid(xi, yi)
from scipy.interpolate import Rbf
rbf = Rbf(XYZ[:,0],XYZ[:,1],XYZ[:,2],function='thin-plate',smooth=0.0)
ZI = rbf(XI,YI)
However this is the result (quite different from that obtained in Matlab):
It's evident that scipy's result does not correspond to a minimal surface.
Is scipy.interpolate.Rbf + thin-plate doing as expected, why does it differ from Matlab's result?
You can use FEniCS:
from fenics import (
UnitSquareMesh,
FunctionSpace,
Expression,
interpolate,
assemble,
sqrt,
inner,
grad,
dx,
TrialFunction,
TestFunction,
Function,
solve,
DirichletBC,
DomainBoundary,
MPI,
XDMFFile,
)
# Create mesh and define function space
mesh = UnitSquareMesh(100, 100)
V = FunctionSpace(mesh, "Lagrange", 2)
# initial guess (its boundary values specify the Dirichlet boundary conditions)
# (larger coefficient in front of the sin term makes the problem "more nonlinear")
u0 = Expression("a*sin(2.5*pi*x[1])*x[0]", a=0.2, degree=5)
u = interpolate(u0, V)
print(
"initial surface area: {}".format(assemble(sqrt(1 + inner(grad(u), grad(u))) * dx))
)
# Define the linearized weak formulation for the Newton iteration
du = TrialFunction(V)
v = TestFunction(V)
q = (1 + inner(grad(u), grad(u))) ** (-0.5)
a = (
q * inner(grad(du), grad(v)) * dx
- q ** 3 * inner(grad(u), grad(du)) * inner(grad(u), grad(v)) * dx
)
L = -q * inner(grad(u), grad(v)) * dx
du = Function(V)
# Newton iteration
tol = 1.0e-5
maxiter = 30
for iter in range(maxiter):
# compute the Newton increment by solving the linearized problem;
# note that the increment has *homogeneous* Dirichlet boundary conditions
solve(a == L, du, DirichletBC(V, 0.0, DomainBoundary()))
u.vector()[:] += du.vector() # update the solution
eps = sqrt(
abs(assemble(inner(grad(du), grad(du)) * dx))
) # check increment size as convergence test
area = assemble(sqrt(1 + inner(grad(u), grad(u))) * dx)
print(
f"iteration{iter + 1:3d} H1 seminorm of delta: {eps:10.2e} area: {area:13.5e}"
)
if eps < tol:
break
if eps > tol:
print("no convergence after {} Newton iterations".format(iter + 1))
else:
print("convergence after {} Newton iterations".format(iter + 1))
with XDMFFile(MPI.comm_world, "out.xdmf") as xdmf_file:
xdmf_file.write(u)
(Modified from http://www-users.math.umn.edu/~arnold/8445/programs/minimalsurf-newton.py.)
Obviously Matlab and SciPy understand TPS in different ways. The Matlab implementation looks correct.
SciPy treats TPS the same way as others RBFs, so you could implement it correctly in Python yourself - it would be enough to form a matrix of the related linear equation system and solve it to receive TPS' coefficients.
This is a bit of a complicated problem, so I'll do my best to break it down into chunks.
I'm writing a 3D Python library for the sake of learning / fun (as opposed to one that I'd intend for others to use). In the system I've developed, three-dimensional points are generally flattened to the image as follows:
Increasing the Z index by width moves the point halfway to the vanishing point in the center.
At Z = 0, the X and Y values correspond directly to the pixel at X, Y.
(There might be a name for this method, but if there is, I'm not familiar with it.)
In Python:
# vx and vy are the vanishing point's coordinates
def flatten_point(width, vx, vy, x, y, z):
distance = (x - vx, y - vy)
flat_distance = [d / (1 + float(z) / width) for d in distance]
return (vx + flat_distance[0], vx + flat_distance[1])
At this point, I'm able to create triangles somewhat efficiently by flattening its vertices and using barycentric coordinates to find and fill in the pixels that fall between those three points. This works well enough if I don't need to know anything about the actual points on the triangle that those pixels correspond to, but if I want to shade the triangle so that deeper points are drawn darker, I need to know what unflattened point on the triangle the pixel corresponds to.
joriki on math.stackexchange recommended using the barycentric coordinates as weights to find the original point. This did appear to work for awhile -- and it probably would work if I were using a linear depth system -- but it falls apart when the depths of the triangle's points differ by enough. The triangle appears to approach the greatest depth more quickly than it actually does, as if it were curved backwards.
So, in short: how can I reverse the point flattening function to get the actual 3D point of an arbitrary 2D pixel on a flattened triangle? Alternatively, if there is a better / more efficient way to flatten triangles without losing the depth of each pixel, that would work too.
You are right that the problem lies in your depth values not being linear. Fortunately, the solution is simple, but a little expensive if calculated per pixels.
Using your barycentric coordinates, rather than interpolating the three Z components directly, you need to interpolate their inverse and reinverse the result. This is called perspective correction.
Example for Z only :
def GetInterpolatedZ(triangle, u, v):
z0 = 1.0 / triangle[0].z
z1 = 1.0 / triangle[1].z
z2 = 1.0 / triangle[2].z
z = z0 + u * (z1-z0) + v * (z2-z0)
return 1.0/z
With triangle a list of three vectors and u and v the barycentric coordinates for triangle[1] and triangle[2] respectively. You will need to remap your Zs before and after the divisions if they are offset.
If you want to interpolate the actual X and Y coordinates, you do something similar. You will need to interpolate x/z and y/z and relinearize the result by multiplying by z.
def GetInterpolatedZ(tri, u, v):
t0 = Vec3(tri[0].x/tri[0].z, tri[0].y/tri[0].z, 1.0/tri[0].z)
t1 = Vec3(tri[1].x/tri[1].z, tri[1].y/tri[1].z, 1.0/tri[1].z)
t2 = Vec3(tri[2].x/tri[2].z, tri[2].y/tri[2].z, 1.0/tri[2].z)
inter = t0 + u * (t1-t0) + v * (t2-t0)
inter.z = 1.0 / inter.z
inter.x *= inter.z
inter.y *= inter.z
return inter
Again, tri is a list of the three vectors and u, v are the barycentric coordinates for tri[1], tri[2]. Vec3 is a regular 3 components Euclidean vector type.