We Keep Coding

Mixed partial dervative w.r.t. tensor in Pytorch

Question:
Is there any working method to calculate gradient of (non-scalar) tensor function?
Example
Given n by n symmetric matrices X, Y and matrix function Z(X, Y) = torch.mm(X.mm(X), Y) calculate d(dZ/dX)/dY.
Expected answer
d(dZ/dX)/dY = d(2*XY)/dY = 2*X
Attempts
Because torch's .backward() works only for scalar variables I've tried to calculate derivative by applying torch.autograd.grad() to each element of tensor Z, but this approach is not correct, because it gives d(X^2)/dX = X + 2*D where D is a diagonal matrix with diagonal values of X. For me it's a bit weird that torch has an ability to build a computational graph, but can't track tensor through it as a variable to get tensor derivative.
Edit
Question was not very clear, so I decided to give more details.
My aim is to get partial derivative of loss function, which involves two matrices as variables. It looks like that:
loss = torch.linalg.norm(my_formula(X, Y) , ord='fro')
And I need to find
d^2(loss)/d(Y^2)
d/dX[d(loss)/dY]
Torch is capable of calculating 1. by using .backward() two times, but it's problematic to find 2. because torch.autograd.grad() expects scalar input and not the tensor

TL;DR
For function f which takes a matrix and gives a scalar:
Find first order derivative, let's name it dX
Take trace: Tr(dX)
To get mixed partial derivative just use the trace from above: d/dY[df/dX] = d/dY[Tr(df/dX)]
Intro
At the moment of posting the question I was not really that good at theory of matrix derivatives, but now I know much more all thanks to this Yandex ml book (unfortunately, I didn't find the english equivalent). This is an attempt to give a full answer to my question.
Basic Theory
Forgive me, Lord, for ugly representation of latex
Let's say you have a function which takes matrix X and returns it's squared Frobenius norm: f(X) = ||X||_F^2
It is a well-known fact that: ||X||_F^2 = Tr(X X^T)
Let's define derivative as shown in same book: D[f] at X_0 = f(X + H) - f(X)
We are ready to find dg(X)/dX:
df(X)/dX = dTr(X X^T)/dX =
(using Trace's feature)
= Tr(d/dX[X X^T]) = Tr(dX/dX X^T + X d[X^T]/dX ) =
(then we should use the definition of derivative from above)
= Tr(HX^T + XH^T) = Tr(HX^T) + Tr(XH^T) =
(now the main trick is to get all matrices H on the right side and get something like
Tr(g(X) H) or Tr(g(X) H^T), where g(X) will be the derivative we are looking for)
= Tr(HX^T) + Tr(XH^T) = Tr(XH^T) + Tr(XH^T) = Tr(2*XH^T)
That means: df(X)/dX = 2X
Second order derivative
Now, after we found out how to get matrix derivatives, let's try to find second order derivative of the same function f(X):
d/dX[df(X)/dX] = d/dX[Tr(2XH_1^T)] = Tr(d/dX[2XH_1^T]) =
= Tr(2I H_2 H_1^T)
We found out that d/dX[df(X)/dX] = 2I where I stands for Identity matrix. But how will it help us to find derivatives in Pytorch?
Trace is the trick
As we can see from the formulas, both first and second order derivatives have Trace inside them, but when we take first order derivative we just instantly get matrix as a result. To get a higher order derivative we just need to take the derivative of trace of first order derivative:
d/dY[df/dX] = d/dY[Tr(df/dX)]
The thing is I was using JAX autograd library when this trick came to my mind, so the code with a function f(X,Y) will look like this:
def scalarized_dy(X, Y):
dY = grad(f, argnums=1)(X, Y)
return jnp.trace(dY)
dYX = grad(scalarized_dy, argnums=0)(X, Y)
dYY = grad(scalarized_dy, argnums=1)(X, Y)
In case of Pytorch I guess we will need to look after tensors' gradients (let loss be a function with X and Y as arguments):
loss = f(X, Y)
loss.backward(create_graph = True)
dX = torch.trace(X.grad)
dX.backward()
dXX = X.grad
dXY = Y.grad
Epilogue
I thought that the question itself is in some way interesting. Also, it took me several months to figure things out, so I decided to give my current point of view on this problem. I will not mark my answer as correct yet in hope that I will get some kind of feedback or, perhaps, even better answers or ideas.

Python - Solve time-dependent matrix differential equation

I am facing some problems solving a time-dependent matrix differential equation.
The problem is that the time-dependent coefficient is not just following some time-dependent shape, rather it is the solution of another differential equation.
Up until now, I have considered the trivial case where my coefficient G(t) is just G(t)=pulse(t) where this pulse(t) is a function I define. Here is the code:
# Matrix differential equation
def Leq(t,v,pulse):
v=v.reshape(4,4) #covariance matrix
M=np.array([[-kappa,0,E_0*pulse(t),0],\. #coefficient matrix
[0,-kappa,0,-E_0*pulse(t)],\
[E_0*pulse(t),0,-kappa,0],\
[0,-E_0*pulse(t),0,-kappa]])
Driff=kappa*np.ones((4,4),float) #constant term
dv=M.dot(v)+v.dot(M)+Driff #solve dot(v)=Mv+vM^(T)+D
return dv.reshape(-1) #return vectorized matrix
#Pulse shape
def Gaussian(t):
return np.exp(-(t - t0)**2.0/(tau ** 2.0))
#scipy solver
cov0=np.zeros((4,4),float) ##initial vector
cov0 = cov0.reshape(-1); ## vectorize initial vector
Tmax=20 ##max value for time
Nmax=10000 ##number of steps
dt=Tmax/Nmax ##increment of time
t=np.linspace(0.0,Tmax,Nmax+1)
Gaussian_sol=solve_ivp(Leq, [min(t),max(t)] , cov0, t_eval= t, args=(Gaussian,))
And I get a nice result. The problem is that is it not exactly what I need. Want I need is that dot(G(t))=-kappa*G(t)+pulse(t), i.e. the coefficient is the solution of a differential equation.
I have tried to implement this differential equation in a sort of vectorized way in Leq by returning another parameter G(t) that would be fed to M, but I was getting problems with the dimensions of the arrays.
Any idea of how should I proceed?
Thanks,

In principle you have the right idea, you just have to split and join the state and derivative vectors.
def Leq(t,u,pulse):
v=u[:16].reshape(4,4) #covariance matrix
G=u[16:].reshape(4,4)
... # compute dG and dv
return np.concatenate([dv.flatten(), dG.flatten()])
The initial vector has likewise to be such a composite.

Calculate a plane from point cloud in Python without Numpy

I've seen several posts on this subject, but I need a pure Python (no Numpy or any other imports) solution that accepts a list of points (x,y,z coordinates) and calculates a normal for the closest plane that to those points.
I'm following one of the working Numpy examples from here: Fit points to a plane algorithms, how to iterpret results?
def fitPLaneLTSQ(XYZ):
# Fits a plane to a point cloud,
# Where Z = aX + bY + c ----Eqn #1
# Rearanging Eqn1: aX + bY -Z +c =0
# Gives normal (a,b,-1)
# Normal = (a,b,-1)
[rows,cols] = XYZ.shape
G = np.ones((rows,3))
G[:,0] = XYZ[:,0] #X
G[:,1] = XYZ[:,1] #Y
Z = XYZ[:,2]
(a,b,c),resid,rank,s = np.linalg.lstsq(G,Z)
normal = (a,b,-1)
nn = np.linalg.norm(normal)
normal = normal / nn
return normal
XYZ = np.array([
[0,0,1],
[0,1,2],
[0,2,3],
[1,0,1],
[1,1,2],
[1,2,3],
[2,0,1],
[2,1,2],
[2,2,3]
])
print fitPLaneLTSQ(XYZ)
[ -8.10792259e-17 7.07106781e-01 -7.07106781e-01]
I'm trying to adapt this code: Basic ordinary least squares calculation to replace np.linalg.lstsq
Here is what I have so far without using Numpy using the same coords as above:
xvals = [0,0,0,1,1,1,2,2,2]
yvals = [0,1,2,0,1,2,0,1,2]
zvals = [1,2,3,1,2,3,1,2,3]
""" Basic ordinary least squares calculation. """
sumx, sumy = map(sum, [xvals, yvals])
sumxy = sum(map(lambda x, y: x*y, xvals, yvals))
sumxsq = sum(map(lambda x: x**2, xvals))
Nsamp = len(xvals)
# y = a*x + b
# a (slope)
slope = (Nsamp*sumxy - sumx*sumy) / ((Nsamp*sumxsq - sumx**2))
# b (intercept)
intercept = (sumy - slope*sumx) / (Nsamp)
a = slope
b = intercept
normal = (a,b,-1)
mag = lambda x : math.sqrt(sum(i**2 for i in x))
nn = mag(normal)
normal = [i/nn for i in normal]
print normal
[0.0, 0.7071067811865475, -0.7071067811865475]
As you can see, the answers come out the same, but that is only because of this particular example. In other examples, they don't match. If you look closely you'll see that in the Numpy example the 'z' values are fed into 'np.linalg.lstsq', but in the non-Numpy version the 'z' values are ignored. How do I work in the 'z' values to the least-squares code?
Thanks

I do not think you can get away without implementing some basic matrix operations. As this is a multivariate linear regression problem, you will definitely need dot product, transpose and norm. These are easy. The difficult part is that you also need matrix inverse or QR decomposition or something similar. People usually use BLAS for these for good reasons, implementing them is not easy - but not impossible either.
With QR decomposition
I would start by creating a Matrix class that has the following methods
dot(m1, m2) (or __matmul__(m1, m2) if you have python 3.5): it is just the sum of products, should be straightforward
transpose(self): swapping matrix elements, should be easy
norm(self): square root of sum of squares (should be only used on vectors)
qr_decomp(self): this one is tricky. For an almost pure python implementation see this rosetta code solution (disclaimer: I have not thoroughly checked this code). It uses some numpy functions, but these are basic functions you can implement for your matrix class (shape, eye, dot, copysign, norm).
leastsqr_ut(R, A): solve the equation Rx = A if R is an upper triangular matrix. Not trivial, but should be easy enough as you can solve it equation by equation from the bottom.
With these, the solution is easy:
Generate the matrix G as detailed in your numpy example
Find the QR decomposition of G
Solve Rb = Q'z for b using that R is an upper triangular matrix
Then the normal vector you are looking for is (b[0], b[1], -1) (or the norm of it if you want a unit length normal vector).
With matrix inverse
The inverse of a 3x3 matrix is relatively easy to calculate, but this method is much less numerically stable than doing QR decomposition. If it is not an important concern, then you can do the following: implement
dot(m1, m2) (or __matmul__(m1, m2) if you have python 3.5): it is just the sum of products, should be straightforward
transpose(self): swapping matrix elements, should be easy
norm(self): square root of sum of squares (should be only used on vectors)
det(self): determinant, but it is enough if it works on 2x2 and 3x3 matrices, and for those simple formulas are available
inv(self): matrix inverse. It is enough if it works on 3x3 matrices, there is a simple formula for example here
Then the formula for b is b = inv(G'G) * (G'z) and your normal vector is again (b[0], b[1], -1).
As you can see, none of these are simple, and most of it is replicating some numpy functionality while making it a lot slower lot slower. So make sure you have absolutely no other choice.

I generated a code with a similar purpose (see "tangentplane_3D" function in the linked code).
In my case I had a scatter cloud of points that define a 3D ellipsoid. For each point I wanted to determine the tangent plane to the ellipsoid containing such point --> Goal: Determination of a 3D plane.
The problem can be seen in the following way: A plane is defined by its normal and the normal can be seen as the eigenvector associated to the minimum of the eigenvalues of a n set of points.
What I did, and you can check it on the code I posted, is to select k points close to the point of interest at which I wanted to calculate the tangent plane. Then, I performed a 3D Single Value Decomposition to these k points. Finally, from these SVD I selected the minimum eigenvalue and its associated eigenvector which is, in fact, the normal of the plane best fitting my set of points, and thus in my case, tangent to the ellipsoid plane. With the normal vector and the point you can subsequently calculate the complete plane equation.
I hope it helps!!
Best wishes.

Estimate Euclidean transformation with python

I want to do something similar to what in image analysis would be a standard 'image registration' using features.
I want to find the best transformation that transforms a set of 2D coordinates A in another one B.
But I want to add an extra constraint being that the transformation is 'rigid/Euclidean transformation' Meaning that there is no scaling but only translation and rotation.
Normally allowing scaling I would do:
from skimage import io, transform
destination = array([[1.0,2.0],[1.0,4.0],[3.0,3.0],[3.0,7.0]])
source = array([[1.2,1.7],[1.1,3.8],[3.1,3.4],[2.6,7.0]])
T = transform.estimate_transform('similarity',source,destination)
I believeestimate_transform under the hood just solves a least squares problem.
But I want to add the constraint of no scaling.
Are there any function in skimage or other packages that solve this?
Probably I need to write my own optimization problem with scipy, CVXOPT or cvxpy.
Any help to phrase/implement this optimization problem?
EDIT:
My implementation thanks to Stefan van der Walt Answer
from matplotlib.pylab import *
from scipy.optimize import *
def obj_fun(pars,x,src):
theta, tx, ty = pars
H = array([[cos(theta), -sin(theta), tx],\
[sin(theta), cos(theta), ty],
[0,0,1]])
src1 = c_[src,ones(src.shape[0])]
return sum( (x - src1.dot(H.T)[:,:2])**2 )
def apply_transform(pars, src):
theta, tx, ty = pars
H = array([[cos(theta), -sin(theta), tx],\
[sin(theta), cos(theta), ty],
[0,0,1]])
src1 = c_[src,ones(src.shape[0])]
return src1.dot(H.T)[:,:2]
res = minimize(obj_fun,[0,0,0],args=(dst,src), method='Nelder-Mead')

With that extra constraint you are no longer solving a linear least squares problem, so you'll have to use one of SciPy's minimization functions. The inner part of your minimization would set up a matrix H:
H = np.array([[np.cos(theta), -np.sin(theta), tx],
[np.sin(theta), np.cos(theta), ty],
[0, 0, 1]])
Then, you would compute the distance
|x_target - H.dot(x_source)|
for all data-points and sum the errors. Now, you have a cost function that you can send to the minimization function. You probably will also want to make use of RANSAC, which is available as skimage.measure.ransac, to reject outliers.

skimage now provides native support in the transform module.
http://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.estimate_transform
Somewhat easier than OpenCV I find. There is an extensive set of functions which covers all use cases.

Get the minimal surface solution of a 3D contour

I have a set of 3D points defining a 3D contour.
What I want to do is to obtain the minimal surface representation corresponding to this contour (see Minimal Surfaces in Wikipedia). Basically, this requires to solve a nonlinear partial differential equation.
In Matlab, this is almost straightforward using the pdenonlinfunction (see Matlab's documentation). An example of its usage for solving a minimal surface problem can be found here: Minimal Surface Problem on the Unit Disk.
I need to make such an implementation in Python, but up to know I haven't found any web resources on how to do this.
Can anyone point me any resources/examples of such implementation?
Thanks,
Miguel.
UPDATE
The 3D surface (ideally a triangular mesh representation) I want to find is bounded by this set of 3D points (as seen in this figure, the points lie in the best-fit plane):
Ok, so doing some research I found that this minimal surface problem is related with the solution of the Biharmonic Equation, and I also found that the Thin-plate spline is the fundamental solution to this equation.
So I think the approach would be to try to fit this sparse representation of the surface (given by the 3D contour of points) using thin-plate splines. I found this example in scipy.interpolate where scattered data (x,y,z format) is interpolated using thin-plate splines to obtain the ZI coordinates on a uniform grid (XI,YI).
Two questions arise:
Would thin-plate spline interpolation be the correct approach for the problem of computing the surface from the set of 3D contour points?
If so, how to perform thin-plate interpolation on scipy with a NON-UNIFORM grid?
Thanks again!
Miguel
UPDATE: IMPLEMENTATION IN MATLAB (BUT IT DOESN'T WORK ON SCIPY PYTHON)
I followed this example using Matlab's tpaps function and obtained the minimal surface fitted to my contour on a uniform grid. This is the result in Matlab (looks great!):
However I need to implement this in Python, so I'm using the package scipy.interpolate.Rbf and the thin-plate function. Here's the code in python (XYZ contains the 3D coordinates of each point in the contour):
GRID_POINTS = 25
x_min = XYZ[:,0].min()
x_max = XYZ[:,0].max()
y_min = XYZ[:,1].min()
y_max = XYZ[:,1].max()
xi = np.linspace(x_min, x_max, GRID_POINTS)
yi = np.linspace(y_min, y_max, GRID_POINTS)
XI, YI = np.meshgrid(xi, yi)
from scipy.interpolate import Rbf
rbf = Rbf(XYZ[:,0],XYZ[:,1],XYZ[:,2],function='thin-plate',smooth=0.0)
ZI = rbf(XI,YI)
However this is the result (quite different from that obtained in Matlab):
It's evident that scipy's result does not correspond to a minimal surface.
Is scipy.interpolate.Rbf + thin-plate doing as expected, why does it differ from Matlab's result?

You can use FEniCS:
from fenics import (
UnitSquareMesh,
FunctionSpace,
Expression,
interpolate,
assemble,
sqrt,
inner,
grad,
dx,
TrialFunction,
TestFunction,
Function,
solve,
DirichletBC,
DomainBoundary,
MPI,
XDMFFile,
)
# Create mesh and define function space
mesh = UnitSquareMesh(100, 100)
V = FunctionSpace(mesh, "Lagrange", 2)
# initial guess (its boundary values specify the Dirichlet boundary conditions)
# (larger coefficient in front of the sin term makes the problem "more nonlinear")
u0 = Expression("a*sin(2.5*pi*x[1])*x[0]", a=0.2, degree=5)
u = interpolate(u0, V)
print(
"initial surface area: {}".format(assemble(sqrt(1 + inner(grad(u), grad(u))) * dx))
)
# Define the linearized weak formulation for the Newton iteration
du = TrialFunction(V)
v = TestFunction(V)
q = (1 + inner(grad(u), grad(u))) ** (-0.5)
a = (
q * inner(grad(du), grad(v)) * dx
- q ** 3 * inner(grad(u), grad(du)) * inner(grad(u), grad(v)) * dx
)
L = -q * inner(grad(u), grad(v)) * dx
du = Function(V)
# Newton iteration
tol = 1.0e-5
maxiter = 30
for iter in range(maxiter):
# compute the Newton increment by solving the linearized problem;
# note that the increment has *homogeneous* Dirichlet boundary conditions
solve(a == L, du, DirichletBC(V, 0.0, DomainBoundary()))
u.vector()[:] += du.vector() # update the solution
eps = sqrt(
abs(assemble(inner(grad(du), grad(du)) * dx))
) # check increment size as convergence test
area = assemble(sqrt(1 + inner(grad(u), grad(u))) * dx)
print(
f"iteration{iter + 1:3d} H1 seminorm of delta: {eps:10.2e} area: {area:13.5e}"
)
if eps < tol:
break
if eps > tol:
print("no convergence after {} Newton iterations".format(iter + 1))
else:
print("convergence after {} Newton iterations".format(iter + 1))
with XDMFFile(MPI.comm_world, "out.xdmf") as xdmf_file:
xdmf_file.write(u)
(Modified from http://www-users.math.umn.edu/~arnold/8445/programs/minimalsurf-newton.py.)

Obviously Matlab and SciPy understand TPS in different ways. The Matlab implementation looks correct.
SciPy treats TPS the same way as others RBFs, so you could implement it correctly in Python yourself - it would be enough to form a matrix of the related linear equation system and solve it to receive TPS' coefficients.