We Keep Coding

Computationally efficient way of maximizing a complicated multivariable function

I've been trying to find an efficient way of maximizing the following monster function in four variables but the program is taking ages to run and I'm not even sure if the results are correct. Can anyone help me code it better in Python?
Here's the function:
where
a=[p,q,r,s].
Y is the measured data sampled at 30 points.
Here's my code.
import numpy as np
import math
Y=Y_t #Y_t is a predefined column vector with 30 entries.
tstep=0.05 #in s
N=30
cov=np.zeros([30,30])
def R(p,q,r,t):
om_D=p*np.sqrt(1-q**2)
return np.pi*r*(np.exp(-q*p*abs(t)))*(np.cos(om_D*t)+(q/(np.sqrt(1-q**2)))*(np.sin(om_D*abs(t))))/(2*q*(p**3))
def I(m,p):
if m==p:
return 1
else:
return 0
def func(a):
a1=a[0] #natural angular frequency bounds=[3,20]
a2=a[1] #damping ratio bounds=[0,1]
a3=a[2] #psd of forcing signal bounds=[300,600]
a4=a[3] #variance of noise bounds=[0,0.0001] in m
#assuming uniform prior for a, we only have to maximise the likelihood function
for i in range(30):
for j in range(30):
cov[i,j]+=R(a1,a2,a3,(j-i)*tstep)+a4*I(i,j)
P=((2*np.pi)**(-N/2)) * ((np.linalg.det(cov))**(-0.5)) * np.exp((-0.5) *np.linalg.multi_dot([np.transpose(Y),np.linalg.inv(cov),Y]))
return (-1)*P[0]
a_start=[5,0.05,100,0.00001]
bnds=((5,20),(0,1),(300,600),(0,0.0001))
result=spo.differential_evolution(func,bounds=bnds)
print(result.x) ```

There is an issue in cov initialization that is why it does not converge. Also an issue on bound for damping ratio, was (0, 1) now (0.0001, 0.999) the ratio should not be 0 or 1 because if it is there will be division by zero error in R(). Code is fixed now see also the output.
Code
import time
import numpy as np
from scipy.optimize import differential_evolution
Y = [[-0.00445551], [-0.01164452], [-0.02171495], [-0.03475491], [-0.00770873], [ 0.0492236 ],
[ 0.07264838], [ 0.03066707], [-0.02457141], [-0.04065968], [-0.01135125], [ 0.02677074], [ 0.06517749],
[ 0.09611112], [ 0.12300657], [ 0.0923581 ], [ 0.03982604], [-0.01473844], [-0.09024497], [-0.14304097],
[-0.17447606], [-0.16926952], [-0.12006193], [-0.00120763], [ 0.11006087], [ 0.19978283], [ 0.24388584],
[ 0.18768875], [ 0.12844553], [ 0.03099409]] #Y_t is a predefined column vector with 30 entries.
tstep = 0.05 #in s
N = 30
def R(p,q,r,t):
om_D = p*np.sqrt(1-q**2)
return np.pi*r*(np.exp(-q*p*abs(t)))*(np.cos(om_D*t)+(q/(np.sqrt(1-q**2)))*(np.sin(om_D*abs(t))))/(2*q*(p**3))
def I(m,p):
if m==p:
return 1
else:
return 0
def func(a):
cov=np.zeros([N,N])
a1=a[0] #natural angular frequency bounds=[3,20]
a2=a[1] #damping ratio bounds=[0,1]
a3=a[2] #psd of forcing signal bounds=[300,600]
a4=a[3] #variance of noise bounds=[0,0.0001] in m
#assuming uniform prior for a, we only have to maximise the likelihood function
for i in range(N):
for j in range(N):
cov[i,j]+=R(a1,a2,a3,(j-i)*tstep)+a4*I(i,j)
P=((2*np.pi)**(-N/2)) * ((np.linalg.det(cov))**(-0.5)) * np.exp((-0.5) *np.linalg.multi_dot([np.transpose(Y),np.linalg.inv(cov),Y]))
return (-1)*P[0]
if __name__ == '__main__':
t0 = time.perf_counter()
a_start = [5, 0.05, 350, 0.00001]
bnds = ((5, 20), (0.0001, 0.999), (300, 600), (0, 0.0001))
result=differential_evolution(func, x0=a_start, bounds=bnds, maxiter=1000)
print(result)
print(f'elapse: {time.perf_counter() - t0:0.0f}s')
Output
fun: array([-2.76736878e+11])
jac: array([-2.91459845e+11, -4.55652161e+12, 1.27377279e+10, 3.34234132e+14])
message: 'Optimization terminated successfully.'
nfev: 3430
nit: 56
success: True
x: array([ 20. , 0.999, 300. , 0. ])
elapse: 55s
Scipy minimize is very fast
Changes:
from scipy.optimize import minimize
result = minimize(func, x0=a_start, bounds=bnds, options={'maxiter': 100, 'disp': True})
Output:
fun: array([-2.76736878e+11])
hess_inv: <4x4 LbfgsInvHessProduct with dtype=float64>
jac: array([-2.91459845e+11, -4.55652161e+12, 1.27377279e+10, 3.34234132e+14])
message: 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
nfev: 30
nit: 4
njev: 6
status: 0
success: True
x: array([ 20. , 0.999, 300. , 0. ])
elapse: 0.5s
Optuna
Optuna after 1000 trials is right there too.
value is positive here because I use maximize direction. In both scipy's DE and minimize values have to be negated.
best param: {'a1': 20, 'a2': 0.9989999999999999, 'a3': 300, 'a4': 0.0}
best value: 276736878140.3103
best trial num: 73
elapse: 22s

Rank issue with spdiags in Python

I am currently trying to create a sparse matrix that will look like this.
[[ 50. -25. 0. 0.]
[-25. 50. -25. 0.]
[ 0. -25. 50. -25.]
[ 0. 0. -25. 50.]]
But when I run it through I keep getting the value error
'data array must have rank 2' in my data array.
I am positive it is a problem with my B variable. I have tried several things but nothing is working. Any advice?
def sparse(a,b,N):
h = (b-a)/(N+1)
e = np.ones([N,1])/h**2
B = np.array([e, -2*e, e])
diags = np.array([-1,0,1])
A = spdiags(B,diags,N,N).toarray()
return A
print(sparse(0,1,4))

Just change to this:
import numpy as np
from scipy.sparse import spdiags
def sparse(a, b, N):
h = (b - a) / (N + 1)
e = np.ones(N) / h ** 2
diags = np.array([-1, 0, 1])
A = spdiags([-1 * e, 2 * e, -1 * e], diags, N, N).toarray()
return A
print(sparse(0, 1, 4))
Output
[[-50. 25. 0. 0.]
[ 25. -50. 25. 0.]
[ 0. 25. -50. 25.]
[ 0. 0. 25. -50.]]
The main change is this:
e = np.ones([N,1])/h**2
by
e = np.ones(N) / h ** 2
Note that toarray transforms the sparse matrix into a dense one, from the documentation:
Return a dense ndarray representation of this matrix.

Calculate pixelwise distance of 3D tensor in tensorflow?

I am trying to create a 3d distance map (size: W * H * D) in tensorflow to be used in a loss function for training. I have a ground truth (binary volume of size W * H * D) that I will use to create the distance map, i.e. the value of each pixel of my distance map will be the minimum distance of that pixel to the positive valued (i.e pixel=1) shape in the ground truth.
Having issues with the 3d shape problem as L2.NORM reduce the axis to a 2D shape and making this problem fully differentiable. Any advice or pointers would be much appreciated.

If I understand correctly, you want to compute the distance from each position in the volume to the closest position of a given class. For simplicity, I will assume that the interesting class is labelled with 1, but hopefully you can adapt it to your case if it is different. The code is for TensorFlow 2.0, but should work the same for 1.x.
The simplest way to do this is to compute the distance between all the coordinates in the volume against every coordinate with a 1, and then pick the smallest distance from there. You can do that like this:
import tensorflow as tf
# Make input data
w, h, d = 10, 20, 30
w, h, d = 2, 3, 4
t = tf.random.stateless_uniform([w, h, d], (0, 0), 0, 2, tf.int32)
print(t.numpy())
# [[[0 1 0 0]
# [0 0 0 0]
# [1 1 0 1]]
#
# [[1 0 0 0]
# [0 0 0 0]
# [1 1 0 0]]]
# Make coordinates
coords = tf.meshgrid(tf.range(w), tf.range(h), tf.range(d), indexing='ij')
coords = tf.stack(coords, axis=-1)
# Find coordinates that are positive
m = t > 0
coords_pos = tf.boolean_mask(coords, m)
# Find every pairwise distance
vec_d = tf.reshape(coords, [-1, 1, 3]) - coords_pos
# You may choose a difference precision type here
dists = tf.linalg.norm(tf.dtypes.cast(vec_d, tf.float32), axis=-1)
# Find minimum distances
min_dists = tf.reduce_min(dists, axis=-1)
# Reshape
out = tf.reshape(min_dists, [w, h, d])
print(out.numpy().round(3))
# [[[1. 0. 1. 2. ]
# [1. 1. 1.414 1. ]
# [0. 0. 1. 0. ]]
#
# [[0. 1. 1.414 2.236]
# [1. 1. 1.414 1.414]
# [0. 0. 1. 1. ]]]
This may work well enough for you, although it may not be the most efficient solution. The smartest thing would be to search for the closest positive position in the neighboring area of each position, but that is complicated to do effectively, both in general and more so in a vectorized way in TensorFlow. There are however a couple of ways we can improve on the code above. On the one hand, we know that positions with a 1 will always have zero distance, so computing for those is unnecessary. On the other hand, if the 1 class in the 3D volume represents some kind of dense shape, then we could save some time if we only computed the distances against the surface of that shape. All other positive positions will have necessarily a greater distance to positions outside the shape. So we can do the same thing we were doing, but computing only distances from non-positive positions to positive surface positions. You can do that like this:
import tensorflow as tf
# Make input data
w, h, d = 10, 20, 30
w, h, d = 2, 3, 4
t = tf.dtypes.cast(tf.random.stateless_uniform([w, h, d], (0, 0)) > .15, tf.int32)
print(t.numpy())
# [[[1 1 1 1]
# [1 1 1 1]
# [1 1 0 0]]
#
# [[1 1 1 1]
# [1 1 1 1]
# [1 1 1 1]]]
# Find coordinates that are positive and on the surface
# (surrounded but at least one 0)
t_pad_z = tf.pad(t, [(1, 1), (1, 1), (1, 1)]) <= 0
m_pos = t > 0
m_surround_z = tf.zeros_like(m_pos)
# Go through the 6 surrounding positions
for i in range(3):
for s in [slice(None, -2), slice(2, None)]:
slices = tuple(slice(1, -1) if i != j else s for j in range(3))
m_surround_z |= t_pad_z.__getitem__(slices)
# Surface points are positive points surrounded by some zero
m_surf = m_pos & m_surround_z
coords_surf = tf.where(m_surf)
# Find coordinates that are zero
coords_z = tf.where(~m_pos)
# Find every pairwise distance
vec_d = tf.reshape(coords_z, [-1, 1, 3]) - coords_surf
dists = tf.linalg.norm(tf.dtypes.cast(vec_d, tf.float32), axis=-1)
# Find minimum distances
min_dists = tf.reduce_min(dists, axis=-1)
# Put minimum distances in output array
out = tf.scatter_nd(coords_z, min_dists, [w, h, d])
print(out.numpy().round(3))
# [[[0. 0. 0. 0.]
# [0. 0. 0. 0.]
# [0. 0. 1. 1.]]
#
# [[0. 0. 0. 0.]
# [0. 0. 0. 0.]
# [0. 0. 0. 0.]]]
EDIT: Here is one way in which you can divide the distance computations in chunks with a TensorFlow loop:
# Following from before
coords_surf = ...
coords_z = ...
CHUNK_SIZE = 1_000 # Choose chunk size
dtype = tf.float32
# If using TF 2.x you can know in advance the size of the tensor array
# (although the element shape will not be constant due to the last chunk)
num_z = tf.shape(coords_z)[0]
arr = tf.TensorArray(dtype, size=(num_z - 1) // CHUNK_SIZE + 1, element_shape=[None], infer_shape=False)
_, arr = tf.while_loop(lambda i, arr: i < num_z,
lambda i, arr: (i + CHUNK_SIZE, arr.write(i // CHUNK_SIZE,
tf.reduce_min(tf.linalg.norm(tf.dtypes.cast(
tf.reshape(coords_z[i:i + CHUNK_SIZE], [-1, 1, 3]) - coords_surf,
dtype), axis=-1), axis=-1))),
[tf.constant(0, tf.int32), arr])
min_dists = arr.concat()
out = tf.scatter_nd(coords_z, min_dists, [w, h, d])

Scipy - Nan when calculating Mahalanobis distance

When I try to calculate the Mahalanobis distance with the following python code I get some Nan entries in the result. Do you have any insight about why this happens?
My data.shape = (181, 1500)
from scipy.spatial.distance import pdist, squareform
data_log = log2(data + 1) # A log transform that I usually apply to my data
data_centered = data_log - data_log.mean(0) # zero centering
D = squareform( pdist(data_centered, 'mahalanobis' ) )
I also tried:
data_standard = data_centered / data_centered.std(0, ddof=1)
D = squareform( pdist(data_standard, 'mahalanobis' ) )
Also got nans.
The input is not corrupted and other distances, such as correlation distance, can be computed just fine.
For some reason when I reduce the number of features I stop getting Nans. E.g the following examples does not get any Nan:
D = squareform( pdist(data_centered[:,:200], 'mahalanobis' ) )
D = squareform( pdist(data_centered[:,180:480], 'mahalanobis' ) )
while those others get Nans:
D = squareform( pdist(data_centered[:,:300], 'mahalanobis' ) )
D = squareform( pdist(data_centered[:,180:600], 'mahalanobis' ) )
Any clue? Is this an expected behaviour if some condition for the input is not satisfied?

You have fewer observations than features, so the covariance matrix V computed by the scipy code is singular. The code doesn't check this, and blindly computes the "inverse" of the covariance matrix. Because this numerically computed inverse is basically garbage, the product (x-y)*inv(V)*(x-y) (where x and y are observations) might turn out to be negative. Then the square root of that value results in nan.
For example, this array also results in a nan:
In [265]: x
Out[265]:
array([[-1. , 0.5, 1. , 2. , 2. ],
[ 2. , 1. , 2.5, -1.5, 1. ],
[ 1.5, -0.5, 1. , 2. , 2.5]])
In [266]: squareform(pdist(x, 'mahalanobis'))
Out[266]:
array([[ 0. , nan, 1.90394328],
[ nan, 0. , nan],
[ 1.90394328, nan, 0. ]])
Here's the Mahalanobis calculation done "by hand":
In [279]: V = np.cov(x.T)
In theory, V is singular; the following value is effectively 0:
In [280]: np.linalg.det(V)
Out[280]: -2.968550671342364e-47
But inv doesn't see the problem, and returns an inverse:
In [281]: VI = np.linalg.inv(V)
Let's compute the distance between x[0] and x[2] and verify that we get the same non-nan value (1.9039) returned by pdist when we use VI:
In [295]: delta = x[0] - x[2]
In [296]: np.dot(np.dot(delta, VI), delta)
Out[296]: 3.625
In [297]: np.sqrt(np.dot(np.dot(delta, VI), delta))
Out[297]: 1.9039432764659772
Here's what happens when we try to compute the distance between x[0] and x[1]:
In [300]: delta = x[0] - x[1]
In [301]: np.dot(np.dot(delta, VI), delta)
Out[301]: -1.75
Then the square root of that value gives nan.
In scipy 0.16 (to be released in June 2015), you will get an error instead of nan or garbage. The error message describes the problem:
In [4]: x = array([[-1. , 0.5, 1. , 2. , 2. ],
...: [ 2. , 1. , 2.5, -1.5, 1. ],
...: [ 1.5, -0.5, 1. , 2. , 2.5]])
In [5]: pdist(x, 'mahalanobis')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-5-a3453ff6fe48> in <module>()
----> 1 pdist(x, 'mahalanobis')
/Users/warren/local_scipy/lib/python2.7/site-packages/scipy/spatial/distance.pyc in pdist(X, metric, p, w, V, VI)
1298 "singular. For observations with %d "
1299 "dimensions, at least %d observations "
-> 1300 "are required." % (m, n, n + 1))
1301 V = np.atleast_2d(np.cov(X.T))
1302 VI = _convert_to_double(np.linalg.inv(V).T.copy())
ValueError: The number of observations (3) is too small; the covariance matrix is singular. For observations with 5 dimensions, at least 6 observations are required.

Python Scipy Error

import scipy.sparse.linalg as scial
import scipy.sparse as scisp
import numpy
def buildB(A,x,col_size_A):
d = numpy.zeros(col_size_A)
for index in xrange(col_size_A):
d[index] = 2*x[index]-1
tmp = scisp.spdiags(d,0,col_size_A,col_size_A)
return scisp.bmat([[A],[tmp]])
def buildQ(l,row_size_A):
q = numpy.zeros(row_size_A)
for index in xrange(row_size_A):
q[index] = 2*l[index]
return scisp.spdiags(q,0,row_size_A,row_size_A)
def buildh(A,x,b,col_size_A):
p = A.dot(x)
p = numpy.subtract(p, b)
quad = numpy.zeros(col_size_A)
for index in xrange(col_size_A):
quad[index] = x[index]*x[index]-x[index]
return numpy.concatenate((p, quad))
def ini():
A = numpy.array([[1,1],[1,-1]])
b = [1, 0]
c = [1, 1]
col_size_A = 2
row_size_A = 2
main(A,b,c,col_size_A,row_size_A)
def main(A,b,c, col_size_A, row_size_A):
x = numpy.zeros(col_size_A)
l = numpy.zeros(row_size_A*2)
eps = 10e-6
k = 0
while True:
B = buildB(A,x,col_size_A)
Q = buildQ(l[row_size_A/2:row_size_A+1], col_size_A)
Bt = B.transpose()
h = buildh(A,x,b,col_size_A)
g = numpy.add(c,Bt.dot(l))
F = numpy.concatenate((g, h))
print "Iteration " + str(k),
tol = numpy.amax(F)
print "- Tol "+ str(tol)
if tol < eps:
print "Done"
break
tF = -numpy.concatenate((c, h))
FGrad2 = scisp.csc_matrix(scisp.bmat([[Q,Bt],[B, None]]))
print FGrad2
print FGrad2.todense()
print " "
print tF
xdelta = scial.spsolve(FGrad2,tF)
print xdelta
x = x + xdelta[0:col_size_A]
l = x[col_size_A:]
k = k + 1
if __name__ == "__main__":
ini()
The output is:
(2, 0) 1.0
(3, 0) 1.0
(4, 0) -1.0
(2, 1) 1.0
(3, 1) -1.0
(5, 1) -1.0
(0, 2) 1.0
(1, 2) 1.0
(0, 3) 1.0
(1, 3) -1.0
(0, 4) -1.0
(1, 5) -1.0
[[ 0. 0. 1. 1. -1. 0.]
[ 0. 0. 1. -1. 0. -1.]
[ 1. 1. 0. 0. 0. 0.]
[ 1. -1. 0. 0. 0. 0.]
[-1. 0. 0. 0. 0. 0.]
[ 0. -1. 0. 0. 0. 0.]]
lda must be >= MAX(N,1): lda=2 N=3BLAS error: Parameter number 7 passed to cblas_dtrsv had an invalid value
[-1. -1. 1. -0. -0. -0.]
So FGrad2 seems to be a valid csc matrix and tF a valid numpy.array.
What is wrong with this code? I don't even know why the error is before the print of tF even so the error is behind at spsolve
Edit
Ok i fixed that, it is because the first guess for the parameters was wrong leading to a singular matrix, but suppling a valid guess for l, leads to wrong calculation of spsolve
as mentioned i labeled all output as you can see spsolve returns the wrong calculation.
$FGrad2 * xdelta != tF$
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import scipy.sparse.linalg as scial
import scipy.sparse as scisp
import numpy
def buildB(A,x,col_size_A):
d = numpy.zeros(col_size_A)
for index in xrange(col_size_A):
d[index] = 2*x[index]-1
tmp = scisp.spdiags(d,0,col_size_A,col_size_A)
return scisp.bmat([[A],[tmp]])
def buildQ(l,row_size_A):
q = numpy.zeros(row_size_A)
for index in xrange(row_size_A):
q[index] = 2*l[index]
return scisp.spdiags(q,0,row_size_A,row_size_A)
def buildh(A,x,b,col_size_A):
p = A.dot(x)
p = numpy.subtract(p, b)
quad = numpy.zeros(col_size_A)
for index in xrange(col_size_A):
quad[index] = x[index]*x[index]-x[index]
return numpy.concatenate((p, quad))
def ini():
A = numpy.array([[1,1],[1,0]])
b = [1, 0]
c = [1, 1]
col_size_A = 2
row_size_A = 2
main(A,b,c,col_size_A,row_size_A)
def main(A,b,c, col_size_A, row_size_A):
x = numpy.zeros(col_size_A)
x[0] = 0
x[1] = 1
l = numpy.ones(row_size_A*2)
eps = 10e-6
k = 0
while True:
B = buildB(A,x,col_size_A)
Q = buildQ(l[row_size_A:], col_size_A)
Bt = B.transpose()
h = buildh(A,x,b,col_size_A)
g = numpy.add(c,Bt.dot(l))
F = numpy.concatenate((g, h))
print "Iteration " + str(k),
tol = numpy.amax(numpy.absolute(F))
print "- Tol "+ str(tol)
if tol < eps:
print "Done"
print x
break
tF = -numpy.concatenate((c, h))
FGrad2 = scisp.csc_matrix(scisp.bmat([[Q,Bt],[B, None]]))
print "FGrad2"
print FGrad2.todense()
print "tF"
print tF
xdelta = scial.spsolve(FGrad2,tF)
print "spsolution"
print xdelta
print ""
x = x + xdelta[0:col_size_A]
l = xdelta[col_size_A:]
k = k + 1
if __name__ == "__main__":
ini()
Output:
Iteration 0 - Tol 3.0
FGrad2
[[ 2. 0. 1. 1. -1. 0.]
[ 0. 2. 1. 0. 0. 1.]
[ 1. 1. 0. 0. 0. 0.]
[ 1. 0. 0. 0. 0. 0.]
[-1. 0. 0. 0. 0. 0.]
[ 0. 1. 0. 0. 0. 0.]]
tF
[-1. -1. -0. -0. -0. -0.]
spsolution
[-1. -1. -0. -0. -0. -0.]

I think this is failing for you because your matrix is singular. E.g. convert to dense and use the regular numpy.linalg.solve:
>>> xdelta = numpy.linalg.solve(FGrad2.todense(), tF)
...
raise LinAlgError('Singular matrix')
numpy.linalg.linalg.LinAlgError: Singular matrix

The error I get is:
File "stack27538259.py", line 62, in main
xdelta = scial.spsolve(FGrad2,tF)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 143, in spsolve
b, flag, options=options)
RuntimeError: superlu failure (singular matrix?) at line 100 in file scipy/sparse/linalg/dsolve/SuperLU/SRC/dsnode_bmod.c
As xnx wrote, FGrad2 is singular.
np.linalg.det(FGrad2.todense()) # 0.0
(scipy version 0.14.0)
after the change I get:
/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/linsolve.py:145: MatrixRankWarning: Matrix is exactly singular
and
spsolution
[ nan nan nan nan nan nan]
and an infinite loop unless I add k counter and break.

Documentation for cblas_dtrsv may be found (here)
Accordingly,
the routine solves a triangular system A*X = B (presumably)
lda is the leading dimension of matrix B
N is the order of the matrix A
the error message says lda = 2 and N = 3 but lda must be >= MAX(N,1)
Perhaps this helps track down the problem.