We Keep Coding

Improving execution time of this Python "angle-of-arrival" estimation algorithm

Background
I've written a python implementation of this answer to my recent question over on Math SE. In short, the problem is as follows:
I have an experimental setup consisting of three receivers, with known rectangular coordinates [xi, yi, zi], and a transmitter with unknown coordinates [x,y,z] emitting a signal at unknown time t with velocity c and arriving at receiver i at known time ti.
This information is insufficient to uniquely determine the rectangular coordinates of the transmitter. We can, however, estimate the angle to the transmitter (i.e. the transmitter's phi and theta in spherical coordinates). This may be done by solving the system of equations given in the linked post.
My goal is to solve these equations for real, experimental data.
Problem
I have been able to write an effective python implementation of the approach described in the linked post. For both simulated and experimental data, this gives satisfactory estimates of the angle to the transmitter.
I now wish to use this in practice. Unfortunately, however, my code runs too slowly to be useful in my desired application.
Ideally, we'd like to be able to solve for on the order of 1 million datapoints per hour in
a near-real-time application. Currently, this takes several hours. While I recognize that, particularly with Python, dramatic performance improvements are not to be expected, any improvement would be helpful.
In short, I'd like to reduce the execution time of this algorithm. Due to the minimal Linux install (and my lack of control over it) being used on the host machine, I'd like to do so by improving my code, and without use of additional modules/external libraries/etc if possible.
Code
import math
import numpy as np
import ROOT
from scipy.optimize import root
from dataclasses import dataclass
from ROOT import TFile, TNtuple
c = 299792
#dataclass
class Vertexer:
roc: list
def fun(self, var, dat):
f0 = var.dot(self.roc[0] - self.roc[1]) - c * (dat[1] - dat[0])
f1 = var.dot(self.roc[1] - self.roc[2]) - c * (dat[2] - dat[1])
n = np.linalg.norm(var) - 1
return [f0, f1, n]
def find(self, dat):
result = root(
self.fun,
(0, 0, 0),
method="lm",
args=dat,
options={
"col_deriv": 1,
"xtol": 1.49012e-08,
"ftol": 1.49012e-08,
"gtol": 0.0,
"maxiter": 0,
"eps": 0.0,
"factor": 100,
"diag": None,
},
)
if result.success:
return result.x
def main():
myVertexer = Vertexer(
[
np.array([3.0085470085470085, 3.7642857142857116, -0.06]),
np.array([2.0034188034188034, 2.0142857142857133, -0.19]),
np.array([1.0324786324786326, 0.27142857142857135, -0.19]),
]
)
data = ROOT.RDataFrame("D", "2018_filtered.root")
colns = data.AsNumpy(columns=["ai", "aj", "ak"])
f = TFile("theta_phi_2.root", "RECREATE")
ntuple = TNtuple("N", "N", "i:j:k:ai:aj:ak")
for (ai, aj, ak) in zip(colns["ai"], colns["aj"], colns["ak"]):
v = myVertexer.find([ai, aj, ak])
if v.any() != None:
ntuple.Fill(v[0], v[1], v[2], ai, aj, ak)
ntuple.Write()
f.Write()
main()
Step-Through
The Vertexer class contains a fairly standard and straightforward SciPy based system of equations solution "algorithm". The function func() contains the system of equations described in the linked post, and find() solves the system given the times of arrival (dat) and the receiver coordinates (roc, provided upon instantiation). I'm using a Levenberg Marquardt solver with coln_deriv set to True in order to improve solution speed. I see similar solution accuracy across all solvers. All other settings are set to default.
The main() function reads time-of-arrival data from a .root file (see) into a dict of NumPy arrays, which I loop over, feed to the algorithm, and record the result in another .root file.
If you like, you may replace the main() function with the following crude "source simulation" code I've written, which simply produces a random point, computes the arrival time of a signal from that point to three randomly-placed "receivers", and feeds those times to the algorithm:
x0 = random.randrange(0,1000); y0 = random.randrange(0,1000); z0 = random.randrange(0,1000)
x1 = random.randrange(0,50); x2 = random.randrange(0,50); x3 = random.randrange(0,50);
y1 = random.randrange(0,50); y2 = random.randrange(0,50); y3 = random.randrange(0,50);
z1 = random.randrange(0,50); z2 = random.randrange(0,50); z3 = random.randrange(0,50);
t1 = math.sqrt((x0-x1)**2 + (y0-y1)**2 + (z0-z1)**2)/c
t2 = math.sqrt((x0-x2)**2 + (y0-y2)**2 + (z0-z2)**2)/c
t3 = math.sqrt((x0-x3)**2 + (y0-y3)**2 + (z0-z3)**2)/c
myVertexer = Vertexer([[x1,y1,z1], [x2,y2,z2], [x3,y3,z3]])
result = myVertexer.find([t1,t2,t3])

You can solve your equations analytically rather than numerically (probably someone will say that this is not the place for this).
Let's analyze your function
def fun(self, var, dat):
f0 = var.dot(self.roc[0] - self.roc[1]) - c * (dat[1] - dat[0])
f1 = var.dot(self.roc[1] - self.roc[2]) - c * (dat[2] - dat[1])
n = np.linalg.norm(var) - 1
return [f0, f1, n]
The solutions of f0 = 0 lie in a plane normal to self.roc[0] - self.roc[1].
The solutions of f1 = 0 lie in a plane normal to self.roc[1] - self.roc[2].
The solutions of n = 0 lies in a sphere.
The intersection of two planes is a line going on the direction give by the cross product between the normal of the two planes.
The intersection between a line and a sphere will either be (a) one tangent point; (b) two points, one entering, one leaving the sphere; (c) no solution if the line passes is far from the sphere.

How to set fixed step size with scipy.integrate?

I am looking for a way to set a fixed step size for solving my initial value problem by Runge-Kutta method in Python. Accordingly, how I can tell the scipy.integrate.RK45 to keep a constant update (step size) for its integration procedure?
Thank you very much.

Scipy.integrate is usually used with changeable step method by controlling the TOL(one step error) while integrating numerically. The TOL is usually computed by checking with another numerical method. For example RK45 uses the 5th order Runge-Kutta to check the TOL of the 4th order Runge-Kutta method to determine the integrating step.
Hence if you must integrate ODEs with fixed step, just turn off the TOL check by setting atol, rtol with a rather large constant. For example, like the form:
solve_ivp(your function, t_span=[0, 10], y0=..., method="RK45", max_step=0.01, atol = 1, rtol = 1)
The TOL check is set to be so large that the integrating step would be the max_step you choose.

It is quite easy to code the Butcher tableau for the Dormand-Prince RK45 method.
0
1/5 | 1/5
3/10 | 3/40 9/40
4/5 | 44/45 −56/15 32/9
8/9 | 19372/6561 −25360/2187 64448/6561 −212/729
1 | 9017/3168 −355/33 46732/5247 49/176 −5103/18656
1 | 35/384 0 500/1113 125/192 −2187/6784 11/84
-----------------------------------------------------------------------------------------
| 35/384 0 500/1113 125/192 −2187/6784 11/84 0
| 5179/57600 0 7571/16695 393/640 −92097/339200 187/2100 1/40
first in a function for a single step
import numpy as np
def DoPri45Step(f,t,x,h):
k1 = f(t,x)
k2 = f(t + 1./5*h, x + h*(1./5*k1) )
k3 = f(t + 3./10*h, x + h*(3./40*k1 + 9./40*k2) )
k4 = f(t + 4./5*h, x + h*(44./45*k1 - 56./15*k2 + 32./9*k3) )
k5 = f(t + 8./9*h, x + h*(19372./6561*k1 - 25360./2187*k2 + 64448./6561*k3 - 212./729*k4) )
k6 = f(t + h, x + h*(9017./3168*k1 - 355./33*k2 + 46732./5247*k3 + 49./176*k4 - 5103./18656*k5) )
v5 = 35./384*k1 + 500./1113*k3 + 125./192*k4 - 2187./6784*k5 + 11./84*k6
k7 = f(t + h, x + h*v5)
v4 = 5179./57600*k1 + 7571./16695*k3 + 393./640*k4 - 92097./339200*k5 + 187./2100*k6 + 1./40*k7;
return v4,v5
and then in a standard fixed-step loop
def DoPri45integrate(f, t, x0):
N = len(t)
x = [x0]
for k in range(N-1):
v4, v5 = DoPri45Step(f,t[k],x[k],t[k+1]-t[k])
x.append(x[k] + (t[k+1]-t[k])*v5)
return np.array(x)
Then test it for some toy example with known exact solution y(t)=sin(t)
def mms_ode(t,y): return np.array([ y[1], sin(sin(t))-sin(t)-sin(y[0]) ])
mms_x0 = [0.0, 1.0]
and plot the error scaled by h^5
for h in [0.2, 0.1, 0.08, 0.05, 0.01][::-1]:
t = np.arange(0,20,h);
y = DoPri45integrate(mms_ode,t,mms_x0)
plt.plot(t, (y[:,0]-np.sin(t))/h**5, 'o', ms=3, label = "h=%.4f"%h);
plt.grid(); plt.legend(); plt.show()
to get the confirmation that this is indeed an order 5 method, as the graphs of the error coefficients come close together.

By looking at the implementation of the step, you'll find that the best you can do is to control the initial step size (within the bounds set by the minimum and maximum step size) by setting the attribute h_abs prior to calling RK45.step:
In [27]: rk = RK45(lambda t, y: t, 0, [0], 1e6)
In [28]: rk.h_abs = 30
In [29]: rk.step()
In [30]: rk.step_size
Out[30]: 30.0

If you are interested in data-wise fix step size, then I highly recommend you to use the scipy.integrate.solve_ivp function and its t_eval argument.
This function wraps up all of the scipy.integrate ode solvers in one function, thus you have to choose the method by giving value to its method argument. Fortunately, the default method is the RK45, so you don't have to bother with that.
What is more interesting for you is the t_eval argument, where you have to give a flat array. The function samples the solution curve at t_eval values and only returns these points. So if you want a uniform sampling by the step size then just give the t_eval argument the following: numpy.linspace(t0, tf, samplingResolution), where t0 is the start and tf is the end of the simulation.
Thusly you can have uniform sampling without having to resort fix step size that causes instability for some ODEs.

You've said you want a fixed-time step behaviour, not just a fixed evluation time step. Therefore, you have to "hack" your way through that if you not want to reimplement the solver yourself. Just set the integration tolerances atol and rtol to 1e90, and max_step and first_step to the value dt of the time step you want to use. This way the estimated integration error will always be very small, thus tricking the solver into not shrinking the time step dynamically.
However, only use this trick with EXPLICIT algortithms (RK23,RK45,DOP853) !
The implicit algorithms from "solve_ivp" (Radau, BDF, maybe LSODA as well) adjust the precision of the nonlinear Newton solver according to atol and rtol, therefore you might end up having a solution which does not make any sense...

I suggest to write your own rk4 fixed step program in py. There are many internet examples to help. That guarantees that you know precisely how each value is being computed. Furthermore, there will normally be no 0/0 calculations and if so they will be easy to trace and prompt another look at the ode's being solved.

Can't find all roots in Newtons-Raphson method (Python)

The code that I am using work fine for 2 of the 4 roots of my function, I can't seem to make it work for all 4 roots.
from numpy import *
from scipy import *
import numpy as np
def f(x):
return np.arctan(2*(x - 1)) - np.log(abs(x))
def fprime(x):
return (16/5)*((x - (5/4))**2)-1
def newtonRaphson(x0,e=1.0e-3):
for val in range(1, 15):
dx = - f(x0)/fprime(x0)
print(dx)
x0 = x0 + dx
if abs(dx) < e:
return x0,val
print('Too many iterations\n')
root,numIter = newtonRaphson(-1.0)
print ('Root =',root)
print ('Number of iterations =',numIter)
The roots should be -0.300098, 0.425412, 1, 4.09946
When I use -1.0, I get the root -0.300098, and when I use 1, I get the root 1, but I can't seem to get the other 2 roots. What do I need to do to my code to get it to work? Thanks

Your code is good, the problem is your function - it's too complicated for the method. You can either use a more sophisticated root finding method or you can decrease dx and increase the number of iterations. For instance you can use dx/1000 and 1.5 million maximum iterations. That will give you all the roots. For roots 1 and 4.0996 you will have to use a very close guess. The code works well for simple functions like simple quadratic functions.

Optimize this function with numpy (or other vectorization methods)

I am computing with Python a classic calculation in the field of population genetics. I am well aware that there exists many algorithm that do the job but I wanted to build my own for some reason.
The below paragraph is a picture because MathJax is not supported on StackOverflow
I would like to have an efficient algorithm to calculate those Fst. For the moment I only manage to make for loops and no calculations are vectorized How can I make this calculation using numpy (or other vectorization methods)?
Here is a code that I think should do the job:
def Fst(W, p):
I = len(p[0])
K = len(p)
H_T = 0
H_S = 0
for i in xrange(I):
bar_p_i = 0
for k in xrange(K):
bar_p_i += W[k] * p[k][i]
H_S += W[k] * p[k][i] * p[k][i]
H_T += bar_p_i*bar_p_i
H_T = 1 - H_T
H_S = 1 - H_S
return (H_T - H_S) / H_T
def main():
W = [0.2, 0.1, 0.2, 0.5]
p = [[0.1,0.3,0.6],[0,0,1],[0.4,0.5,0.1],[0,0.1,0.9]]
F = Fst(W,p)
print("Fst = " + str(F))
return
main()

There's no reason to use loops here. And you really shouldn't use Numba or Cython for this stuff - linear algebra expressions like the one you have are the whole reason behind vectorized operations in Numpy.
Since this type of problem is going to pop up again and again if you keep using Numpy, I would recommend getting a basic handle on linear algebra in Numpy. You might find this book chapter helpful:
https://www.safaribooksonline.com/library/view/python-for-data/9781449323592/ch04.html
As for your specific situation: start by creating numpy arrays from your variables:
import numpy as np
W = np.array(W)
p = np.array(p)
Now, your \bar p_i^2 are defined by a dot product. That's easy:
bar_p_i = p.T.dot(W)
Note the T, for the transpose, because the dot product takes the sum of the elements indexed by the last index of the first matrix and the first index of the second matrix. The transpose inverts the indices so the first index becomes the last.
You H_t is defined by a sum. That's also easy:
H_T = 1 - bar_p_i.sum()
Similarly for your H_S:
H_S = 1 - ((bar_p_i**2).T.dot(W)).sum()

Solve a system of differential equations using Euler's method

I'm trying to solve a system of ordinary differential equations with Euler's method, but when I try to print velocity I get
RuntimeWarning: overflow encountered in double_scalars
and instead of printing numbers I get nan (not a number). I think the problem might be when defining the acceleration, but I don't know for sure, I would really appreciate if someone could help me.
from numpy import *
from math import pi,exp
d=0.0005*10**-6
a=[]
while d<1.0*10**-6 :
d=d*2
a.append(d)
D=array(a)
def a_particula (D,x, v, t):
cc=((1.00+((2.00*lam)/D))*(1.257+(0.400*exp((-1.10*D)/(2.00*lam)))))
return (g-((densaire*g)/densparticula)-((mu*18.0*v)/(cc*densparticula* (D**2.00))))
def euler (acel,D, x, v, tv, n=15):
nv, xv, vv = tv.size, zeros_like(tv), zeros_like(tv)
xv[0], vv[0] = x, v
for k in range(1, nv):
t, Dt = tv[k-1], (tv[k]-tv[k-1])/float(n)
for i in range(n):
a = acel(D,x, v, t)
t, v = t+Dt, v+a*Dt
x = x+v*Dt
xv[k], vv[k] = x, v
return (xv, vv)
g=(9.80)
densaire= 1.225
lam=0.70*10**-6
densparticula=1000.00
mu=(1.785*10**-5)
tv = linspace(0, 5, 50)
x, v = 0, 0 #initial conditions
for j in range(len(D)):
xx, vv = euler(a_particula, D[j], x, v, tv)
print(D[j],xx,vv)

In future it would be helpful if you included the full warning message in your question - it will contain the line where the problem occurs:
tmp/untitled.py:15: RuntimeWarning: overflow encountered in double_scalars
return (g-((densaire*g)/densparticula)-((mu*18.0*v)/(cc*densparticula* (D**2.00))))
Overflow occurs when the magnitude of a variable exceeds the largest value that can be represented. In this case, double_scalars refers to a 64 bit float, which has a maximum value of:
print(np.finfo(float).max)
# 1.79769313486e+308
So there is a scalar value in the expression:
(g-((densaire*g)/densparticula)-((mu*18.0*v)/(cc*densparticula* (D**2.00))))
that is exceeding ~1.79e308. To find out which one, you can use np.errstate to raise a FloatingPointError when this occurs, then catch it and start the Python debugger:
...
with errstate(over='raise'):
try:
ret = (g-((densaire*g)/densparticula)-((mu*18.0*v)/(cc*densparticula* (D**2.00))))
except FloatingPointError:
import pdb
pdb.set_trace()
return ret
...
From within the debugger you can then check the values of the various parts of this expression. The overflow seems to occur in:
(mu*18.0*v)/(cc*densparticula* (D**2.00))
The first time the warning occurs, (cc*densparticula* (D**2.00) evaluates as 2.3210168586496022e-12, whereas (mu*18.0*v) evaluates as -9.9984582297025182e+299.
Basically you are dividing a very large number by a very small number, and the magnitude of the result is exceeding the maximum value that can be represented. This might be a problem with your math, or it might be that your inputs to the function are not reasonably scaled.

Your system reduces to
dv/dt = a = K - L*v
with K about 10 and L ranging between, at first glance 1e+5 to 1e+10. The actual coefficients used confirm that:
D=1.0000e-09 K= 9.787995 L=1.3843070e+08
D=3.2000e-08 K= 9.787995 L=4.2570244e+06
D=1.0240e-06 K= 9.787995 L=9.0146813e+04
The Euler step for the velocity is
v[j+1]=v[j]+(K-L*v[j])*dt =(1-L*dt)*v[j] + K*dt
For anything resembling the intended friction effect, i.e., the velocity falling to K/L one needs that abs(1-L*dt)<1, and if possible 0<1-L*dt<1, that is, dt < 1/L. Which means here that dt < 1e-10.
To be able to use larger time steps you need to use methods for stiff differential equations, which means implicit methods. The most simple ones are the implicit Euler method, the midpoint method and the trapezoidal method.
Because of the linearity, the midpoint and trapezoidal method amount to the same formula
v[j+1] = v[j] + dt * ( K - L*(v[j]+v[j+1])/2 )
or
v[j+1] = ( (1-L*dt/2)*v[j] + K*dt ) / (1+L*dt/2)
Of course, the most simple method is to just exactly integrate the ODE
(-L*v')/(K-L*v)=-L => K-L*v(t)=C*exp(-L*t), C=K-L*v(0)
v(t)=K/L + exp(-L*t)*(v(0)-K/L)
which integrates to
x(t)=x(0)+K/L*t+(1-exp(-L*t))/L*(v(0)-K/L).
Alternatively it is possible that you made an error in transcribing the physical laws into formulas so that the magnitudes of the constants are all wrong.