We Keep Coding

Finding alpha and beta of beta-binomial distribution with scipy.optimize and loglikelihood

A distribution is beta-binomial if p, the probability of success, in a binomial distribution has a beta distribution with shape parameters α > 0 and β > 0. The shape parameters define the probability of success.
I want to find the values for α and β that best describe my data from the perspective of a beta-binomial distribution. My dataset players consist of data about the number of hits (H), the number of at-bats (AB) and the conversion (H / AB) of a lot of baseball players. I estimate the PDF with the help of the answer of JulienD in Beta Binomial Function in Python
from scipy.special import beta
from scipy.misc import comb
pdf = comb(n, k) * beta(k + a, n - k + b) / beta(a, b)
Next, I write a loglikelihood function that we will minimize.
def loglike_betabinom(params, *args):
"""
Negative log likelihood function for betabinomial distribution
:param params: list for parameters to be fitted.
:param args: 2-element array containing the sample data.
:return: negative log-likelihood to be minimized.
"""
a, b = params[0], params[1]
k = args[0] # the conversion rate
n = args[1] # the number of at-bats (AE)
pdf = comb(n, k) * beta(k + a, n - k + b) / beta(a, b)
return -1 * np.log(pdf).sum()
Now, I want to write a function that minimizes loglike_betabinom
from scipy.optimize import minimize
init_params = [1, 10]
res = minimize(loglike_betabinom, x0=init_params,
args=(players['H'] / players['AB'], players['AB']),
bounds=bounds,
method='L-BFGS-B',
options={'disp': True, 'maxiter': 250})
print(res.x)
The result is [-6.04544138 2.03984464], which implies that α is negative which is not possible. I based my script on the following R-snippet. They get [101.359, 287.318]..
ll <- function(alpha, beta) {
x <- career_filtered$H
total <- career_filtered$AB
-sum(VGAM::dbetabinom.ab(x, total, alpha, beta, log=True))
}
m <- mle(ll, start = list(alpha = 1, beta = 10),
method = "L-BFGS-B", lower = c(0.0001, 0.1))
ab <- coef(m)
Can someone tell me what I am doing wrong? Help is much appreciated!!

One thing to pay attention to is that comb(n, k) in your log-likelihood might not be well-behaved numerically for the values of n and k in your dataset. You can verify this by applying comb to your data and see if infs appear.
One way to amend things could be to rewrite the negative log-likelihood as suggested in https://stackoverflow.com/a/32355701/4240413, i.e. as a function of logarithms of Gamma functions as in
from scipy.special import gammaln
import numpy as np
def loglike_betabinom(params, *args):
a, b = params[0], params[1]
k = args[0] # the OVERALL conversions
n = args[1] # the number of at-bats (AE)
logpdf = gammaln(n+1) + gammaln(k+a) + gammaln(n-k+b) + gammaln(a+b) - \
(gammaln(k+1) + gammaln(n-k+1) + gammaln(a) + gammaln(b) + gammaln(n+a+b))
return -np.sum(logpdf)
You can then minimize the log-likelihood with
from scipy.optimize import minimize
init_params = [1, 10]
# note that I am putting 'H' in the args
res = minimize(loglike_betabinom, x0=init_params,
args=(players['H'], players['AB']),
method='L-BFGS-B', options={'disp': True, 'maxiter': 250})
print(res)
and that should give reasonable results.
You could check How to properly fit a beta distribution in python? for inspiration if you want to rework further your code.

Solve an implicit function

I have an implicit function to solve:
So I tried root finding functions from scipy.optimize:
- fsolve : RuntimeWarning: The iteration is not making good progress, as
measured by the improvement from the last ten iterations.
- excitingmixing : NoConvergence
-brent: RuntimeWarning: invalid value encountered in double_scalars (but
without a chance to set constraints)
I have to confess that I'm not sure about the correct solver. Could someone please help?
Minimal working example:
from scipy.optimize import fsolve, newton, brent, excitingmixing
import numpy as np
def func_zeta(zeta):
k= 1.2
Re = 5000.
d = 0.03
return (2.51 /Re/np.sqrt(zeta)+k/d/3.71) + 10**(0.5/ np.sqrt(zeta))
zeta = fsolve(func_zeta, 64/Re)

There were problems in your translation of the formula. Shouldn't it rather be the following return statement?
return 2.51 / (Re * np.sqrt(zeta)) + k / (d * 3.71) - 10 ** (-0.5 / np.sqrt(zeta))
But even then we get again a RuntimeWarning. Let's try again and substitute zeta:
from scipy.optimize import fsolve
import numpy as np
def zeta_in_disguise(x):
global k, d, Re
return x + 2 * np.log10(2.51 * x / Re + k / (d * 3.71))
k = 1.2
Re = 5000
d = 0.03
#x = 1 / np.sqrt(zeta)
x = fsolve(zeta_in_disguise, 0)
print(x)
#let's test, if x is really the solution to the equation
print(-2 * np.log10(2.51 * x / Re + k / d / 3.71))
Ah, now it is convergent and produces the following output
-2.06528864
And there we have the problem - there is no solution for your given parameters, at least not, when we only consider float numbers.

dblquad in scipy vs. matlab giving different results

I want to double integrate a function. But I get different results when using dblquad over scipy.integrate and matlab. A python implementation of my function to double integrate is like this:
###Python implementation##
import numpy as np
from scipy.integrate import dblquad
def InitialCondition(x_b, y_b, m10, m20, N0):
IC = np.zeros((len(x_b)-1,len(y_b)-1))
for i in xrange(len(x_b) - 1):
for j in xrange(len(y_b) - 1):
IC[i,j], abserr = dblquad(ExponenIC, x_b[i], x_b[i + 1], lambda x: y_b[j], lambda x: y_b[j+1], args=(m10, m20, N0), epsabs=1.49e-15, epsrel=1.49e-15)
return IC
def ExponenIC(x, y, m10, m20, N0):
retVal = (16 * N0) / (m10 * m20) * (x / m10)* (y / m20) * np.exp(-2 * (x / m10) - 2 * (y / m20))
return retVal
if __name__=='__main__':
x_min, x_max = 0.0004, 20.0676
x_b = np.exp(np.linspace(np.log(x_min), np.log(x_max), 4))
y_b = np.copy(x_b)
m10, m20, N0 = 0.04, 0.04, 1
print InitialCondition(x_b, y_b, m10, m20, N0)
But if I repeat the same in matlab, with equivalent implementation and same input as shown below:
%%%Matlab equivalent%%%
function IC = test(x_b, y_b, m10, m20, N0)
for i = 1:length(x_b)-1
for j = 1:length(y_b)-1
IC(i, j) = dblquad(#ExponenIC, x_b(i), x_b(i+1), y_b(j), y_b(j+1), 1e-6, #quad, m10, m20, N0);
end
end
return
function retVal = ExponenIC(x, y, m10, m20, N0)
retVal = (16 * N0) / (m10*m20) * (x / m10) .* (y / m20) .* exp(-2*(x/m10) - 2 * (y/m20));
return
% for calling
x_min = 0.0004;
x_max = 20.0676;
x_b = exp(linspace(log(x_min), log(x_max), 4));
y_b = x_b;
m10 = 0.04;
m20 = 0.04;
N0 = 1;
I = test(x_b, y_b, m10, m20, N0)
Scipy dblquad returns:
[[ 2.84900512e-02 1.40266599e-01 7.34019842e-12]
[ 1.40266599e-01 6.90582083e-01 3.61383932e-11]
[ 7.28723691e-12 3.58776449e-11 1.89113430e-21]]
and Matlab dblquad returns:
IC =
28.4901e-003 140.2666e-003 144.9328e-012
140.2666e-003 690.5820e-003 690.9716e-012
144.9328e-012 690.9716e-012 737.2926e-021
I have tried to change tolerances and order of input but two solutions remained always different. Thus, I am not able to understand which one is accurate and I would like to have it correct in python. Can someone suggest if this is a bug in either of the dblquadsolver or in my code somewhere?

From a glance at the results, the repetition of 690 in the Matlab output (in places where Python has different results) casts a doubt on Matlab's performance.
One of problems with using the (deprecated) function dblquad in Matlab is that the tolerance you specify to it is absolute (to my understanding). This is why when you specify 1e-6, the integrals of order 1e-11 come out wrong. When you replace it by 1e-12, the computation takes a lot longer (because the larger integrals must now be computed to a great precision), yet the smallest of integrals, of size 1e-21, is still wrong.
Hence, you should use a routine that supports relative error tolerance, such as integral2.
Replacing the Matlab line with dblquad by
IC(i, j) = integral2(#(x,y) ExponenIC(x,y, m10, m20, N0), x_b(i), x_b(i+1), y_b(j), y_b(j+1), 'RelTol', 1e-12);
I get
0.0284900512006556 0.14026659933722 7.10653215130477e-12
0.14026659933722 0.690582082532588 3.51109000906259e-11
7.10653215130476e-12 3.5110900090626e-11 1.78512164747727e-21
which roughly agrees with Python output. Still, there remains a substantial difference. To settle the matter definitely, I calculated the integrals analytically. The exact result is
0.0284900512006717 0.140266599337199 7.28723691243472e-12
0.140266599337199 0.690582082532677 3.58776449039036e-11
7.28723691243472e-12 3.58776449039036e-11 1.86394265998016e-21
Neither package achieved the desired accuracy, but Python/scipy was much closer.
For completeness, the loop that outputs the analytic solution:
function IC = test(x_b, y_b, m10, m20, N0)
F = #(x,a) -0.25*exp(-2*x/a)*(2*x+a);
for i = 1:length(x_b)-1
for j = 1:length(y_b)-1
IC(i,j) = (16 * N0) / (m10*m20) *(F(x_b(i+1),m10)-F(x_b(i),m10)) * (F(y_b(j+1),m20)-F(y_b(j),m20));
end
end
end

Solve an implicit ODE (differential algebraic equation DAE)

I'm trying to solve a second order ODE using odeint from scipy. The issue I'm having is the function is implicitly coupled to the second order term, as seen in the simplified snippet (please ignore the pretend physics of the example):
import numpy as np
from scipy.integrate import odeint
def integral(y,t,F_l,mass):
dydt = np.zeros_like(y)
x, v = y
F_r = (((1-a)/3)**2 + (2*(1+a)/3)**2) * v # 'a' implicit
a = (F_l - F_r)/mass
dydt = [v, a]
return dydt
y0 = [0,5]
time = np.linspace(0.,10.,21)
F_lon = 100.
mass = 1000.
dydt = odeint(integral, y0, time, args=(F_lon,mass))
in this case I realise it is possible to algebraically solve for the implicit variable, however in my actual scenario there is a lot of logic between F_r and the evaluation of a and algebraic manipulation fails.
I believe the DAE could be solved using MATLAB's ode15i function, but I'm trying to avoid that scenario if at all possible.
My question is - is there a way to solve implicit ODE functions (DAE) in python( scipy preferably)? And is there a better way to pose the problem above to do so?
As a last resort, it may be acceptable to pass a from the previous time-step. How could I pass dydt[1] back into the function after each time-step?

Quite Old , but worth updating so it may be useful for anyone, who stumbles upon this question. There are quite few packages currently available in python that can solve implicit ODE.
GEKKO (https://github.com/BYU-PRISM/GEKKO) is one of the packages, that specializes on dynamic optimization for mixed integer , non linear optimization problems, but can also be used as a general purpose DAE solver.
The above "pretend physics" problem can be solved in GEKKO as follows.
m= GEKKO()
m.time = np.linspace(0,100,101)
F_l = m.Param(value=1000)
mass = m.Param(value =1000)
m.options.IMODE=4
m.options.NODES=3
F_r = m.Var(value=0)
x = m.Var(value=0)
v = m.Var(value=0,lb=0)
a = m.Var(value=5,lb=0)
m.Equation(x.dt() == v)
m.Equation(v.dt() == a)
m.Equation (F_r == (((1-a)/3)**2 + (2*(1+a)/3)**2 * v))
m.Equation (a == (1000 - F_l)/mass)
m.solve(disp=False)
plt.plot(x)

if algebraic manipulation fails, you can go for a numerical solution of your constraint, running for example fsolve at each timestep:
import sys
from numpy import linspace
from scipy.integrate import odeint
from scipy.optimize import fsolve
y0 = [0, 5]
time = linspace(0., 10., 1000)
F_lon = 10.
mass = 1000.
def F_r(a, v):
return (((1 - a) / 3) ** 2 + (2 * (1 + a) / 3) ** 2) * v
def constraint(a, v):
return (F_lon - F_r(a, v)) / mass - a
def integral(y, _):
v = y[1]
a, _, ier, mesg = fsolve(constraint, 0, args=[v, ], full_output=True)
if ier != 1:
print "I coudn't solve the algebraic constraint, error:\n\n", mesg
sys.stdout.flush()
return [v, a]
dydt = odeint(integral, y0, time)
Clearly this will slow down your time integration. Always check that fsolve finds a good solution, and flush the output so that you can realize it as it happens and stop the simulation.
About how to "cache" the value of a variable at a previous timestep, you can exploit the fact that default arguments are calculated only at the function definition,
from numpy import linspace
from scipy.integrate import odeint
#you can choose a better guess using fsolve instead of 0
def integral(y, _, F_l, M, cache=[0]):
v, preva = y[1], cache[0]
#use value for 'a' from the previous timestep
F_r = (((1 - preva) / 3) ** 2 + (2 * (1 + preva) / 3) ** 2) * v
#calculate the new value
a = (F_l - F_r) / M
cache[0] = a
return [v, a]
y0 = [0, 5]
time = linspace(0., 10., 1000)
F_lon = 100.
mass = 1000.
dydt = odeint(integral, y0, time, args=(F_lon, mass))
Notice that in order for the trick to work the cache parameter must be mutable, and that's why I use a list. See this link if you are not familiar with how default arguments work.
Notice that the two codes DO NOT produce the same result, and you should be very careful using the value at the previous timestep, both for numerical stability and precision. The second is clearly much faster though.

solving for a compound interest between PV and "summation" FV?

Given inputs of:
present value = 11, SUMMATIONS of future values that = 126, and n = 7 (periods of change)
how can I solve for a rate of chain that would create a chain that sums into being the FV? This is different from just solving for a rate of return between 11 and 126. This is solving for the rate of return that allows the summation to 126. Ive been trying different ideas and looking up IRR and NPV functions but the summation aspect is stumping me.
In case the summation aspect isn't clear, if I assume a rate of 1.1, that would turn PV = 11 into a list like so (that adds up to nearly the FV 126), how can I solve for r only knowing n, summation fv and pv?:
11
12.1
13.31
14.641
16.1051
17.71561
19.487171
21.4358881
total = 125.7947691
Thank you.
EDIT:
I attempted to create a sort of iterator, but it's hanging after the first loop...
for r in (1.01,1.02,1.03,1.04,1.05,1.06,1.07,1.08,1.09,1.10,1.11,1.12):
print r
test = round(11* (1-r**8) / (1 - r),0)
print test
while True:
if round(126,0) == round(11* (1-r**8) / (1 - r),0):
answer = r
break
else:
pass
EDIT 2:
IV = float(11)
SV = float(126)
N = 8
# sum of a geometric series: (SV = IV * (1 - r^n) / (1 - r )
# r^n - (SV/PV)*r + ((SV - IV)/IV) = 0
# long form polynomial to be solved, with an n of 3 for example:
# 1r^n + 0r^n + 0r^n + -(SV/PV)r + ((SV - IV)/IV)
# each polynomial coefficient can go into numpy.roots to solve
# for the r that solves for the abcd * R = 0 above.
import numpy
array = numpy.roots([1.,0.,0.,0.,0.,0.,0.,(-SV)/IV,(SV-IV)/IV])
for i in array:
if i > 1:
a = str(i)
b = a.split("+")
answer = float(b[0])
print answer
I'm getting a ValueError that my string "1.10044876702" cant be converted to float. any ideas?
SOLVED: i.real gets the real part of it. no need for split or string conversion ie:
for i in array:
if i > 1:
a = i.real
answer = float(a)
print answer

Sum of a geometric series
Subbing in,
126 = 11 * (1 - r**8) / (1 - r)
where we need to solve for r. After rearranging,
r**8 - (126/11)*r + (115/11) = 0
then using NumPy
import numpy as np
np.roots([1., 0., 0., 0., 0., 0., 0., -126./11, 115./11])
gives
array([-1.37597528+0.62438671j, -1.37597528-0.62438671j,
-0.42293755+1.41183514j, -0.42293755-1.41183514j,
0.74868844+1.1640769j , 0.74868844-1.1640769j ,
1.10044877+0.j , 1.00000000+0.j ])
where the first six roots are imaginary and the last is invalid (gives a div-by-0 in the original equation), so the only useable answer is r = 1.10044877.
Edit:
Per the Numpy docs, np.root expects an array-like object (aka a list) containing the polynomial coefficients. So the parameters above can be read as 1.0*r^8 + 0.*r^7 + 0.*r^6 + 0.*r^5 + 0.*r^4 + 0.*r^3 + 0.*r^2 + -126./11*r + 115./11, which is the polynomial to be solved.
Your iterative solver is pretty crude; it will get you a ballpark answer, but the calculation time is exponential with the desired degree of accuracy. We can do much better!
No general analytic solution is known for an eighth-order equation, so some numeric method is needed.
If you really want to code your own solver from scratch, the simplest is Newton-Raphson method - start with a guess, then iteratively evaluate the function and offset your guess by the error divided by the first derivative to hopefully converge on a root - and hope that your initial guess is a good one and your equation has real roots.
If you care more about getting good answers quickly, np.root is hard to beat - it computes the eigenvectors of the companion matrix to simultaneously find all roots, both real and complex.
Edit 2:
Your iterative solver is hanging because of your while True clause - r never changes in the loop, so you will never break. Also, else: pass is redundant and can be removed.
After a good bit of rearranging, your code becomes:
import numpy as np
def iterative_test(rng, fn, goal):
return min(rng, key=lambda x: abs(goal - fn(x)))
rng = np.arange(1.01, 1.20, 0.01)
fn = lambda x: 11. * (1. - x**8) / (1. - x)
goal = 126.
sol = iterative_test(rng, fn, goal)
print('Solution: {} -> {}'.format(sol, fn(sol)))
which results in
Solution: 1.1 -> 125.7947691
Edit 3:
Your last solution is looking much better, but you must keep in mind that the degree of the polynomial (and hence the length of the array passed to np.roots) changes as the number of periods changes.
import numpy as np
def find_rate(present_value, final_sum, periods):
"""
Given the initial value, sum, and number of periods in
a geometric series, solve for the rate of growth.
"""
# The formula for the sum of a geometric series is
# final_sum = sum_i[0..periods](present_value * rate**i)
# which can be reduced to
# final_sum = present_value * (1 - rate**(periods+1) / (1 - rate)
# and then rearranged as
# rate**(periods+1) - (final_sum / present_value)*rate + (final_sum / present_value - 1) = 0
# Build the polynomial
poly = [0.] * (periods + 2)
poly[ 0] = 1.
poly[-2] = -1. * final_sum / present_value
poly[-1] = 1. * final_sum / present_value - 1.
# Find the roots
roots = np.roots(poly)
# Discard unusable roots
roots = [rt for rt in roots if rt.imag == 0. and rt.real != 1.]
# Should be zero or one roots left
if len(roots):
return roots[0].real
else:
raise ValueError('no solution found')
def main():
pv, fs, p = 11., 126., 7
print('Solution for present_value = {}, final_sum = {}, periods = {}:'.format(pv, fs, p))
print('rate = {}'.format(find_rate(pv, fs, p)))
if __name__=="__main__":
main()
This produces:
Solution for present_value = 11.0, final_sum = 126.0, periods = 7:
rate = 1.10044876702

Solving the polynomial roots is overkill. This computation is usually made with a solver such as the Newton method directly applied to the exponential formula. Works for fractional durations too.
For example, https://math.stackexchange.com/questions/502976/newtons-method-annuity-due-equation