Related
I wish to do a curve fit to some tabulated data using my own objective function, not the in-built normal least squares.
I can make the normal curve_fit work, but I can't understand how to properly formulate my objective function to feed it into the method.
I am interested in knowing the values of my fitted curve at each tabulated x value.
x = np.array([-5.0,-4.5,-4.0,-3.5,-3.0,-2.5,-2.0,-1.5,-1.0,-0.5,0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0,4.5,5.0,5.5,6.0,6.5,7.0,7.5,8.0,8.5,9.0,9.5,10.0])
y = np.array([300,300,1000,350,340,1230,500,360,360,920,365,365,350,1000,375,1050,380,385,385,390,400,395,780,410,420,420,415,435,440,435,455])
e = np.array([math.sqrt(i) for i in y]) #uncertainty in y values
def test_func(x, a0, a1):
""" This is the function I want to fit to my data """
return a0 + a1*x
def norm_residual(test_func, x, y, e, params):
""" This calculates the normalised residuals, given the tabulated data and function parameters"""
yhat = test_func(x,*params)
z = (y-yhat)/e
return z
def f(z):
""" This modifies the normalised residual value, depending on it's sign."""
if z <= 0:
return z**2
else:
return 6*np.log(2*z/(np.sqrt(math.pi) * sp.special.erf(z/np.sqrt(2))))-3*np.log(2)
def objective(test_func, x, y, e, params):
"""This returns the sum of the modified normalised residuals. Smaller is better"""
z = norm_residual(test_func, x, y, e, params)
return np.sum(np.array([f(i) for i in z]))
#normal scipy curve fit
params, params_covariance = sp.optimize.curve_fit(test_func, x, y, p0=[0,0])
plt.scatter(x, y, label='Data')
plt.plot(x, test_func(x, params[0], params[1]), label='Fitted function', color="orange")
plt.legend(loc='best')
plt.show()
#how do I use my objective function to do my curve fit?
This is what I came up with, for my slightly more realistic requirements.
Lesson: vectorise everything! Don't just wrap it in a np.vectorize function call. I got a speed-up of ~100x by doing so.
Some guidance taken from https://hernandis.me/2020/04/05/three-examples-of-nonlinear-least-squares-fitting-in-python-with-scipy.html
import numpy as np
import scipy as sp
from scipy import optimize
import matplotlib.pyplot as plt
import math
x_data = np.array([-5.0,-4.5,-4.0,-3.5,-3.0,-2.5,-2.0,-1.5,-1.0,-0.5,0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0,4.5,5.0,5.5,6.0,6.5,7.0,7.5,8.0,8.5,9.0,9.5,10.0])
y_data = np.array([300,300,1000,350,340,1230,500,360,360,920,365,365,350,1000,375,1050,380,385,385,390,400,395,780,410,420,420,415,435,440,435,455])
e_data = np.array([math.sqrt(i) for i in y_data]) #uncertainty in y values
# https://hernandis.me/2020/04/05/three-examples-of-nonlinear-least-squares-fitting-in-python-with-scipy.html
def model(params, x):
"""Calculates the model, given params and x. Is vectorised; can be used with numpy.arrays"""
a0, a1 = params
return a0 + a1 * x
def v_f(z):
"""Modifies the residual. Used when you need to calc your own chi2 value. Is vectorised; can be used with numpy.arrays"""
return np.where(z <= 0, np.square(z), 6*np.log(z/sp.special.erf(z*0.7071067811865475)) - 1.3547481158683645)
def v_f_2(z):
"""Modifies the residual. Used when chi2 is calc'd for you. Is vectorised; can be used with numpy.arrays"""
return np.where(z <= 0, z, np.sqrt(6*np.log(z/sp.special.erf(z*0.7071067811865475)) - 1.3547481158683645))
def objective(params, model_func, data, v_modify_residuals_func = None):
""" Calculates the residuals given a model and data. Is vectorised; can be used with numpy.arrays """
if len(data) == 3:
xd, yd, ed = data
elif len(data) == 2:
xd, yd = data
ed = np.ones(len(xd))
r = (yd - model_func(params, xd)) / ed # r is an array of residuals
if v_modify_residuals_func is not None:
r = v_modify_residuals_func(r)
return r
def objective_sum(params, model_func, data, modify_residuals_func = None):
""" Calculates the sum of the residuals given a model and data. Used when you need to calc your own chi2 value. Is vectorised; can be used with numpy.arrays """
r = objective(params, model_func, data, modify_residuals_func)
return np.sum(r)
def v_cheb(n, x):
""" Calculate a chebyshev polynomial. -1.0 <= x <= 1.0, n >= 0, int. Is vectorised; can be used with numpy.arrays """
return np.cos(n * np.arccos(x))
def bkg_model(params, x):
""" Calculate a bkg curve from a number of chebyshev polynomials. Polynomial degree given by len(params). Is vectorised; can be used with numpy.arrays """
r = 0
for i, p in enumerate(params):
r += p * v_cheb(i, x)
return r
def v_normaliseList(nparray):
""" Given a monotonically increasing x-ordinate array, normalise values in the range -1 <= x <= 1. Is vectorised; can be used with numpy.arrays """
min_ = nparray[0]
max_ = nparray[-1]
r = (2*(nparray - min_)/(max_ - min_)) - 1
return r
initial_params = [0,0]
""" least_squares takes an array of residuals, r, and minimises Sum(r**2) """
results1 = sp.optimize.least_squares(objective,
initial_params,
method = 'lm',
args = [bkg_model,
[v_normaliseList(x_data), y_data, e_data],
v_f_2])
""" minimize takes a scalar, r, and minimises r """
results2 = sp.optimize.minimize(objective_sum,
initial_params,
#method = 'SLSQP',
args = (bkg_model,
(v_normaliseList(x_data), y_data, e_data),
v_f))
print(results1.x)
print(results2.x)
I have difficulty with writing the bounds of parameters in basinhopping.
(x0)=(a, b, c )
a = (0, 100)
b = (0, 0.100)
c = (0, 10)
from scipy.optimize import basinhopping
minimizer_kwargs = { "method": "Nelder-Mead" }
min = basinhopping(rmse, x0, minimizer_kwargs=minimizer_kwargs, T=0.5, stepsize=0.1, niter=200, disp=True)
There are multiple approaches for this, each potentially behaving
differently (common in non-convex global-optimization). The best approach always takes a-priori information about the optimization-problem into consideration!
The most robust general approach (and in my opinion the best) would be a combination of:
A: inner level: bound-constrained local-search
B: outer level: bound-constrained steps
The original author of this optimizer says, relying only on A (as done in both other answers as of now) might fail!
Code:
import numpy as np
from scipy.optimize import basinhopping
""" Example problem
https://docs.scipy.org/doc/scipy-0.19.1/reference/generated/scipy.optimize.basinhopping.html
"""
def func2d(x):
f = np.cos(14.5 * x[0] - 0.3) + (x[1] + 0.2) * x[1] + (x[0] + 0.2) * x[0]
df = np.zeros(2)
df[0] = -14.5 * np.sin(14.5 * x[0] - 0.3) + 2. * x[0] + 0.2
df[1] = 2. * x[1] + 0.2
return f, df
""" Example bounds """
bx0 = (-0.175, 1.)
bx1 = (-0.09, 1.)
bounds = [bx0, bx1]
""" Solve without bounds """
minimizer_kwargs = {"method":"L-BFGS-B", "jac":True}
x0 = [1.0, 1.0]
ret = basinhopping(func2d, x0, minimizer_kwargs=minimizer_kwargs, niter=200)
print(ret.message)
print("unconstrained minimum: x = [%.4f, %.4f], f(x0) = %.4f" % (ret.x[0], ret.x[1],ret.fun))
""" Custom step-function """
class RandomDisplacementBounds(object):
"""random displacement with bounds: see: https://stackoverflow.com/a/21967888/2320035
Modified! (dropped acceptance-rejection sampling for a more specialized approach)
"""
def __init__(self, xmin, xmax, stepsize=0.5):
self.xmin = xmin
self.xmax = xmax
self.stepsize = stepsize
def __call__(self, x):
"""take a random step but ensure the new position is within the bounds """
min_step = np.maximum(self.xmin - x, -self.stepsize)
max_step = np.minimum(self.xmax - x, self.stepsize)
random_step = np.random.uniform(low=min_step, high=max_step, size=x.shape)
xnew = x + random_step
return xnew
bounded_step = RandomDisplacementBounds(np.array([b[0] for b in bounds]), np.array([b[1] for b in bounds]))
""" Custom optimizer """
minimizer_kwargs = {"method":"L-BFGS-B", "jac":True, "bounds": bounds}
""" Solve with bounds """
x0 = [1.0, 1.0]
ret = basinhopping(func2d, x0, minimizer_kwargs=minimizer_kwargs, niter=200, take_step=bounded_step)
print(ret.message)
print("constrained minimum: x = [%.4f, %.4f], f(x0) = %.4f" % (ret.x[0], ret.x[1],ret.fun))
Output:
['requested number of basinhopping iterations completed successfully']
unconstrained minimum: x = [-0.1951, -0.1000], f(x0) = -1.0109
['requested number of basinhopping iterations completed successfully']
constrained minimum: x = [-0.1750, -0.0900], f(x0) = -0.9684
You should use "bounds" parameter in minimizer_kwargs which is passing to scipy.optimize.minimize(). Here is an example:
bnds = ((1, 100), (1, 100), (1,100))# your bounds
def rmse(X):# your function
a,b,c = X[0],X[1],X[2]
return a**2+b**2+c**2
x0 = [10., 10., 10.]
minimizer_kwargs = { "method": "L-BFGS-B","bounds":bnds }
ret = basinhopping(rmse, x0, minimizer_kwargs=minimizer_kwargs,niter=10)
print("global minimum: a = %.4f, b = %.4f c = %.4f | f(x0) = %.4f" % (ret.x[0], ret.x[1], ret.x[2], ret.fun))
The result is
global minimum: a = 1.0000, b = 1.0000 c = 1.0000 | f(x0) = 3.0000
Without boundaries it is (clear) 0,0,0
I know this question is old (the accepted answer is from about 5 years ago) and I wish I could comment to it rather than provide a new "answer" (I lack the reputation to do that, unfortunately), but I would like to ask for some clarification on the accepted answer provided by #sascha on Nov 1, 2017.
Specifically, by looking at the code of the function __call__ of class RandomDisplacementBounds(object), I don't understand why x_new is defined as xnew = x + random_step rather than simply as xnew = random_step.
Probably, I am missing out on something and/or misunderstanding the stated problem and context. My reasoning is that if xmin and xmax passed to the function __init__ are indeed the lower and upper bounds of the vector x and random_step is a point randomly drawn from a uniform distribution between xmin and xmax, then what is called random_step is indeed the new point for the local optimization routine. Isn't this true? I would greatly appreciate any help with this.
I am using Pymc to run a Gibbs sampler on a simple model with the data set as a list with 110 elements (55 observations in each dimension).
log y[i,j,k] = alpha[i,k] + beta[j,k] + mu[k]
where log y follows a multivariate normal distribution (because k = 2) with some covariance matrix that is modeled as rho, sigma1 and sigma2.
After taking log-transformation, the data becomes a list of 110 numbers ranging from 6 to 15.
This is the piece of code that I have used:
import pymc as pm
from pymc import Normal, Uniform, MvNormal, Exponential, Gamma,InverseGamma
from pymc import MCMC
mu = np.zeros(2, dtype=object)
alpha = np.zeros([10,2], dtype = object)
beta = np.zeros([10,2], dtype = object)
for k in range(2):
mu[k] = Normal('mu_{}'.format(k), 0,1000)
for i in range(0,10):
alpha[i][k] = Normal('alpha_{}_{}'.format(i,k), 0, 1000)
beta[i][k] = Normal('beta_{}_{}'.format(i,k), 0, 1000)
rho = Uniform('rho', lower = -1, upper = 1)
sigma1 = InverseGamma('sigma1', 2.0001,1) #sigma squared
sigma2 = InverseGamma('sigma2', 2.0001,1)
#pm.deterministic
def PRECISION():
PREC = [[sigma2/(sigma1*sigma2*(1-rho)),(-rho*
(sigma1*sigma2)**0.5)/(sigma1*sigma2*(1-rho))],[(-rho*
(sigma1*sigma2)**0.5)/(sigma1*sigma2*(1-rho)), sigma1/(sigma1*sigma2*(1-
rho))]]
return PREC
mean = np.zeros([10,10,2])
mean_list_1 = []
mean_list_2 = []
for i in range(10):
for j in range(10):
mean[i,j,0] = mu[0] + alpha[i][0] + beta[j][0]
mean_list_1.append(mean[i,j,0])
mean[i,j,1] = mu[1] + alpha[i][1] + beta[j][1]
mean_list_2.append(mean[i,j,1])
#Restructure the vector
bi_mean = np.zeros(55, dtype = object)
bi_data = np.zeros(55, dtype = object)
log_Y = np.zeros(55, dtype = object)
for i in range(55):
bi_mean[i] = [mean_list_1[i], mean_list_2[i]]
bi_data[i] = [data[i], data[i+55]]
log_Y = [pm.MvNormal('log-Y_{}'.format(i), bi_mean[i], PRECISION, value =
bi_data[i], observed = True) for i in range(55)]
monitor_list = [sigma1, sigma2, rho,mu, alpha, beta,log_Y]
model = MCMC([monitor_list],calc_deviance=True)
model.sample(iter=10000, burn=5000, thin=5)
I tried running it in Pymc but the resulting values of alpha and beta is too small to match the magnitude of the observations. Is there a way that I can check where I go wrong? Thank you.
just learned gradient desc. algo and i tried to implement it, input is set of cordinates on 2D plane, and aim is to predict the line that passes through most of given input points.
using python, i wrote :
def cost( theta, data ):
X, y = data[:, 0], data[:, 1]
m = shape(X)[0]
y = y.reshape(m, 1)
X = c_[ones((m, 1)), X]
J = X.dot(theta) - y
J = J.T.dot(J) / (m)
# print(J[0, 0])
return J
def gradDesc(theta, data):
X = data[:, 0]
y = data[:, 1]
m = shape(X)[0]
X = c_[ones((m, 1)), X]
y = y.reshape(m, 1)
hypo = X.dot(theta)
grad = X.T.dot(hypo - y)/m
# print(grad)
return grad
def run(theta, data ):
result = scipy.optimize.fmin_cg( f = cost, fprime=gradDesc, x0=theta, \
args = (data), maxiter=50, disp=False, full_output=True )
theta = result[0]
minCost = result[1]
return theta, minCost
def main():
data = genfromtxt('in.txt', delimiter=',')
theta = zeros((2, 1))
# plot_samples(data)
run(theta, data)
i tried using fmin_cg() to minimize cost, but one of its parameters 'args' causes an error :
line 282, in function_wrapper
return function(*(wrapper_args + args))
TypeError: gradDesc() takes 2 positional arguments but 5 were given
in docs i read that args is the list of params passed to f and fprime other than the one to be altered to minimize f, which here is data. need help to know where i am going wrong ..
full code : http://ideone.com/E22yzl
Extra arguments need to be a tuple. If you mean to have a single parameter called data, you need to construct a one-element tuple (data,) -- notice the comma. This is not the same as (data), where brackets are effectively ignored.
Example:
>>> def f(x, data):
... return (x - data.sum())**2
...
>>> import numpy as np
>>> data = np.asarray([1., 2., 3.])
>>> fmin_cg(f, x0=11., args=(data,))
Optimization terminated successfully.
Current function value: 0.000000
Iterations: 2
Function evaluations: 15
Gradient evaluations: 5
array([ 6.])
I am writing an automated curve fitting routine for 2D data based on scipy's optimize.leastsq, and it works. However when adding many curves with starting values slighly off I get non-physical results (negative amplitude, for example).
I found this post Scipy: bounds for fitting parameter(s) when using optimize.leastsq and was trying to use the parameter transformation according to Minuit from Cern. In the above mentioned question somebody provided a link to some python code.
code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/leastsqbound.py
I wrote this minimal working example (extending the code)
"""
http://code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/leastsqbound.py
Constrained multivariate Levenberg-Marquardt optimization
"""
from scipy.optimize import leastsq
import numpy as np
import matplotlib.pyplot as plt #new
def internal2external_grad(xi, bounds):
"""
Calculate the internal to external gradiant
Calculates the partial of external over internal
"""
ge = np.empty_like(xi)
for i, (v, bound) in enumerate(zip(xi, bounds)):
a = bound[0] # minimum
b = bound[1] # maximum
if a == None and b == None: # No constraints
ge[i] = 1.0
elif b == None: # only min
ge[i] = v / np.sqrt(v ** 2 + 1)
elif a == None: # only max
ge[i] = -v / np.sqrt(v ** 2 + 1)
else: # both min and max
ge[i] = (b - a) * np.cos(v) / 2.
return ge
def i2e_cov_x(xi, bounds, cov_x):
grad = internal2external_grad(xi, bounds)
grad = grad = np.atleast_2d(grad)
return np.dot(grad.T, grad) * cov_x
def internal2external(xi, bounds):
""" Convert a series of internal variables to external variables"""
xe = np.empty_like(xi)
for i, (v, bound) in enumerate(zip(xi, bounds)):
a = bound[0] # minimum
b = bound[1] # maximum
if a == None and b == None: # No constraints
xe[i] = v
elif b == None: # only min
xe[i] = a - 1. + np.sqrt(v ** 2. + 1.)
elif a == None: # only max
xe[i] = b + 1. - np.sqrt(v ** 2. + 1.)
else: # both min and max
xe[i] = a + ((b - a) / 2.) * (np.sin(v) + 1.)
return xe
def external2internal(xe, bounds):
""" Convert a series of external variables to internal variables"""
xi = np.empty_like(xe)
for i, (v, bound) in enumerate(zip(xe, bounds)):
a = bound[0] # minimum
b = bound[1] # maximum
if a == None and b == None: # No constraints
xi[i] = v
elif b == None: # only min
xi[i] = np.sqrt((v - a + 1.) ** 2. - 1)
elif a == None: # only max
xi[i] = np.sqrt((b - v + 1.) ** 2. - 1)
else: # both min and max
xi[i] = np.arcsin((2.*(v - a) / (b - a)) - 1.)
return xi
def err(p, bounds, efunc, args):
pe = internal2external(p, bounds) # convert to external variables
return efunc(pe, *args)
def calc_cov_x(infodic, p):
"""
Calculate cov_x from fjac, ipvt and p as is done in leastsq
"""
fjac = infodic['fjac']
ipvt = infodic['ipvt']
n = len(p)
# adapted from leastsq function in scipy/optimize/minpack.py
perm = np.take(np.eye(n), ipvt - 1, 0)
r = np.triu(np.transpose(fjac)[:n, :])
R = np.dot(r, perm)
try:
cov_x = np.linalg.inv(np.dot(np.transpose(R), R))
except LinAlgError:
cov_x = None
return cov_x
def leastsqbound(func, x0, bounds, args = (), **kw):
"""
Constrained multivariant Levenberg-Marquard optimization
Minimize the sum of squares of a given function using the
Levenberg-Marquard algorithm. Contraints on parameters are inforced using
variable transformations as described in the MINUIT User's Guide by
Fred James and Matthias Winkler.
Parameters:
* func functions to call for optimization.
* x0 Starting estimate for the minimization.
* bounds (min,max) pair for each element of x, defining the bounds on
that parameter. Use None for one of min or max when there is
no bound in that direction.
* args Any extra arguments to func are places in this tuple.
Returns: (x,{cov_x,infodict,mesg},ier)
Return is described in the scipy.optimize.leastsq function. x and con_v
are corrected to take into account the parameter transformation, infodic
is not corrected.
Additional keyword arguments are passed directly to the
scipy.optimize.leastsq algorithm.
"""
# check for full output
if "full_output" in kw and kw["full_output"]:
full = True
else:
full = False
# convert x0 to internal variables
i0 = external2internal(x0, bounds)
# perfrom unconstrained optimization using internal variables
r = leastsq(err, i0, args = (bounds, func, args), **kw)
# unpack return convert to external variables and return
if full:
xi, cov_xi, infodic, mesg, ier = r
xe = internal2external(xi, bounds)
cov_xe = i2e_cov_x(xi, bounds, cov_xi)
# XXX correct infodic 'fjac','ipvt', and 'qtf'
return xe, cov_xe, infodic, mesg, ier
else:
xi, ier = r
xe = internal2external(xi, bounds)
return xe, ier
# new below
def _evaluate(x, p):
'''
Linear plus Lorentzian curve
p = list with three parameters ([a, b, I, Pos, FWHM])
'''
return p[0] + p[1] * x + p[2] / (1 + np.power((x - p[3]) / (p[4] / 2), 2))
def residuals(p, y, x):
err = _evaluate(x, p) - y
return err
if __name__ == '__main__':
data = np.loadtxt('constraint.dat') # read data
p0 = [5000., 0., 500., 2450., 3] #Start values for a, b, I, Pos, FWHM
constraints = [(4000., None), (-50., 20.), (0., 2000.), (2400., 2451.), (None, None)]
p, res = leastsqbound(residuals, p0, constraints, args = (data[:, 1], data[:, 0]), maxfev = 20000)
print p, res
plt.plot(data[:, 0], data[:, 1]) # plot data
plt.plot(data[:, 0], _evaluate(data[:, 0], p0)) # plot start values
plt.plot(data[:, 0], _evaluate(data[:, 0], p)) # plot fit values
plt.show()
Thats the plot output, where green is the starting conditions and red the fit result:
Is this the correct usage? External2internal conversion just throws a nan if outside the bounds. leastsq seems to be able to handle this?
I uploaded the fitting data here. Just paste into a text file named constraint.dat.
There is already an existing popular constrained Lev-Mar code
http://adsabs.harvard.edu/abs/2009ASPC..411..251M
with the implementation in python
http://code.google.com/p/astrolibpy/source/browse/mpfit/mpfit.py
I would suggest not to reinvent the wheel.
Following sega_sai's answer I came up with this minimal working example using mpfit.py
import matplotlib.pyplot as plt
from mpfit import mpfit
import numpy as np
def _evaluate(p, x):
'''
Linear plus Lorentzian curve
p = list with three parameters ([a, b, I, Pos, FWHM])
'''
return p[0] + p[1] * x + p[2] / (1 + np.power((x - p[3]) / (p[4] / 2), 2))
def residuals(p, fjac = None, x = None, y = None, err = None):
status = 0
error = _evaluate(p, x) - y
return [status, error / err]
if __name__ == '__main__':
data = np.loadtxt('constraint.dat') # read data
x = data[:, 0]
y = data[:, 1]
err = 0 * np.ones(y.shape, dtype = 'float64')
parinfo = [{'value':5000., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'a'},
{'value':0., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'b'},
{'value':500., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'I'},
{'value':2450., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'Pos'},
{'value':3., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'FWHM'}]
fa = {'x':x, 'y':y, 'err':err}
m = mpfit(residuals, parinfo = parinfo, functkw = fa)
print m
The fit results are:
mpfit.py: 3714.97545, 0.484193283, 2644.47271, 2440.13385, 22.1898496
leastsq: 3714.97187, 0.484194545, 2644.46890, 2440.13391, 22.1899295
So conclusion: Both methods work, both allow constraints. But as mpfit comes from a very established sourceI trust it more. Also it honors error values, if available.
Try lmfit-py - https://github.com/newville/lmfit-py
It also uses the Levenberg-Marquardt (LM) algorithm via scipy.optimize.leastsq. Uncertaintes are OK.
It allows you not only to constrain your fitting parameters with bounds but also with mathematical expressions between them without modification of your fitting function.
Forget about using those awful p[0], p[1] ... in fitting function. Just use names of the fitting parameters via the Parameters class.