I have difficulty with writing the bounds of parameters in basinhopping.
(x0)=(a, b, c )
a = (0, 100)
b = (0, 0.100)
c = (0, 10)
from scipy.optimize import basinhopping
minimizer_kwargs = { "method": "Nelder-Mead" }
min = basinhopping(rmse, x0, minimizer_kwargs=minimizer_kwargs, T=0.5, stepsize=0.1, niter=200, disp=True)
There are multiple approaches for this, each potentially behaving
differently (common in non-convex global-optimization). The best approach always takes a-priori information about the optimization-problem into consideration!
The most robust general approach (and in my opinion the best) would be a combination of:
A: inner level: bound-constrained local-search
B: outer level: bound-constrained steps
The original author of this optimizer says, relying only on A (as done in both other answers as of now) might fail!
Code:
import numpy as np
from scipy.optimize import basinhopping
""" Example problem
https://docs.scipy.org/doc/scipy-0.19.1/reference/generated/scipy.optimize.basinhopping.html
"""
def func2d(x):
f = np.cos(14.5 * x[0] - 0.3) + (x[1] + 0.2) * x[1] + (x[0] + 0.2) * x[0]
df = np.zeros(2)
df[0] = -14.5 * np.sin(14.5 * x[0] - 0.3) + 2. * x[0] + 0.2
df[1] = 2. * x[1] + 0.2
return f, df
""" Example bounds """
bx0 = (-0.175, 1.)
bx1 = (-0.09, 1.)
bounds = [bx0, bx1]
""" Solve without bounds """
minimizer_kwargs = {"method":"L-BFGS-B", "jac":True}
x0 = [1.0, 1.0]
ret = basinhopping(func2d, x0, minimizer_kwargs=minimizer_kwargs, niter=200)
print(ret.message)
print("unconstrained minimum: x = [%.4f, %.4f], f(x0) = %.4f" % (ret.x[0], ret.x[1],ret.fun))
""" Custom step-function """
class RandomDisplacementBounds(object):
"""random displacement with bounds: see: https://stackoverflow.com/a/21967888/2320035
Modified! (dropped acceptance-rejection sampling for a more specialized approach)
"""
def __init__(self, xmin, xmax, stepsize=0.5):
self.xmin = xmin
self.xmax = xmax
self.stepsize = stepsize
def __call__(self, x):
"""take a random step but ensure the new position is within the bounds """
min_step = np.maximum(self.xmin - x, -self.stepsize)
max_step = np.minimum(self.xmax - x, self.stepsize)
random_step = np.random.uniform(low=min_step, high=max_step, size=x.shape)
xnew = x + random_step
return xnew
bounded_step = RandomDisplacementBounds(np.array([b[0] for b in bounds]), np.array([b[1] for b in bounds]))
""" Custom optimizer """
minimizer_kwargs = {"method":"L-BFGS-B", "jac":True, "bounds": bounds}
""" Solve with bounds """
x0 = [1.0, 1.0]
ret = basinhopping(func2d, x0, minimizer_kwargs=minimizer_kwargs, niter=200, take_step=bounded_step)
print(ret.message)
print("constrained minimum: x = [%.4f, %.4f], f(x0) = %.4f" % (ret.x[0], ret.x[1],ret.fun))
Output:
['requested number of basinhopping iterations completed successfully']
unconstrained minimum: x = [-0.1951, -0.1000], f(x0) = -1.0109
['requested number of basinhopping iterations completed successfully']
constrained minimum: x = [-0.1750, -0.0900], f(x0) = -0.9684
You should use "bounds" parameter in minimizer_kwargs which is passing to scipy.optimize.minimize(). Here is an example:
bnds = ((1, 100), (1, 100), (1,100))# your bounds
def rmse(X):# your function
a,b,c = X[0],X[1],X[2]
return a**2+b**2+c**2
x0 = [10., 10., 10.]
minimizer_kwargs = { "method": "L-BFGS-B","bounds":bnds }
ret = basinhopping(rmse, x0, minimizer_kwargs=minimizer_kwargs,niter=10)
print("global minimum: a = %.4f, b = %.4f c = %.4f | f(x0) = %.4f" % (ret.x[0], ret.x[1], ret.x[2], ret.fun))
The result is
global minimum: a = 1.0000, b = 1.0000 c = 1.0000 | f(x0) = 3.0000
Without boundaries it is (clear) 0,0,0
I know this question is old (the accepted answer is from about 5 years ago) and I wish I could comment to it rather than provide a new "answer" (I lack the reputation to do that, unfortunately), but I would like to ask for some clarification on the accepted answer provided by #sascha on Nov 1, 2017.
Specifically, by looking at the code of the function __call__ of class RandomDisplacementBounds(object), I don't understand why x_new is defined as xnew = x + random_step rather than simply as xnew = random_step.
Probably, I am missing out on something and/or misunderstanding the stated problem and context. My reasoning is that if xmin and xmax passed to the function __init__ are indeed the lower and upper bounds of the vector x and random_step is a point randomly drawn from a uniform distribution between xmin and xmax, then what is called random_step is indeed the new point for the local optimization routine. Isn't this true? I would greatly appreciate any help with this.
Related
In order to circumvent the cauchy principle value, I tried to integrate an integral using a small shift iε into the complex plane to evade the pole. However, as can be inferred from the figure below, the result is pretty bad. The code for this result is shown below. Do you have ideas how to improve this method? Why is it not working? I already tried changing ε or the limit in the integral.
Edit: I included the method "cauchy" with the principle value, which seems not to work at all.
import matplotlib.pyplot as plt
from scipy.integrate import quad
import numpy as np
def cquad(func, a, b, **kwargs):
real_integral = quad(lambda x: np.real(func(x)), a, b, limit = 200,**kwargs)
imag_integral = quad(lambda x: np.imag(func(x)), a, b, limit = 200,**kwargs)
return (real_integral[0] + 1j*imag_integral[0], real_integral[1:], imag_integral[1:])
def k_(a):
ϵ = 1e-32
return (cquad(lambda x: np.exp(-1j*x)/(x**2 - a**2 - 1j*ϵ),-np.inf,np.inf)[0])
def k2_(a):
return (cquad(lambda x: np.exp(-1j*x)/(x**2 - a**2),-1e6,1e6, weight='cauchy', wvar = a)[0])
k = np.vectorize(k_)
k2 = np.vectorize(k2_)
fig, ax = plt.subplots()
a = np.linspace(-10,10,300)
ax.plot(a,np.real(k(a)),".-",label = "numerical result")
ax.plot(a,np.real(k2(a)),".-",label = "numerical result (cauchy)")
ax.plot(a, - np.pi*np.sin(a)/a,"-",label="analytical result")
ax.set_ylim(-5,5)
ax.set_ylabel("f(x)")
ax.set_xlabel("x")
ax.set_title(r"$\mathcal{P}\int_{-\infty}^{\infty} \frac{e^{-i y}}{y^2 - x^2}\mathrm{d}y = -\frac{\pi\sin(x)}{x}$")
plt.legend()
plt.savefig("./bad_result.png")
plt.show()
The main problem is that the integrand has poles at both x=a and
x=-a. ev-br's post show how
to deal with a pole at x=a. All that's needed then is to find a way to
massage the integral into a form that avoids integrating through the other pole
at x=-a. Taking advantage of evenness allows us to "fold the integral over",
so instead of having two poles we just need to deal with one pole at x=a.
The real part of
np.exp(-1j*x) / (x**2 - a**2) = (np.cos(x) - 1j * np.sin(x)) / (x**2 - a**2)
is an even function of x so integrating the real part from x = -infinity to
infinity would equal twice the integral from x = 0 to infinity. The
imaginary part of the integrand is an odd function of x. The integral from x = -infinity to infinity equals the integral from x = -infinity to 0, plus
the integral from x = 0 to infinity. These two parts cancel each other out
since the (imaginary) integrand is odd. So the integral of the imaginary part equals 0.
Finally, using ev-br's suggestion, since
1 / (x**2 - a**2) = 1 / ((x - a)(x + a))
using weight='cauchy', wvar=a implicitly weights the integrand by 1 / (x - a) thus allowing us to reduce the explicit integrand to
np.cos(x) / (x + a)
Since the integrand is an even function of a, we can assume without loss of generality that a is positive:
a = abs(a)
Now integrating from x = 0 to infinity avoids the pole at x = -a.
import matplotlib.pyplot as plt
from scipy.integrate import quad
import numpy as np
def cquad(func, a, b, **kwargs):
real_integral = quad(lambda x: np.real(func(x)), a, b, limit=200, **kwargs)
imag_integral = quad(lambda x: np.imag(func(x)), a, b, limit=200, **kwargs)
return (real_integral[0] + 1j*imag_integral[0], real_integral[1:], imag_integral[1:])
def k2_(a):
a = abs(a)
# return 2*(cquad(lambda x: np.exp(-1j*x)/(x + a), 0, 1e6, weight='cauchy', wvar=a)[0]) # also works
# return 2*(cquad(lambda x: np.cos(x)/(x + a), 0, 1e6, weight='cauchy', wvar=a)[0]) # also works, but not necessary
return 2*quad(lambda x: np.cos(x)/(x + a), 0, 1e6, limit=200, weight='cauchy', wvar=a)[0]
k2 = np.vectorize(k2_)
fig, ax = plt.subplots()
a = np.linspace(-10, 10, 300)
ax.plot(a, np.real(k2(a)), ".-", label="numerical result (cauchy)")
ax.plot(a, - np.pi*np.sin(a)/a, "-", label="analytical result")
ax.set_ylim(-5, 5)
ax.set_ylabel("f(x)")
ax.set_xlabel("x")
ax.set_title(
r"$\mathcal{P}\int_{-\infty}^{\infty} \frac{e^{-i y}}{y^2 - x^2}\mathrm{d}y = -\frac{\pi\sin(x)}{x}$")
plt.legend()
plt.show()
You can use instead the weight="cauchy" argument to quad.
https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.quad.html
I have the same problem as in this question but don't want to add only one but several constraints to the optimization problem.
So e.g. I want to maximize x1 + 5 * x2 with the constraints that the sum of x1 and x2 is smaller than 5 and x2 is smaller than 3 (needless to say that the actual problem is far more complicated and cannot just thrown into scipy.optimize.minimize as this one; it just serves to illustrate the problem...).
I can to an ugly hack like this:
from scipy.optimize import differential_evolution
import numpy as np
def simple_test(x, more_constraints):
# check wether all constraints evaluate to True
if all(map(eval, more_constraints)):
return -1 * (x[0] + 5 * x[1])
# if not all constraints evaluate to True, return a positive number
return 10
bounds = [(0., 5.), (0., 5.)]
additional_constraints = ['x[0] + x[1] <= 5.', 'x[1] <= 3']
result = differential_evolution(simple_test, bounds, args=(additional_constraints, ), tol=1e-6)
print(result.x, result.fun, sum(result.x))
This will print
[ 1.99999986 3. ] -16.9999998396 4.99999985882
as one would expect.
Is there a better/ more straightforward way to add several constraints than using the rather 'dangerous' eval?
An example is something like this::
additional_constraints = [lambda(x): x[0] + x[1] <= 5., lambda(x):x[1] <= 3]
def simple_test(x, more_constraints):
# check wether all constraints evaluate to True
if all(constraint(x) for constraint in more_constraints):
return -1 * (x[0] + 5 * x[1])
# if not all constraints evaluate to True, return a positive number
return 10
There is a proper solution to the problem described in the question, to enforce multiple nonlinear constraints with scipy.optimize.differential_evolution.
The proper way is by using the scipy.optimize.NonlinearConstraint function.
Here below I give a non-trivial example of optimizing the classic Rosenbrock function inside a region defined by the intersection of two circles.
import numpy as np
from scipy import optimize
# Rosenbrock function
def fun(x):
return 100*(x[1] - x[0]**2)**2 + (1 - x[0])**2
# Function defining the nonlinear constraints:
# 1) x^2 + (y - 3)^2 < 4
# 2) (x - 1)^2 + (y + 1)^2 < 13
def constr_fun(x):
r1 = x[0]**2 + (x[1] - 3)**2
r2 = (x[0] - 1)**2 + (x[1] + 1)**2
return r1, r2
# No lower limit on constr_fun
lb = [-np.inf, -np.inf]
# Upper limit on constr_fun
ub = [4, 13]
# Bounds are irrelevant for this problem, but are needed
# for differential_evolution to compute the starting points
bounds = [[-2.2, 1.5], [-0.5, 2.2]]
nlc = optimize.NonlinearConstraint(constr_fun, lb, ub)
sol = optimize.differential_evolution(fun, bounds, constraints=nlc)
# Accurate solution by Mathematica
true = [1.174907377273171, 1.381484428610871]
print(f"nfev = {sol.nfev}")
print(f"x = {sol.x}")
print(f"err = {sol.x - true}\n")
This prints the following with default parameters:
nfev = 636
x = [1.17490808 1.38148613]
err = [7.06260962e-07 1.70116282e-06]
Here is a visualization of the function (contours) and the feasible region defined by the nonlinear constraints (shading inside the green line). The constrained global minimum is indicated by the yellow dot, while the magenta one shows the unconstrained global minimum.
This constrained problem has an obvious local minimum at (x, y) ~ (-1.2, 1.4) on the boundary of the feasible region which will make local optimizers fail to converge to the global minimum for many starting locations. However, differential_evolution consistently finds the global minimum as expected.
Trying to optimize using scipy.optimize.linprog a cost function, where the cost coefficients are function of the variables; e.g.
Cost = c1 * x1 + c2 * x2 # (x1,x2 are the variables)
for example
if x1 = 1, c1 = 0.5
if x1 = 2, c1 = 1.25
etc.
* Just to clarify *
we are looking for a minimum cost of variables; xi; i=1,2,3,...
xi are positive integers.
however, the cost coefficient per xi, is a function of the value of xi.
cost is x1*f1(x1) + x2*f2(x2) + ... + c0
fi - is a "rate" table; e.g. - f1(0) = 0; f1(1) = 2.00; f1(2) = 3.00, etc.
the xi are under constrains, and they can't be negative and can't be over qi =>
0 <= xi <= qi
fi() values are calculated for each possible value of xi
I hope it clarifies the model.
Here is some prototype-code to show you how, that your problem is quite hard (regarding formulation and performance; the former is visible in the code).
The implementation uses cvxpy for modelling (convex-programming only) and is based on the mixed-integer approach.
Code
import numpy as np
from cvxpy import *
"""
x0 == 0 -> f(x) = 0
x0 == 1 -> f(x) = 1
...
x1 == 0 -> f(x) = 1
x1 == 1 -> f(x) = 4
...
"""
rate_table = np.array([[0, 1, 3, 5], [1, 4, 5, 6], [1.3, 1.7, 2.25, 3.0]])
bounds_x = (0, 3) # inclusive; bounds are needed for linearization!
# Vars
# ----
n_vars = len(rate_table)
n_values_per_var = [len(x) for x in rate_table]
I = Bool(n_vars, n_values_per_var[0]) # simplified assumption: rate-table sizes equal
X = Int(n_vars)
X_ = Variable(n_vars, n_values_per_var[0]) # X_ = mul_elemwise(I*X) broadcasted
# Constraints
# -----------
constraints = []
# X is bounded
constraints.append(X >= bounds_x[0])
constraints.append(X <= bounds_x[1])
# only one value in rate-table active (often formulated with SOS-type-1 constraints)
for i in range(n_vars):
constraints.append(sum_entries(I[i, :]) <= 1)
# linearization of product of BIN * INT (INT needs to be bounded!)
# based on Erwin's answer here:
# https://www.or-exchange.org/questions/10775/how-to-linearize-product-of-binary-integer-and-integer-variables
for i in range(n_values_per_var[0]):
constraints.append(bounds_x[0] * I[:, i] <= X_[:, i])
constraints.append(X_[:, i] <= bounds_x[1] * I[:, i])
constraints.append(X - bounds_x[1]*(1-I[:, i]) <= X_[:, i])
constraints.append(X_[:, i] <= X - bounds_x[0]*(1-I[:, i]))
# Fix chosings -> if table-entry x used -> integer needs to be x
# assumptions:
# - table defined for each int
help_vec = np.arange(n_values_per_var[0])
constraints.append(I * help_vec == X)
# ONLY FOR DEBUGGING -> make simple max each X solution infeasible
constraints.append(sum_entries(mul_elemwise([1, 3, 2], square(X))) <= 15)
# Objective
# ---------
objective = Maximize(sum_entries(mul_elemwise(rate_table, X_)))
# Problem & Solve
# ---------------
problem = Problem(objective, constraints)
problem.solve() # choose other solver if needed, e.g. commercial ones like Gurobi, Cplex
print('Max-objective: ', problem.value)
print('X:\n' + str(X.value))
Output
('Max-objective: ', 20.70000000000001)
X:
[[ 3.]
[ 1.]
[ 1.]]
Idea
Transform the objective max: x0*f(x0) + x1*f(x1) + ...
into: x0*f(x0==0) + x0*f(x0==1) + ... + x1*f(x1==0) + x1*f(x1==1)+ ...
Introduce binary-variables to formulate:
f(x0==0) as I[0,0]*table[0,0]
f(x1==2) as I[1,2]*table[0,2]
Add constraints to limit the above I to have one nonzero entry only for each variable x_i (only one of the expanded objective-components will be active)
Linearize the product x0*f(x0==0) == x0*I[0,0]*table(0,0) (integer * binary * constant)
Fix the table-lookup: using table-entry with index x (of x0) should result in x0 == x
assuming, that there are no gaps in the table, this can be done formulated as I * help_vec == X) where help_vec == vector(lower_bound, ..., upper_bound)
cvxpy is automatically (by construction) proving, that our formulation is convex, which is needed for most solvers (and in general not easy to recognize).
Just for fun: bigger-problem and commercial-solver
Input generated by:
def gen_random_growing_table(size):
return np.cumsum(np.random.randint(1, 10, size))
SIZE = 100
VARS = 100
rate_table = np.array([gen_random_growing_table(SIZE) for v in range(VARS)])
bounds_x = (0, SIZE-1) # inclusive; bounds are needed for linearization!
...
...
constraints.append(sum_entries(square(X)) <= 150)
Output:
Explored 19484 nodes (182729 simplex iterations) in 129.83 seconds
Thread count was 4 (of 4 available processors)
Optimal solution found (tolerance 1.00e-04)
Warning: max constraint violation (1.5231e-05) exceeds tolerance
Best objective -1.594000000000e+03, best bound -1.594000000000e+03, gap 0.0%
('Max-objective: ', 1594.0000000000005)
I'm trying to maximize a sum of two monotonically increasing functions f(x) = f1(x) + f2(x) within the given bounds, say x = 0 to 6. The curves of the two functions are:
To solve this I'm using basinhopping function from scipy package.
I would like to specify a constraint for using the bounds. Specifically, I want the summation of bounds to be lesser than or equal to a constant value. i.e. In my implementation below, I want x[0] + x[1] <= C where C = 6.
In the above figure, for C = 6, approximately x[0] = 2 and x[1] = 4 (4 + 2 =<=6) will yield the maximum value. My question is how to specify this constraint? If it's not possible, is there another optimization function that is better suited for this problem?
from scipy.optimize import basinhopping
from math import tanh
def f(x):
return -(f1(x[0]) + f2(x[1])) # -ve sign for maximization
def f1(x):
return tanh(x)
def f2(x):
return (10 ** (0.2 * x))
# Starting point
x0 = [1., 1.]
# Bounds
xmin = [1., 1.]
xmax = [6., 6.]
# rewrite the bounds in the way required by L-BFGS-B
bounds = [(low, high) for low, high in zip(xmin, xmax)]
minimizer_kwargs = dict(method="L-BFGS-B", bounds=bounds)
result = basinhopping(f,
x0,
minimizer_kwargs=minimizer_kwargs,
niter=200)
print("global max: x = [%.4f, %.4f], f(x0) = %.4f" % (result.x[0], result.x[1], result.fun))
This is possible with COBYLA constrained optimization, though I don't know how to do it with L-BFGS-B. You can add constraints like the following:
def constraint1(x):
return 6 - x[0]
def constraint2(x):
return 6 - x[0] - x[1]
def constraint3(x):
return x[0] - 1
To add these constraints to the minimizer, use:
c1={"type":"ineq","fun":constraint1}
c2={"type":"ineq","fun":constraint2}
c3={"type":"ineq","fun":constraint3}
minimizer_kwargs = dict(method="COBYLA",constraints=(c1,c2,c3))
When I run this example, I get the result global max: x = [1.0000, 5.0000], f(x0) = -10.7616
Note, I haven't tested whether you need to add a constraint4 that x[1]>1.
For more reading, see the documentation for cobyla:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_cobyla.html
I am writing an automated curve fitting routine for 2D data based on scipy's optimize.leastsq, and it works. However when adding many curves with starting values slighly off I get non-physical results (negative amplitude, for example).
I found this post Scipy: bounds for fitting parameter(s) when using optimize.leastsq and was trying to use the parameter transformation according to Minuit from Cern. In the above mentioned question somebody provided a link to some python code.
code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/leastsqbound.py
I wrote this minimal working example (extending the code)
"""
http://code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/leastsqbound.py
Constrained multivariate Levenberg-Marquardt optimization
"""
from scipy.optimize import leastsq
import numpy as np
import matplotlib.pyplot as plt #new
def internal2external_grad(xi, bounds):
"""
Calculate the internal to external gradiant
Calculates the partial of external over internal
"""
ge = np.empty_like(xi)
for i, (v, bound) in enumerate(zip(xi, bounds)):
a = bound[0] # minimum
b = bound[1] # maximum
if a == None and b == None: # No constraints
ge[i] = 1.0
elif b == None: # only min
ge[i] = v / np.sqrt(v ** 2 + 1)
elif a == None: # only max
ge[i] = -v / np.sqrt(v ** 2 + 1)
else: # both min and max
ge[i] = (b - a) * np.cos(v) / 2.
return ge
def i2e_cov_x(xi, bounds, cov_x):
grad = internal2external_grad(xi, bounds)
grad = grad = np.atleast_2d(grad)
return np.dot(grad.T, grad) * cov_x
def internal2external(xi, bounds):
""" Convert a series of internal variables to external variables"""
xe = np.empty_like(xi)
for i, (v, bound) in enumerate(zip(xi, bounds)):
a = bound[0] # minimum
b = bound[1] # maximum
if a == None and b == None: # No constraints
xe[i] = v
elif b == None: # only min
xe[i] = a - 1. + np.sqrt(v ** 2. + 1.)
elif a == None: # only max
xe[i] = b + 1. - np.sqrt(v ** 2. + 1.)
else: # both min and max
xe[i] = a + ((b - a) / 2.) * (np.sin(v) + 1.)
return xe
def external2internal(xe, bounds):
""" Convert a series of external variables to internal variables"""
xi = np.empty_like(xe)
for i, (v, bound) in enumerate(zip(xe, bounds)):
a = bound[0] # minimum
b = bound[1] # maximum
if a == None and b == None: # No constraints
xi[i] = v
elif b == None: # only min
xi[i] = np.sqrt((v - a + 1.) ** 2. - 1)
elif a == None: # only max
xi[i] = np.sqrt((b - v + 1.) ** 2. - 1)
else: # both min and max
xi[i] = np.arcsin((2.*(v - a) / (b - a)) - 1.)
return xi
def err(p, bounds, efunc, args):
pe = internal2external(p, bounds) # convert to external variables
return efunc(pe, *args)
def calc_cov_x(infodic, p):
"""
Calculate cov_x from fjac, ipvt and p as is done in leastsq
"""
fjac = infodic['fjac']
ipvt = infodic['ipvt']
n = len(p)
# adapted from leastsq function in scipy/optimize/minpack.py
perm = np.take(np.eye(n), ipvt - 1, 0)
r = np.triu(np.transpose(fjac)[:n, :])
R = np.dot(r, perm)
try:
cov_x = np.linalg.inv(np.dot(np.transpose(R), R))
except LinAlgError:
cov_x = None
return cov_x
def leastsqbound(func, x0, bounds, args = (), **kw):
"""
Constrained multivariant Levenberg-Marquard optimization
Minimize the sum of squares of a given function using the
Levenberg-Marquard algorithm. Contraints on parameters are inforced using
variable transformations as described in the MINUIT User's Guide by
Fred James and Matthias Winkler.
Parameters:
* func functions to call for optimization.
* x0 Starting estimate for the minimization.
* bounds (min,max) pair for each element of x, defining the bounds on
that parameter. Use None for one of min or max when there is
no bound in that direction.
* args Any extra arguments to func are places in this tuple.
Returns: (x,{cov_x,infodict,mesg},ier)
Return is described in the scipy.optimize.leastsq function. x and con_v
are corrected to take into account the parameter transformation, infodic
is not corrected.
Additional keyword arguments are passed directly to the
scipy.optimize.leastsq algorithm.
"""
# check for full output
if "full_output" in kw and kw["full_output"]:
full = True
else:
full = False
# convert x0 to internal variables
i0 = external2internal(x0, bounds)
# perfrom unconstrained optimization using internal variables
r = leastsq(err, i0, args = (bounds, func, args), **kw)
# unpack return convert to external variables and return
if full:
xi, cov_xi, infodic, mesg, ier = r
xe = internal2external(xi, bounds)
cov_xe = i2e_cov_x(xi, bounds, cov_xi)
# XXX correct infodic 'fjac','ipvt', and 'qtf'
return xe, cov_xe, infodic, mesg, ier
else:
xi, ier = r
xe = internal2external(xi, bounds)
return xe, ier
# new below
def _evaluate(x, p):
'''
Linear plus Lorentzian curve
p = list with three parameters ([a, b, I, Pos, FWHM])
'''
return p[0] + p[1] * x + p[2] / (1 + np.power((x - p[3]) / (p[4] / 2), 2))
def residuals(p, y, x):
err = _evaluate(x, p) - y
return err
if __name__ == '__main__':
data = np.loadtxt('constraint.dat') # read data
p0 = [5000., 0., 500., 2450., 3] #Start values for a, b, I, Pos, FWHM
constraints = [(4000., None), (-50., 20.), (0., 2000.), (2400., 2451.), (None, None)]
p, res = leastsqbound(residuals, p0, constraints, args = (data[:, 1], data[:, 0]), maxfev = 20000)
print p, res
plt.plot(data[:, 0], data[:, 1]) # plot data
plt.plot(data[:, 0], _evaluate(data[:, 0], p0)) # plot start values
plt.plot(data[:, 0], _evaluate(data[:, 0], p)) # plot fit values
plt.show()
Thats the plot output, where green is the starting conditions and red the fit result:
Is this the correct usage? External2internal conversion just throws a nan if outside the bounds. leastsq seems to be able to handle this?
I uploaded the fitting data here. Just paste into a text file named constraint.dat.
There is already an existing popular constrained Lev-Mar code
http://adsabs.harvard.edu/abs/2009ASPC..411..251M
with the implementation in python
http://code.google.com/p/astrolibpy/source/browse/mpfit/mpfit.py
I would suggest not to reinvent the wheel.
Following sega_sai's answer I came up with this minimal working example using mpfit.py
import matplotlib.pyplot as plt
from mpfit import mpfit
import numpy as np
def _evaluate(p, x):
'''
Linear plus Lorentzian curve
p = list with three parameters ([a, b, I, Pos, FWHM])
'''
return p[0] + p[1] * x + p[2] / (1 + np.power((x - p[3]) / (p[4] / 2), 2))
def residuals(p, fjac = None, x = None, y = None, err = None):
status = 0
error = _evaluate(p, x) - y
return [status, error / err]
if __name__ == '__main__':
data = np.loadtxt('constraint.dat') # read data
x = data[:, 0]
y = data[:, 1]
err = 0 * np.ones(y.shape, dtype = 'float64')
parinfo = [{'value':5000., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'a'},
{'value':0., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'b'},
{'value':500., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'I'},
{'value':2450., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'Pos'},
{'value':3., 'fixed':0, 'limited':[0, 0], 'limits':[0., 0.], 'parname':'FWHM'}]
fa = {'x':x, 'y':y, 'err':err}
m = mpfit(residuals, parinfo = parinfo, functkw = fa)
print m
The fit results are:
mpfit.py: 3714.97545, 0.484193283, 2644.47271, 2440.13385, 22.1898496
leastsq: 3714.97187, 0.484194545, 2644.46890, 2440.13391, 22.1899295
So conclusion: Both methods work, both allow constraints. But as mpfit comes from a very established sourceI trust it more. Also it honors error values, if available.
Try lmfit-py - https://github.com/newville/lmfit-py
It also uses the Levenberg-Marquardt (LM) algorithm via scipy.optimize.leastsq. Uncertaintes are OK.
It allows you not only to constrain your fitting parameters with bounds but also with mathematical expressions between them without modification of your fitting function.
Forget about using those awful p[0], p[1] ... in fitting function. Just use names of the fitting parameters via the Parameters class.