I'm using lmfit to fit some data to a two reaction system in order to estimate the rate constants. My data are the changes in concentration of x1, x2, and x3 species in x1 -> x2 -> x3
In other tools, I generally use a global optimizer followed by a local optimizer so I can more easily get access to the Hessian etc. In lmfit I thought I could do something like:
minimizer = lmfit.Minimizer(self._residuals, params)
result = minimizer.minimize(method='differential_evolution')
result = minimizer.minimize(method='leastsqr')
I assumed that the parameters fitted by the differential evolution would remain in the minimized object and get picked up automatically by the second minimize function call.
However, I am not sure because I colleague of mine suggested otherwise. If anyone knows the lmfit package better, does the second minimize pick up the parameters where the first minimize left off?
Update 1: I added the Minimize call to ensure you that only one Minnizmize object is created. My current tests appear to indicate that the parameter values do get passed from one minimize call to another (which is what I'd expect).
Update 2: Further experiments indicate if the system is non-identifiable, then there is a difference, meaning that the first call to minimize doesn't appear to pass on its fitted parameters to the second minimize call.
no, the fitted parameters from the first method will not be used in the second minimization with the code you provided.
If you don't specify anything for minimizer.minimize() it will start from the params you supplied to initialize the Minimizer class. The code below should do what you want:
minimizer = lmfit.Minimizer(self._residuals, params)
result_de = minimizer.minimize(method='differential_evolution')
result = minimizer.minimize(params=result_de.params, method='leastsqr')
(I am assuming here that self._residuals is your fitting function, i.e., what you want to be minimized). Please check the documentation here.
Related
I am setting up to use SciPy's basin-hopping global optimizer. Its documentation for parameter T states
T: float, optional
The “temperature” parameter for the accept or reject criterion. Higher “temperatures” mean that larger jumps in function value will be accepted. For best results T should be comparable to the separation (in function value) between local minima.
When it says "function value", does that mean the expected return value of the cost function func? Or the value passed to it? Or something else?
I read the source, and I see where T is passed to the Metropolis acceptance criterion, but I do not understand how it is used when converted to "beta".
I'm unfamiliar with the algorithm, but if you keep reading the documentation on the link you posted you'll find this:
Choosing T: The parameter T is the “temperature” used in the Metropolis criterion. Basinhopping steps are always accepted if func(xnew) < func(xold). Otherwise, they are accepted with probability:exp( -(func(xnew) - func(xold)) / T ). So, for best results, T should to be comparable to the typical difference (in function values) between local minima. (The height of “walls” between local minima is irrelevant.)
So I believe T should take on the value of the function which you are trying to optimize, func. This makes sense if you look at that probability expression -- you'd be comparing a difference in function values to what is meant to be a type of "upper bound" for the step. For example, if one local minima is func = 10 and another is func = 14, you might consider T = 4 to be an appropriate value.
Currently, I am looking for a Nelder-Mead optimizer in python that also accepts bounds and constraints for the variables.
Scipy has a Nelder-Mead optimizer, but it does not allow any constraints.
During my search I came across the package constrNMPy, which should make this possible.
Here is an example of how to use constrNMPy:
# Define initial guess
x0=[2.5,2.5]
# Define lower and upper bounds (None indicates no bound)
LB=[2,2]
UB=[None,3]
# Call optimizer
import constrNMPy as cNM
res=cNM.constrNM(cNM.test_funcs.rosenbrock,x0,LB,UB,full_output=True)
# Print results
cNM.printDict(res)
However, this example only explains how to define bounds, but cannot define constraints. In the example above I would like to have the following constraint, so that the variables only accept values where the sum is 5:
cons = {'type':'eq', 'fun':lambda x0: 5 - sum(x0)}
How do I pass this constraint to the constrNM call?
Or are there other packages for a Nelder-Mead optimizer with constraints?
You can always add a a constraint by defining a "penalty" function.
the missio of this function is to have a value above zero when this constraint is broken. and adding penalty function's value to the objective function (needed to become minimum) will garantee that the optimum solution would be reached with all constraints (if many are used) are satisfied.
I’m trying to solve a simple ODE to visualise the temporal response, which works well for constant input conditions using the new solve_ivp integration API in SciPy. For example:
def dN1_dt_simple(t, N1):
return -100 * N1
sol = solve_ivp(fun=dN1_dt_simple, t_span=[0, 100e-3], y0=[N0,])
However, I wonder is it possible to plot the response to a time-varying input? For instance, rather than having y0 fixed at N0, can I find the response to a simple sinusoid?
Is there a compatible way to pass time-varying input conditions into the API?
The function you pass to solve_ivp has a t in its signature for exactly this reason. You can do with it whatever you like¹. For example, to get a smooth, one-time pulse, you can do:
from numpy import pi, cos
def fun(t,N1):
input = 1-cos(t) if 0<t<2*pi else 0
return -100*N1 + input
sol = solve_ivp(fun=fun, t_span=[0,20], y0=[N0])
Note that y0 is not the input in your use of the term, but the initial condition. It is defined and makes sense for one point in time only – where you start your integration/simulation.
With ODEs, you typically model external inputs as forces or similar (affecting the time derivative of the system like in the above example) rather than direct changes to the state.
With this approach and in your context of an excitable system, N0 is already the outcome of some external input.
¹ As long as it is sufficiently smooth for the needs of the respective integrator, usually continuously differentiable (C¹). If you want to implement a step, better use a very sharp sigmoid instead.
EDIT: looks like this was already answered before here
It didn't show up in my searches because I didn't know the right nomenclature. I'll leave the question here for now in case someone arrives here because of the constraints.
I'm trying to optimize a function which is flat on almost all points ("steps function", but in a higher dimension).
The objective is to optimize a set of weights, that must sum to one, and are the parameters of a function which I need to minimize.
The problem is that, as the function is flat at most points, gradient techniques fail because they immediately converge on the starting "guess".
My hypothesis is that this could be solved with (a) Annealing or (b) Genetic Algos. Scipy sends me to basinhopping. However, I cannot find any way to use the constraint (the weights must sum to 1) or ranges (weights must be between 0 and 1) using scipy.
Actual question: How can I solve a minimization problem without gradients, and also use constraints and ranges for the input variables?
The following is a toy example (evidently this one could be solved using the gradient):
# import minimize
from scipy.optimize import minimize
# define a toy function to minimize
def my_small_func(g):
x = g[0]
y = g[1]
return x**2 - 2*y + 1
# define the starting guess
start_guess = [.5,.5]
# define the acceptable ranges (for [g1, g2] repectively)
my_ranges = ((0,1),(0,1))
# define the constraint (they must always sum to 1)
def constraint(g):
return g[0] + g[1] - 1
cons = {'type':'eq', 'fun': constraint}
# minimize
minimize(my_small_func, x0=start_guess, method='SLSQP',
bounds=rranges, constraints=cons)
I usually use R so maybe this is a bad answer, but anyway here goes.
You can solve optimization problems like the using a global optimizer. An example of this is Differential Evolution. The linked method does not use gradients. As for constraints, I usually build them manually. That looks something like this:
# some dummy function to minimize
def objective.function(a, b)
if a + b != 1 # if some constraint is not met
# return a very high value, indicating a very bad fit
return(10^90)
else
# do actual stuff of interest
return(fit.value)
Then you simply feed this function to the differential evolution package function and that should do the trick. Methods like differential evolution are made to solve in particular very high dimensional problems. However the constraint you mentioned can be a problem as it will likely result in very many invalid parameter configurations. This is not necessarily a problem for the algorithm, but is simply means you need to do a lot of tweaking and need to expect a lot of waiting time. Depending on your problem, you could try optimizing weights/ parameters in blocks. That means, optimize parameters given a set of weights, then optimize weights given the previous set of parameters and repeat that many times.
Hope this helps :)
I'm trying to use scipy.optimize.minimize to minimize a complicated function. I noticed in hindsight that the minimize function takes the objective and derivative functions as separate arguments. Unfortunately, I've already defined a function which returns the objective function value and first-derivative values together -- because the two are computed simultaneously in a for loop. I don't think there is a good way to separate my function into two without the program essentially running the same for loop twice.
Is there a way to pass this combined function to minimize?
(FYI, I'm writing an artificial neural network backpropagation algorithm, so the for loop is used to loop over training data. The objective and derivatives are accumulated concurrently.)
Yes, you can pass them in a single function:
import numpy as np
from scipy.optimize import minimize
def f(x):
return np.sin(x) + x**2, np.cos(x) + 2*x
sol = minimize(f, [0], jac=True, method='L-BFGS-B')
Something that might work is: you can memoize the function, meaning that if it gets called with the same inputs a second time, it will simply return the same outputs corresponding to those inputs without doing any actual work the second time. What is happening behind the scenes is that the results are getting cached. In the context of a nonlinear program, there could be thousands of calls which implies a large cache. Often with memoizers(?), you can specify a cache limit and the population will be managed FIFO. IOW you still benefit fully for your particular case because the inputs will be the same only when you are needing to return function value and derivative around the same point in time. So what I'm getting at is that a small cache should suffice.
You don't say whether you are using py2 or py3. In Py 3.2+, you can use functools.lru_cache as a decorator to provide this memoization. Then, you write your code like this:
#functools.lru_cache
def original_fn(x):
blah
return fnvalue, fnderiv
def new_fn_value(x):
fnvalue, fnderiv = original_fn(x)
return fnvalue
def new_fn_deriv(x):
fnvalue, fnderiv = original_fn(x)
return fnderiv
Then you pass each of the new functions to minimize. You still have a penalty because of the second call, but it will do no work if x is unchanged. You will need to research what unchanged means in the context of floating point numbers, particularly since the change in x will fall away as the minimization begins to converge.
There are lots of recipes for memoization in py2.x if you look around a bit.
Did I make any sense at all?