I am looking for a differential evolution algorithm (hopefully the one from Scipy) I could use in an unorthodox way. I would like that for each generation, the DE gives me all the child members of the new generation in advance and that I evaluate them all at once in my objective function.
The reason is that my objective function calls COMSOL. I can do a batch of calculations in a COMSOL that COMSOl is going to parallelize carefully, so I don't want the DE to parallelize it itself. So in the end, I want to calculate all the members in one call of COMSOL. Do you have any idea of a package in Python with this kind of freedom?
Thank you for your help!
You can vectorise differential_evolution by using the ability of the workers keyword to accept a map-like callable that is sent the entire population and is expected to return an array with the function values evaluated for the entire population:
from scipy.optimize import rosen, differential_evolution
bounds=[(0, 10), (0, 10)]
def maplike_fun(func, x):
# x.shape == (S, N), where S is the size of the population and N
# is the number of parameters. This is where you'd call out from
# Python to COMSOL, instead of the following line.
return func(x.T)
res = differential_evolution(rosen, bounds, workers=maplike_fun, polish=False, updating='deferred')
From scipy 1.9 there will also be a vectorized keyword, which will send the entire population to the objective function at each iteration.
Related
I'm trying to solve a DAE in Gekko, where some of the components will depend upon the solution to a convolution integral
This requires a constant dt, but I'm sure that's somewhere in the options. Consequently, what I want to do is use a function to record the current value of a state variable in an array, and return the sum up to that point. Here was my attempt using one of the simple ODE examples:
import numpy as np
from gekko import GEKKO
import matplotlib.pyplot as plt
m = GEKKO()
class adder:
def __init__(self,):
self.i=m.Array(m.Param, 10)
self.count=0
def f(self, y):
self.i[self.count]=y
self.count+=1
return sum(self.i)
a=adder()
m.Equation(y.dt()==-y+1+a.f(y))
m.time = np.linspace(0,5)
m.options.IMODE = 4
m.solve()
but the solution 1) looks incorrect and 2) I can't print anything about the solution using the a.f() function and 3) even when I set the size of the self.i array to 1, it doesn't throw an out of bounds error so I expect it's not being called in the way I think. I've also seen people suggest using the delay() function, but I don't know how to access which timestep I'm currently in and to loop over each previous timestep.
Mixing continuous differential equations with discrete equations can be a challenge. Two recommendations are to either (1) convert the DAEs to a discrete form (such as with Orthogonal Collocation on Finite Elements) or (2) use a continuous integral form of the convolution integral. The m.integral() function is available in Gekko to help with any integral expressions. See How to solve integral in Python Gekko? and Integral Objective (Luus) for two examples of solving problems with continuous integrals.
Gekko calls functions only once when building the model. The model file is found in m.path as gk0_model.apm (text file) or by opening the folder with m.open_folder(). The model must be expressed symbolically so that it can be compiled for automatic differentiation.
I am trying to implement a numerical integration. Specifically,
I am using the trapezoidal rule one gets the following analytical expression
When I try to implement it numerically.
gamma=0.1 #parameter
Tmax=100
N=100000 #number of steps
t=np.linspace(0,Tmax,N)
B=np.random.rand(len(t))
def A(t):
A=np.zeros(len(t))
trapezi=0
for i in range(1,N-1):
trapezi+=(np.exp(-gamma*(Tmax-t[i]))*B[i])
h=Tmax/N
A=h*(trapezi+1/2*(B[-1]+B[0]*np.exp(-gamma*Tmax)))
return A
where I have created a random B matrix.
The problem is that right now the output A(t) is just an single value (i.e. for this t I get A(t)=0.47), but what I need is an array of the function going from t=0 to t=t. This is my main problem: how to use the variable t as both a parameter in the integration and as a variable.
Any advice would be highly welcomed.
EDIT: looks like this was already answered before here
It didn't show up in my searches because I didn't know the right nomenclature. I'll leave the question here for now in case someone arrives here because of the constraints.
I'm trying to optimize a function which is flat on almost all points ("steps function", but in a higher dimension).
The objective is to optimize a set of weights, that must sum to one, and are the parameters of a function which I need to minimize.
The problem is that, as the function is flat at most points, gradient techniques fail because they immediately converge on the starting "guess".
My hypothesis is that this could be solved with (a) Annealing or (b) Genetic Algos. Scipy sends me to basinhopping. However, I cannot find any way to use the constraint (the weights must sum to 1) or ranges (weights must be between 0 and 1) using scipy.
Actual question: How can I solve a minimization problem without gradients, and also use constraints and ranges for the input variables?
The following is a toy example (evidently this one could be solved using the gradient):
# import minimize
from scipy.optimize import minimize
# define a toy function to minimize
def my_small_func(g):
x = g[0]
y = g[1]
return x**2 - 2*y + 1
# define the starting guess
start_guess = [.5,.5]
# define the acceptable ranges (for [g1, g2] repectively)
my_ranges = ((0,1),(0,1))
# define the constraint (they must always sum to 1)
def constraint(g):
return g[0] + g[1] - 1
cons = {'type':'eq', 'fun': constraint}
# minimize
minimize(my_small_func, x0=start_guess, method='SLSQP',
bounds=rranges, constraints=cons)
I usually use R so maybe this is a bad answer, but anyway here goes.
You can solve optimization problems like the using a global optimizer. An example of this is Differential Evolution. The linked method does not use gradients. As for constraints, I usually build them manually. That looks something like this:
# some dummy function to minimize
def objective.function(a, b)
if a + b != 1 # if some constraint is not met
# return a very high value, indicating a very bad fit
return(10^90)
else
# do actual stuff of interest
return(fit.value)
Then you simply feed this function to the differential evolution package function and that should do the trick. Methods like differential evolution are made to solve in particular very high dimensional problems. However the constraint you mentioned can be a problem as it will likely result in very many invalid parameter configurations. This is not necessarily a problem for the algorithm, but is simply means you need to do a lot of tweaking and need to expect a lot of waiting time. Depending on your problem, you could try optimizing weights/ parameters in blocks. That means, optimize parameters given a set of weights, then optimize weights given the previous set of parameters and repeat that many times.
Hope this helps :)
I'm trying to use scipy.optimize.minimize to minimize a complicated function. I noticed in hindsight that the minimize function takes the objective and derivative functions as separate arguments. Unfortunately, I've already defined a function which returns the objective function value and first-derivative values together -- because the two are computed simultaneously in a for loop. I don't think there is a good way to separate my function into two without the program essentially running the same for loop twice.
Is there a way to pass this combined function to minimize?
(FYI, I'm writing an artificial neural network backpropagation algorithm, so the for loop is used to loop over training data. The objective and derivatives are accumulated concurrently.)
Yes, you can pass them in a single function:
import numpy as np
from scipy.optimize import minimize
def f(x):
return np.sin(x) + x**2, np.cos(x) + 2*x
sol = minimize(f, [0], jac=True, method='L-BFGS-B')
Something that might work is: you can memoize the function, meaning that if it gets called with the same inputs a second time, it will simply return the same outputs corresponding to those inputs without doing any actual work the second time. What is happening behind the scenes is that the results are getting cached. In the context of a nonlinear program, there could be thousands of calls which implies a large cache. Often with memoizers(?), you can specify a cache limit and the population will be managed FIFO. IOW you still benefit fully for your particular case because the inputs will be the same only when you are needing to return function value and derivative around the same point in time. So what I'm getting at is that a small cache should suffice.
You don't say whether you are using py2 or py3. In Py 3.2+, you can use functools.lru_cache as a decorator to provide this memoization. Then, you write your code like this:
#functools.lru_cache
def original_fn(x):
blah
return fnvalue, fnderiv
def new_fn_value(x):
fnvalue, fnderiv = original_fn(x)
return fnvalue
def new_fn_deriv(x):
fnvalue, fnderiv = original_fn(x)
return fnderiv
Then you pass each of the new functions to minimize. You still have a penalty because of the second call, but it will do no work if x is unchanged. You will need to research what unchanged means in the context of floating point numbers, particularly since the change in x will fall away as the minimization begins to converge.
There are lots of recipes for memoization in py2.x if you look around a bit.
Did I make any sense at all?
I'm using scipy.optimize.leastsq in conjunction with a simulator. leastsq calls a user-defined objective function and passes an input vector to it. In turn, the objective function returns an error vector. leastsq optimizes the input vector in such a way that the sum of the squares of the error vector is minimized.
In my case the objective function will run a whole simulation each time it is called. The employed simulator is single-threaded and needs several minutes for each run. I'd therefore like to run multiple instances of the simulator at once. However, calls to the objective function are performed serially.
How can I get leastsq to perform multiple calls to the objective function at once?
There's a good opportunity to speed up leastsq by supplying your own function to calculate the derivatives (the Dfun parameter), providing you have several parameters. If this function is not supplied, leastsq iterates over each of the parameters to calculate the derivative each time, which is time consuming. This appears to take the majority of the time in the fitting.
You can use your own Dfun function which calculates the derivatives for each parameter using a multiprocessing.Pool to do the work. These derivatives can be calculated independently and should be trivially parallelised.
Here is a rough example, showing how to do this:
import numpy as np
import multiprocessing
import scipy.optimize
def calcmod(params):
"""Return the model."""
return func(params)
def delta(params):
"""Difference between model and data."""
return calcmod(params) - y
pool = multiprocessing.Pool(4)
def Dfun(params):
"""Calculate derivatives for each parameter using pool."""
zeropred = calcmod(params)
derivparams = []
delta = 1e-4
for i in range(len(params)):
copy = np.array(params)
copy[i] += delta
derivparams.append(copy)
results = pool.map(calcmod, derivparams)
derivs = [ (r - zeropred)/delta for r in results ]
return derivs
retn = scipy.optimize.leastsq(leastfuncall, inputparams, gtol=0.01,
Dfun=Dfun, col_deriv=1)
The algorithm used by leastsq, Levenberg-Marquardt, needs to know the value of the objective function at the current point before determining the next point. In short, there is no straightforward way to parallelize such a serial algorithm.
You can, however, parallelize your objective function in some cases. This can be done, if it's of the form:
def objective_f(params):
r = np.zeros([200], float)
for j in range(200):
r[j] = run_simulation(j, params)
return
def run_simulation(j, params):
r1 = ... compute j-th entry of the result ...
return r1
Here, you can clearly parallelize across the loop over j, for instance using the multiprocessing module. Something like this: (untested)
def objective_f(params):
r = np.zeros([200], float)
def parameters():
for j in range(200):
yield j, params
pool = multiprocessing.Pool()
r[:] = pool.map(run_simulation, parameters())
return r
Another opportunity for parallelization occurs if you have to fit multiple data sets --- this is an (embarassingly) parallel problem, and the different data sets can be fitted in parallel.
If this does not help, you can look into discussion on parallelization of the LM algorithm in the literature. For instance: http://dl.acm.org/citation.cfm?id=1542338 The main optimization suggested in this paper seems to be parallelization of the numerical computation of the Jacobian. You can do this by supplying your own parallelized Jacobian function to leastsq. The remaining suggestion of the paper, speculatively parallelizing Levenberg-Marquardt search steps, is however more difficult to implement and requires changes in the LM algorithm.
I'm not aware of Python (or other language) libraries implementing optimization algorithms targeted for parallel computation, although there may be some. If you manage to implement/find one of them, please advertise this on the Scipy users mailing list --- there is certainly interest in one of these!
Does this help?
http://docs.python.org/library/multiprocessing.html
I've always found Pool to be the simplest to multiprocess with python.
NumPy/SciPy's functions are usually optimized for multithreading. Did you look at your CPU utilization to confirm that only one core is being used while the simulation is being ran? Otherwise you have nothing to gain from running multiple instances.
If it is, in fact, single threaded, then your best option is to employ the multiprocessing module. It runs several instances of the Python interpreter so you can make several simultaneous calls to SciPy.
Have you used scipy.least_squares, it is a much better option, and when I use it to optimize a function it uses all the available threads. Therefore exactly what you asked