I am trying to optimize a function using the mystic library in Python, and have a general question about the gtol parameter.
The parameter space I'm searching in, is of high dimension (30-40), and for this reason, it seemed fitting for me to set gtol = 40000. Strangely, the algorithm seems to stop after 30000 iterations, although I specified the gtol as 40000, which (by my knowledge) means that the algorithm should have 40000 identical iterations before it stops running.
My function call is pretty basic:
stepmon = VerboseLoggingMonitor(1, 1)
result = diffev2(utilss, x0=[1/value]*value, bounds=[(0,0.2)] * value, npop=150, gtol=40000, disp=True, full_output=True, itermon=stepmon
I checked the merit function's evolution, and it is flat around the 29000th iteration. The last 1000 iterations are identical, but it already stops there instead of running the rest of the 39000 required gtol iterations.
Am I misinterpreting the gtol parameter? Or what am I missing here?
Thanks in advance.
I'm the mystic author. That is the correct interpretation for gtol, however, you are probably ignoring maxiter (and maxfun) which is the maximum number of iterations (and function evaluations) to perform.
If you don't set maxiter (default is None), then the I believe that the default setting for diffev2 is 10 * nDim * nPop. Try setting maxiter=100000, or something like that.
If the termination message says Warning: Maximum number of iterations has been exceeded, then maxiter caused it to stop.
Related
I have a loss function that opens an external .exe file for calculation, and running this executable file usually takes around 2 or 3 minutes. It works very well, but When I want to pass this function to the Genetic Algorithm for optimization, I face this error:
AssertionError: After 10.0 seconds delay func_timeout: the given function
does not provide any output.
So it obviously says that GA waited 10 sec and did not receive anything so bye-bye... Since the function needs more time to return the result (loss), it will not continue.
Is there any way to set time_out to more than 10.0 sec? I couldn't find the option in algorithm_param or ga_model itself. These are my parameters:
from geneticalgorithm import geneticalgorithm as ga
algorithm_param = {'max_num_iteration': None,\
'population_size':100,\
'mutation_probability':0.1,\
'elit_ratio': 0.01,\
'crossover_probability': 0.5,\
'parents_portion': 0.3,\
'crossover_type':'uniform',\
'max_iteration_without_improv':None}
ga_model = ga(function=loss_fn,
dimension=(2),
variable_type='real',
variable_boundaries=varbound,
algorithm_parameters=algorithm_param)
ga_model.run()
I searched here and other forums, but almost all the related questions are for actually limiting a function to a particular time (manually setting a time out for a process), not the opposite thing; I want to remove the restriction.
From the geneticalgorithm library Github page:
Function timeout
geneticalgorithm is designed such that if the given function does not
provide any output before timeout (the default value is 10 seconds),
the algorithm would be terminated and raise the appropriate error. In
such a case make sure the given function works correctly (i.e. there
is no infinite loop in the given function). Also if the given function
takes more than 10 seconds to complete the work make sure to increase
function_timeout in arguments.
From the Arguments section:
#param function_timeout - if the given function does not provide
output before function_timeout (unit is seconds) the algorithm raise
error. For example, when there is an infinite loop in the given
function.
So, if you want a time out of, say, 20 sec, you should change your ga function to:
ga_model = ga(function=loss_fn,
dimension=(2),
variable_type='real',
variable_boundaries=varbound,
algorithm_parameters=algorithm_param,
function_timeout = 20) # <- added
I'm trying to use scipy's ode solver to plot the interaction between a 2D system of equations. I'm attempting to alter the parameters passed to the solver by the following block of code:
# define maximum number of iteration steps for ode solver iteration
m = 1 #power of iteration
N = 2**m #number of steps
# setup a try-catch formulation to increase the number of steps as needed for solution to converge
while True:
try:
z = ode(stateEq).set_integrator("vode",nsteps=N,method='bdf',max_step=5e5)
z.set_initial_value(x0, t0)
for i in range(1, t.size):
if i%1e3 == 0:
print 'still integrating...'
x[i, :] = z.integrate(t[i]) # get one more value, add it to the array
if not z.successful():
raise RuntimeError("Could not integrate")
break
except:
m += 1
N = 2**m
if m%2 == 0:
print 'increasing nsteps...'
print 'nsteps = ', N
Running this never breaks the while loop. It keeps increasing the nsteps forever and the system never gets solved. If I don't put it in the while loop, the system gets solved, I think, because the solution gets plotted. Is the while loop necessary? Am I formulating the solver incorrectly?
The parameter nsteps regulates how many integration steps can be maximally performed during one sampling step (i.e., a call of z.integrate). Its default value is okay if your sampling step is sufficiently small to capture the dynamics. If you want to integrate over a huge time span in one large sampling step (e.g., to get rid of transient dynamics), the value can easily be too small.
The point of this parameter is to avoid problems arising from unexpectedly very long integrations. For example, if you want to perform a given integration for 100 values of a control parameter in a loop over night, you do not want to see on the next morning that the No. 14 was pathological and is still running.
If this is not relevant to you, just set nsteps to a very high value and stop worrying about it. There is certainly no point to successively increase nsteps, you are just performing the same calculations all over again.
Running this never breaks the while loop. It keeps increasing the nsteps forever and the system never gets solved.
This suggests that you have a different problem than nsteps being exceeded, most likely that the problem is not well posed. Carefully read the error message produced by the integrator. I also recommend that you check your differential equations. It may help to look at the solutions until the integration fails to see what is going wrong, i.e., plot x after running this:
z = ode(stateEq)
z.set_integrator("vode",nsteps=1e10,method='bdf',max_step=5e5)
z.set_initial_value(x0, t0)
for i,time in enumerate(t):
x[i,:] = z.integrate(time)
if not z.successful():
break
Your value for max_step is very high (this should not be higher than the time scale of your dynamics). Depending on your application, this may very well be reasonable, but then it suggests that you are working with large numbers. This in turn may mean that the default values of the parameters atol and first_step are not suited for your situation and you need to adjust them.
I've been working on the following code which sort of maximizes the number of unique (in lowest common denominator) p by q blocks with some constraints. It is working perfectly. For small inputs. E.g. input 50000, output 1898.
I need to run it on numbers greater than 10^18, and while I have a different solution that gets the job done, this particular version gets super slow (made my desktop reboot at one point), and this is what my question is about.
I'm trying to figure out what is causing the slowdown in the following code, and to figure out in what order of magnitude they are slow.
The candidates for slowness:
1) the (-1)**(i+1) term? Does Python do this efficiently, or is it literally multiplying out -1 by itself a ton of times?
[EDIT: still looking for how operation.__pow__ works, but having tested setting j=-j: this is faster.]
2) set instantiation/size? Is the set getting too large? Obviously this would impact membership check if the set can't get built.
3) set membership check? This indicates O(1) behavior, although I suppose the constant continues to change.
Thanks in advance for insight into these processes.
import math
import time
a=10**18
ti=time.time()
setfrac=set([1])
x=1
y=1
k=2
while True:
k+=1
t=0
for i in xrange(1,k):
mo = math.ceil(k/2.0)+((-1)**(i+1))*(math.floor(i/2.0)
if (mo/(k-mo) not in setfrac) and (x+(k-mo) <= a and y+mo <= a):
setfrac.add(mo/(k-mo))
x+=k-mo
y+=mo
t+=1
if t==0:
break
print len(setfrac)+1
print x
print y
to=time.time()-ti
print to
My program seems to be crashing almost arbitrarily.
My code includes these two lines:
z[i, j] = -math.exp(oneminusbeta[j, i])
weights[i,j] = math.exp(beta[j,i] + oneminusbeta[j,i])
I've run my whole code before on data that had 2 dimensions, it was 7341 x 648. I had no issues at all running that code.
But now the data I'm using is about ten times as big. It's 71678 x 648, and I'm getting this error:
OverflowError: math range error
And I'm not getting this on any specific point. I'm logging comments before every line of code so that I can see what's causing the crash, and it appears the crash is happening more often on the second line mentioned above (weights[i,j] = math.exp(beta[j,i] + oneminusbeta[j,i])).
The thing is, it crashes at different times.
At first, it crashed at weights[30816, 42]. Then at weights[55399, 43]. Then at z[33715,45]. But the data is the same in all 3 cases.
What could the problem be? Is this a memory related issue with python? I'm using Numba too, by the way.
Edit
I forgot to mention, I've put thresholds so that what goes into the exp() function doesn't exceed what 709 or -708, so technically there shouldn't be an overflow.
The result of your calculation cannot be represented on your computer. This probably means that math.exp(...) is greater than about 10308, or the argument passed to math.exp() is greater than about 710.
Try printing the values of beta[j,i] and oneminusbeta[j,i] before each calculation.
In fact, you don't have to print comments before every line of code. Instead, try wrapping the calculations with a try block, like so:
try:
weights[i,j] = math.exp(beta[j,i] + oneminusbeta[j,i])
except OverflowError:
print "Calculation failed! j=%d i=%d beta=%f oneminusbeta=%f"%(j,i,beta[j,i],oneminusbeta[j,i])
raise
Your overflow is almost certainly a real overflow; one of your values is too large to fit in a Python float (meaning a C double).
So, why does it happen in different places each time?
Because you're using Numba.
Numba JIT-compiles your code. If it detects that there's no contention, it can reorder your code—or even, at least in theory, run it in parallel on multiple cores or on the GPU (although I think at present you only get GPU computation if you code it explicitly with numba.cuda).
At any rate, this means that the path through your code can be nondeterministic. If there's more than one place an overflow could happen, you can't predict which one will fail and trigger the exception.
At any rate, that doesn't really matter. If your calculations are overflowing, you have to fix that. And the fact that different ones overflow each time shouldn't make it that much harder to debug—especially given that it apparently usually happens in a single place, just not always.
I wrote a Python (2.7) program to evaluate some scientific data. Its main task is to fit this data to a certain function (1). Since this is quite a lot of data the program distributes the jobs (= "fit one set of data") to several cores using multiprocessing. In a first attempt I implemented the fitting process using curve_fit from scipy.optimize which works pretty well.
So far, so good. Then we saw that the data is more precisely described by a convolution of function (1) and a gaussian distribution. The idea was to fit the data first to function (1), get the guess values as a result of this and then fit the data again to the convolution. Since the data is quite noisy and I am trying to fit it to a convolution with seven parameters, the results were this time rather bad. Especially the gaussian parameters were in some extend physically impossible.
So I tried implementing the fitting process in PyMinuit because it allows to limit the parameters within certain borders (like a positive amplitude). As I never worked with Minuit before and I try to start small, I rewrote the first ("easy") part of the fitting process. The code snippet doing the job looks like this (simplified):
import minuit
import numpy as np
# temps are the x-values
# pol are the y-values
def gettc(temps, pol, guess_values):
try:
efunc_chi2 = lambda a,b,c,d: np.sum( (efunc(temps, a,b,c,d) - pol)**2 )
fit = minuit.Minuit(efunc_chi2)
fit.values['a'] = guess_values[0]
fit.values['b'] = guess_values[1]
fit.values['c'] = guess_values[2]
fit.values['d'] = guess_values[3]
fit.fixed['d'] = True
fit.maxcalls = 1000
#fit.tol = 1000.0
fit.migrad()
param = fit.args
popc = fit.covariance
except minuit.MinuitError:
return np.zeros(len(guess_values))
return param,popc
Where efunc() is function (1). The parameter d is fixed since I don't use it at the moment.
PyMinuit function reference
Finally, here comes the actual problem: When running the script, Minuit prints for almost every fit
VariableMetricBuilder: Tolerance is not sufficient - edm is 0.000370555 requested 1e-05 continue the minimization
to stdout with different values for edm. The fit still works fine but the printing slows the program considerably down. I tried to increase fit.tol but there are a lot of datasets which return even higher edm's. Then I tried just hiding the output of fit.migrad() using this solution which actually works. And now something strange happens: Somewhere in the middle of the program, the processes on all cores fail simultaneously. Not at the first fit but in the middle of my whole dataset. The only thing I changed, was
with suppress_stdout_stderr():
fit.migrad()
I know this is quite a long introduction, but I think it helps you more when you know the whole framework. I would be very grateful if someone has any idea on how to approach this problem.
Note: Function (1) is defined as
def efunc(x,a,b,c,d):
if x < c:
return a*np.exp(b*x**d)
else:
return a*np.exp(b*c**d) # therefore constant
efunc = np.vectorize(efunc)