Cross posted from csexchange:
Most versions of simulated annealing I've seen are implemented similar to what is outlined in the wikipedia pseudocode below:
Let s = s0
For k = 0 through kmax (exclusive):
T ← temperature( 1 - (k+1)/kmax )
Pick a random neighbour, snew ← neighbour(s)
If P(E(s), E(snew), T) ≥ random(0, 1):
s ← snew
Output: the final state s
I am having trouble understanding how this algorithm does not get stuck in a local optima as the temperature cools. If we jump around at the start while the temp is high, and eventually only take uphill moves as the temp cools, then isn't the solution found highly dependent on where we just so happened to end up in the search space as the temperature started to cool? We may have found a better solution early on, jumped off of it while the temp was high, and then be in a worse-off position as the temp cools and we transition to hill climbing.
An often listed modification to this approach is to keep track of the best solution found so far. I see how this change mitigates the risk of "throwing away" a better solution found in the exploratory stage when the temp is high, but I don't see how this is any better than simply performing repeated random hill-climbing to sample the space, without the temperature theatrics.
Another approach that comes to mind is to combine the ideas of keeping track of the "best so far" with repeated hill climbing and beam search. For each temperature, we could perform simulated annealing and track the best 'n' solutions. Then for the next temperature, start from each of those local peaks.
UPDATE:
It was rightly pointed out I didn't really ask a specific question, I've updated in the comments but want to reflect it here:
I don't need a transcription of the pseudo-code into essay form, I understand how it works - how the temperature balances exploration vs exploitation, and how this (theoretically) helps avoid local optima.
My specific questions are:
Without modification to keep track of the global best solution,
wouldn't this algorithm be extremely prone to local optima, despite
the temperature component? Yes, I understand the probability of taking a worse move declines as the temperature cools (e.g. it transitions to pure hill-climbing), but it would be possible that you had found a better solution early on in the exploitation portion (temp hot), jumped off of it, and then as the temperature cools you have no way back because you're on a path that leads to a new local peak.
With the addition of tracking the global optima, I can absolutely
see how this mitigates getting stuck at local peaks avoiding the problem described above. But how does
this improve upon simple random search? One specific example being
repeated random hill climbing. If you're tracking the global optima and you happen to hit it while in the high-temp portion, then that essentially is a random-search.
In what circumstances would this algorithm be preferable to something like repeated random hill-climbing and why? What properties does a problem have that makes it particularly suited to SA vs an alternative.
By my understanding, simulated annealing is not guaranteed to not get stuck in a local maxima (for maximization problems), especially as it "cools" where later in the cycle as k -> kmax.
So as you say, we "jump all over the place" at the start, but we are still selective in whether or not we accept that jump, as the P() function that determines the probability of acceptance is a function of the goal, E(). Later in the same wikipedia article, they describe the P() function a bit, and suggest that if e_new > e, then perhaps P=1, always taking the move if it is an improvement, but perhaps not always if it isn't an improvement. Later in the cycle, we aren't as willing to take random leaps to lesser results, so the algorithm tends to settle into a max (or min), which may or may not be the global.
The challenge in global function evaluation is to obtain good efficiency, i.e. find the optimal solution or one that is close to the optimum with as little function evaluations as possible. So many "we could perform..." reasonings are correct in terms of convergence to a good solution, but may fail to be efficient.
Now the main difference between pure climbing and annealing is that annealing temporarily admits a worsening of the objective to avoid... getting trapped in a local optimum. Hill climbing only searches the subspace where the function takes larger values than the best so far, which may fail to be connected to the global optimum. Annealing adds some "tunneling" effect.
Temperature decrease is not the key ingredient here and by the way, hill climbing methods such as the Simplex or Hooke-Jeeves patterns also work with diminishing perturbations. Temperature schemes are more related to multiscale decomposition of the target function.
The question as asked doesn't seem very coherent, it seems to ask about guarantees offered by the vanilla simulated annealing algorithm, then proceeds to propose multiple modifications with little to no justification.
In any case, I will try to be coherent in this answer.
The only known rigorous proof of simulated annealing that I know of is described as
The concept of using an adaptive step-size has been essential to the development of
better annealing algorithms. The Classical Simulated Annealing (CSA) was the first annealing algorithm with a rigorous mathematical proof for its global convergence (Geman and
Geman, 1984). It was proven to converge if a Gaussian distribution is used for GXY (TK),
coupled with an annealing schedule S(Tk) that decreases no faster than T = To/(log k).
The integer k is a counter for the external annealing loop, as will be detailed in the next
section. However, a logarithmic decreasing schedule is considered to be too slow, and for
many problems the number of iterations required is considered as “overkill” (Ingber, 1993).
Where the proof of this can be found in
Geman, Stuart, and Donald Geman. "Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images." IEEE Transactions on pattern analysis and machine intelligence 6 (1984): 721-741.
In terms of successful modifications of SA, if you peruse literature you will find many empirical claims of superiority, but few proofs. Having studied this problem in the past, I can say from the various modified versions of SA I have tried empirically for the problems I was interested in at the time, the only SA modification that consistently gave better results than classical simulated annealing is known as parallel tempering.
Parallel tempering is the modification of SA whereby instead of mixing one Markov-Chain (i.e. the annealing step) multiple chains are mixed concurrently at very specific temperatures, with exchanges between chains occurring on a dynamic or static schedule.
Original paper on parallel tempering:
Swendsen, Robert H., and Jian-Sheng Wang. "Replica Monte Carlo simulation of spin-glasses." Physical review letters 57.21 (1986): 2607.
There are various free and proprietary implementations of parallel tempering available due to its effectiveness; I would highly recommend starting with a a high quality implementation instead of rolling your own.
Note also, parallel tempering is extensively used throughout various physical sciences for modelling purposes due to its efficacy, but for some reason has a very small representation in other areas of literature (such as computer science,) which actually baffles me somewhat. Perhaps computer scientists general disinterest with SA modifications may stem from their belief that SA is unlikely to be improved upon.
This area is still very new to me, so forgive me if I am asking dumb questions. I'm utilizing MCTS to run a model-based reinforcement learning task. Basically I have an agent foraging in a discrete environment, where the agent can see out some number of spaces around it (I'm assuming perfect knowledge of its observation space for simplicity, so the observation is the same as the state). The agent has an internal transition model of the world represented by an MLP (I'm using tf.keras). Basically, for each step in the tree, I use the model to predict the next state given the action, and I let the agent calculate how much reward it would receive based on the predicted change in state. From there it's the familiar MCTS algorithm, with selection, expansion, rollout, and backprop.
Essentially, the problem is that this all runs prohibitively slowly. From profiling my code, I notice that a lot of time is spent doing the rollout, likely I imagine because the NN needs to be consulted many times and takes some nontrivial amount of time for each prediction. Of course, I can probably stand to clean up my code to make it run faster (e.g. better vectorization), but I was wondering:
Are there ways to speed up/work around the traditional random walk done for rollout in MCTS?
Are there generally other ways to speed up MCTS? Does it just not mix well with using an NN in terms of runtime?
Thanks!
I am working on a similar problem and so far the following have helped me:
Make sure you are running tensorflow on you GPU (You will have to install CUDA)
Estimate how many steps into the future your agent needs to calculate to still get good results
(The one I am currently working on) parallelize
I am trying to run a differential equation system solver with GEKKO in Python 3 (via Jupyter notebook).
For bad initial parameters it of course immediately halts and says solution not found. Or immediately concludes with no changes done to the parameters / functions.
For other initial parameters the CPU (APM.exe in taskmanager) is busy for 30 seconds (depending on problem size), then it goes back down to 0% usage. But some calculation is still running and does not produce output. The only way to stop it (other than killing python completely) is to stop the APM.exe. I have not get a solution.
When doing this, I get the disp=True output me it has done a certain number of iterations (same for same initial parameters) and that:
No such file or directory: 'C:\\Users\\MYUSERNAME\\AppData\\Local\\Temp\\tmpfetrvwwwgk_model7\\options.json'
The code is very lengthy and depends on loading data from file. So I can not really provide a MWE.
Any ideas?
Here are a few suggestions on troubleshooting your application:
Sequential Simulation: If you are simulating a system with IMODE=4 then I would recommend that you switch to m.options.IMODE=7 and leave m.solve(disp=True) to see where the model is failing.
Change Solver: Sometimes it helps to switch solvers as well with m.options.SOLVER=1.
Short Time Step Solution: You can also try using a shorter time horizon initially with m.time=[0,0.0001] to see if your model can converge for even a short time step.
Coldstart: For optimization problems, you can try m.options.COLDSTART=2 to help identify equations or constraints that lead to an infeasible solution. It breaks the problem down into successive pieces that it can solve and tries to find the place that is causing convergence problems.
There are additional tutorials on simulation with Gekko and a troubleshooting guide (see #18) that helps you dig further into the application. When there are problems with convergence it is typically from one of the following issues:
Divide by zero: If you have an equation such as m.Equation(x.dt()==(x+1)/y) then replace it with m.Equation(y*x.dt()==x+1).
Non-continuously differentiable functions: Replace m.abs(x) with m.abs2(x) or m.abs3(x) that do not have a problem with x=0.
Infeasible: Remove any upper or lower bounds on variables.
Too Many Time Points: Try fewer time points or a shorter horizon initially to verify the model.
Let us know if any of this helps.
I have a system of non-linear differential equations that model memory processes in the brain. I am using scipy.integrate.odeint to solve this system of ODEs. This is the first step and this executes fine. I get a good oscillating system.
The next step, however, is to find the equilibrium points and eigen values of this oscillating system which will prove what kind of bifurcation I am seeing. In python, this seems to be an impossible task. As far as I have googled, people speak about using AUTO in xppaut but I am not comfortable using any other tool as that involves lot of redundant work.
Is it possible to find the equilibrium points and eigen values of a system of ODEs in python?
Please help.
Thanks in advance.
Esash
I apply optimization tool to solve pratical production planning problem.
My current problem is doing planning for a factory with various items in a unique production flow stage. In each stage, there are few parallel machines as graph below.
I have done maths MILP model, and try to solve by CPLEX but it too hard to handle the big scale model by itself.
Currently, I prepare to use Genetic Algorithm to solve it, but don't know where to start.
I have some knowdlege in Python Language. My friends, please advise how I start to deal with this problem?
Do someone have a similar solved problem with code, that I can have a reference?
Before you start writing code(or using someone else's) you must understand the theory behind the scene.
What are the main entities of your Production System?
You need to formulate optimization objectives,optimal schedule but defined in terms of optimization problem.
One example that comes to my mind
https://github.com/jpuigcerver/jsp-ga
Take a look at this thesis
http://lancet.mit.edu/~mbwall/phd/thesis/thesis.pdf