LESSON.md

@@ -36,27 +36,31 @@ $$\argmin_{x\in[0,1]^n} f(x),\quad\text{s.t.}\quad g(x)=0$$
 [^argminmax]: In the metaheuristics literature, $\argmax$ is often assumed for evolutionary algorithms, whether $\argmin$ is often assumed for local search or simulated annealing.
 
 
+
+Complete VS approximation VS heuristics.
+
+
 Algorithmics
 ============
 
 > Those algorithms are randomized and iteratives (hence stochastics) and manipulates a sample (synonym population)
 > of solutions (s. individual) to the problem, each one being associated with its quality (s. cost, fitness).
-> 
+>
 > Thus, algorithms have a main loop, and articulate functions that manipulates the sample (called "operators").
-> 
+>
 > Main design problem: exploitation/exploration compromise (s. intensification/diversification).
 > Main design goal: raise the abstraction level.
 > Main design tools: learning (s. memory) + heuristics (s. bias).
-> 
+>
 > Forget metaphors and use mathematical descriptions.
-> 
+>
 > Seek a compromise between complexity, performances and explainability.
-> 
+>
 > The is no better "method".
 > Difference between model and instance, for problem and algorithm.
 > No Free Lunch Theorem.
 > But there is a "better algorithm instances on a given problem instances set".
-> 
+>
 > The better you understand it, the better the algorithm will be.
 
 
@@ -70,13 +74,13 @@ Problem modelization
 ## Main models
 
 > Encoding:
-> 
+>
 > - continuous (s. numeric),
 > - discrete metric (integers),
 > - combinatorial (graph, permutation).
-> 
+>
 > Fitness:
-> 
+>
 > - mono-objective,
 > - multi-modal,
 > - multi-objectives.
@@ -93,30 +97,81 @@ Problem modelization
 Performance evaluation
 ======================
 
-## What is performance
+What is performance
+-------------------
 
 > Main performances axis:
-> 
+>
 > - time,
 > - quality,
 > - probability.
-> 
+>
 > Additional performance axis:
-> 
+>
 > - robustness,
 > - stability.
-> 
+>
 > Golden rule: the output of a metaheuristic is a distribution, not a solution.
 
 
-## Empirical evaluation
+### Run time and target quality
+
+One may think that the obvious objective of an optimization algorithm is to find the location of the optimum.
+While this is true for deterministic and provable optimization, it is more complex in the case of metaheuristics.
+When dealing with search heuristics, the quality of the (sub)optimum found is also a performance metric,
+as one want to maximize the quality of the best solution found during the search.
+
+The two main performance metrics are thus the runtime that is necessary to find a solution of a given quality,
+and conversely, the quality of the solution which can be found in a given runtime.
+Of course, those two metrics tend to be contradictory: the more time is given to the search, the best the solution,
+and conversely, the better the solution one want, the longer the run should be.
+
+
+### Measuring time and quality
+
+To measure the run time, a robust measure of time should be available.
+However, measuring run time on a modern computer is not necessarily robust, for instance because one cannot easily
+control the context switching managed by the scheduler, or because the CPU load can produce memory access contentions.
+However, in practical application, the call to the objective function largely dominates any other part of the algorithm.
+The number of calls to the objective function is thus a robust proxy measure of the run time.
+
+To measure the quality of solutions, the obvious choice is to rely on absolute values.
+However, this may vary a lot across problem instances and be read differently if one has a minimization or a
+maximization problem.
+It may thus be useful to use the error against a known bound of the problem.
+
+
+### Probability of attainment
+
+For metaheuristics which are based on randomized process (which is the vast majority of them), measuring time and
+quality is not enough to estimate their performance.
+Indeed, if one run several time the same "algorithm", one will get different results, hence different performances.
+
+That's why it's more useful to consider that the "output" of an "algorithm" is not a single solution, but a random
+variable, a distribution of several solutions.
+If one define an "algorithm" with fuzzy concepts, like "simulated annealing" or "genetic algorithm", the reason is
+obvious, because the terms encompass a large variety of possible implementations.
+But one should keep in mind that even a given implementation has (a lot of) parameters, and that metaheuristics are
+usually (very) sensitive to their parameter setting.
+
+In order to have a good mental image of how to asses the performance of a solver,
+one should relize that we can only *estimates* the performances, considering *at least* run time, quality
+*and probability* of attaining a fixed target.
+
+
+### Robustness and stability
+
+
+
+Empirical evaluation
+--------------------
 
 > Proof-reality gap is huge, thus empirical performance evaluation is gold standard.
-> 
+>
 > Empirical evaluation = scientific method.
-> 
+>
 > Basic rules of thumb:
-> 
+>
 > - randomized algorithms => repetition of runs,
 > - sensitivity to parameters => design of experiments,
 > - use statistical tools,
@@ -126,43 +181,43 @@ Performance evaluation
 ## Useful statistical tools
 
 > Statistical tests:
-> 
+>
 > - classical null hypothesis: test equality of distributions.
 > - beware of p-value.
-> 
+>
 > How many runs?
-> 
+>
 > - not always "as many as possible",
 > - maybe "as many as needed",
 > - generally: 15 (min for non-parametric tests) -- 20 (min for parametric-gaussian tests).
-> 
+>
 > Use robust estimators: median instead of mean, Inter Quartile Range instead of standard deviation.
 
 
 ## Expected Empirical Cumulative Distribution Functions
 
 > On Run Time: ERT-ECDF.
-> 
+>
 > $$ERTECDF(\{X_0,\dots,X_i,\dots,X_r\}, \delta, f, t) := \#\{x_t \in X_t | f(x_t^*)>=\delta \}$$
-> 
+>
 > $$\delta \in \left[0, \max_{x \in \mathcal{X}}(f(x))\right]$$
-> 
+>
 > $$X_i := \left\{\left\{ x_0^0, \dots, x_i^j, \dots, x_p^u | p\in[1,\infty[ \right\} | u \in [0,\infty[ \right\} \in \mathcal{X}$$
-> 
+>
 > with $p$ the sample size, $r$ the number of runs, $u$ the number of iterations, $t$ the number of calls to the objective
 > function.
-> 
+>
 > The number of calls to the objective function is a good estimator of time because it dominates all other times.
-> 
+>
 > The dual of the ERT-ECDF can be easily computed for quality (EQT-ECDF).
-> 
+>
 > 3D ERT/EQT-ECDF may be useful for terminal comparison.
 
 
 ## Other tools
 
 > Convergence curves: do not forget the golden rule and show distributions:
-> 
+>
 > - quantile boxes,
 > - violin plots,
 > - histograms.
@@ -174,35 +229,35 @@ Algorithm Design
 ## Neighborhood
 
 > Convergence definition(s):
-> 
+>
 > - strong,
 > - weak.
-> 
+>
 > Neighborhood: subset of solutions atteinable after an atomic transformation:
-> 
+>
 > - ergodicity,
 > - quasi-ergodicity.
-> 
+>
 > Relationship to metric space in the continuous domain.
 
 
 ## Structure of problem/algorithms
 
 > Structure of problems to exploit:
-> 
+>
 > - locality (basin of attraction),
 > - separability,
 > - gradient,
 > - funnels.
-> 
+>
 > Structure with which to capture those structures:
-> 
+>
 > - implicit,
 > - explicit,
 > - direct.
-> 
+>
 > Silver rule: choose the algorithmic template that adhere the most to the problem model.
-> 
+>
 > - taking constraints into account,
 > - iterate between problem/algorithm models.
 
@@ -210,14 +265,14 @@ Algorithm Design
 ## Grammar of algorithms
 
 > Parameter setting < tuning < control.
-> 
+>
 > Portfolio approaches.
 > Example: numeric low dimensions => Nelder-Mead Search is sufficient.
-> 
+>
 > Algorithm selection.
-> 
+>
 > Algorithms are templates in which operators are interchangeable.
-> 
+>
 > Most generic way of thinking about algorithms: grammar-based algorithm selection with parameters.
 > Example: modular CMA-ES.
 
@@ -242,9 +297,9 @@ Design tools:
 
 > Fitness landscapes: structure of problems as seen by an algorithm.
 > Features: tool that measure one aspect of a fitness landscape.
-> 
+>
 > We can observe landscapes, and learn which algorithm instance solves it better.
 > Examples: SAT, TSP, BB.
-> 
+>
 > Toward automated solver design.
 

sho/algo.py

@@ -36,13 +36,12 @@ def greedy(func, init, neighb, again):
     return best_val, best_sol
 
 
-def simulated_annealing(func, init, neighb, again):
+def simulated_annealing(func, init, neighb, temp, again):
     """Iterative randomized simulated-annealing template."""
     best_sol = init()
     best_val = func(best_sol)
     val,sol = best_val,best_sol
     i = 1
-    T = 1
     while again(i, best_val, best_sol):
         sol = neighb(best_sol)
         val = func(sol)
@@ -50,7 +49,7 @@ def simulated_annealing(func, init, neighb, again):
         if val >= best_val:
             best_val = val
             best_sol = sol
-        elif np.random.random() < np.exp(-(best_val-val)/T):
+        elif np.random.random() < np.exp(-(best_val-val)/temp(i, best_val, best_sol)):
             best_val = val
             best_sol = sol
         i += 1

sho/bit.py

@@ -71,8 +71,8 @@ def neighb_square(sol, scale, domain_width):
                 nx = np.random.randint(px-w,px+w)
                 ny = min(max(0,ny),domain_width-1)
                 nx = min(max(0,nx),domain_width-1)
-                
-                if new[nx][ny] != 1:
+
+                if new[ny][nx] != 1:
                     new[py][px] = 0 # Remove original position.
                     new[ny][nx] = 1
                 # else pass

sho/temp.py

+
+########################################################################
+# Temperature programme
+########################################################################
+
+class continus:
+    """Continus decrease temperature from T_0 and with T_{i+1} = \alpha x T_i"""
+
+    def __init__(self, T_0, alpha):
+        self.T = T_0
+        self.alpha = alpha
+
+    def __call__(self, i, val, sol):
+        T = self.T
+        self.T *= self.alpha
+        return T
+
+class level:
+    """Level decrease temperature from T_0 and with T_{i+1} = T_i - \alpha after \beta changes"""
+
+    def __init__(self, T_0, alpha, beta):
+        self.T = T_0
+        self.alpha = alpha
+        self.beta = beta
+        self.val = None
+        self.count = 0
+
+    def __call__(self, i, val, sol):
+        T = self.T
+        if val != self.val:
+            self.count += 1
+            self.val = val
+        if self.count > self.beta:
+            self.count = 0
+            self.T -= self.alpha
+        print()
+        print("T = "+str(T))
+        return T
\ No newline at end of file

snp.py

@@ -3,7 +3,7 @@ import math
 import numpy as np
 import matplotlib.pyplot as plt
 
-from sho import make, algo, iters, plot, num, bit, pb
+from sho import make, algo, iters, plot, num, bit, pb, temp
 
 ########################################################################
 # Interface
@@ -48,6 +48,9 @@ if __name__=="__main__":
     can.add_argument("-a", "--variation-scale", metavar="RATIO", default=0.3, type=float,
             help="Scale of the variation operators (as a ration of the domain width)")
 
+    can.add_argument("-o", "--output-file", metavar="NAME", default=None, type=str,
+            help="Output file where log are put")
+
     the = can.parse_args()
 
     # Minimum checks.
@@ -63,6 +66,10 @@ if __name__=="__main__":
     # Weird numpy way to ensure single line print of array.
     np.set_printoptions(linewidth = np.inf)
 
+    # Default filename
+    if the.output_file == None:
+        the.output_file = the.solver
+
 
     # Common termination and checkpointing.
     history = []
@@ -72,7 +79,7 @@ if __name__=="__main__":
                     make.iter(iters.max,
                         nb_it = the.iters),
                     make.iter(iters.save,
-                        filename = the.solver+".csv",
+                        filename = the.output_file+".csv",
                         fmt = "{it} ; {val} ; {sol}\n"),
                     make.iter(iters.log,
                         fmt="\r{it} {val}"),
@@ -133,6 +140,7 @@ if __name__=="__main__":
                 make.neig(num.neighb_square,
                     scale = the.variation_scale,
                     domain_width = the.domain_width),
+                temp.continus(100,0.99),
                 iters
             )
         sensors = num.to_sensors(sol)
@@ -149,6 +157,7 @@ if __name__=="__main__":
                 make.neig(bit.neighb_square,
                     scale = the.variation_scale,
                     domain_width = the.domain_width),
+                temp.continus(100,0.99),
                 iters
             )
         sensors = bit.to_sensors(sol)