8.4 Outro

8.4.1 Remarks

For any \(p\ge 1\), the search space type determines the problem class:

  • \(D\subseteq\mathbb{R}^p\)continuous optimisation

    In particular:
    • \(D=\mathbb{R}^p\) – continuous unconstrained
    • \(D=[a_1,b_1]\times\dots\times[a_n,b_n]\) – continuous with box constraints
    • constrained with \(k\) linear inequality constraints

      \(a_{1,1} x_1 + \dots + a_{1,p} x_p \le b_1\), …, \(a_{k,1} x_1 + \dots + a_{k,p} x_p \le b_k\)

However, there are other possibilities as well:

  • \(D\subseteq\mathbb{Z}^p\) (\(\mathbb{Z}\) – the set of integers) – discrete optimisation

    In particular:
    • \(D=\{0,1\}^p\) – 0–1 optimisation (hard!)
  • \(D\) is finite (but perhaps large, its objects can be enumerated) – combination optimisation

    For example:
    • \(D=\) all possible routes between two points on a map.

These optimisation tasks tend to be much harder than the continuous ones.

Genetic algorithms might come in handy in such cases.

Specialised methods, customised to solve a specific problem (like Lloyd’s algorithm) will often outperform generic ones (like SGD, genetic algorithms) in terms of speed and reliability.

All in all, we prefer a suboptimal solution obtained by means of heuristics to no solution at all.

Problems that you could try solving with GAs include variable selection in multiple regression – finding the subset of features optimising the AIC (this is a hard problem to and forward selection was just a simple greed heuristic).

Other interesting algorithms:

  • Hill Climbing (a simple variation of GD with no gradient use)
  • Simulated annealing
  • CMA-ES
  • Tabu search
  • Particle swarm optimisation
  • Artificial bee/ant colony optimisation
  • Cuckoo Search

8.4.2 Further Reading

Recommended further reading:


  • (Simon 2013)
  • (Boyd & Vandenberghe 2004)