Solution 5: Use a simplex of points to find a good search direction without derivatives.
A simplex in $\mathbb{R}^n$ is a set of $n+1$ non-colinear points. In 2D, it's a triangle; in 3D, a tetrahedron.
Order the vertices by function value:
$$f(\mathbf{x}_1) \le f(\mathbf{x}_2) \le \cdots \le f(\mathbf{x}_{n+1})$$The simplex gives us a local "linear" model of the function: the function values at the vertices tell us which direction is "downhill."
The key insight: the line from the worst point $\mathbf{x}_{n+1}$ through the centroid of the best points is a reasonable search direction.
The centroid of the best $n$ points is:
$$\bar{\mathbf{x}} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{x}_i$$We search along the parametric line:
$$\bar{\mathbf{x}}(t) = \frac{1}{n}\sum_{i=1}^{n}\mathbf{x}_i + t\left(\mathbf{x}_{n+1} - \frac{1}{n}\sum_{i=1}^{n}\mathbf{x}_i\right)$$Nelder-Mead replaces the worst point $\mathbf{x}_{n+1}$ with a new point $\bar{\mathbf{x}}(t)$ for specific values of $t$:
| Operation | $t$ | When |
|---|---|---|
| Reflect | $-1$ | First try — mirror the worst point through the centroid |
| $-2$ | Reflection was the best point so far — try going further | |
| Outside contract | $-\tfrac{1}{2}$ | Reflection worse than second-worst but better than worst |
| Inside contract | $+\tfrac{1}{2}$ | Reflection worse than worst — contract toward centroid |
| Shrink | — | Nothing worked — shrink all points toward $\mathbf{x}_1$ |
Every iteration starts by computing the reflection $\bar{\mathbf{x}}(-1)$. What happens next depends on how $f_r = f(\bar{\mathbf{x}}(-1))$ compares to the sorted vertex values $f_1 \le f_2 \le \cdots \le f_{n+1}$:
Most steps are reflections, requiring only 1 new function evaluation. Expansion and contraction also need 1 additional evaluation (2 total, counting the reflection). Only shrink needs $n$ evaluations.