Why a Small Gradient Can Stall at a Saddle

Introduce the driving question with a contour plot of f(x,y) = x² - y², highlighting the saddle at the origin where ∇f = 0. Ask why gradient descent, given such a tiny gradient, can still crawl out — or fail to.

• Saddle points have zero gradient but are not minima
• Gradient size alone seems to suggest 'near a minimum'
• We want to know what actually controls exit speed

Single multiple-choice question forcing the learner to predict what governs fast escape from a saddle: gradient magnitude, curvature, step size, or problem dimension.

• Make one explicit commitment before evidence is shown

Simulation of gradient descent on f(x,y) = x² - y² from points near the origin. Slider controls initial perturbation and step size; trajectories and per-step progress are plotted. Learners observe that with small steps the iterates wiggle along the flat x-direction and exit is slow.

• Gradient near origin is proportional to (2x, -2y)
• Tiny x-perturbation makes the x-component of the gradient tiny
• Each step reduces x by only a small amount — visible slow drift

Show the Hessian of f = x² - y²: eigenvalues +1 and -1. The eigenvector with eigenvalue -1 is the direction in which the function curves downward — the escape route.

• Hessian eigenvalues encode curvature along eigenvectors
• Negative eigenvalue ⇒ direction of strict local decrease
• Gradient descent must align enough with this direction to exit quickly

Walk through the local Taylor expansion f(z) ≈ f(z*) + ½ (z - z*)ᵀ H (z - z*). At a saddle, the linear term vanishes, so progress in one step is roughly ½ gᵀ d = ½ (H d) · d, which is controlled by curvature along the step, not by ‖g‖.

• Linear term disappears at critical points
• Quadratic term in the Hessian governs one-step descent
• Small gradient can coexist with large negative curvature

Interactive decomposition: for a 2D quadratic with eigenvalues λ₁, λ₂ (one positive, one negative), let the learner choose the direction d and step size η. The widget plots how much function decrease occurs per step as ½ η² |component along negative eigenvector|² |λ_neg|.

• Progress scales with η² and alignment to the negative eigenvector
• Random or gradient-only directions have limited alignment
• Curvature controls the magnitude of escape, not gradient norm

Contrast with a strict local minimum: here every eigenvalue of the Hessian is non-negative, so no negative-curvature direction exists and small gradient genuinely signals near-convergence.

• Minima: all eigenvalues ≥ 0, escape impossible
• Saddles: at least one negative eigenvalue, escape always exists
• Gradient size is informative only when paired with curvature information

New widget: a rotated, anisotropic saddle with eigenvalues λ = 0.01 and λ = -1 along skewed axes. The learner must pick a step direction that exploits the negative eigenvector within a limited budget of iterations and watch how gradient-only choices stall while eigenvector-aligned choices escape quickly.

• Rotation does not change the eigenvalues, only the coordinate frame
• Gradient descent on its own struggles to find the escape direction
• Detecting or approximating the negative eigenvector accelerates exit

Resolve the puzzle: a small gradient does not guarantee fast exit because progress is governed by the curvature in the descent direction, not the gradient's magnitude. At a saddle, the linear signal vanishes and the quadratic — specifically the negative eigenvector — dictates how quickly we can leave.

• Gradient magnitude is a first-order signal; saddles zero it out
• Second-order information (negative curvature) is what unlocks fast escape
• Algorithms like perturbed gradient descent or Newton's method add this curvature signal explicitly