Saddle Points in the Wild
A saddle point is a stationary point with both upward and downward curvature, and its escape routes lie along the negative-curvature directions revealed by the Hessian's eigenvalues.
A complete interactive classroom, not just a preview.
Start when you are ready to enter this Stage's 7 scenes and explore, respond, and learn as you go.
Where do saddle points and their escape routes appear in real optimization landscapes?
Modern deep learning models are huge, yet training often stalls in places that are not minima at all — they are saddle points.
Intuition says optimization gets stuck at peaks or valleys, but in high dimensions the geometry is different: most 'flat' regions are saddles, and each one has escape routes.
A manipulable 2D landscape lets the learner drop a ball, see it rest at a saddle, then nudge it along different directions to reveal which escape routes exist and which are blocked.
Saddle points and their directions of escape are the dominant stall mechanism of high-dimensional optimization, and a single sketch of Hessian eigenvalues explains exactly why.
Optimization gets stuck at high peaks or deep valleys, so saddle points are rare and not really a practical problem.
- Stochastic noise and mini-batch dynamics beyond a brief mention
- Full proofs of convergence rates for saddle-point escape algorithms
- Convex versus non-convex taxonomy of loss surfaces in general
- 01The Wrong Picture of Getting StuckslideQuestion
Opens with the driving question. Contrasts two visuals: a 1D ball stuck in a valley, and a high-dimensional loss surface where stationary regions look flat. Asks where a descent method would actually stop.
- Most intuition about optimization comes from 1D valleys and peaks
- In high dimensions, flat regions dominate the surface
- The driving question: are those flat regions minima, maxima, or something else?
- 02Predict Where the Ball Will StopinteractivePrediction
A 2D landscape widget shows a curved surface with a clearly marked stationary point. The learner must predict, before dropping the ball, whether it will come to rest at a minimum, a maximum, or a saddle point.
- Read the local shape of the surface around the marked point
- Decide: minimum, maximum, or saddle?
- Predict whether there is a direction in which the ball could escape
- 03Evidence: Dropping Balls on the SurfaceslideEvidence
Shows the same 2D surface with several balls dropped from different starting points. Some roll into a bowl-shaped minimum; one settles on a flat-looking ridge where a small side nudge sends it sliding off in a clear direction.
- Balls from different starts reach different resting points
- One rest point is shaped like a ridge, not a bowl
- A tiny perpendicular push causes the ball to escape along the ridge
- 04Why It Is a Saddle: Reading the HessianslideExplanation
Introduces the Hessian matrix at the stationary point and reads its eigenvalues as directions of curvature. Positive eigenvalue curves up like a bowl; negative eigenvalue curves down like a ridge. A saddle has both signs, so one direction is an escape route.
- The Hessian is the matrix of second derivatives at a point
- Each eigenvalue measures curvature along an eigenvector direction
- Positive eigenvalue = valley, negative eigenvalue = escape ramp
- Saddle point = at least one positive and one negative eigenvalue
- 05Transfer: Apply It to a New LandscapeinteractiveTransfer
A new widget shows a loss surface where the learner can read the curvature directions around three unknown stationary points and classify each as minimum, maximum, or saddle, then identify which direction a stuck optimizer could escape along.
- Classify three stationary points using only local curvature
- Mark the negative-eigenvalue direction where escape is possible
- Decide which point a perturbed gradient method would leave first
- 06When Saddles Disappear: The Low-Dimensional BoundaryslideBoundary
Shows the same logic restricted to 1D and 2D landscapes, where saddle points are rare and easy to visualize, contrasting with the high-D regime where saddles dominate. Prevents the learner from overgeneralizing the saddle story to every loss surface.
- In 1D, stationary points are only minima or maxima, no saddles
- In 2D, saddles are visible but uncommon
- In high dimensions, saddles dominate because there are exponentially more ways for curvature to mix signs
- 07Answering the Driving QuestionslideResolution
Directly resolves the opening question. Saddle points appear wherever the Hessian has both positive and negative eigenvalues, they are the typical stall in high-dimensional training, and their escape routes are precisely the negative-eigenvalue directions.
- Saddles live in any landscape with mixed-sign curvature at a stationary point
- They dominate high-dimensional loss surfaces rather than minima
- Escape routes are the directions of negative curvature from the Hessian
- Modern optimizers that ignore negative curvature stall at these saddles
Discussion threads for a Stage aren't available yet.
This path ends here.
