Why Gradient Descent Slows at Saddles
Gradient descent slows near a saddle because one direction has weak downhill slope and weak curvature, so the update becomes very small in that direction.
A complete interactive classroom, not just a preview.
Start when you are ready to enter this Stage's 4 scenes and explore, respond, and learn as you go.
Why does gradient descent slow down near a saddle point?
A loss surface can look nearly flat at a saddle point even though it is not a minimum in every direction.
If the gradient is small, one might expect every step to become tiny, but the decisive factor is the surface's curvature around the saddle.
Compare a nearly flat valley with a saddle-shaped surface, showing how the update behaves along the upward-curving direction.
Near a saddle, small gradients and weak effective curvature produce short, hesitant updates; increasing the step size, using momentum, or exploiting curvature can help the optimizer pass through.
- does not cover convergence proofs
- does not compare all saddle-point optimization methods
- does not discuss local minima broadly
- 01A Surface That Is Not Flat EverywhereslideSlot 1Hook
Show a saddle-shaped loss surface: the path rises in one direction and falls in another, with the center appearing nearly level.
- A saddle is flat in some directions
- Its curvature changes with direction
- The gradient can be close to zero without the surface being a minimum
PhenomenonGradient descent appears to creep along the nearly level center of a saddle-shaped surface.
QuestionIf the slope is almost zero, why does the optimizer take such short steps?
- 02Small Slope Does Not Mean Fast EscapeslideSlot 2Tension
Contrast a gently sloping valley with the saddle's weakly curved direction. A small gradient alone does not reveal how rapidly the slope changes.
- The step is proportional to the gradient
- The direction may also have very small curvature
- Both effects can make the update shrink
PredictionNear the center, the update should become much smaller and progress should visibly stall.
Tempting intuitionA small gradient always means a small step, so the slowdown is simply caused by the loss changing very little.
- 03Curvature Controls the Effective StepslideSlot 3Reveal
Visualize the optimizer crossing a direction whose slope increases only slowly. After a small move, the gradient reverses or nearly disappears, so the next update cannot travel far.
- The gradient supplies the direction and magnitude of the step
- Weak curvature makes the gradient change only slowly
- The resulting updates repeatedly overshoot and correct themselves
EvidenceAlong the saddle's weakly curved direction, the gradient is small and changes gradually as the optimizer moves.
ConclusionThe slowdown comes from the combination of a small gradient and weak curvature, not from a small gradient alone.
Mechanism- 1A small gradient produces a small displacement from the saddle center.
- 2Because curvature is weak, the gradient remains small for a while, so the next displacement is again short.
- 04How to Recognize the BottleneckslideSlot 4Takeaway
Show that a larger step, momentum, or curvature-aware method can move farther across the weakly curved direction instead of repeatedly making tiny corrections.
- Look for directions with small gradients and weak curvature
- A larger effective step can cross the saddle region
- The diagnosis applies to other nearly flat, sharply changing directions too
TransferWhen a training curve slows near a nearly flat region, consider whether the optimizer is spending many updates escaping a weak-curvature direction.
Expected inferenceA small gradient can be harmless in a strongly curved direction, but near a saddle it can signal tiny updates and stalled progress.
Discussion threads for a Stage aren't available yet.
