When a Learning Rate Diverges
A learning rate that worked can diverge when the loss landscape shifts under it — when gradients spike, curvature grows, or accumulated updates push the model into a steeper region — and you can read early warning signs before the explosion.
A complete interactive classroom, not just a preview.
Start when you are ready to enter this Stage's 12 scenes and explore, respond, and learn as you go.
Why does a learning rate that trained smoothly yesterday suddenly make the loss explode today?
- effective-step-size
- The true parameter move per step equals learning rate times gradient; changes in gradient magnitude change effective step size even if LR is fixed.
- gradient-spike
- Sudden large gradients from a rare batch, bad sample, or instability in an earlier layer can multiply the fixed LR into a destructive step.
- curvature-shift
- As training moves through parameter space, local curvature (sharpness) can grow, so a fixed step size that was safe in a flat region becomes too large on a sharp ridge.
- schedule-and-warmup
- Warmup keeps early steps small enough to survive an unfamiliar region; schedules pre-shrink LR to stay below the local safe step size later.
- batch-size-effect
- Large-batch gradients have lower variance but the same mean, yet loss curves still shift, so the same nominal LR interacts with a changed effective step.
- diagnostic-signals
- Loss spikes, gradient-norm explosions, and NaN propagation are the early warnings that a working LR has become unsafe.
If a learning rate worked for 10,000 steps, it will keep working.
Show that the loss landscape moves under the optimizer, so a step size safe in one region can become unsafe in another.
Divergence means the learning rate is too high in absolute terms.
Reframe divergence as the effective step (LR × gradient) exceeding the local safe step; the LR can stay the same while the gradient explodes.
Lowering the learning rate is the only fix for divergence.
Show warmup, gradient clipping, and LR schedules as alternative interventions that target the cause, not just the symptom.
- Familiarity with gradient descent
- Basic understanding of learning rate as a step size
- Concept of training loss
- optimizer implementation details
- specific architecture-level fixes
- full derivation of adaptive optimizers
- Learner can name at least three conditions under which a previously safe learning rate becomes destructive.
- Learner can read a loss curve and gradient-norm trace and identify the early-warning pattern of an impending explosion.
- Learner can choose an appropriate intervention (LR drop, warmup, gradient clip, schedule change) for a given failure mode.
- Diagnose a new divergence event from a training log and propose a targeted fix without blindly halving the learning rate.
ML practitioners who understand gradient descent and learning rates but have not built a mental model of loss-surface dynamics or training-instability diagnostics.
- 01The Mystery of the Working RateslideOrientationObserve
Frame the question: a learning rate that trained stably can suddenly explode. Preview the cases we will investigate.
- Set up the opening question
- Show a loss curve that was smooth then diverged
- Preview: gradients, curvature, schedules, batches
- 02Step Size on a Moving TerraininteractiveModel buildingPredict
Let the learner drive a ball across a 1D loss landscape with a fixed step size and see when it overshoots the valley.
- Drag the starting position
- Observe fixed step size across flat and steep regions
- Watch overshoot when the valley narrows
- 03Effective Step Size: LR × GradientslideModel buildingExplain
Formalize that the true parameter move per step is LR times gradient, so changes in gradient magnitude change the effective step even when LR is fixed.
- Effective step = learning rate × gradient
- Gradient can spike without LR changing
- Why a fixed LR is not a fixed step size
- 04Case 1: The Gradient SpikeslideModel buildingExplain
Walk through how a rare batch, bad sample, or earlier-layer instability produces a sudden large gradient that multiplies a fixed LR into a destructive step.
- Bad batches and rare tokens can spike gradients
- Earlier-layer instability amplifies later gradients
- Result: one oversized parameter jump
- 05Case 2: The Sharper ValleyslideModel buildingExplain
Show that as the optimizer moves, the local curvature can grow; a step size safe in a flat basin becomes too large on a sharp ridge.
- Loss surface is non-stationary
- Local sharpness can rise late in training
- Safe step size shrinks as sharpness grows
- 06Which Case Broke It?quizAssessmentChoose
Force the learner to commit to a diagnosis for three short loss-curve snippets before the explanation.
- Classify each snippet as gradient spike, curvature shift, or other
- Justify in one line
- 07Case 3: Batch Size and the Linear Scaling RuleslideModel buildingExplain
Show why changing batch size effectively changes the step even when LR is held or rescaled, and why large-batch training often needs warmup.
- Large batches reduce gradient variance
- Linear scaling rule: LR scales with batch size
- Edge of stability differs at large batch
- 08Early Warning SignsslideModel buildingObserve
List the diagnostic signals that appear before a full divergence: loss spikes, rising gradient norm, weight-norm jumps, NaN propagation.
- Loss spike followed by recovery is a warning
- Gradient norm trending up precedes explosion
- NaN in one layer corrupts downstream layers
- 09Read the Training LoginteractivePracticeApply
Give the learner a short log with loss and gradient norm and let them flag the step where intervention is needed.
- Scan loss and gradient-norm traces
- Click the step to intervene
- See the consequence of intervening early vs late
- 10Interventions That Match the CauseslideApplicationApply
Map each failure mode to its targeted fix: warmup for early curvature, schedules for growing sharpness, gradient clipping for spikes, LR drop as the blunt fallback.
- Warmup handles early unfamiliar region
- Schedules pre-shrink LR as sharpness grows
- Gradient clipping bounds spike damage
- 11Pick the Right FixquizAssessmentChoose
Present three divergence scenarios and ask the learner to choose the most targeted intervention for each.
- Choose warmup, schedule, gradient clip, or LR drop
- Explain why the chosen fix targets the cause
- 12Synthesis: Why a Working Rate BreaksslideSynthesisExplain
Unify the three cases under one mental model: divergence happens when the effective step exceeds the local safe step, and the safe step can shrink while the effective step grows.
- Divergence = effective step exceeds local safe step
- Gradient spikes and curvature shifts both raise effective step or lower safe step
- Diagnose by reading loss and gradient-norm traces
Discussion threads for a Stage aren't available yet.
