Learning Rate or Bad Model? Diagnosing Training Failure

Open with the scenario: a model trained with gradient descent converges to poor performance. Two suspects — learning rate and the model itself — produce identical-looking symptoms on the loss curve. Frame the driving question visually.

• Two runs, same final loss, different root causes
• Loss curves alone are ambiguous
• Ask: which lever do I pull first?

Force one committed prediction before any evidence. The learner must choose which signal best separates learning-rate failure from model failure.

• Commit to one diagnostic signal
• Justify which signal matters most

A small simulation that lets the learner run the same toy network with three settings — LR too high, LR too low, and a genuinely too-small model — and observe loss curves, gradient norms, and update-to-weight ratios simultaneously.

• Run 1: LR = 1.0 — loss diverges, gradient norm explodes
• Run 2: LR = 0.0001 — loss creeps down, updates are 10^-6 of weights
• Run 3: bad model, LR = 0.01 — loss plateaus high, everything looks calm

Walk through the mechanism: gradient descent steps of size lr·∇L. If lr is huge, ∇L stays bounded but updates overshoot — loss oscillates or diverges. If lr is tiny, updates are negligible — loss stalls. If the model is wrong, ∇L itself saturates at a non-zero value — a stable equilibrium, not a stuck one.

• Update magnitude = lr × gradient magnitude
• Too-high LR: gradient fine, update catastrophic
• Too-low LR: gradient fine, update negligible
• Bad model: gradient itself settles to a nonzero plateau

Give the learner four unlabeled training logs — loss curve, gradient norm trace, and final accuracy — and let them classify each as LR-too-high, LR-too-low, bad model, or healthy. Reveals the diagnostic pattern in a new context.

• Apply the three-signal rule to fresh cases
• Distinguish 'stuck' from 'plateaued'
• Catch the healthy run to confirm the rule

Acknowledge the limits: noisy loss curves can mask divergence; Adam normalizes updates so the update-to-weight ratio behaves differently; data leakage and label noise look like 'bad model' but live in a different layer. The rule is a first triage, not a full autopsy.

• Adam-style optimizers hide the raw update-size signal
• Loss noise can disguise divergence as 'just bumpy'
• Data and label issues mimic bad-model symptoms
• Use this rule to choose the next experiment, not to conclude

Resolve the driving question directly. Lay out the checklist: (1) is loss diverging or oscillating? → LR too high. (2) Is loss flat with tiny gradient norm? → LR too low. (3) Is loss stable at a high plateau with normal gradients? → model is the bottleneck. End with the one-sentence answer.

• Oscillation or divergence → reduce LR
• Stalled loss with tiny gradients → raise LR
• Stable plateau with healthy gradients → change the model
• Always read three signals, never just the loss curve