Why Stochastic Gradient Descent Beats Full-Batch

Set up the paradox: full-batch GD computes the exact descent direction using all data, yet SGD — which uses tiny, noisy gradient estimates — wins in practice for training modern neural networks.

• Full-batch GD uses all N examples per step to compute the 'true' gradient direction
• SGD uses mini-batches of size B ≪ N, producing noisy but cheap gradient estimates
• Intuition says noisier directions should produce worse solutions
• Empirically, SGD generalizes better — so where does the advantage come from?

Let the learner commit to the dominant reason they think SGD outperforms full-batch GD before the evidence is shown.

• Pick the mechanism you believe is most responsible

Simulate full-batch GD and SGD side-by-side on a 2D non-convex loss surface with saddle points. Learners adjust mini-batch size and learning rate and observe trajectories.

• Full-batch GD often stalls at saddle points or plateaus
• SGD's noisy updates push the trajectory off saddles and continue descent
• Smaller batch size → more noise → stronger exploration of the landscape
• SGD trajectories wander more but find lower-loss regions

Show classic empirical evidence: training loss is similar for both, but SGD achieves noticeably lower test loss — the signature of better generalization.

• Training loss curves for full-batch GD and SGD are nearly identical
• Test loss curves diverge: SGD reaches a lower plateau
• Lower training loss ≠ better generalization; SGD trades a bit of training fit for test fit
• This gap is the fingerprint of implicit regularization

Explain the mechanism: SGD's gradient noise is a stochastic perturbation that acts like Langevin dynamics — it injects exploration proportional to learning rate and inversely proportional to batch size.

• SGD update ≈ true gradient + isotropic noise scaled by η/B
• Noise magnitude grows with learning rate and shrinks with batch size
• This noise is equivalent to sampling from a tempered posterior — implicit Bayesian regularization
• The 'error' in SGD's gradient is doing useful exploration work

Explain why SGD prefers wide, flat minima: noise kicks the trajectory off sharp minima easily but lets it settle in wide basins where small perturbations don't dislodge it.

• Wide minima have similar loss across a neighborhood of parameters — robust
• Sharp minima sit on steep walls — small input changes cause large loss changes
• Sharp minima memorize training data; wide minima capture the underlying function
• SGD's noise filters out sharp minima naturally; full-batch GD can settle into them

Transfer test: learners are given a loss surface and optimizer behavior, and must choose whether to shrink the batch size, lower the learning rate, or switch optimizers — applying the noise-as-regularizer principle.

• Identify whether the optimizer is stuck at a saddle or in a sharp minimum
• Choose the intervention that adds the right amount of gradient noise
• Connect intervention back to the learning rate × 1/batch size scaling law

Bound the claim: too much noise destabilizes training. Very small batches can prevent convergence, and very large learning rates blow up the implicit regularization term.

• Below a critical batch size, gradient noise overwhelms the signal
• Above a critical learning rate, SGD diverges rather than regularizes
• The benefit of noise is regime-dependent — it is not 'more is always better'
• Full-batch GD wins on small, clean convex problems where noise adds no value

Resolve the opening paradox by tying evidence and mechanisms together: SGD's 'worse' gradients are the source of its generalization advantage.

• SGD noise escapes saddles that full-batch GD is trapped on
• SGD noise biases convergence toward wide minima that generalize
• Gradient noise acts as implicit regularization, equivalent to a tempered posterior
• The noisier, 'less accurate' gradient is the feature — not the bug — of modern deep learning