Gradient Noise vs. Parameter Noise
Gradient noise and parameter noise shape optimization through fundamentally different mechanisms: gradient noise produces an implicit regularization that scales with learning rate, while parameter noise produces an explicit perturbation that scales with weight magnitude.
A complete interactive classroom, not just a preview.
Start when you are ready to enter this Stage's 8 scenes and explore, respond, and learn as you go.
What happens to optimization when noise is added to gradients versus directly to the parameters?
Two ways to inject randomness into neural network training, and they do not behave the same way.
It feels natural to assume that noise is noise, whether it is added during the gradient computation or directly to the weights, the effect on optimization should be similar.
Side-by-side loss landscapes and optimization trajectories that show how each noise type biases the path taken by the optimizer.
Gradient noise acts as an implicit, scale-invariant regularizer that anneals with the learning rate; parameter noise acts as an explicit, scale-dependent perturbation that biases toward wider minima.
Adding noise is adding noise, so the two strategies should produce similar smoothing or regularization effects on the loss landscape.
- Stochastic gradient descent as a baseline noise source
- Specific noise schedules such as cosine or linear annealing
- Adam versus SGD optimizer-specific dynamics
- Generative model noise injection such as diffusion processes
- 01Two Doors Into the OptimizerslideQuestion
Open the investigation by contrasting the two injection points: noise added to the gradient signal versus noise added directly to the parameters, and frame the driving question.
- Noise can enter before or after the loss surface is probed
- Gradient noise perturbs the direction of the update step
- Parameter noise perturbs the location where the gradient is evaluated
- Both strategies are used in practice: SGD minibatch noise versus weight perturbation methods
- 02Commit to a First GuessquizPrediction
Ask the learner to predict which noise type acts like implicit regularization and which acts like explicit perturbation, before any mechanism is revealed.
- Make one independent choice between the two mechanisms
- Justify the prediction in terms of where the noise enters the update rule
- 03Watch Two Optimizers ClimbinteractiveEvidence
A simulation widget where the learner adjusts noise variance and learning rate, and observes how gradient-noise versus parameter-noise trajectories differ on a 2D loss surface.
- Adjust noise scale independently for each injection point
- Adjust learning rate and observe trajectory shape
- Compare the two paths on the same loss landscape
- 04Why the Two Paths DivergeslideExplanation
Walk through the update equations to show that gradient noise is multiplied by the learning rate while parameter noise is effectively multiplied by the gradient magnitude, producing fundamentally different bias.
- Update rule separates noise that enters the gradient from noise that enters the weights
- Gradient noise contribution to the step scales with learning rate
- Parameter noise contribution to the step scales with local gradient magnitude
- Learning rate annealing therefore kills gradient noise but not parameter noise
- 05Anneal the Learning RateinteractiveEvidence
Let the learner decay the learning rate over training and observe that gradient-noise-induced exploration collapses while parameter-noise-induced exploration persists.
- Run a short training schedule with a decaying learning rate
- Measure trajectory spread under each noise type
- Notice which noise type loses its smoothing effect
- 06When the Difference DisappearsslideBoundary
Show the edge case where the two noise types become equivalent: a linear model with unit gradient magnitude, where the learning-rate scale and weight scale align.
- On linear models, gradient noise and parameter noise are equivalent up to rescaling
- Scale-invariance of parameter noise requires careful per-layer tuning
- In deep nonlinear networks the difference is large and practically important
- 07Apply It to a Real RecipeinteractiveTransfer
A transfer widget where the learner chooses between two training tricks, SAM-style parameter perturbation and minibatch gradient noise, and predicts which one survives learning rate warmup-and-decay.
- Recognize which noise type is hidden inside each training trick
- Predict the effect of a learning rate schedule on each trick
- Justify the prediction using the update-rule reasoning
- 08The Answer in One LineslideResolution
Close the investigation by directly answering the driving question and contrasting the two mechanisms in a single comparison frame.
- Gradient noise is an implicit, learning-rate-scaled regularizer that anneals away
- Parameter noise is an explicit, gradient-magnitude-scaled perturbation that biases toward flat minima
- The optimizer sees them very differently because of where they enter the update
Discussion threads for a Stage aren't available yet.
This path ends here.
