What Really Moves the Weights?

Open with the SGD update rule and pose the driving question: is the step size just the learning rate, or does the gradient itself stretch it?

• SGD update: parameter ← parameter − learning_rate × gradient
• Step size is not the same as the learning-rate knob alone
• Driving question: how do learning rate and gradient magnitude combine?

Before revealing the rule, let the learner drag the slope (gradient magnitude) and the learning-rate knob on a 1D loss curve, and guess the resulting step length before measuring it.

• Manipulate slope and learning rate
• Predict step length before it is measured
• Notice surprise when slope dominates

Ask the learner to choose the formula that best matches their intuition: step size equals learning rate, gradient magnitude, their sum, or their product.

• Single multiple-choice commitment
• Forces an explicit hypothesis

Show the actual step taken on a visible 1D loss curve as slope and learning rate change, plotting measured step length against the product of the two inputs.

• Read step length directly off the parameter axis
• Sweep slope while holding learning rate fixed
• Sweep learning rate while holding slope fixed

Plot the measured step length as a function of learning rate times gradient magnitude; the points fall on a straight line through the origin, confirming multiplication.

• Step length vs. (learning rate × gradient magnitude) is linear
• Doubling either input roughly doubles the step
• Direction of motion is given by the sign of the gradient

Walk through the algebra: the update subtracts the scaled gradient, so the distance moved along the parameter axis is the magnitude of that vector — the learning rate times the gradient magnitude.

• Update vector has magnitude |learning rate × gradient|
• Direction is set by the sign of the gradient
• Magnitude is set by the product

Push the slope to extreme values and let the step overshoot the minimum or even diverge, showing that the product rule still holds but the linear approximation of the loss no longer applies.

• Linear approximation breaks when the step is too large
• Product rule still gives the attempted step
• Real training diverges when attempted step is huge

Give the learner a 2D loss landscape with a much steeper direction than the other; let them choose a single learning rate that keeps steps safe in the steep direction while still moving in the flat one.

• Same learning rate, different gradient magnitudes per direction
• Effective step differs per direction
• Motivates why a single learning rate is a global compromise

Close by stating the answer directly: effective step size = learning rate × gradient magnitude, and recap why the two multiply rather than add or trade off.

• Effective step size = learning rate × gradient magnitude
• Gradient magnitude is as influential as the learning-rate knob
• Direction comes from the sign of the gradient