Capacity and the Interpolation Threshold
A capacity-controlled threshold separates approximation from interpolation: once a network has enough parameters to fit every training point exactly, the optimization problem shifts from finding shared patterns to finding any solution, and the learned function changes character accordingly.
A complete interactive classroom, not just a preview.
Start when you are ready to enter this Stage's 8 scenes and explore, respond, and learn as you go.
How does a network's capacity control whether it interpolates the training data or only approximates it?
Why does a model that is too large memorize perfectly, while a slightly smaller one fails to fit the same data?
Intuition suggests more capacity should always fit data better, yet there is a sharp threshold where networks switch from approximating patterns to memorizing every example.
Side-by-side decision boundaries on the same dataset for under-parameterized, critically-parameterized, and over-parameterized networks, showing the transition from smooth approximation to perfect interpolation.
Capacity is not just 'how big' a network is — it controls whether the optimizer is forced to compress the data into shared patterns or is free to carve a unique path through every point.
A plausible first guess is that capacity is a continuous dial — bigger networks fit data a little better each time, with no qualitative change in behavior.
- Generalization to unseen test data
- Specific optimizers and learning rate schedules
- Architectural details beyond parameter count
- 01Two Ways to Fit DataslideQuestion
Open with the driving question and present two contrasting pictures of what 'fitting' can mean: a smooth curve that captures a trend versus a wiggly curve that passes through every point.
- Pose the driving question about capacity controlling interpolation vs. approximation
- Show a clean smooth fit alongside a wildly oscillating interpolating fit on the same scatter
- Tease that a single number — capacity — separates these two regimes
- 02Predict the Capacity EffectquizPrediction
Let the learner commit to a single prediction about how the training error changes as the number of parameters grows from very small to very large.
- Choose one hypothesis about the error-vs-capacity curve before seeing evidence
- 03Sweep Capacity and Watch the BoundaryinteractiveEvidence
A small 2D classifier whose hidden width can be dragged from narrow to wide on a fixed 2D point cloud. The learner watches the decision boundary morph and the training error drop.
- Drag the capacity slider from low to high
- Observe training error stay above zero, then collapse to zero at a visible threshold
- Observe the boundary change from smooth rule-following to intricate per-point carving
- 04The Error Curve Has a KneeslideEvidence
Plot training error as a function of parameter count for a fixed dataset and summarize the three regimes visible in the interactive: approximation, threshold, interpolation.
- Show a non-monotonic-feeling curve: error falls gradually, then drops to zero and stays at zero
- Mark the interpolation threshold on the x-axis
- Note that adding capacity above the threshold changes the solution's character, not its training error
- 05Why a Threshold, Not a DialslideExplanation
Explain that fitting N points exactly with K parameters is a system of N equations in K unknowns. The qualitative change happens when K crosses the rank needed to solve the system.
- Approximation regime: system is underdetermined for exact fit, so the optimizer is forced toward low-complexity solutions
- At the threshold: the system first admits exact interpolants; the smallest-capacity exact fit is still structured
- Interpolation regime: many exact fits exist; the optimizer picks one shaped by implicit biases, and extra parameters enlarge that solution space rather than shrinking error
- 06When Capacity Is Not the LeverslideBoundary
Pin down what the threshold explanation does and does not claim, so the learner does not overgeneralize.
- Capacity is necessary but not sufficient: optimization dynamics and architecture also shape which interpolant is found
- Boundary applies to the training-data question only; test-error behavior is a separate phenomenon
- With structured data, the threshold can be reached at surprisingly small capacities because labels lie on a low-dimensional manifold
- 07Predict the New ThresholdinteractiveTransfer
Change the dataset to a harder one — more points, higher intrinsic dimension — and ask the learner to predict where the interpolation threshold now sits before running the sweep.
- Adjust the dataset difficulty on screen
- Make a prediction about the new threshold location
- Run the sweep and compare the predicted vs. observed threshold
- 08Capacity as a Phase BoundaryslideResolution
Resolve the driving question by reframing capacity as a threshold parameter between two qualitatively distinct fitting regimes.
- Directly answer: capacity controls a threshold between approximation and interpolation, not a smooth dial
- Below the threshold, the network is forced to share parameters across points and approximate the underlying rule
- Above the threshold, the network can interpolate exactly, and extra capacity reshapes the interpolant rather than improving a single shared solution
- Tie the answer back to the opening tension about 'bigger should be better' and replace it with 'bigger unlocks a different regime'
Discussion threads for a Stage aren't available yet.
This path ends here.
