Interpolation vs. Approximation in Neural Networks
Interpolation forces the network to pass exactly through every training point, producing high-frequency oscillations between them; approximation sacrifices exact fit for a smoother function that captures the underlying trend and behaves reasonably on unseen inputs.
A complete interactive classroom, not just a preview.
Start when you are ready to enter this Stage's 8 scenes and explore, respond, and learn as you go.
What changes in a neural network's learned function when it interpolates its training data versus merely approximating it?
A network that memorizes its examples looks identical on the outside to one that has genuinely learned the underlying rule — until you test it in between.
Modern deep networks can fit random labels perfectly, yet also generalize from real data. What is actually happening to the function when it interpolates versus when it approximates?
A side-by-side comparison of a network trained on smooth data (approximation) and the same network forced to pass through every noisy training point (interpolation), visualized through loss curves, predicted curves, and out-of-sample behavior.
Interpolation memorizes the training points and oscillates wildly between them, while approximation trades exact fit for a smooth function that captures the underlying trend.
If a neural network fits the training data perfectly, it has learned the pattern — interpolation and approximation should look the same from the outside.
- Specific architectures like transformers or CNNs
- Training algorithms such as SGD variants
- Generalization theory and statistical learning bounds
- 01Two Ways to Fit DataslideQuestion
Frame the driving question by contrasting two ways a network can satisfy its training objective: passing exactly through every point, or capturing the general trend.
- Interpolation: the network's curve passes through every training point exactly.
- Approximation: the network's curve follows the underlying trend and tolerates small errors on individual points.
- Both can reach near-zero training loss — so what actually differs between them?
- 02Predict the BehaviorquizPrediction
Ask the learner to commit to an intuition about what an interpolating network will do on points it has never seen, before any evidence is shown.
- Choose one expectation for out-of-sample behavior of an interpolating network.
- 03Fit Two Networks Side by SideinteractiveEvidence
A simulation widget where the learner fits one neural network to noisy data with a standard loss (approximation) and another forced to interpolate every training point, then observes the two predicted curves.
- Toggle interpolation mode on and off.
- Observe how the curve changes shape between training points.
- Compare training loss between the two modes.
- 04Oscillations Between PointsslideEvidence
Show a static zoomed-in comparison diagram of the two learned curves between two training points, highlighting that the interpolating curve swings above and below while the approximating curve stays smooth.
- Between training points, an interpolating network is unconstrained.
- Approximating networks inherit smoothness from their architecture and loss.
- The same training loss can hide very different functions.
- 05Why Interpolation OscillatesslideExplanation
Explain why an interpolating network is free to oscillate between points: the loss only constrains the function at training inputs, and without a smoothness bias the network can take any path it likes in between.
- Loss is a sum over training points — there is no penalty between them.
- High-capacity networks can fit the points with arbitrarily wild shapes in between.
- Regularization, architecture, and noise during training bias the function toward smoothness, producing approximation.
- 06Test on New InputsinteractiveTransfer
A simulation widget where the learner evaluates both the interpolating and approximating networks on fresh test inputs and observes which one tracks the true underlying function better.
- Query both networks at new x-values.
- Compare predictions to the true underlying trend.
- See that lower training loss does not imply better generalization.
- 07When Interpolation Is FineslideBoundary
Clarify the boundary of the claim: interpolation is harmful when the underlying function is smooth and we care about unseen inputs, but on a finite discrete task like classification it is often the natural regime.
- For tabular or classification problems, exact fit at seen points is acceptable.
- Interpolation hurts when the true function varies smoothly between training inputs.
- The contrast is about between-point behavior, not training accuracy.
- 08Answering the Driving QuestionslideResolution
Directly resolve the driving question by stating what changes: the function's shape between points, its smoothness, and its behavior on unseen inputs — not its training loss.
- Interpolation pins the function at training points; approximation trades exact fit for a smooth shape.
- Between training points, interpolating networks oscillate; approximating networks stay close to the trend.
- The change is invisible in training loss but visible in the learned function and its out-of-sample predictions.
Discussion threads for a Stage aren't available yet.
