Machine LearningInformation Theory

Interpolation Can Be Harmless

Survival and contamination

A model can fit every training example, including its noise, and still predict new examples well. How can fitting the noise coexist with learning the pattern?

With enough overparameterization, contamination—the contribution of signal and noise assigned to wrong directions—can become small. The less understood side is survival: how much remains in the true signal direction. Minimum-norm optimization can make extra features act as regularizers, shrinking that useful component. The signal-processing perspective lets us separate and calculate these two effects.

When I started on this in 2017, the puzzle was not that a model with more candidate features than samples could work — sparsity-seeking methods had shown that for years. The puzzle was why more features helped with nothing enforcing sparsity, and what was standing in for it. Those are the two questions the bookkeeping below answers: understanding contamination tells us why more features can help, and understanding survival tells us what has to play sparsity's role in making sure that the true pattern is recovered.

Overparameterization is undersampling

Our decisive conceptual move, in the ISIT 2019 paper and its expanded JSAIT version, was to turn a high-dimensional puzzle into an elementary problem from signal processing. Take nn regularly spaced samples of a one-dimensional signal, and fit them using more Fourier frequencies than samples. Overparameterization is now literally undersampling. Different frequencies can agree at every training sample while behaving differently between samples. Those indistinguishable frequencies are called aliases.

Extra features spread fitted noise across aliases; preference for the true directions preserves signal. In the classification bridge, the same interpolant can fail on amplitude yet succeed on sign.

The regular-grid Fourier Ultra Toy, alias allocation, survival, and contamination come from Harmless Interpolation of Noisy Data in Regression (ISIT 2019; expanded JSAIT 2020 treatment). The n=49n=49, s=7s=7, d=441d=441 presets and λH\lambda_H values reproduce the finite examples in Classification vs regression in overparameterized regimes: Does the loss function matter? (JMLR 2021).

Frequencies separated by multiples of the sampling rate, ff and f+knf + kn, are exact aliases. Under the toy's uniform test distribution, however, these Fourier features are exactly orthogonal. Minimum-ℓ2\ell_2-norm interpolation divides the coefficient of each sampled Fourier mode among its aliases. With equal feature weights, even a noiseless true mode retains less of its coefficient as its alias class grows: extra features regularize the signal.

Parseval's identity makes the noise bookkeeping exact. Spreading fitted noise over an expanding set of orthogonal directions can make its mean-squared test contribution vanish, even though every noisy training sample is fit perfectly. The noise has been dissipated into individually tiny, mutually orthogonal directions.

Where the instrument came from

The Fourier toy was in my thoughts due to teaching. In 2015 and 2016 I spent two semesters engaged in thinking about lower-division courses — specifically, about how the Shannon–Nyquist sampling theorem ought to be taught to students with no measure theory: discretely, with the DFT as polynomial evaluation and sampling as Lagrange interpolation. It raised questions nobody assigns, like what an antialiasing filter is for polynomials — in a sequence of courses that was also teaching least squares, the SVD, and minimum-norm solutions. So all of these tools were primed years before we needed this instrument.

Keeping the signal alive

This led us to two complementary bookkeeping quantities. Survival measures how much of the true signal direction remains in the learned predictor; signal bleed is the true coefficient leaking into its aliases. Contamination measures how much the false directions, carrying shares of signal and noise, perturb a prediction at a test point. Signal that bleeds away can thus return as contamination even with noiseless labels — a directional distinction that aggregate bias and variance can hide.

Harmless interpolation requires enough extra directions to absorb the noise and enough inductive preference for the true directions — for instance, feature weights favoring low frequencies — to keep the signal alive. Which frequencies are aliased, and how the interpolator allocates energy among them, decide whether interpolation is benign or catastrophic.

What this work added

The contribution was not the observation that interpolation of noise can be benign — that was being established concurrently, and in more general settings. What we added was a Fourier model in which overparameterization becomes undersampling, making exact aliasing and exact orthogonality available as instruments for calculating rather than merely describing the effect; and a separation of that calculation into survival and contamination, which turns one question about generalization into two questions with different answers.

A toy that made conjectures possible

This Fourier construction was part of the work from the beginning, not an illustration we attached afterward.

In early talks I would show the best interpolating degree-6 polynomial through a handful of noisy points next to the best degree-40 one. The degree-40 fit is closer to what we want, and once people saw that, they accepted it. What shifted was a prior nobody had examined: that nature would spend the extra degrees of freedom against you, when it could as easily spend them kindly, and more of them make kindness easier (Mr. Burns's diagnosis).

In the Fourier lab, open the attenuated predictor and then judge it by sign. The fitted curve stays the same; what counts as a successful prediction changes.

What stays fixed

The training labels and the minimum-norm rule do not change when the evaluation ruler changes. Regression asks whether the amplitude is right; classification asks whether the sign is right. That single change creates the intermediate regime.

Classification is not regression

Exact aliasing and exact orthogonality collapse the mechanism to a calculation simple enough to serve as a conjecture engine. The Fourier "ultra-toy" suggested — and we then established for Gaussian features in the JMLR 2021 paper written with Mikhail Belkin and Daniel Hsu (along with students Vidya Muthukumar, Adhyyan Narang, and Vignesh Subramanian) — that regression and classification obey different thresholds.

Regression needs the predicted value to be right: survival must stay near one and contamination near zero. Classification only needs the surviving signed signal to dominate the contamination — a weakened signal still classifies correctly as long as it outvotes the noise at the test point. The consequence is a genuine intermediate regime in which the very same minimum-norm predictor fails as a regressor while succeeding as a classifier.

There is a second, stranger consequence. With sufficient effective overparameterization, every training point becomes a support vector, and the hard-margin SVM — trained on a completely different loss — coincides exactly with the minimum-norm least-squares interpolator. In these regimes, the choice of training loss can matter far less than the geometry both losses inherit.

Six panels in three rows showing low-frequency feature weights and minimum-norm interpolation of binary step labels: accurate amplitude, attenuated but mostly correct sign, and severe signal loss.

Three finite regimes in the Fourier Ultra Toy, controlled by λH\lambda_H — the weight placed on the favored low-frequency features, plotted in the left column. Top: the learned function retains enough amplitude for regression and classification. Middle: its amplitude is badly attenuated, but its sign remains right almost everywhere. Bottom: even the sign is lost in this displayed finite example. Open each row in the Fourier lab: both succeed, classification only, or finite failure. The JMLR paper proves the corresponding three-way separation for its Gaussian model; it explicitly leaves the matching asymptotic Fourier lower bound in the third regime open.

How the conjecture found its collaborators

By 2019 the field had precise results for regression, and during the Simons program that summer we asked what the Fourier toy said about classification instead. The calculation said the thresholds differ. We took it to Daniel Hsu and Mikhail Belkin, which is how the JMLR paper became joint. Support-vector proliferation emerged during that work, first in the Fourier toy; we had not gone looking for it.

Turning mechanisms into phase maps

Our second major tool was a deliberately physics-style use of asymptotics. Instead of sending sample size and dimension to infinity along one fixed aspect ratio, we write each structural quantity as a power of the number of training points nn: the number of features scales as npn^p, the number of favored directions as nrn^r, the strength of the favoring enters through an exponent qq, and — in the later work — the number of classes grows as ntn^t.

Every competing mechanism now owns an exponent, and comparing exponents turns complicated non-asymptotic bounds into linear boundaries between regions: a phase map. One region where regression and classification both succeed; one where classification alone succeeds; one where nothing does. We progressively filled in these maps across the multiclass papers: NeurIPS 2022 established achievable regions, ISIT 2023 supplied a matching converse, and the NeurIPS 2023 paper sharpened the lower bounds into strong converses that pin the boundaries exactly.

Changing the growth exponents moves the boundary between vanishing and nonvanishing error for interpolation. Open the averaging comparison to see how averaging the positive examples of each class can allow more classes even where regression works. The averaging overlay follows the paper’s scaling calculation; these are asymptotic Gaussian-model comparisons, not finite-sample measurements.

Bi-level ensemble and exact multiclass boundary from Definition 1 and Theorem 3.2 of Precise asymptotic generalization for multiclass classification with overparameterized linear models (NeurIPS 2023, revised 2025); averaging comparison from Appendix G. Regression and binary comparisons follow the accompanying phase-map framework.

Compare with the original paper figure
The method, not just the map

The maps were a method of discovery: they told us which theorem ought to be true in each region before we could prove it.

The loop closed in a classroom

In autumn 2020 I taught the machine-learning course with this material in it, and hired Alex Tsigler as a teaching assistant on purpose — he is a co-author of the benign-overfitting characterization for linear regression — with Vignesh Subramanian alongside. Teaching binary classification to a class exposed how little we could say about many classes. The multiclass program, and eventually the strong converses above, began as the gap between what we could prove and what we had to teach.

Ridge regression, seen wider

The JSAIT regression paper also gives an exact matrix construction of this regularizing effect. It went into an appendix because of page limits, and because it was not what the paper was about, but it earns a place here. The familiar "pseudo-observation" view of ridge regression makes the data matrix taller: append fake training examples with zero targets, and the ridge solution appears.

The alternative view makes it wider: append one sample-specific feature per training point — a scaled identity block — and solve an ordinary minimum-norm interpolation problem. Those private features can absorb each training point's residual, but they contribute nothing at a test point, where they are never active. The two constructions impose exactly the same spectral shift, so ridge regression is minimum-norm interpolation with extra feature directions. The feature model really has become wider, and the minimum-norm rule now selects a regularized shared predictor. The extra parameters let the training residuals be fit exactly while bringing shrinkage to the part of the model used at test time — the same effect that alias sharing makes visible in the Fourier model.

Private features absorb every training residual and contribute zero to a fresh test sample—even one at the same input coordinate. The shared prediction is exactly the ridge solution.

An illustrative quadratic-feature example of the wide-matrix ridge construction in Harmless Interpolation of Noisy Data in Regression (JSAIT 2020).

Adversarial sensitivity with clean labels

We showed that adversarial sensitivity can arise even when every training label is correct, the true function is available in the feature set, and the input is only one-dimensional. Belkin, Hsu, and Mitra (2018) had already connected fitting noisy labels to adversarial sensitivity; our example requires no label noise. Minimum-norm interpolation spreads the true signal into aliases. They form sharp positive peaks at the training inputs, with negative side lobes beside them: clean signal bleed alone can create wrong-sign predictions.

The negative regions occupy little of the input domain but recur near every training point. In the paper's asymptotic regime, ordinary classification error tends to zero while adversarial error tends to one for perturbations on the scale of the sample spacing. The lab starts with a finite clean-label example of this mechanism; the noisy-label comparison shows a different route to sensitivity.

The two strips compare where predictions are wrong with where a wrong prediction can be reached within the chosen radius. Clean labels are the starting point. The noisy-label comparison also uses stronger feature preference, so the presets compare mechanisms rather than changing noise alone. Distances wrap around the periodic domain.

Clean-label mechanism from Lemma 1, Figure 1, and Theorem 1 of Classification and Adversarial Examples in an Overparameterized Linear Model: A Signal-Processing Perspective. Both presets are independently calculated finite examples, not reproductions of the paper’s plotted parameters or asymptotic limits.

Compare with the paper’s clean-label figure

Weak-to-strong generalization

In weak-to-strong generalization, a framework introduced by Burns et al. (2023), a weaker model supplies imperfect labels for training a stronger student. The question is whether the student can become more accurate than its teacher. OpenAI's call for superalignment research prompted us to study this using survival and contamination.

A Fourier counterpart illustrates the mechanism. A weak alias in the teacher's predictor enters the pseudo-labels, so the student has an opportunity to learn it. But the student's inductive bias need not preserve every teacher direction equally: the true direction can survive while the alias remains weak. Our formal result establishes weak-to-strong generalization in a Gaussian-feature model, where survival and contamination quantify this competition. In its successful regime, the student fits the imperfect supervision exactly yet predicts more accurately than the teacher on fresh inputs.

The weak teacher learns from n clean labels, then labels many new inputs. A student with stronger features fits those imperfect pseudo-labels exactly. On fresh inputs, successful generalization occurs when survival of the true direction dominates contamination from other directions. Evaluation compares the student with true labels, not its teacher.

Exact agreement in training need not mean the same errors at test time. The student fits every pseudo-label, including the teacher’s mistakes. Its feature geometry can nevertheless favor the true direction enough that survival dominates contamination on fresh inputs. This depicts the successful asymptotic regime of the Gaussian-feature theorem; it does not assert that every teacher error disappears, or that any stronger model will improve on its teacher.

Original schematic of Procedure 1, Theorem 3.3, and the survival/contamination decomposition in Section 3.1 of Provable Weak-to-Strong Generalization via Benign Overfitting (ICLR 2025). Gaussian covariates, bi-level feature scaling, and the weak-to-strong subset ensemble.

What the method makes possible

The through-line of the whole story is a method as much as a result: build the smallest world in which a mechanism operates exactly, calculate it there, and let the exponents say where each phenomenon begins and ends. I think of it as controlled handwaving — keep the toy world fully rigorous, and hand-wave only the claim that it extends. The idea is not mine; it is how theorists have always built models and how physics has long been taught. What teaching added was doing it on purpose, and twice in this story teaching set up what came next: the tools in 2015 and 2016, the multiclass program in 2020. The Fourier toy and the phase maps remain live instruments for us, and this arc is still being extended.

Scope and caveats

These are linear and Fourier toy models by design. The claim is not that they explain every neural network; it is that phenomena often bundled together as "overfitting" can be separated, calculated, and proved to have different phase boundaries.

The papers

Our ISIT 2019 paper introduced the harmless-interpolation problem and already pointed toward the Fourier setting. Its expanded arXiv/JSAIT version developed the Fourier Ultra Toy, exact alias calculations, survival and contamination, and the wide-feature view of ridge. The JMLR 2021 paper used that Ultra Toy to conjecture the classification/regression separation and support-vector proliferation, then established the corresponding results in the Gaussian model. The 2021 signal-processing paper made the mechanism spatially visible, including its adversarial consequences. The NeurIPS 2022 and ISIT 2023 papers opened up the multiclass phase maps, the NeurIPS 2023 paper closed them with precise asymptotics and strong converses, and the ICLR 2025 paper carried benign overfitting to weak-to-strong generalization.

Related Papers