Interpolation Can Be Harmless

With enough overparameterization, contamination from fitted noise can become small. The less understood side is survival: minimum-norm optimization can make extra features act as regularizers, shrinking the useful signal. The signal-processing perspective lets us separate and calculate these two effects.

When I started on this in 2017, the puzzle was not that a model with more candidate features than samples could work — sparsity-seeking methods had shown that for years. The puzzle was why more features helped with nothing enforcing sparsity, and what was standing in for it. Those are the two questions the bookkeeping below answers: understanding contamination tells us why more features can help, and understanding survival tells us what has to play sparsity's role in making sure that the true pattern is recovered.

Our decisive conceptual move, in the ISIT 2019 paper and its expanded JSAIT version, was to turn a high-dimensional puzzle into an elementary problem from signal processing. Take $n$ regularly spaced samples of a one-dimensional signal, and fit them using more Fourier frequencies than samples. Overparameterization is now literally undersampling.

Sampling theory says exactly what goes wrong — and here, what goes right. Frequencies separated by multiples of the sampling rate, $f$ and $f + kn$, are exact aliases: on the training grid they are indistinguishable, taking identical values at every sample. Under the uniform test distribution used in the toy, however, the corresponding Fourier features are exactly orthogonal. The sampled data can be decomposed into discrete Fourier modes, and minimum-$\ell_2$-norm interpolation responds to the degeneracy in the most even-handed way possible: it divides the coefficient required for each sampled mode among the frequencies in that mode's alias class. Even with noiseless data, a true Fourier mode must share its coefficient with its aliases. With equal feature weights, the coefficient retained in the true direction shrinks as its alias class grows. That is the regularizing effect of the extra features under the minimum-norm rule.

Parseval's identity then makes the bookkeeping exact. Noise that has been spread over an expanding set of orthogonal directions contributes almost nothing to mean-squared test error — even though every noisy training sample was fit perfectly. The fitted noise is not gone; it has been dissipated into directions that are individually tiny and mutually orthogonal under the test distribution.

This loss of survival is a consequence of the optimization rule. A larger feature set allows more exact fits, but minimum-norm interpolation chooses the allocation with the smallest norm. The true low-frequency component must share its coefficient with its high-frequency aliases, so its retained coefficient can become smaller even as the feature model becomes richer.

This led us to a pair of complementary bookkeeping quantities. Survival measures how much of the true signal direction remains in the learned predictor; what limits it is what we called signal bleed, the true coefficient leaking into its aliases. Contamination measures how much the many false directions, carrying their shares of signal and noise, perturb a prediction at a test point. The two are linked but distinct: signal that bleeds away lands in the aliases and can return as contamination. A bias–variance reading obscures this. It suggests that only label noise contaminates, but in high dimensions even noiseless labels do. Harmless interpolation requires both sides of a bargain: enough extra directions to absorb the noise, together with enough inductive preference for the true directions — for instance, feature weights that favor low frequencies — to keep the signal alive while the noise dissipates.

This is why the story cannot be compressed to "noise gets spread across many dimensions." Which directions are aliased with which, and how the interpolator allocates energy among them, is precisely the sampling geometry — and it is that geometry, not high dimensionality alone, that decides whether interpolation is benign or catastrophic.

This Fourier construction was part of the work from the beginning, not an illustration we attached afterward. The ISIT 2019 paper already pointed toward the Fourier setting; the full arXiv/JSAIT treatment developed the regular-grid Ultra Toy and its exact alias calculations. Its value is that nothing important is hidden behind a large random matrix: one can watch minimum-norm interpolation allocate every sampled Fourier mode among features that are identical on the training grid and orthogonal under the test-distribution inner product.

In early talks I would show the best interpolating degree-6 polynomial through a handful of noisy points next to the best degree-40 one. The degree-40 fit is closer to what we want, and once people saw that, they accepted it. What shifted was a prior nobody had examined: that nature would spend the extra degrees of freedom against you, when it could as easily spend them kindly, and more of them make kindness easier (Mr. Burns's diagnosis).

The regression side is not confined to one carefully chosen waveform. In the lab below, every smooth target in the menu is supported entirely on the default $s=7$ favored low-frequency directions. Widen the favored band to see precisely which additional Fourier directions receive the same preference; change the target and the same aliasing mechanism persists. Then switch to the clean $-1$-to-$+1$ step: that deliberately misspecified target is the natural bridge to the classification paper.

Exact aliasing and exact orthogonality collapse the mechanism to a calculation simple enough to serve as a conjecture engine. The Fourier "ultra-toy" suggested — and we then established for Gaussian features in the JMLR 2021 paper written with Mikhail Belkin and Daniel Hsu (along with students Vidya Muthukumar, Adhyyan Narang, and Vignesh Subramanian) — that regression and classification obey different thresholds.

Regression needs the predicted value to be right: survival must stay near one and contamination near zero. Classification only needs the surviving signed signal to dominate the contamination — a weakened signal still classifies correctly as long as it outvotes the noise at the test point. The consequence is a genuine intermediate regime in which the very same minimum-norm predictor fails as a regressor while succeeding as a classifier.

There is a second, stranger consequence. With sufficient effective overparameterization, every training point becomes a support vector, and the hard-margin SVM — trained on a completely different loss — coincides exactly with the minimum-norm least-squares interpolator. In these regimes, the choice of training loss can matter far less than the geometry both losses inherit.

Our second major tool was a deliberately physics-style use of asymptotics. Instead of sending sample size and dimension to infinity along one fixed aspect ratio, we write each structural quantity as a power of the number of training points $n$: the number of features scales as $n^p$, the number of favored directions as $n^r$, the strength of the favoring enters through an exponent $q$, and — in the later work — the number of classes grows as $n^t$.

Every competing mechanism now owns an exponent, and comparing exponents turns complicated non-asymptotic bounds into linear boundaries between regions: a phase map. One region where regression and classification both succeed; one where classification alone succeeds; one where nothing does. The maps did not merely summarize finished theorems — they organized the search for theorems, separating effects that tangle together at any single problem size and identifying which result should be true in each region. We then progressively filled in these maps across the multiclass papers: NeurIPS 2022 established achievable regions, ISIT 2023 supplied a matching converse, and the NeurIPS 2023 paper sharpened the lower bounds into strong converses that pin the boundaries exactly.

The JSAIT regression paper also gives an exact matrix construction of this regularizing effect. It went into an appendix because of page limits, and because it was not what the paper was about, but it earns a place here. The familiar "pseudo-observation" view of ridge regression makes the data matrix taller: append fake training examples with zero targets, and the ridge solution appears.

The alternative view makes it wider: append one sample-specific feature per training point — a scaled identity block — and solve an ordinary minimum-norm interpolation problem. Those private features can absorb each training point's residual, but they contribute nothing at a test point, where they are never active. The two constructions impose exactly the same spectral shift, so ridge regression is minimum-norm interpolation with extra feature directions. The feature model really has become wider, and the minimum-norm rule now selects a regularized shared predictor. The extra parameters let the training residuals be fit exactly while bringing shrinkage to the part of the model used at test time — the same effect that alias sharing makes visible in the Fourier model.

Once survival, contamination, and the phase-map method exist, they travel.

Adversarial examples. The aliased component is not spread uniformly in space but concentrated in sharp, Gibbs-like spikes near the training samples. Contamination can be small on average — so ordinary classification succeeds — while being locally enormous, so that tiny input perturbations near training data flip predictions. Low test error and proliferating adversarial failures are not in tension; in this model they are the same geometry read at two different scales.

Many classes. Multiclass classification asks the surviving signal to win more comparisons with less margin. We extended the phase maps to a growing number of classes, located the new boundaries, and — in the NeurIPS 2023 paper — proved matching lower bounds and strong converses, so the map's frontiers are exact rather than conjectured.

Weak-to-strong generalization. A Fourier counterpart makes the connection vivid. If a weak alias is present in the teacher model, then giving the student more pseudo-labeled samples puts that alias into the samples themselves. The student could learn it. But the inductive bias does not preserve every teacher-side direction equally: the weak alias does not survive strongly enough to be learned, while the true direction does. In the formal Gaussian-feature model, the same competition is measured through survival and contamination. The student can fit the imperfect supervision exactly without inheriting every error that produced it, and can therefore generalize beyond the teacher. OpenAI's call for superalignment research is what prompted us to look; the framework already had the answer in it.

The through-line of the whole story is a method as much as a result: build the smallest world in which a mechanism operates exactly, calculate it there, and let the exponents say where each phenomenon begins and ends. I think of it as controlled handwaving — keep the toy world fully rigorous, and hand-wave only the claim that it extends. The idea is not mine; it is how theorists have always built models and how physics has long been taught. What teaching added was doing it on purpose, and twice in this story teaching set up what came next: the tools in 2015 and 2016, the multiclass program in 2020. The Fourier toy and the phase maps remain live instruments for us, and this arc is still being extended.

Our ISIT 2019 paper introduced the harmless-interpolation problem and already pointed toward the Fourier setting. Its expanded arXiv/JSAIT version developed the Fourier Ultra Toy, exact alias calculations, survival and contamination, and the wide-feature view of ridge. The JMLR 2021 paper used that Ultra Toy to conjecture the classification/regression separation and support-vector proliferation, then established the corresponding results in the Gaussian model. The 2021 signal-processing paper made the mechanism spatially visible, including its adversarial consequences. The NeurIPS 2022 and ISIT 2023 papers opened up the multiclass phase maps, the NeurIPS 2023 paper closed them with precise asymptotics and strong converses, and the ICLR 2025 paper carried benign overfitting to weak-to-strong generalization.