TPV: Parameter Perturbations Through the Lens of Test Prediction Variance

摘要

We identify test prediction variance (TPV)—the first-order sensitivity of model outputs to parameter perturbations around a trained solution—as a unifying quantity that links several classical observations about generalization in deep networks. TPV is a fully label-free object whose trace form $\mathrm{Tr}(H_{\mathrm{eff}} C)$ separates the geometry of the trained model $H_{\mathrm{eff}}$ from the specific perturbation mechanism $C$, allowing a broad family of parameter perturbations like SGD noise, label noise, finite-precision noise, and other post-training perturbations to be analyzed under a single framework. Theoretically, we show that TPV estimated on the training set converges to its test-set value in the overparameterized limit, providing the first result that prediction variance under local parameter perturbations can be inferred from training inputs alone, and this stability is decoupled from generalization performance. Empirically, TPV exhibits a striking stability across datasets and architectures even for extremely narrow networks. Further, TPV correlates well with test loss, serving as a training-set based predictive metric for generalization.