Reducing cross-sample prediction churn in scientific machine learning

arXiv:2605.13826v1 Announce Type: new Abstract: Scientific machine learning reports predictive performance. It does not report whether the same prediction would survive a different draw of training data. Across 9 chemistry benchmarks, two classifiers trained on independent bootstraps of the same training set agree on aggregate accuracy to within 1.3ext{--}4.2 percentage points but disagree on the class label of 8.0ext{--}21.8% of test molecules. We call this gap cross-sample prediction churn. The standard parameter-side techniques (deep ensembles, MC dropout, stochastic weight averaging) do not reduce this gap; two data-side methods do. The first is K-bootstrap bagging, which cuts the rate 40ext{--}54% on every dataset at no accuracy cost (K{imes}-ERM compute). The second is twin-bootstrap, our proposal: two networks trained jointly on independent bootstraps with a sym-KL consistency loss between their predictions, which at matched 2{imes}-ERM compute reduces churn a further median 45% beyond bagging-K{=}2. Cross-sample prediction churn deserves a column alongside predictive performance in scientific-ML benchmark reports, because without it the parameter-side and data-side methods are indistinguishable on the metric they actually differ on.

Source: arXiv cs.LG | 2026-05-14