Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification

arXiv:2607.23856v1 Announce Type: new Abstract: A Last-Layer Ensemble (LLE), K linear units on one shared frozen feature map, is an efficient single-pass approach to the disagreement-based epistemic uncertainty for out-of-distribution (OOD) detection. Its weakness is that members share the backbone gradient and can converge toward the same function, collapsing the inter-member diversity the signal depends on. Whether last-layer diversity can be restored, and what mitigates the collapse, is an open question. The weight-orthonormality defining Orthonormal Certificates (OC), the weight-orthonormal special case of the LLE, is only an indirect correction; it decorrelates the weights of the members, not their predictions. Here, we instead target the collapse directly in function space, with a Covariance Last-Layer Ensemble (cov-LLE) that places a direct covariance penalty on member activations. Cov-LLE restores the function-space diversity that weight-orthonormality cannot, and at matched K recovers much of the diversity and calibration of a deep ensemble at 1imes backbone cost (in-distribution prediction variance 0.05!o!9.3 vs. 22.1 (imes10^{-3}), and ECE 0.135!o!0.090 vs. 0.035, for a Kimes-cost deep ensemble), at no cost to accuracy. Viewing OC as a last-layer ensemble also organizes detectors into a two-axis taxonomy (by how their units are trained and how their outputs are scored) and exposes the OC score as a magnitude, motivating a scale-invariant, label-free direction score that repairs its near-OOD failure, adding +0.16 to +0.18 ROC AUC on every backbone.

Related

• Is the Last Layer Sufficient for Uncertainty Quantification?
• Variance-Gated Ensembles: An Epistemic-Aware Framework for Uncertainty Estimation
• Efficient Bayesian Deep Ensembles via Analytic Predictive Inference

Source: arXiv cs.LG | 2026-07-28