Research
Structural Limits of the Information-Theoretic Uncertainty Decomposition
arXiv:2609.39591v1 Announce Type: new Abstract: Uncertainty estimation in machine learning typically decomposes uncertainty into aleatoric uncertainty (AU) and epistemic uncertainty (EU) using the sta
arXiv:2609.39591v1 Announce Type: new Abstract: Uncertainty estimation in machine learning typically decomposes uncertainty into aleatoric uncertainty (AU) and epistemic uncertainty (EU) using the standard information-theoretic framework. However, in practice, two critical issues arise: entanglement (AU and EU are highly correlated) and epistemic collapse (EU magnitude shrinks with increasing model capacity). We analyze this framework on a functional level and discover that significant portions of the assumed AU, EU range are infeasible in finite settings, and cannot be attained with any class probabilities. We characterize how this infeasible region scales with the number of classes and Monte Carlo samples N (e.g., from ensembles with N members), revealing it is bounded by ext{AU} leq log(2)/N. Crucially, the infeasible region's boundary helps explain epistemic collapse: when model confidence is high, ext{AU} > ext{EU} is guaranteed by this fundamental structural limitation. Our findings show that increasing ensemble size mitigates epistemic collapse by reducing the infeasible area. Lastly, we caution against interpreting AU and EU as independent quantities in low AU regimes, since we show they are coupled when ext{AU} leq log(2)/N.
Related
- It depends: Incorporating correlations for joint aleatoric and epistemic uncertainties of high-dimensional output spaces
- AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow
- CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation
Source: arXiv cs.CV | 2026-10-01