BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs

arXiv:2608.30646v1 Announce Type: cross Abstract: Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.

Related

• Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Security
• Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs
• GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedback

Source: arXiv cs.AI | 2026-09-01