ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification

arXiv:2609.10122v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance across a broad range of classification settings, yet the reliability of their predictions remains a major obstacle to deployment in high-stakes scenarios. Although confidence estimation for LLMs has been widely studied, confidence calibration for LLM-based classification remains underexplored. We introduce ProbPlug, a lightweight confidence estimation framework for LLM-based binary classification, which predicts whether an output is correct using internal token features extracted from a frozen LLM. ProbPlug employs a self-attention module to aggregate hidden representations and can be integrated into the original inference pipeline without modifying the base model. Experiments across multiple tasks involving both text-based and multimodal large models show that ProbPlug provides more reliable confidence estimates, improves classification performance with negligible additional overhead, and exhibits strong generalization across tasks. These results indicate that ProbPlug serves as a practical solution for confidence estimation in LLM-based classification. Our code is publicly available at Github.

Related

• Adaptive Multi-Expert Reasoning via Difficulty-Aware Routing and Uncertainty-Guided Aggregation
• When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information
• Confidence Estimation in Automatic Short Answer Grading with LLMs
• CGES: Confidence-Guided Early Stopping for Efficient and Accurate Self-Consistency

Source: arXiv cs.CL | 2026-09-10