Calibrating Small Language Models for Claim Check-Worthiness Detection

arXiv:2608.30731v1 Announce Type: cross Abstract: Assessing claim check-worthiness is an essential first step in automated fact-checking pipelines. This work is motivated by a real deployment challenge at an early-stage startup: running large language models (LLMs) over every incoming claim is cost- and latency-prohibitive, yet smaller models sacrifice accuracy. We propose NN-PPI, a pointwise extension of Prediction-Powered Inference (PPI) that calibrates model predictions at inference time as a lightweight post-hoc layer, without re-training the underlying model. NN-PPI achieves weighted F1 gains ranging from 12% to 33.80% depending on the size and performance of the baseline model, bringing SLMs on par with larger LLMs. Beyond few-shot SLMs, NN-PPI further improves a production-deployed fine-tuned model, demonstrating that residual calibration is complementary to supervised fine-tuning. By recovering LLM-level accuracy from models that are an order of magnitude cheaper to serve, it makes accurate check-worthiness detection substantially cheaper to operate at scale. Our code and data can be found at https://anonymous.4open.science/r/arr-claim-worthiness-F237.

Related

• Improved Confidence Estimates for Black-Box Large Language Models
• Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis
• Graphical Models of False Information and Fact Checking Ecosystems
• A Cost-Efficient Routing Pipeline for Multilingual Short-Text Classification Using Small Language Models

Source: arXiv cs.AI | 2026-09-01