StepGap: A Hybrid NLI-LLM Checker for Step-Level Evidence-Gap Detectionin Multi-Hop Question Answering

arXiv:2605.24733v1 Announce Type: new Abstract: We present extbf{StepGap}, a hybrid NLI-LLM decision tree that detects step-level evidence gaps in multi-hop QA and emits one of three typed labels: extsc{Contradicted Claim} (CC), extsc{Irrelevant Evidence} (IE), or extsc{Missing Bridge} (MB), each tied to a concrete repair action. On 82 multi-hop questions (181 annotated steps, kappa{=}0.704), StepGap reaches sF1=72.0, within the bootstrap confidence interval of an LLM-only baseline (70.1) but with a more decomposable structure: every StepGap stage hurts F1 when removed, while three of four LLM-only removals improve F1 -- a sign of competing-error cancellation, where internal stages mask each other's errors. We further expose a Q-F1 trap: question-level F1 is mechanically inflated by checkers that flag every step, making step-level F1 the necessary diagnostic. Used as a typed GRPO process reward, StepGap improves Qwen2.5-7B-Instruct Exact Match from 32.1{pm}0.3 to 35.4{pm}0.9 across three seeds, with the single-run comparison showing a +5.6 Avg EM gain over the matched Search-R1 GRPO reproduction.

Source: arXiv cs.CL | 2026-05-26