BOW: Training Language Models to Reason Over Plausible Next Words

arXiv:2506.13502v3 Announce Type: replace Abstract: Next-word prediction (NWP) trains language models against a single observed continuation, even though many contexts admit multiple plausible next words. Recent RL-based next-word reasoning methods make this tension explicit: they reward a model for producing a rationale that supports one context-conditioned continuation, which can turn a pre-existing preference into a confident, self-justifying trajectory. We introduce BOW, an RL framework that instead trains models to produce self-contained, neutral, and comprehensive descriptions of the plausible next-word space. BOW's core reward is mediated by the generated trajectory. The policy conditions on the full context, but a frozen scorer assigns the core reward from the trajectory alone, without receiving the original context as a separate input. The trajectory may restate relevant context; the bottleneck is the missing direct context-to-scorer path in the core reward. BOW-Reg adds a lightweight breadth regularizer around this core term to discourage premature collapse. Across ten general reasoning benchmarks, BOW remains competitive with the original instruction models and often outperforms trained baselines. On both backbones, BOW-Reg achieves the highest SharedRef correctness and the lowest HoWN-Simple single-sense collapse. Human evaluation further shows that BOW-Reg elicits broader next-word reasoning trajectories, while intrinsic NWP results show that these trajectories remain predictive.

Source: arXiv cs.CL | 2026-08-05