Learning to Reason with Selective Internal Feedback

arXiv:2610.21392v1 Announce Type: new Abstract: Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers. However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy. We introduce SIF, a reinforcement learning method that combines comparative reward construction with selective policy updates. SIF compares ranked token-level self-certainty profiles across multiple windows and peers, assigning positive rewards only when both levels of voting provide sufficient support. Sample and group level filtering determines which responses contribute to policy optimization. Experiments on three backbones across 12 mathematical and scientific reasoning benchmarks show that SIF's final checkpoints achieve higher average accuracy than the best-validation checkpoints of Intuitor and Vigor, with a gain of 4.71 percentage points over Intuitor on Qwen2.5-Math-1.5B. Further analyses show better retention of accuracy during continued optimization, while test-time training experiments demonstrate sustained improvements with as few as four rollouts per prompt.

Related

• Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models
• Reliability-Aware LLM Alignment from Inconsistent Human Feedback
• Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
• Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
• STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

Source: arXiv cs.AI | 2026-10-16