Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning

arXiv:2603.09803v2 Announce Type: replace Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that arrive at correct answers by chance. We observe that better reasoning makes better demonstrations: high-quality solutions serve as more effective in-context examples than low-quality ones. We term this teaching ability extbf{Demonstration Utility}, and show that the policy model's own in-context learning ability provides an efficient way to measure it, yielding a quality signal termed extbf{Evidence Gain}. To leverage this signal during training, we introduce extbf{In-Context RLVR}, which prepends demonstrations before each rollout. Theoretically, we prove that this simple input modification implicitly reweights rewards by a factor approximately proportional to Evidence Gain, assigning higher weights to high-quality traces without requiring costly computation. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements in both accuracy and reasoning quality over standard RLVR baselines. Our codes and datasets are available at https://github.com/Mithas-114/IC-DAPO.

Source: arXiv cs.LG | 2026-06-04