Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking

arXiv:2610.17767v1 Announce Type: cross Abstract: Large reasoning models (LRMs) achieve strong performance by externalizing explicit reasoning traces before producing the answer, yet suffer from overthinking challenge that allocates uniformly heavy computation to queries of varying difficulty. While proprietary models mitigate this via opaque routing, open-source LRMs still lack an efficient mechanism to internalize adaptive reasoning due to both expensive training cost and limited disclosure of training recipes. In response, we introduce RPO (Root-token Policy Optimization), a framework that enables LRMs to self-determine when to reason by training only the initial root token (e.g., whether to invoke the tag) via group relative reward and group-wise advantages. By focusing on this pivotal branching point, RPO drastically reduces training overhead and VRAM usage. Across multiple model families and scales, RPO learns difficulty-aware adaptive thinking at just ~2% of the training compute of prior adaptive-reasoning methods.

Related

• TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
• First Token Matters: Understanding Safety Collapse in Large Reasoning Models
• Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
• Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

Source: arXiv cs.AI | 2026-10-15