Sharpen Before You Adapt: Data-Free Entry-State Sharpening for Test-Time Reinforcement Learning

arXiv:2610.00903v1 Announce Type: new Abstract: Test-time reinforcement learning (TTRL) adapts language models on unlabeled test problems using supervision derived from their own samples. This makes the checkpoint's entry state consequential: a diffuse policy provides noisier self-supervision and may spend much of a limited adaptation budget merely concentrating probability mass before reliably expressing capability it already possesses. We propose extbf{entry-state sharpening}: use data-free training before TTRL to prepare a general-purpose checkpoint in a state that subsequent label-free adaptation can exploit more efficiently. The idea is not tied to one training recipe; different data-free objectives can move the same base model to different entry states. Across five data-free checkpoints derived from Qwen3-4B and evaluated under an identical 15-step TTRL protocol, entry policy entropy strongly rank-orders endpoint conversion efficiency, a reliability-to-reachability measure (Spearman rho=-0.90; rho=-0.99 after controlling for entry reachability). The contrast across objectives is striking: R-Zero remains diffuse at 3.39 nats and finishes below the untuned base in 6/6 matched comparisons across MATH, GPQA, and AMC, whereas SPIRAL reaches 0.07 nats and achieves the highest post-TTRL accuracy on MATH and GPQA despite its self-play stage using no math training data. An in-domain label-free self-distillation intervention further shows that the entry state can be deliberately sharpened. These results motivate treating checkpoint preparation as a state-control problem: use data-free training to improve TTRL readiness, with entry entropy as a label-free control signal and reachable capability as the constraint.

Related

• ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment
• Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
• Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning
• TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models
• Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning

Source: arXiv cs.LG | 2026-10-02