Safety
How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum
arXiv:2604.25907v1 Announce Type: new Abstract: Adapting reasoning models to new tasks during post-training with only output-level supervision stalls under reinforcement learning from verifiable rewar
arXiv:2604.25907v1 Announce Type: new Abstract: Adapting reasoning models to new tasks during post-training with only output-level supervision stalls under reinforcement learning from verifiable rewards (RLVR) when the initial success probability p_0 is small. Using the Tsallis q-logarithm, we define a loss family J_Q that interpolates between RLVR (at q{=}0, the exploitation pole) and the log-marginal-likelihood over latent trajectories (at q{=}1, the density-estimation pole). All members share the same per-example gradient direction, differing only by a scalar amplification P_{heta^{-q}} that reweights each instance independently of the learning rate. This amplification is the mechanism that addresses cold-start stalling: under gradient flow, the exploitation pole requires Omega(frac{1}{p_0}) time to escape cold start, while the density-estimation pole escapes in Thetaig(log(frac{1}{p_0})ig); intermediate q trades escape speed against noise memorization. Because P_heta is intractable, we derive two Monte Carlo estimators from the two factorizations of the gradient: Gradient-Amplified RL (GARL) samples from the prior and amplifies the RL gradient, and Posterior-Attenuated Fine-Tuning (PAFT) importance-resamples from the posterior and runs standard SFT. Both have bias Oig(frac{q}{M P_{heta}^{q+1}}ig); GARL has lower variance, PAFT has semantically coherent gradients. On FinQA, HotPotQA, and MuSiQue, GARL at q{=}0.75 substantially mitigates cold-start stalling, escaping cold start where GRPO fails entirely. In warm start, GARL at low q dominates FinQA where training is stable; on HotPotQA and MuSiQue, GARL destabilizes during training, and PAFT at q{=}0.75 provides stable gradients (best overall on HotPotQA at 47.9 maj@16, +14.4 over GRPO).
Related
- PubSwap: Public-Data Off-Policy Coordination for Federated RLVR
- Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
- Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error
- When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Source: arXiv cs.LG | 2026-04-29