Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR

arXiv:2610.37553v1 Announce Type: new Abstract: Label-free reinforcement learning with verifiable rewards (RLVR) rewards a language model for agreeing with its own majority vote. It improves reasoning at first but eventually collapses: the self-assessed reward goes to one while accuracy falls, often until the model gives the same answer to every problem. The collapse appears only when training uses many problems. We explain this dataset-size dependence as a collective instability. In a solvable many-problem model in which problems share parameters, a population whose problems prefer different answers amplifies any shared perturbation for as long as the population-averaged slope langle F'rangle of the single-problem self-reinforcement drift F is positive. This holds while problems are still uncertain about their answers. Per-problem sharpening, slowed by 1/N for N training problems, races against the shared mode, which grows at a rate proptolambda^2 independent of N, where lambda is the strength of the shared coupling. This yields a critical number of problems N_csimeqln(1/elta)/(lambda^2G), where elta seeds the shared mode and G decreases with the initial confidence. Simulations of the model confirm the predictions: a sharp transition, N_cproptolambda^{-2} up to a logarithmic correction, a single critical amplification lambda^2NG across couplings, confidence profiles and training variants, a logarithmic dependence on batch size, and no collapse for confident initial models. A kernel version of the theory identifies a protective coupling (generalization of sharpening) and a marginal-entropy bonus that stabilizes the uniform collective mode. In label-free RL of a small Transformer, removing the shared direction prevents collapse entirely, restoring it causes collapse, and collapse grows with training-set size and base-model entropy.

Source: arXiv cs.LG | 2026-10-28