SPIRAL: Learning to Search and Aggregate

arXiv:2606.23595v2 Announce Type: replace Abstract: Language model reasoning can be substantially improved at test time via scaffolds that scale inference compute across different primitives -- sequential reasoning within a trace, independently sampled parallel traces, and aggregation of multiple reasoning traces into a final response. During post-training, however, language models are optimized only for sequential reasoning within a single trace. We introduce Sequential-Parallel-Aggregative Reinforcement Learning (SPIRAL), a framework in which a language model is trained to use all three primitives, as part of a unified inference compute pipeline. Concretely, the language model first samples a set of independent traces in parallel, each produced through sequential chain-of-thought reasoning, and then generates a final aggregation trace conditioned on those traces; all components are optimized end-to-end against the reward of the final aggregated response. To train this system, SPIRAL uses set reinforcement learning to teach models to produce a set of traces that are collectively useful for an aggregator and standard reinforcement learning to teach models to aggregate the set into improved final responses. Our experiments on reasoning tasks show that SPIRAL effectively scales with inference compute, outperforming GRPO by up to 11imes scaling efficiency and 15% higher performance when all three compute primitives are scaled.

Related

• EAGer: Entropy-Aware GEneRation for Adaptive Inference-Time Scaling
• Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
• Beyond the Frontier: Stochastic Backtracking for Efficient Test-Time Scaling
• When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
• Provable Test-Time Scaling for Beam Search in LLM Reasoning

Source: arXiv cs.AI | 2026-10-02