SPaCe: Unlocking Sample-Efficient Large Language Models Training With Self-Pace Curriculum Learning

arXiv:2508.05015v2 Announce Type: replace Abstract: Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL). However, such methods require extensive data and compute, making them impractical under many realistic training budgets. Many existing pipelines sample training examples uniformly across steps or epochs, ignoring differences in difficulty, redundancy, and learning value, which slows learning and wastes computation. We propose extbf{SPaCe}, a self-paced learning framework that enables efficient learning based on the capability of the model being trained through optimizing which data to use and when. First, we apply cluster-based data reduction to partition training data by semantics and difficulty, extracting a compact yet diverse subset that reduces redundancy. Then, a extit{multi-armed bandit} treats data clusters as arms, allocating training samples based on the model's solve rates and learning progress. Experiments across multiple reasoning benchmarks show that SPaCe achieves comparable or better accuracy than state-of-the-art baselines while using up to (100imes) fewer samples. Ablation studies and analyses further highlight the importance of both data clustering and adaptive selection. Our results demonstrate that carefully curated, performance-driven training curricula can unlock strong reasoning abilities in LLMs with minimal resources.

Related

• Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models
• RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs
• Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
• Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning

Source: arXiv cs.LG | 2026-04-17