AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin

arXiv:2505.14264v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has emerged as an effective approach for enhancing the reasoning capabilities of large language models (LLMs), especially in scenarios where supervised fine-tuning (SFT) falls short due to limited chain-of-thought (CoT) data. Among RL-based post-training methods, group relative advantage estimation, as exemplified by Group Relative Policy Optimization (GRPO), has attracted considerable attention for eliminating the dependency on the value model, thereby simplifying training compared to traditional approaches like Proximal Policy Optimization (PPO). However, we observe that exsiting group relative advantage estimation method still suffers from training inefficiencies, particularly when the estimated advantage approaches zero. To address this limitation, we propose Advantage-Augmented Policy Optimization (AAPO), a novel RL algorithm that optimizes the cross-entropy (CE) loss using advantages enhanced through a margin-based estimation scheme. This approach effectively mitigates the inefficiencies associated with group relative advantage estimation. Experimental results on multiple mathematical reasoning benchmarks demonstrate the superior performance of AAPO. Code is available at https://github.com/JianxXiong/AAPO.

Related

• A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
• Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
• Reinforcement-aware Knowledge Distillation for LLM Reasoning
• Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation
• StaRPO: Stability-Augmented Reinforcement Policy Optimization
• Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Source: arXiv cs.CL | 2026-04-15