Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning

arXiv:2604.18978v1 Announce Type: cross Abstract: Scaling critic capacity is a promising direction for enhancing off-policy reinforcement learning (RL). However, larger critics are prone to overfitting and unstable in replay-buffer-based bootstrap training. This paper leverages Low-Rank Adaptation (LoRA) as a structural-sparsity regularizer for off-policy critics. Our approach freezes randomly initialized base matrices and solely optimizes low-rank adapters, thereby constraining critic updates to a low-dimensional subspace. Built on top of SimbaV2, we further develop a LoRA formulation, compatible with SimbaV2, that preserves its hyperspherical normalization geometry under frozen-backbone training. We evaluate our method with SAC and FastTD3 on DeepMind Control locomotion and IsaacLab robotics benchmarks. LoRA consistently achieves lower critic loss during training and stronger policy performance. Extensive experiments demonstrate that adaptive low-rank updates provide a simple, scalable, and effective structural regularization for critic learning in off-policy RL.

Related

• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training
• CROP: Conservative Reward for Model-based Offline Policy Optimization
• SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
• FASTER: Value-Guided Sampling for Fast RL
• Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning

Source: arXiv cs.AI | 2026-04-22