Safety
From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection
arXiv:2608.25243v1 Announce Type: new Abstract: Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fi
arXiv:2608.25243v1 Announce Type: new Abstract: Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fine-tuning (SFT), which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning. To address this, we propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection. Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on-policy rollouts fail on novel facts. We further introduce Blank and Counter, two document-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single-fact recall, multi-source retrieval, and inferential reasoning. Our experiments establish a clear empirical claim: mixed-policy reinforcement learning enables knowledge absorption beyond what supervised fine-tuning can achieve. GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.
Related
- Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning
- Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning
- MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
- Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
Source: arXiv cs.CL | 2026-08-27