GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models

arXiv:2506.10946v3 Announce Type: replace Abstract: Unlearning large language models (LLMs) is important due to copyright and privacy concerns. A key challenge is unintended forgetting, where removing data inadvertently impairs model utility. While prior work has focused on architectural innovations, the influence of data-level factors remains underexplored. We introduce GUARD, a framework for Guided Unlearning And Retention via Data Attribution. GUARD uses a lightweight proxy data attribution metric tailored to LLM unlearning, which efficiently quantifies the "alignment" between the Forget and Retain (or synthetic supplement) Sets. This metric leads to a new unlearning objective that assigns adaptive, nonuniform weights to samples, decaying exponentially with their proxy attribution scores, and thereby provably mitigates unintended retention loss. On the TOFU benchmark with a 10% forget set, applying GUARD to Gradient Ascent (GA) improves Retain Set ROUGE-L from 0.18 to 22.40 (a 124x increase) and Retain Probability from 0.06% to 17.50%, a substantial recovery from the GA collapse. Furthermore, GUARD combined with NPO (Negative Preference Optimization) outperforms Representation Misdirection for Unlearning (RMU) with 33.80% and 53.10% reductions in the Retain Set Sacrifice Rates computed from ROUGE-L and Probability (rho_r and rho_p), and a 16.01% improvement in knowledge retention on the MUSE NEWS Retain Set. These findings persist when attribution scores are computed using synthetic or highly subsampled (20%) Retain Sets, where NPO+GUARD still reduces rho_r and rho_p by up to 43.89% and 48.13% over NPO. Similar improvements are observed on MUSE, with a tolerable absolute privacy leakage degradation of 13.7 compared to the best-performing existing method in the privacy category.

Source: arXiv cs.LG | 2026-10-14