From Domains to Instances: Dual-Granularity Data Synthesis for LLM Unlearning

arXiv:2601.04278v2 Announce Type: replace Abstract: Although machine unlearning is essential for removing private, harmful, or copyrighted content from LLMs, current benchmarks often fail to faithfully represent the true ``forgetting scope'' learned by the model. We formalize two distinct unlearning granularities, domain-level and instance-level, and propose BiForget, an automated framework for synthesizing high-quality forget sets. Unlike prior work relying on external generators, BiForget exploits the target model per se to elicit data that matches its internal knowledge distribution through seed-guided and adversarial prompting. Our experiments across diverse benchmarks show that it achieves a superior balance of relevance, diversity, and efficiency. Quantitatively, in the Harry Potter domain, it improves relevance by {sim}20 and diversity by {sim}0.05 while halving the total data size compared to SOTAs. Ultimately, it facilitates more robust forgetting and better utility preservation, providing a more rigorous foundation for evaluating LLM unlearning.

Related

• Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMs
• CiPO: Counterfactual Unlearning for Large Reasoning Models through Iterative Preference Optimization
• Beyond Static Benchmarks: Synthesizing Harmful Content via Persona-based Simulation for Robust Evaluation

Source: arXiv cs.CL | 2026-04-21