GRASP: Reinforcing Language Model Anonymizers with Group Relative Policy Optimization

arXiv:2608.06526v1 Announce Type: new Abstract: Large language models can infer sensitive personal attributes, such as age, location, and occupation, from ordinary text, turning everyday writing into a privacy risk. Adversarial anonymization defends against this by rewriting a text with a capable language model that also plays the attacker, but it needs a powerful model at inference time and thus sends private text to a third party, the very exposure anonymization should prevent. Recent work distills this behavior into a small on-device model using supervised fine-tuning and direct preference optimization (DPO), but DPO only imitates the teacher's offline choices and never directly optimizes the privacy--utility objective we care about. We introduce extbf{GRASP} (extbf{G}roup-extbf{R}elative extbf{A}nonymization via extbf{S}elf-refinement extbf{P}olicy-optimization), which reinforces the local anonymizer online with Group Relative Policy Optimization. A single small model acts as anonymizer, adversary, and utility judge, trained against a self-generated reward that hides attributes while preserving meaning, with a design that guards against reward hacking. Trained on Llama-3.1-8B, ours{} improves the privacy--utility trade-off over the DPO-distilled baseline, consistently across three independent LLM judges. Against adversarial anonymization driven by frontier models such as Gemini2.5Flash and Claude, it achieves a comparable or better overall trade-off while removing substantially more private information, and it runs entirely on-device at roughly 1% of the GPT-4o teacher's cost.

Related

• N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization
• Predictable GRPO: A Closed-Form Model of Training Dynamics

Source: arXiv cs.CL | 2026-08-10