Distillation Traps and Guards: A Calibration Knob for LLM Distillability

arXiv:2604.18963v1 Announce Type: cross Abstract: Knowledge distillation (KD) transfers capabilities from large language models (LLMs) to smaller students, yet it can fail unpredictably and also underpins model leakage risks. Our analysis revealed several distillation traps: tail noise, off-policy instability, and, most fundamentally, the teacher-student gap, that distort training signals. These traps manifest as overconfident hallucinations, self-correction collapse, and local decoding degradation, causing distillation to fail. Motivated by these findings, we propose a post-hoc calibration method that, to the best of our knowledge, for the first time enables control over a teacher's distillability via reinforcement fine-tuning (RFT). Our objective combines task utility, KL anchor, and across-tokenizer calibration reward. This makes distillability a practical safety lever for foundation models, connecting robust teacher-student transfer with deployment-aware model protection. Experiments across math, knowledge QA, and instruction-following tasks show that students distilled from distillable calibrated teachers outperform SFT and KD baselines, while undistillable calibrated teachers retain their task performance but cause distilled students to collapse, offering a practical knob for both better KD and model IP protection.

Related

• MoBiE: Efficient Inference of Mixture of Binary Experts under Post-Training Quantization
• Optimizing Large Language Models: Metrics, Energy Efficiency, and Case Study Insights
• ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference
• R^2-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction
• DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LM Inference

Source: arXiv cs.AI | 2026-04-22