Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs

arXiv:2602.01064v2 Announce Type: replace Abstract: Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, more efficient models. However, traditional distillation approaches face challenges related to knowledge conflicts and high resource demands, particularly when leveraging multiple teacher models. In this paper, we introduce the concept of extbf{Knowledge Purification}, which consolidates the rationales from multiple teacher LLMs into a single rationale, thereby mitigating conflicts and enhancing efficiency. To investigate the effectiveness of knowledge purification, we further propose five purification methods from various perspectives. Our experiments demonstrate that these methods not only improve the performance of the distilled model but also effectively alleviate knowledge conflicts. Moreover, router-based methods exhibit robust generalization capabilities, underscoring the potential of innovative purification techniques in optimizing multi-teacher distillation and facilitating the practical deployment of powerful yet lightweight models.

Related

• Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Models
• Adaptive Multi-Expert Reasoning via Difficulty-Aware Routing and Uncertainty-Guided Aggregation
• Task-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarios
• Self-Calibrating Language Models via Test-Time Discriminative Distillation
• Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight

Source: arXiv cs.CL | 2026-04-14