What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2

arXiv:2610.37157v1 Announce Type: new Abstract: Knowledge distillation compresses large language models into compact students that preserve aggregate metrics, yet the mechanistic consequences of depth-reduction distillation remain largely uncharacterized. This work applies linear CKA, logit lens analysis, attention pattern decomposition, sparse autoencoder feature statistics, and activation patching to compare GPT-2 and its six-layer student DistilGPT-2 across two canonical circuits: the Indirect Object Identification (IOI) circuit and the Greater-Than (GT) circuit. The results reveal a compression-collapse dichotomy: the IOI circuit suffers complete mechanistic collapse (DistilGPT-2: 35.2% accuracy vs. 94.8%, zero name-mover preservation), while the GT circuit survives with 87.7% task performance recovery through active remapping of the teacher's layer-9 comparison head into student layer~4 (mean Pearson r = 0.945). These findings demonstrate that circuit compressibility governs mechanistic survival under distillation, and that aggregate metrics are insufficient diagnostics for circuit integrity.

Related

• GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning
• ReAD: Reinforcement-Guided Capability Distillation for Large Language Models
• Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs
• InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion

Source: arXiv cs.CL | 2026-10-28