Safety
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
arXiv:2608.29662v1 Announce Type: new Abstract: Knowledge distillation effectively transfers reasoning capabilities from large language models to lightweight student models. To enable knowledge transf
arXiv:2608.29662v1 Announce Type: new Abstract: Knowledge distillation effectively transfers reasoning capabilities from large language models to lightweight student models. To enable knowledge transfer across disparate model families, researchers increasingly explore cross-tokenizer distillation. However, cross-tokenizer distillation remains challenging due to vocabulary and sequence misalignment, while approximate vocabulary alignment can introduce additional noise into distillation. To address these challenges, we propose Anchor-Based Cross-Tokenizer Distillation with Residual Regularization (ACTD). ACTD bridges structural heterogeneity through vocabulary and sequence alignment, while mitigating alignment noise via a novel anchor loss with residual regularization. We further extend this framework to a multi-teacher setting. Evaluated across five reasoning benchmarks with three distinct teacher models, ACTD achieves state-of-the-art performance. Moreover, its multi-teacher extension outperforms the strongest single-teacher and multi-teacher baselines, further demonstrating the robustness of our method.
Related
- SRA: Span Representation Alignment for Large Language Model Distillation
- Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
- RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer
- A Survey of On-Policy Distillation for Large Language Models
Source: arXiv cs.CL | 2026-09-01