Safety
Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model
arXiv:2603.16622v2 Announce Type: replace Abstract: Instead of directly distilling a language model, this study addresses the problem of aligning a base model with a target model in distribution by de
arXiv:2603.16622v2 Announce Type: replace Abstract: Instead of directly distilling a language model, this study addresses the problem of aligning a base model with a target model in distribution by designing the domain mixture of training data for pretraining or continued pretraining as a fixed training recipe. We propose a method for determining domain weights by viewing models as points in log-likelihood space and aligning the training update direction with the direction toward the target model. Experiments with NanoGPT show that the proposed method consistently reduces the KL divergence between the trained base model and the target model relative to training with Pile-original weighting. Although knowledge distillation remains more effective when available, the proposed method achieves meaningful alignment, and downstream task performance also tends to become closer to that of the target model.
Related
- LM-mixup: Text Data Augmentation via Language Model based Mixup
- RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
- GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization
- AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling
Source: arXiv cs.CL | 2026-09-01