Research
Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings
arXiv:2505.15353v3 Announce Type: replace Abstract: Log-likelihood vectors define a common space for comparing language models as probability distributions, enabling unified comparisons across heterog
arXiv:2505.15353v3 Announce Type: replace Abstract: Log-likelihood vectors define a common space for comparing language models as probability distributions, enabling unified comparisons across heterogeneous settings. We extend this framework to training checkpoints and intermediate layers, and establish a consistent scale for KL divergence across pretraining, model size, random seeds, quantization, fine-tuning, and layers. Analysis of Pythia pretraining trajectories further shows that changes in log-likelihood space, as measured by the scaling behavior of KL divergence, are much smaller than in weight space, resulting in subdiffusive learning trajectories and early stabilization of language-model behavior despite weight drift.
Related
- Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight
- DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning
- Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs
- Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models
Source: arXiv cs.CL | 2026-04-21