VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation

arXiv:2604.12044v1 Announce Type: cross Abstract: Deep learning models may converge to suboptimal solutions despite strong validation accuracy, masking an optimization failure we term Trajectory Deviation. This is because as training proceeds, models can abandon high generalization states for specific data sub-populations, thus discarding previously learned latent features without triggering classical overfitting signals. To address this problem we introduce VISTA, an online self-distillation framework that enforces consistency along the optimization trajectory. Using a validation-informed Marginal Coverage score, VISTA identifies expert anchors, which are earlier model states that retain specialized competence over distinct data regions. A coverage-weighted ensemble of these anchors is integrated online during training, regularizing the loss landscape and preserving mastered knowledge. When evaluated across multiple benchmarks, VISTA demonstrates improved robustness and generalization over standard training and prior self-distillation methods, while a lightweight implementation reduces storage overhead by 90% without performance loss.

Related

• Training Deep Visual Networks Beyond Loss and Accuracy Through a Dynamical Systems Approach
• A Minimal Model of Representation Collapse: Frustration, Stop-Gradient, and Dynamics
• Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
• WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
• Distilling Genomic Models for Efficient mRNA Representation Learning via Embedding Matching
• Training Deep Visual Networks Beyond Loss and Accuracy Through a Dynamical Systems Approach

Source: arXiv cs.AI | 2026-04-15