No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse

arXiv:2610.01493v1 Announce Type: cross Abstract: Iterative fine-tuning on synthetic data causes model collapse: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we develop a new approach grounded in mathematical information theory: the non-parametric Kontoyiannis entropy rate estimator h_k, computed entirely from raw text via match-length statistics, with no model of any kind. We show that this is in fact a superior training-data filter on text-diversity metrics in a fully-synthetic, single-lineage fine-tuning setting. In a six-generation QLoRA collapse experiment on Llama-3.1-8B, logprob-based filtering (the most established model-access-requiring baseline) provides no significant text-diversity benefit on any metric (p > 0.23), whereas h_k-filtering yields +42% unique trigrams, +30% vocabulary, and -19% repetition (all p < 0.001). We validate h_k as a cross-domain entropy proxy (eta = 0.924, R^2 = 0.746) and collapse detector (rho = +0.454, p < 0.0001) across 4domains, 2temperatures, 2~generator--scorer model pairs, and 1{,}520 generated documents. Our results demonstrate that information theoretic approaches to collapse mitigation are efficient, and suggest new approaches for maintaining multi-agent diversity.

Related

• Persona-Model Collapse in Emergent Misalignment
• Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
• Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigation
• Not All Synthetic Data Is Yours to Learn From

Source: arXiv cs.AI | 2026-10-02