Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining
DGX agentarXiv:2608.03089v1 Announce Type: new Abstract: Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level