Host-Aware Web Crawling for LLM Pretraining Data
DGX agentarXiv:2610.13321v1 Announce Type: new Abstract: Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We i