Model Releases
Size Matters: Foundation Model for Czech HTML documents
arXiv:2609.18494v1 Announce Type: new Abstract: Creating universal, high-quality representations of web documents in high-traffic industrial environments requires models that are both performant and e
arXiv:2609.18494v1 Announce Type: new Abstract: Creating universal, high-quality representations of web documents in high-traffic industrial environments requires models that are both performant and economic. Existing approaches, however, often depend on large models, overlook the structural information inherent in HTML, or are constrained by short context windows, limiting their ability to process real-world web pages. We present HTML-LM, a compact foundation model with 154 million parameters that addresses these limitations through HTML-aware training and a ModernBERT-based architecture. It was trained on 100 million web documents using multiple objectives, including masked language modeling, bag-of-words prediction, and contrastive distillation from large language models. Consequently, HTML-LM sets a new state-of-the-art for classification and regression applications in the Czech Internet domain, surpassing both larger encoders and small-sized LLMs. The model is deployed in production, processing thousands of web documents per second, and released to the community under the CC BY-NC 4.0. https://huggingface.co/Seznam/html-lm.
Related
Source: arXiv cs.CL | 2026-09-17