Neurograf v1: An End-to-End Romanian-First Language Model Family

arXiv:2610.13134v1 Announce Type: new Abstract: We present Neurograf v1, an experimental end-to-end Romanian-first language-model pipeline developed to investigate the feasibility of training language models from scratch for a lower-resource European language. The pipeline comprises RoSfera, a 635 GB Romanian corpus containing 175.7 million documents; an 80k byte-level BPE tokenizer; five decoder-only models ranging from 160M to 6B parameters, trained on 131 billion tokens; three instruction-tuned assistants; and two contrastively trained embedding models. Training and evaluation were conducted using computing infrastructure at ELI-NP/IFIN-HH. The model family exhibits consistent scaling of validation loss and increasing Romanian factual and cultural knowledge with model size. On the 1,000-question WWTBM benchmark, Neurograf-6B achieves 69.89% accuracy on Romanian-context questions, while Neurograf-3.2B reaches 67.38%. Compute-optimal analysis yields a parameter-scaling exponent of approximately 0.423. Neurograf v1 serves as an experimental demonstrator rather than a production system. The study identifies practical challenges involving diacritic preservation, corpus quality, data composition, and tokenization efficiency, while demonstrating the feasibility of developing Romanian-first base models, instruction-following assistants, and semantic embedding models within a unified training pipeline.

Related

• TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages
• From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
• The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese
• Training a Language Model End-to-End in Rust: An Experience Report

Source: arXiv cs.CL | 2026-10-12