Research
Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text
arXiv:2610.19294v1 Announce Type: new Abstract: This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge. We improve on our baseline using data that has been summarize
arXiv:2610.19294v1 Announce Type: new Abstract: This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge. We improve on our baseline using data that has been summarized and translated by an LLM, showing that higher-quality data can come from synthetic sources. We further improve with 2 simple changes: using a larger model, and using a character-level vocabulary. Both of these changes require LoRA-style fine-tuning for realizing their full potential. We dub our final Qwen3-based model Qwenfant, which, at the time of submission, ranks 1st in the Multilingual track.
Related
- Relational Attention for Data-Efficient Language Modeling
- XITE: Cross-lingual Interpolation for Transfer using Embeddings
- A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books
- Understanding the Ability of LLMs to Handle Character-Level Perturbation
Source: arXiv cs.CL | 2026-10-15