Model Releases
Four small architecture decisions can cost up to 47% of a model's long-context performance. New research from Ai2, Carnegie Mellon, and the …
Four small architecture decisions can cost up to 47% of a model's long-context performance. New research from Ai2, Carnegie Mellon, and the University of Washington isolates them. Normalization, GQA,
Four small architecture decisions can cost up to 47% of a model's long-context performance. New research from Ai2, Carnegie Mellon, and the University of Washington isolates them. Normalization, GQA, pretraining context length, and sliding window attention. Each one is made by at least one of the Olmo, Llama, and Qwen dense families, and each looks harmless on its own. Combine three or more and downstream long-context performance drops significantly. None of it shows up in short-context loss or validation sets, which is how the choices survive into shipped models. Applying context extension early in pretraining exposes the problem. They release OlmPool, 26 comparable 7B models with checkpoints before and after extension, from over 170,000 GPU hours of training. Paper: https://arxiv.org/abs/2608.10296 Track more trending AI papers in our academy: https://academy.dair.ai/
Related
- Banger paper from NVIDIA. Agentic reasoning needs models that are not just capable, but efficient at long-context inference. The agent model…
- New research from Meta and CMU. This one is on agentic context management for long horizon tasks. (bookmark it) Production agents accumulate…
- On benchmarking long-context agentic instruction following. Agent benchmarks mostly reward reaching the answer. This new benchmark measures …
Source: DAIR.AI (X) | 2026-08-12