Staying on Task: Testing the Foundations of Long-Horizon Agent Reliability

arXiv:2609.38712v1 Announce Type: new Abstract: Long-horizon agentic workflows require models to sustain repeated state-dependent actions all while the context grows, sub-task complexity changes, and new data arrives. Each situation represents an independent axis along which an agent may fail. An agent reconciling a long ledger, for example, must repeatedly read its state, update the correct record, and preserve alignment across thousands of outputs. A model may accept the entire ledger yet lose its place or stop applying the operation consistently as generation proceeds. We introduce Long-Transduction, a controlled diagnostic that tests a model's ability to stay on task during long generation while continuously reading, mutating, and outputting input-context dependent operations such as arithmetic, sorting, variable lookups, and table transformations. Long-Transduction evaluation independently varies local task complexity, input data formatting, and context length isolate failures along each axis. We evaluate seven open-weight models, finding a 62.8% decrease when scaling context length from 4-128K, a 36.5% decrease when varying input format, and a 39.9% decrease by increasing local task complexity. Together, these failures represent critical liabilities in long-horizon agentic workflows.

Related

• Beyond Policy Optimization: A Data Curation Flywheel for Sparse-Reward Long-Horizon Planning
• Toward Reliable Design of LLM-Enabled Agentic Workflows: Optimizing Latency-Reliability-Cost Tradeoffs
• SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents
• Mnemosyne: Agentic Transaction Processing for Validating and Repairing AI-generated Workflows
• DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training

Source: arXiv cs.AI | 2026-10-01