Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems

arXiv:2608.13571v1 Announce Type: cross Abstract: When a language model fails to answer a query on the first attempt, an agentic system retries, consuming additional tokens each time. This retry overhead creates a gap between what a model's per-token price implies and what a full workflow actually costs. We call this gap token inflation and define it as the ratio of true workflow cost to single-call cost. Systems like FrugalGPT route based on the latter, which can underestimate real cost by more than 2imes on difficult tasks. We address this with InflationAgent, a four-stage router that (1) measures token inflation systematically across model tiers and task types, finding inflation as high as 4.25imes for a 7B model on multi-hop question answering; (2) introduces CoT Branching Entropy (CBE), a pre-execution difficulty signal computed entirely from local inference, which predicts high inflation with AUROC 0.887; and (3) selects models by maximizing a Semantic Exchange Rate (SER) that divides expected accuracy by predicted true cost, with a fresh-escalation policy that discards failed chains before routing to a stronger model. On GSM8K under a fixed budget, InflationAgent achieves 94.7% accuracy versus 91.0% for FrugalGPT while using 31% fewer tokens, and we show that forwarding a failed reasoning chain to GPT-4o reduces its accuracy by up to 34.8 percentage points, validating the fresh-escalation design.

Related

• Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry
• FutureBridge: Token Selection Beyond Local Preference in Collaborative Decoding
• Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

Source: arXiv cs.AI | 2026-08-17