Cost-Aware Model Orchestration for LLM-based Systems

arXiv:2512.01099v2 Announce Type: replace Abstract: As modern artificial intelligence (AI) systems become more advanced and capable, they can leverage a wide range of tools and models to perform complex tasks. The task of orchestrating these models is increasingly performed by Large Language Models (LLMs) that rely on qualitative descriptions of models for decision-making. However, the descriptions provided to existing LLM-based orchestrators frequently do not reflect true model capabilities and performance characteristics, leading to suboptimal model selection, reduced task accuracy, and increased cost. In this paper, we conduct an empirical analysis of LLM-based orchestration limitations and propose a cost-aware model selection method that accounts for performance-cost trade-offs by incorporating quantitative model performance characteristics within decision-making. Initial experimental results demonstrate that our proposed method increases accuracy by 0.90%-11.92% across various evaluated tasks, achieves up to a 54% energy efficiency improvement, and reduces orchestrator model selection latency from 4.51 s to 7.2 ms.

Related

• LLMbench: A Comparative Close Reading Workbench for Large Language Models
• TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning
• LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification
• Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models

Source: arXiv cs.AI | 2026-04-20