Lightweight, Rubric-Guided Trajectory Evaluation for Production AI Agents

arXiv:2610.03315v1 Announce Type: new Abstract: Trajectory evaluation is essential for improving the reliability of LLM-based agents, but production use makes it expensive to run repeatedly. Modern agents generate long traces containing tool calls, observations, retries, and external outputs, while not all raw tokens are equally useful for diagnosis. We present extit{LiteTrajEval}, a lightweight architecture for budget-bounded trajectory evaluation. LiteTrajEval derives compact domain-specific rule profiles offline, then preprocesses each trajectory online, marks heuristic failure signals, serializes it under a fixed global budget, and invokes a single rubric-guided LLM judge to produce structured diagnostic reports. Evaluated on public Magentic-One-style and au-bench-style trajectory datasets, LiteTrajEval improves failure-localization alignment with human annotations by roughly 20--35 percentage points on Magentic-One and up to 23 percentage points on au-retail compared with AgentRx, while reducing cost by about 6imes and evaluation time by more than 8imes. This solution has also been deployed in our enterprise agentic platform.

Related

• trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
• Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
• Governing Execution Risk in Agentic AI Systems: A Trajectory-Guided Framework for Red Teaming
• ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents

Source: arXiv cs.AI | 2026-10-05