Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems

arXiv:2606.23403v2 Announce Type: replace Abstract: As agentic LLM systems move from prototypes to deployment, evaluating them has become more important and more difficult. Evaluation goals are often left implicit. Without a clear account of what a system is expected to do, how it can fail, and which failures matter, metric choices become difficult to justify, interpret, or validate. We present Litmus, a zero-label system that designs evaluation and monitoring metrics for AI pipelines by eliciting evaluation intent from source code and targeted interrogation. Litmus first identifies what must be measured and why, then converts those answers into constraints for a justified, per-stage metric portfolio. We evaluate Litmus on three real, code-defined pipelines against AutoMetrics and three DynamicRubric baselines. Litmus achieves broader concern coverage on account grouping at matched size, spans more stages, produces a near-zero-redundancy portfolio, and, under a zero-label design-time constraint, ranks first in validity against per-row quality labels on all three pipelines, decisively on scientific QA (rho{=}0.72 vs. 0.47 for the strongest baseline on the stitched single-vector score), with overlapping intervals on the other two. Results support a shift from automatic metric implementation to automatic metric specification: before asking which metric to compute, evaluation systems should ask what must be measured and why.

Related

• Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System
• Humanize: Judgement Engineering for Agentic Coding
• Multi-Dimensional Behavioral Evaluation of Agentic Stock Prediction Systems Using Large Language Model Judges with Closed-Loop Reinforcement Learning Feedback
• Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation

Source: arXiv cs.AI | 2026-10-15