Selection-Aware Stress Testing for Interactive Agents

arXiv:2608.30916v1 Announce Type: new Abstract: Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selected from the same data. We introduce Selection-Aware Semantic Stress Testing (SASST{}), which learns a task reweighting from pre-execution features on discovery tasks and evaluates the same paired comparison on separate confirmation tasks. The protocol checks support and stability, uses joint bounds for all planned claims, and can return no claim. We prove conditional asymptotic validity under stated cluster assumptions. A forty-cluster audit finds Gaussian undercoverage and conservative Bonferroni t bounds. In one 480-episode au-bench study, a 3.75 point discovery gain vanished on confirmation. A second-model study likewise confirmed neither a workflow benefit nor a stable stress rule.

Related

• Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions
• Prescribe-then-Select: Adaptive Policy Selection for Contextual Stochastic Optimization
• When Do Local Score Models Extrapolate Across Size? A Diagnostic Theory and Benchmark
• Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions

Source: arXiv cs.LG | 2026-09-01