HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation

arXiv:2610.19184v1 Announce Type: new Abstract: Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce extsc{HERO} (extbf{H}arnessing extbf{E}vidence for extbf{R}efinement and extbf{O}ptimization), which combines offline and test-time harness optimization. At the first stage, method{} learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, method{} achieves the best offline performance, improving pass@1 by (7.6) points over the initial prompt, while test-time refinement adds another (2.0) points and improves execution efficiency.

Related

• Self-Reflective Generation at Test Time
• LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimization
• REVA: Reusable Evidence View Aggregation for Context-Efficient RAG Serving
• DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode

Source: arXiv cs.CL | 2026-10-15