How SWE-Serve Exposes the Gap Between Local Tests and Live Serving

SWE‑Serve evaluates AI coding agents on 53 inference‑engineering tasks derived from SGLang pull requests across six functional domains. Among 19 tasks that launch a live server, patches that pass local tests succeed only 45.9 % of the time when full serving verification is applied—about 23 percentage points lower than the 69.4 % success rate without live‑serving checks—and multi‑domain tasks show a 21.3 percentage‑point drop compared with single‑domain tasks. Eleven models were benchmarked, achieving pass@1 scores between 34.6 % and 75.5 %, with Claude Opus 5 and GPT‑5.6 Sol topping the list while exhibiting varied cost and wall‑time characteristics.

Source: NVIDIA Developer | 2026-09-23