LLM agents patch security bugs, pass all tests, but still leave the vulnerability open [R]

Research demonstrates that LLM-based agents can generate functionally correct patches that pass all tests while still containing security vulnerabilities, challenging the assumption that test-passing patches are inherently secure. Current evaluation methods primarily focus on functional correctness rather than security risks in automated program repair. Developer test suites verify expected behavior but do not defend against adversarial inputs, meaning patches passing all tests may still leave systems vulnerable.

Source: r/MachineLearning | 2026-06-02