SecPRBench: agent code review recall ranges from 32.5% to 78.1%
SecPRBench scores security code review on the real commits that introduced flaws in nine C, C++ and Rust projects.
· arXiv · Research by Yihan Yang, Chenyuan Yang, Zijie Zhao, Anh Nguyen Hoang, Ziqi Zhang, Weidong Wang
SecPRBench measures whether LLM agents catch a security flaw in the commit that introduced it. Across eight frontier models the authors report recall from 32.5% to 78.1%, on 265 cases taken from nine C, C++ and Rust projects.
01What happened
Yihan Yang, Chenyuan Yang and six co-authors posted SecPRBench to arXiv on October 24, 2026, along with the benchmark code, a review sandbox and a results site. The paper reports numbers for eight frontier models and for two vendor security harnesses.
02How it works
Each case is an upstream commit that introduced a real vulnerability, not the later fix, which is what earlier benchmarks used. The reviewer gets the diff, the commit message and a source snapshot with no usable git history, but no description of the flaw, and must name the flaw and explain its root cause. A paired task shows the repaired change, so a reviewer that flags everything scores badly. An LLM judge matches reported findings against the known vulnerability.
03Who is affected
Teams that gate merges on an AI reviewer are the audience. The repository counts 265 cases and 530 tasks drawn from curl, FFmpeg, Firefox, FreeType, OpenSSL, Redis, SQLite, V8 and Wasmtime. The authors report that the two vendor harnesses they tested move recall in opposite directions, listed below.
04What to do
Treat an agentic reviewer as a filter, not a gate: the strongest model in the paper reached 78.1% recall, so roughly a fifth of the planted flaws still went through. The harness is installable and ships frozen results, so a team can score its own reviewer configuration before trusting it.
Sources 3 links, all fetched
- 1arxiv.orgarXiv Paper
- 2github.comSecPRBench Code
- 3secprbench.github.ioResults Site