# SecPRBench: agent code review recall ranges from 32.5% to 78.1%

> SecPRBench scores security code review on the real commits that introduced flaws in nine C, C++ and Rust projects.

- **Severity:** Info: Research, tools, evals or lab claims with no direct exposure. Context, not an alert.
- **Category:** Research, Benchmark
- **Published by:** arXiv
- **Disclosed:** 2026-10-24
- **Affects:** AI coding agents, OpenAI Codex Security, Anthropic Defending Code, Grok 4.6, GPT-6 sol
- **Research by:** Yihan Yang, Chenyuan Yang, Zijie Zhao, Anh Nguyen Hoang, Ziqi Zhang, Weidong Wang
- **Primary source:** https://arxiv.org/abs/2610.33509
- **Page:** https://theinjection.dev/items/arxiv-secprbench/

SecPRBench measures whether LLM agents catch a security flaw in the commit that introduced it. Across eight frontier models the authors report recall from 32.5% to 78.1%, on 265 cases taken from nine C, C++ and Rust projects.

## What happened

Yihan Yang, Chenyuan Yang and six co-authors posted SecPRBench to arXiv on October 24, 2026, along with the benchmark code, a review sandbox and a results site. The paper reports numbers for eight frontier models and for two vendor security harnesses.

## How it works

Each case is an upstream commit that introduced a real vulnerability, not the later fix, which is what earlier benchmarks used. The reviewer gets the diff, the commit message and a source snapshot with no usable git history, but no description of the flaw, and must name the flaw and explain its root cause. A paired task shows the repaired change, so a reviewer that flags everything scores badly. An LLM judge matches reported findings against the known vulnerability.

## Who is affected

Teams that gate merges on an AI reviewer are the audience. The repository counts 265 cases and 530 tasks drawn from curl, FFmpeg, Firefox, FreeType, OpenSSL, Redis, SQLite, V8 and Wasmtime. The authors report that the two vendor harnesses they tested move recall in opposite directions, listed below.

## What to do

Treat an agentic reviewer as a filter, not a gate: the strongest model in the paper reached 78.1% recall, so roughly a fifth of the planted flaws still went through. The harness is installable and ships frozen results, so a team can score its own reviewer configuration before trusting it.

## Quick facts

- **Benchmark:** 265 cases, 530 tasks, nine projects
- **Recall range:** 32.5% to 78.1% across eight frontier models
- **Top models:** Grok 4.6 and GPT-6 sol, per the paper
- **Codex Security:** Recall 8.3 points lower, mostly by abstaining
- **Defending Code:** Recall 7.7 points higher, at higher cost
- **Near misses:** 68% of misses name the right file, wrong mechanism
- **Released:** Benchmark, review sandbox and judge

## Sources

1. [arXiv Paper](https://arxiv.org/abs/2610.33509)
2. [SecPRBench Code](https://github.com/secprbench/secprbench)
3. [Results Site](https://secprbench.github.io/)