Snyk VulnBench 2.0: SAST and AI agents each miss what the other finds
Snyk VulnBench 2.0 measures where deterministic scanners and AI security agents disagree, and finds the two are strongest together.
· Snyk · Research by Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark
Snyk Labs reports that Snyk Code found 82.4% of 108 attacker-reachable vulnerabilities and Claude Code's security review found 79.3%, and that the two together reached 101 of 108, the best result of any configuration tested.
01What happened
Snyk Labs published VulnBench 2.0 on arXiv on October 28, 2026, a benchmark of 108 hand-curated, attacker-reachable vulnerabilities across 20 open-source web apps in Go, JavaScript and TypeScript, Python and Java. The authors ran nine productized configurations under one blind protocol, among them Snyk Code, Claude Code's security review, Codex Security and the DeepSec scanner.
02How it works
A finding counts only when the tool locates both the attacker-controlled source and the dangerous sink within two lines, which is a stricter test than naming the right file. Sixty-two of the 108 vulnerabilities span more than one file. The authors also ran Claude Code without its built-in review policy, isolating how much the harness, rather than the model, decides what gets reported.
03Who is affected
Teams that rely on a single scanner for pull request security review are the audience. The authors report that Snyk Code and Claude Code disagreed on 28 vulnerabilities, split 16 to 12, which they say is too few to separate the two tools on accuracy alone. Codex Security running GPT-5.6 Sol located 80 vulnerabilities at file level but only 34 at line level, and DeepSec's model-free file selection kept a relevant file for only 56 of the 108 flows.
04What to do
Run a deterministic scanner and an AI reviewer together rather than choosing between them, since the authors reach their ceiling of 101 of 108 only in combination. Check what your agent's review policy excludes: without Claude Code's built-in policy the same model found 16 of 20 vulnerabilities in the excluded classes instead of 9.
Measured results
Macro recall, Snyk VulnBench 2.0
Higher is better for defenders
| Result | Score |
|---|---|
| Snyk Code | 82.4% |
| Claude Code, Opus 5 xhigh | 79.3% |
Scale 0 to 100% · Source: arxiv.org, as published
Questions people ask
- Did Claude Code beat Snyk Code on VulnBench 2.0?
- No. Snyk Code reached 82.4% macro recall, 89 of 108, and Claude Code running Opus 5 at xhigh effort reached 79.3%, 85 of 108. The authors state the two disagreed on only 28 vulnerabilities, split 16 to 12, which is too small a margin to separate them. They present the pair as complementary rather than ranked.
- How much does Claude Code's security review policy suppress?
- In a supplementary run across 19 of the 20 apps, removing Claude Code's built-in review policy raised results from 9 to 16 of the 20 vulnerabilities in the classes that policy excludes or restricts. Snyk Labs frames this as a harness effect rather than a model limit, meaning the review configuration, not the model's ability, decided what went unreported.
- Can I reproduce the VulnBench 2.0 results myself?
- Yes. Snyk Labs published the benchmark harness, fixtures, reference sets and run data at github.com/snyk-labs/snyk-vulnbench under Apache-2.0. The repository README warns that the fixture projects contain deliberately vulnerable code and should not be deployed. The harness is written in TypeScript and built on the Claude Agent SDK.
Sources 2 links, all fetched
- 1arxiv.orgarXiv Paper
- 2github.comBenchmark Repo