CWE-bench v2: coding agents fix at most 68% of real security bugs

01What happened

Collinear AI published CWE-bench v2 on November 3, 2026, releasing 100 defensive security tasks that were previously held out. Each task hands an agent a checkout of a real open-source repository and a short instruction to fix a security weakness without breaking existing functionality. The authors report results for 11 models, with Gemini 4 Argon leading at 67.8% pass@1 and Claude Opus 5.5 close behind at 67.3%.

02How it works

Scoring runs as code rather than as a model judgment, so a run is deterministic and anyone can reproduce it. Each task ships as a Dockerized Harbor task. The authors say every task was reviewed by hand and checked four ways: that the instruction reads realistically, that the graded behavior can be inferred from it, that the starter code fails while the reference fix passes, and that verifiers reject a memorized upstream patch.

03Who is affected

Teams choosing a model to review or repair their own code are the audience, across 8 programming languages and 91 distinct CWE types. Collinear AI keeps a separate held-out set for tracking progress on unseen tasks, and access to it requires contacting the company. The public dataset is Apache 2.0 licensed and gated behind accepting its terms.

04What to do

Run the benchmark against the model your team actually uses before trusting it with security fixes: Harbor executes the suite straight from the Hugging Face dataset repository. Read the cost column next to the score, since DeepSeek V4.1 Flash reaches 57% at $0.04 per rollout against $3.92 for the leader.

Measured results

Sources 3 links, all fetched

1. 1huggingface.coCWE-bench v2 Write-up
2. 2huggingface.coCWE-bench Dataset
3. 3huggingface.coFull Leaderboard