# CWE-bench v2: coding agents fix at most 68% of real security bugs

> CWE-bench v2 is Collinear AI's open benchmark for whether a coding agent can find and fix a real security weakness in a real repository.

- **Severity:** Info: Research, tools, evals or lab claims with no direct exposure. Context, not an alert.
- **Category:** Benchmark, Research
- **Published by:** Collinear AI
- **Disclosed:** 2026-11-03
- **Affects:** Claude Opus 5.5, Gemini 4 Argon, DeepSeek V4.1 Flash
- **Research by:** Gonzalo Gonzalez, Ajay Jagannath, Nazneen Rajani, Nathan Habib
- **Primary source:** https://huggingface.co/blog/cwe-bench
- **Page:** https://theinjection.dev/items/collinear-cwe-bench-v2/

Collinear AI released CWE-bench v2 on November 3, 2026: 100 open tasks across 91 CWE types where an agent must find and fix a real security weakness without breaking the code. The best model passes 67.8% on the first attempt.

## What happened

Collinear AI published CWE-bench v2 on November 3, 2026, releasing 100 defensive security tasks that were previously held out. Each task hands an agent a checkout of a real open-source repository and a short instruction to fix a security weakness without breaking existing functionality. The authors report results for 11 models, with Gemini 4 Argon leading at 67.8% pass@1 and Claude Opus 5.5 close behind at 67.3%.

## How it works

Scoring runs as code rather than as a model judgment, so a run is deterministic and anyone can reproduce it. Each task ships as a Dockerized Harbor task. The authors say every task was reviewed by hand and checked four ways: that the instruction reads realistically, that the graded behavior can be inferred from it, that the starter code fails while the reference fix passes, and that verifiers reject a memorized upstream patch.

## Who is affected

Teams choosing a model to review or repair their own code are the audience, across 8 programming languages and 91 distinct CWE types. Collinear AI keeps a separate held-out set for tracking progress on unseen tasks, and access to it requires contacting the company. The public dataset is Apache 2.0 licensed and gated behind accepting its terms.

## What to do

Run the benchmark against the model your team actually uses before trusting it with security fixes: Harbor executes the suite straight from the Hugging Face dataset repository. Read the cost column next to the score, since DeepSeek V4.1 Flash reaches 57% at $0.04 per rollout against $3.92 for the leader.

## Task resolve rate, CWE-bench v2 open-weight leaderboard

Higher is better: share of the 100 tasks where the agent fixed the flaw

| Result | Score |
|---|---|
| DeepSeek-V4.1-Flash-1017 | 57% |
| tencent/Hy4-preview | 52% |
| Mistral-Large-4-1T-A52B | 46.3% |
| GLM-5.3 | 44% |

Source: https://huggingface.co/datasets/collinear-ai/cwe-bench

## Quick facts

- **Tasks:** 100 public, across 91 CWE types and 8 languages
- **Leader:** Gemini 4 Argon, 67.8% pass@1 and 81.0% pass@4
- **Runner-up:** Claude Opus 5.5, 67.3% pass@1 and 78.0% pass@4
- **Unsolved:** 5 of 100 tasks solved by no model; the leader fails 19% on every attempt
- **Cost per rollout:** $0.04 DeepSeek V4.1 Flash, $1.83 Claude Opus 5.5, $3.92 Gemini 4 Argon
- **Scoring:** Programmatic and deterministic, run as Dockerized Harbor tasks
- **License:** Apache 2.0, gated dataset

## Sources

1. [CWE-bench v2 Write-up](https://huggingface.co/blog/cwe-bench)
2. [CWE-bench Dataset](https://huggingface.co/datasets/collinear-ai/cwe-bench)
3. [Full Leaderboard](https://huggingface.co/spaces/collinear-ai/cwe-bench-leaderboard)