# Last Secure Code Benchmark: agents pass the tests, miss the security

> Last Secure Code Benchmark measures whether the code a coding agent writes to a plain functional spec is also secure.

- **Severity:** Info: Research, tools, evals or lab claims with no direct exposure. Context, not an alert.
- **Category:** Research, Benchmark
- **Published by:** arXiv
- **Disclosed:** 2026-10-10
- **Affects:** Claude Fable 5, GPT-5.6-Sol, Qwen3.8-Max, GLM-5.3, DeepSeek-V4-Pro, Kimi-K3, Claude Code, Codex
- **Research by:** Tianyi Wu, Chao Ge, Peng Chen, Fengji Zhang, Yongxing Wu, Qwen Team (Alibaba Group), National University of Singapore
- **Primary source:** https://arxiv.org/abs/2610.14766
- **Page:** https://theinjection.dev/items/arxiv-last-secure-code-benchmark/

Last Secure Code Benchmark asks six frontier coding agents to rebuild code that once held a known flaw, never mentioning security. The authors report up to 91.3% of tasks work and at most 25.1% are both working and secure.

## What happened

Fifteen authors from the Qwen Team, Alibaba Cloud and the National University of Singapore posted Last Secure Code Benchmark to arXiv on October 10, 2026. The benchmark builds 450 tasks from 150 scenarios drawn from real vulnerabilities, across seven languages and 24 CWE classes. The team released the code, the task dataset and 2,700 agent run traces under Apache 2.0.

## How it works

Each task removes code that once contained a known vulnerability and asks the agent to rebuild it from a specification that mentions only functionality, never security. Hidden functional tests then check that the result works, and a second suite replays a proof-of-concept attack against it. A task scores as functional and secure only when it passes both suites, and a scenario counts only when the model passes it at all three granularities: function, file and repository.

## Who is affected

Teams shipping code written by the six agents measured here are the audience: Claude Fable 5 run through Claude Code, GPT-5.6-Sol through Codex, plus Qwen3.8-Max, GLM-5.3, DeepSeek-V4-Pro and Kimi-K3. Coverage spans Java, C, C++, JavaScript, Python, PHP and Go. The scores come from the authors' own benchmark, not from a reported incident.

## What to do

Do not read a passing test suite as evidence that generated code is secure, because the authors report most functional output still carries the vulnerability it replaced. Asking for secure code helps without solving it: appending one sentence that requires secure, vulnerability-free code moved the functional-and-secure rate by 15.6 to 21.5 points for three of four retested models, and by 1.1 points for DeepSeek-V4-Pro.

## Functional and secure, 450 tasks

Share of tasks passing both the functional and the security suite, higher is better

| Result | Score |
|---|---|
| Claude Fable 5 | 25.1% |
| GPT-5.6-Sol | 24.7% |
| Qwen3.8-Max | 22% |
| GLM-5.3 | 21.3% |
| DeepSeek-V4-Pro | 18% |
| Kimi-K3 | 16.4% |

Source: https://arxiv.org/html/2610.14766v1

## Functional pass rate, 450 tasks

Share of tasks whose code works, with security ignored

| Result | Score |
|---|---|
| Claude Fable 5 | 91.1% |
| GPT-5.6-Sol | 89.1% |
| Qwen3.8-Max | 89.3% |
| GLM-5.3 | 87.1% |
| DeepSeek-V4-Pro | 73.6% |
| Kimi-K3 | 91.3% |

Source: https://arxiv.org/html/2610.14766v1

## Quick facts

- **Benchmark:** 450 tasks from 150 scenarios, 24 CWE classes
- **Languages:** Java, C, C++, JavaScript, Python, PHP, Go
- **Scenario level:** Claude Fable 5 leads at 12.0% of scenarios passed at all three granularities
- **Code:** Apache 2.0, with the Harbor dataset and 2,700 agent run traces
- **Affiliations:** Qwen Team and Alibaba Cloud, with the National University of Singapore
- **arXiv v1:** October 10, 2026

## Sources

1. [arXiv Abstract](https://arxiv.org/abs/2610.14766)
2. [Full Text HTML](https://arxiv.org/html/2610.14766v1)
3. [Benchmark Code](https://github.com/alibaba/last-secure-code-benchmark)