Last Secure Code Benchmark: agents pass the tests, miss the security
Last Secure Code Benchmark measures whether the code a coding agent writes to a plain functional spec is also secure.
· arXiv · Research by Tianyi Wu, Chao Ge, Peng Chen, Fengji Zhang, Yongxing Wu, Qwen Team (Alibaba Group), National University of Singapore
Last Secure Code Benchmark asks six frontier coding agents to rebuild code that once held a known flaw, never mentioning security. The authors report up to 91.3% of tasks work and at most 25.1% are both working and secure.
01What happened
Fifteen authors from the Qwen Team, Alibaba Cloud and the National University of Singapore posted Last Secure Code Benchmark to arXiv on October 10, 2026. The benchmark builds 450 tasks from 150 scenarios drawn from real vulnerabilities, across seven languages and 24 CWE classes. The team released the code, the task dataset and 2,700 agent run traces under Apache 2.0.
02How it works
Each task removes code that once contained a known vulnerability and asks the agent to rebuild it from a specification that mentions only functionality, never security. Hidden functional tests then check that the result works, and a second suite replays a proof-of-concept attack against it. A task scores as functional and secure only when it passes both suites, and a scenario counts only when the model passes it at all three granularities: function, file and repository.
03Who is affected
Teams shipping code written by the six agents measured here are the audience: Claude Fable 5 run through Claude Code, GPT-5.6-Sol through Codex, plus Qwen3.8-Max, GLM-5.3, DeepSeek-V4-Pro and Kimi-K3. Coverage spans Java, C, C++, JavaScript, Python, PHP and Go. The scores come from the authors' own benchmark, not from a reported incident.
04What to do
Do not read a passing test suite as evidence that generated code is secure, because the authors report most functional output still carries the vulnerability it replaced. Asking for secure code helps without solving it: appending one sentence that requires secure, vulnerability-free code moved the functional-and-secure rate by 15.6 to 21.5 points for three of four retested models, and by 1.1 points for DeepSeek-V4-Pro.
Measured results
Functional and secure, 450 tasks
Share of tasks passing both the functional and the security suite, higher is better
| Result | Score |
|---|---|
| Claude Fable 5 | 25.1% |
| GPT-5.6-Sol | 24.7% |
| Qwen3.8-Max | 22% |
| GLM-5.3 | 21.3% |
| DeepSeek-V4-Pro | 18% |
| Kimi-K3 | 16.4% |
Scale 0 to 100% · Source: arxiv.org, as published
Functional pass rate, 450 tasks
Share of tasks whose code works, with security ignored
| Result | Score |
|---|---|
| Claude Fable 5 | 91.1% |
| GPT-5.6-Sol | 89.1% |
| Qwen3.8-Max | 89.3% |
| GLM-5.3 | 87.1% |
| DeepSeek-V4-Pro | 73.6% |
| Kimi-K3 | 91.3% |
Scale 0 to 100% · Source: arxiv.org, as published
Sources 3 links, all fetched
- 1arxiv.orgarXiv Abstract
- 2arxiv.orgFull Text HTML
- 3github.comBenchmark Code