HarnessSecurity-Bench: auto-approve lifts attack success to 95.6%
HarnessSecurity-Bench measures whether the security mechanisms in six coding agent harnesses actually stop attacks.
· arXiv · Research by Zhengyang Zhu, Liming Huang, Runmin Ji, Mingxi Ye, Zihan Zhou, Hanyang Guo, Jingwen Wu, Yuhan Ye, Yuming Feng, Hong-Ning Dai, Zibin Zheng
HarnessSecurity-Bench tests nine security mechanisms across six coding agent harnesses and reports that enabling auto-approve raises attack success from 29.2% to 95.6%, while command denylisting cut attack effects with a utility gain.
01What happened
HarnessSecurity-Bench, posted to arXiv on October 6, 2026, tests the security mechanisms of six coding agent harnesses: Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code and GitHub Copilot. The authors build a ten-mechanism taxonomy, assess 400 harness-mechanism cells using researcher and LLM-judge ratings, and run 23 tasks across five attack surfaces. The reported scale is 2,500 trials, 81,155 tool calls and over 2.2 billion tokens, with GLM-5.2 as the baseline model.
02How it works
The benchmark evaluates nine mechanisms across the six harnesses and measures both attack success and task utility. The paper reports that enabling auto-approve raises attack success from 29.2% to 95.6%. Network isolation and read-only mode reduced attack effects but cost substantial utility, command allowlisting reduced attack effects with a small utility loss, and command denylisting reduced attack effects while giving a utility gain.
03Who is affected
Teams running any of the six harnesses, and anyone treating a harness setting as a security control. The authors report that about half of confirmed mechanism implementations are opt-in, that closed-source harnesses show substantial evidence gaps, and that allowed tools or commands can still leave unauthorized operations reachable through alternative execution paths.
04What to do
Leave auto-approve off, since the paper measures attack success rising from 29.2% to 95.6% when it is on. Check which mechanisms your harness enables by default rather than assuming they are active, and note the paper's finding that command denylisting and allowlisting cost less utility than network isolation or read-only mode.
Sources 2 links, all fetched
- 1arxiv.orgarXiv Abstract
- 2tsingpig.github.ioProject Website