AutoRedClaw: auto-generated attacks beat hand-built agent benchmarks
AutoRedClaw writes both the attack and the environment it needs, so a red team does not have to hand-build every agent test case.
· arXiv · Research by Hao Li, Tongyan Hu, Weidi Luo, Shanghao Shi, Xiaogeng Liu, SaFoLab
AutoRedClaw generates agent attacks and the environments to run them, validated by execution. The authors report most of seven harnesses exceeded a 50% attack success rate, and their 330-case benchmark draws higher rates than AgentDojo.
01What happened
AutoRedClaw went up on arXiv on October 19, 2026, from Hao Li and colleagues at SaFoLab and collaborating groups. The paper targets what the authors call claw-style agents: persistent assistants such as OpenClaw that keep config files, long-term memory and reusable skills between sessions. The authors release the framework and a benchmark built from its output, RedClaw-Bench, as code.
02How it works
The framework generates end-to-end attacks together with the environment each one needs, then confirms success by running them rather than by asking a model to judge. Generation is organized along four axes, risk category, attack strategy, attack stage and attack surface, and an experience-guided search steers toward combinations it has not covered yet. Because the poisoned state lives in files, memory and skills the agent reloads, one successful write keeps firing on later tasks and sessions.
03Who is affected
The repository names seven harnesses the framework drives, listed below. Its defaults use DeepSeek-V4-Pro-0423 to generate and probe and gpt-5-mini to evaluate, and a figure in the paper names GPT-5.4, Claude-Sonnet-4.5 and Gemini-3.1-Pro among the models tested. The authors say existing evaluations rely on hand-built cases that go stale as agents change.
04What to do
Treat an agent's saved config, memory and skill files as untrusted input that is re-read at the start of every session, not as settings a user wrote once. Teams that currently measure agent robustness with hand-built suites should retest against the released benchmark, which the authors report draws higher attack success rates than the suites it replaces.
Sources 2 links, all fetched
- 1arxiv.orgarXiv Paper
- 2github.comCode And Benchmark