# AutoRedClaw: auto-generated attacks beat hand-built agent benchmarks

> AutoRedClaw writes both the attack and the environment it needs, so a red team does not have to hand-build every agent test case.

- **Severity:** Info: Research, tools, evals or lab claims with no direct exposure. Context, not an alert.
- **Category:** Research, Benchmark
- **Published by:** arXiv
- **Disclosed:** 2026-10-19
- **Affects:** OpenClaw, Claude Code, OpenAI Codex, nanobot, nemoclaw, seclaw, zeroclaw
- **Research by:** Hao Li, Tongyan Hu, Weidi Luo, Shanghao Shi, Xiaogeng Liu, [SaFoLab](https://github.com/SaFo-Lab)
- **Primary source:** https://arxiv.org/abs/2610.27244
- **Page:** https://theinjection.dev/items/arxiv-autoredclaw/

AutoRedClaw generates agent attacks and the environments to run them, validated by execution. The authors report most of seven harnesses exceeded a 50% attack success rate, and their 330-case benchmark draws higher rates than AgentDojo.

## What happened

AutoRedClaw went up on arXiv on October 19, 2026, from Hao Li and colleagues at SaFoLab and collaborating groups. The paper targets what the authors call claw-style agents: persistent assistants such as OpenClaw that keep config files, long-term memory and reusable skills between sessions. The authors release the framework and a benchmark built from its output, RedClaw-Bench, as code.

## How it works

The framework generates end-to-end attacks together with the environment each one needs, then confirms success by running them rather than by asking a model to judge. Generation is organized along four axes, risk category, attack strategy, attack stage and attack surface, and an experience-guided search steers toward combinations it has not covered yet. Because the poisoned state lives in files, memory and skills the agent reloads, one successful write keeps firing on later tasks and sessions.

## Who is affected

The repository names seven harnesses the framework drives, listed below. Its defaults use DeepSeek-V4-Pro-0423 to generate and probe and gpt-5-mini to evaluate, and a figure in the paper names GPT-5.4, Claude-Sonnet-4.5 and Gemini-3.1-Pro among the models tested. The authors say existing evaluations rely on hand-built cases that go stale as agents change.

## What to do

Treat an agent's saved config, memory and skill files as untrusted input that is re-read at the start of every session, not as settings a user wrote once. Teams that currently measure agent robustness with hand-built suites should retest against the released benchmark, which the authors report draws higher attack success rates than the suites it replaces.

## Quick facts

- **Harnesses tested:** OpenClaw, nanobot, nemoclaw, seclaw, zeroclaw, Codex and Claude Code
- **Reported result:** Most agent systems above 50% attack success across seven harnesses and four frontier models
- **Generation cost:** Effective attack rate above 30%, at under $0.10 per successful sample
- **Benchmark:** RedClaw-Bench, 330 cases, reported above AgentDojo, AgentDyn and DTap
- **Attack type:** Poisoning of persistent agent config, memory and skills
- **Found by:** Hao Li and colleagues, SaFoLab
- **Published:** 2026-10-19, arXiv v1

## Sources

1. [arXiv Paper](https://arxiv.org/abs/2610.27244)
2. [Code And Benchmark](https://github.com/SaFo-Lab/AutoRedClaw)