DecisionRobust: typed decision models still fall to prompt injection
DecisionRobust measures how far typed decision models bend under text edits, image changes and injected instructions.
· The Pennsylvania State University · Research by Yifan Lan, Zhongye Liu, Yurui Chang, Hanyu Wang, Yongkang Du, The Pennsylvania State University
DecisionRobust is a Penn State benchmark that attacks JEV-style decision models with text edits, image perturbations and injected instructions. Target attack success ranges from 18.3% to 82.5% across six text decision models.
01What happened
Researchers at The Pennsylvania State University published DecisionRobust on arXiv on October 13, 2026. The benchmark targets decision models, which return typed choices, judgments and scores instead of free text, and which teams increasingly wire into automated decisions. Data and evaluation tools are published on the project site.
02How it works
Each decision task is paired with four attack families: meaning-preserving text edits, common and optimized image perturbations, and prompt injection placed either in the text or inside the image. Attacks are built against six text and five vision selection models, then replayed against held-out models they were never tuned on. A label-consistency check discards cases where the model was already wrong before the attack, separating genuine flips from noise.
03Who is affected
Teams routing decisions through typed decision models are in scope, starting with the six text models and five vision models the paper tests. The authors report that attacks built on those models also transfer to held-out models, including Perplexity Decider v1.1-27B and Strands Decider-2B, so absence from the benchmark is not evidence of safety.
04What to do
Add a label-consistency check before acting on a decision model's output: the authors report it substantially lowers measured attack success in both text and vision. Measure injection resistance separately from perturbation resistance, because the paper finds that one does not predict the other.
Measured results
Text prompt injection: target attack success rate
Higher is worse for defenders. Construction pool, either injection location.
| Result | Score |
|---|---|
| Jev | 49.6% |
| KEV | 18.3% |
| AutoTrust | 60.7% |
| OpenJEV | 82.5% |
| Bosun | 41.1% |
| Liquid D1 | 23.2% |
Scale 0 to 100% · Source: arxiv.org, as published
Questions people ask
- Which decision models does DecisionRobust test?
- DecisionRobust evaluates 30 text configurations and 17 vision configurations. Six text models, Jev, KEV, AutoTrust, OpenJEV, Bosun and Liquid D1, were used to build the text attacks, alongside five vision models. The remaining 24 text and 12 vision models were held out, letting the authors check whether attacks transfer to models they never targeted.
- Does resisting text perturbation mean a decision model resists prompt injection?
- No, according to the DecisionRobust authors, who write that resistance to perturbations does not necessarily imply resistance to prompt injection. Their example is Perplexity Decider v1.1-27B, which shows a 3.2% perturbation attack success rate but 45.7% target success under instruction injection. Security teams should measure both attack families separately rather than treat one score as a proxy.
- Can I run the DecisionRobust benchmark myself?
- Yes. The DecisionRobust project site publishes the dataset on Hugging Face and the evaluation tools on GitHub, plus a downloadable CSV of leaderboard results. The curated benchmark holds 1,058 examples across five suites: 141 text adversarial, 374 text injection, 200 image perturbation, 223 image overlay and 120 image instruction cases.
Sources 4 links, all fetched
- 1arxiv.orgarXiv Paper
- 2yifan-lan.github.ioBenchmark Site
- 3arxiv.orgFull Paper (HTML)
- 4huggingface.coHugging Face Paper