FlipJev: short suffixes flip a commercial decision model's answers
FlipJev finds short adversarial suffixes that flip a typed decision model's confident answers, where ordinary prompt injection failed.
· Johns Hopkins University · Research by Xiaogeng Liu, Xinyan Wang (University of Wisconsin-Madison), Chaowei Xiao
FlipJev is a black-box attack that appends suffixes of 5 to 31 tokens to untrusted text and flips the answers of Jev, TypeSafe's commercial decision model. The authors report one suffix per dataset flips 43% of unseen documents.
01What happened
Xiaogeng Liu, Xinyan Wang and Chaowei Xiao posted FlipJev to arXiv on October 28, 2026, red teaming Jev, a closed commercial typed decision model from TypeSafe that returns probabilities over fixed options instead of text. The authors report that common prompt injection did nothing: across 10,240 paired tests, no strategy flipped a correct answer. Their own query-based search did, and the project page states TypeSafe received a report before publication.
02How it works
Typed decision models answer in one cheap query and expose the probabilities behind the choice, and FlipJev treats those probabilities as a fitness signal. An evolutionary search makes word-level edits to a suffix appended to the untrusted content, never to the developer's question, running 64 children from 16 elites per generation inside a 2,000-query budget. Suffixes are capped at 32 tokens, and a flip counts only when at least two of three fresh queries repeat it. No attacker language model is involved.
03Who is affected
Teams that route security-sensitive decisions through typed decision models are in scope, starting with Jev, which the project page identifies as typesafe/jev-1.13 and the first deployed model of its kind. The attack also moved agent safety verdicts: the authors report a universal suffix turned 9 of 30 held-out unsafe trajectories in the ATBench set to a safe verdict.
04What to do
Add adaptive, query-based attacks to any evaluation of a decision model before trusting its verdicts, which is the authors' own recommendation. Do not rely on an instruction in the trusted question to hold the boundary: the authors tested one and report the suffixes still worked.
Questions people ask
- Is Jev vulnerable to ordinary prompt injection?
- FlipJev's authors report no. Across 10,240 paired tests on email comprehension, naive, context-ignoring, fake-completion and combined injections, the InjecAgent instruction and a forged Jev-style JSON answer produced zero strict flips, and the forged answer moved mean confidence in the correct option by at most 0.004. The suffix search, which reads the returned probabilities, is what succeeded.
- Do FlipJev suffixes work on documents and tasks they were not built on?
- Yes, by the authors' measurements. A single suffix per dataset flipped 114 of 264 held-out documents, or 43%, while no suffix and a neutral note flipped none of them. Across 72 ordered dataset pairs the mean flip rate was 18%, and a suffix found on full arXiv papers flipped 90% to 100% of held-out documents in five other datasets.
- Has TypeSafe responded to FlipJev?
- The FlipJev project page states the authors reported their findings to TypeSafe before publication. Neither the paper nor the project page records a vendor fix, a patched version or a CVE, so no remediation is documented. The paper proposes no defense either, and notes that a boundary instruction placed in the trusted question did not weaken the suffixes.
Sources 4 links, all fetched
- 1arxiv.orgarXiv Paper
- 2flipjev.github.ioProject Page
- 3github.comCode on GitHub
- 4arxiv.orgFull Paper (HTML)