# Jev as a judge: a counterfeit policy cuts injection detection to 3.88%

> Jev returns the verdict an attacker wants when the text it judges carries a counterfeit policy that redefines the criteria.

- **Severity:** Medium: Real and reproducible, but needs unusual setup, user help or a narrow audience. Plan a fix.
- **Category:** Research, Attack
- **Published by:** Datadog Security Labs
- **Disclosed:** 2026-10-15
- **Affects:** TypeSafe Jev, LangChain
- **Research by:** Nick Frichette (Datadog), Feras Al-Kassar (Datadog)
- **Primary source:** https://securitylabs.datadoghq.com/articles/jev-as-a-judge-how-counterfeit-policy-steers-jev/
- **Page:** https://theinjection.dev/items/datadog-jev-counterfeit-policy/

Counterfeit policy is a prompt-injection technique that hands a security classifier its own answer key. Datadog Security Labs reports it cut TypeSafe Jev's injection detection from 82.90% recall to 3.88% across 12,295 samples.

## What happened

Datadog Security Labs published this research on October 15, 2026. Nick Frichette and Feras Al-Kassar tested Jev, the first public System One decision model from TypeSafe AI, both as a prompt-injection classifier and as a gate on tool calls. They measured it on four public datasets, then repeated every test with a counterfeit policy prepended to the input.

## How it works

A counterfeit policy is text an attacker places inside the content the classifier is about to read. The text supplies the answers the attacker wants, quotes the attack itself as a benign exception, and redefines each attack category as something unrelated, so that credential access applies only to a door key. Jev then answers the criteria the input carries rather than the criteria the application intended, and the verdict flips while the attack text stays in place.

## Who is affected

Applications that ask Jev to judge untrusted input and then act on the answer are exposed, including LangChain's experimental AutoModeMiddleware, which refuses a tool call when the risk probability reaches 0.5. Any field an attacker can edit and the classifier later reads can carry the policy. Datadog used a user-editable profile description in its test application.

## What to do

Enforce permissions in the tool or service layer rather than in a model's verdict, as Datadog advises. Trace which user-editable fields reach a classifier, and treat tool arguments, tool descriptions and stored state as untrusted data. Test any classifier against adversarial content before it gates real actions.

## Prompt injection recall, 12,295 samples

Recall over 4,643 attack samples. Higher is better for defenders.

| Result | Score |
|---|---|
| Noul question, raw input | 82.9% |
| Choice question, raw input | 85.46% |
| **Noul question, counterfeit policy** (this story) | 3.88% |
| Choice question, counterfeit policy | 9.69% |

Source: https://securitylabs.datadoghq.com/articles/jev-as-a-judge-how-counterfeit-policy-steers-jev/

## Quick facts

- **Affected:** TypeSafe Jev used as a classifier or a tool-risk gate
- **Attack type:** Counterfeit policy prepended to untrusted content
- **Found by:** Nick Frichette and Feras Al-Kassar, Datadog Security Labs
- **Disclosed:** October 15, 2026
- **Test set:** 12,295 samples from four public datasets, 4,643 of them attacks
- **Gate bypass:** Mean risk score fell from 0.893 to 0.290 over 40 runs, under LangChain's 0.5 block
- **Vendor response:** None published in the post

## Questions

### Am I affected if my application uses Jev to screen untrusted input?

Applications that pass untrusted text to Jev and act on its verdict are exposed, because Datadog Security Labs showed the verdict can be written by the same text being judged. Exposure depends on whether an attacker controls any field the classifier reads. Datadog's test used a user-editable profile description, a pattern common in agent applications.

### Is there a fix for the Jev counterfeit policy bypass?

No patch exists, because Datadog Security Labs describes a limit of model-based classification rather than a software bug with a version number. TypeSafe's documentation for jev-1.13 states that state is data and that the model does not treat it as hostile by default. Datadog recommends enforcing authorization deterministically in the tool or service layer instead.

### How does counterfeit policy differ from ordinary prompt injection against Jev?

Ordinary injection did not work in Datadog's tests. Messages impersonating a system or assistant turn and claiming editor rights produced a mean risk score of 0.905 over 40 runs, slightly above the 0.893 baseline. Counterfeit policy instead rewrites the criteria the model is asked to apply, and that pushed the mean score down to 0.290.

## Sources

1. [Datadog Research](https://securitylabs.datadoghq.com/articles/jev-as-a-judge-how-counterfeit-policy-steers-jev/)
2. [Tool Risk Gating Docs](https://docs.langchain.com/oss/python/integrations/providers/typesafe)
3. [TypeSafe Jev 1.13 Docs](https://docs.typesafe.ai/model-jaggedness/jev-1.13)