Jev as a judge: a counterfeit policy cuts injection detection to 3.88%
Jev returns the verdict an attacker wants when the text it judges carries a counterfeit policy that redefines the criteria.
· Datadog Security Labs · Research by Nick Frichette (Datadog), Feras Al-Kassar (Datadog)
Counterfeit policy is a prompt-injection technique that hands a security classifier its own answer key. Datadog Security Labs reports it cut TypeSafe Jev's injection detection from 82.90% recall to 3.88% across 12,295 samples.
- securitylabs.datadoghq.comDatadog Research
- docs.langchain.comTool Risk Gating Docs
- docs.typesafe.aiTypeSafe Jev 1.13 Docs
01What happened
Datadog Security Labs published this research on October 15, 2026. Nick Frichette and Feras Al-Kassar tested Jev, the first public System One decision model from TypeSafe AI, both as a prompt-injection classifier and as a gate on tool calls. They measured it on four public datasets, then repeated every test with a counterfeit policy prepended to the input.
02How it works
A counterfeit policy is text an attacker places inside the content the classifier is about to read. The text supplies the answers the attacker wants, quotes the attack itself as a benign exception, and redefines each attack category as something unrelated, so that credential access applies only to a door key. Jev then answers the criteria the input carries rather than the criteria the application intended, and the verdict flips while the attack text stays in place.
03Who is affected
Applications that ask Jev to judge untrusted input and then act on the answer are exposed, including LangChain's experimental AutoModeMiddleware, which refuses a tool call when the risk probability reaches 0.5. Any field an attacker can edit and the classifier later reads can carry the policy. Datadog used a user-editable profile description in its test application.
04What to do
Enforce permissions in the tool or service layer rather than in a model's verdict, as Datadog advises. Trace which user-editable fields reach a classifier, and treat tool arguments, tool descriptions and stored state as untrusted data. Test any classifier against adversarial content before it gates real actions.
Measured results
Prompt injection recall, 12,295 samples
Recall over 4,643 attack samples. Higher is better for defenders.
| Result | Score |
|---|---|
| Noul question, raw input | 82.9% |
| Choice question, raw input | 85.5% |
| Noul question, counterfeit policythis story | 3.9% |
| Choice question, counterfeit policy | 9.7% |
Scale 0 to 100% · Source: securitylabs.datadoghq.com, as published
Questions people ask
- Am I affected if my application uses Jev to screen untrusted input?
- Applications that pass untrusted text to Jev and act on its verdict are exposed, because Datadog Security Labs showed the verdict can be written by the same text being judged. Exposure depends on whether an attacker controls any field the classifier reads. Datadog's test used a user-editable profile description, a pattern common in agent applications.
- Is there a fix for the Jev counterfeit policy bypass?
- No patch exists, because Datadog Security Labs describes a limit of model-based classification rather than a software bug with a version number. TypeSafe's documentation for jev-1.13 states that state is data and that the model does not treat it as hostile by default. Datadog recommends enforcing authorization deterministically in the tool or service layer instead.
- How does counterfeit policy differ from ordinary prompt injection against Jev?
- Ordinary injection did not work in Datadog's tests. Messages impersonating a system or assistant turn and claiming editor rights produced a mean risk score of 0.905 over 40 runs, slightly above the 0.893 baseline. Counterfeit policy instead rewrites the criteria the model is asked to apply, and that pushed the mean score down to 0.290.
Sources 3 links, all fetched
- 1securitylabs.datadoghq.comDatadog Research
- 2docs.langchain.comTool Risk Gating Docs
- 3docs.typesafe.aiTypeSafe Jev 1.13 Docs