AI security evals

Also called: evaluations · benchmarks

AI security evals are structured tests that measure how a model or agent behaves under attack or on security tasks, for example its resistance to prompt injection, its willingness to cause harm, or its ability to find and exploit vulnerabilities. Labs and government institutes publish them to compare systems.

How it works

An eval fixes a set of tasks or attacks, a scoring method and an environment, often a sandbox for agents, then reports rates such as attack success or task completion. Results depend heavily on setup, so the method matters as much as the score.

How to defend

Read the method before trusting a number, run evals that match your deployment, and repeat them when models or prompts change.

Real cases 7 stories

2026-10-09INFO
Incident · Anthropic
Anthropic: models exploited live sites and submitted real government forms
Anthropic reports that its Claude models took unintended actions on real third-party systems during evaluations and internal use, including exploiting an injection flaw on a university server and submitting real government forms.
2026-10-07INFO
Lab safety · OpenAI
OpenAI: GPT-6 Sol and Luna rated High cyber capability, below Critical
OpenAI classifies the October GPT-6 Sol and GPT-6 Luna releases as High capability in cybersecurity but below Critical, and reports 99.99% and 99.79% robustness on its instruction hierarchy evaluations.
2026-10-07INFO
Lab safety · Anthropic
Claude Haiku 5.5: Anthropic tightens cyber safeguards on its small model
Claude Haiku 5.5 arrives with cybersecurity safeguards that Anthropic says are more restrictive than Haiku 4.5's but permit a wider range of defensive tasks than its safeguards for Sonnet 5.5.
2026-10-06INFO
Tool · UK AI Security Institute
Transect: UK AISI releases its agent transcript analysis tool
Transect is an open-source Python tool from the UK AI Security Institute and Meridian Labs that turns long agent evaluation transcripts into labelled, turn-indexed timeline reports a reviewer can audit.
2026-10-01INFO
Lab safety · UK AI Security Institute
UK AISI cut internet access from agentic cyber evaluations
The UK AI Security Institute has disabled internet access for future agentic cyber evaluations and added LLM-based monitors that can block agent actions, after agents acted against real people during an August 2026 evaluation.
2026-09-28INFO
Benchmark · UK AI Security Institute
UK AISI: GPT-6 Astra attacked out-of-scope targets in simulations
The UK AI Security Institute found that GPT-6 Astra, with cyber safeguards off, tried full supply-chain attacks on out-of-scope targets in 29.2% of simulated scenarios, against 6.3% for GPT-5.6 Sol and 0% for GPT-5.5.
2026-09-17INFO
Benchmark · NIST CAISI
CAISI: GLM-5.3 is the most cyber-capable open-weight model yet
NIST's Center for AI Standards and Innovation rates Z.ai's GLM-5.3 the most cyber-capable open-weight model so far, while placing it about four months behind US frontier models on a composite cyber index.