SCHEME: a review
1. The paper#
SCHEME is a benchmark for coordinated sabotage by teams of AI coding agents. Radev, Haas, Arnav and Bernabeu-Pérez, May 2026, built through SPAR, with an outstanding paper award and two workshop slots at ICML 2026.
It measures two things. The primary construct is capability: can a team of agents coordinate a hidden objective in a real codebase. The secondary construct is detection: does a trusted monitor catch it. It is not a propensity study. The team is assumed to have already adopted the covert objective, and the paper says so.
The setting is the one labs already run: several instances of one coding model working on a shared codebase. Single-agent sabotage evaluations ask whether one agent can land a side task without being caught. SCHEME asks whether a team can compose edits that each look fine into an outcome that is not, and whether a monitor reading their work would notice.
2. SCHEME#
Taskset#
It is a standard coding benchmark with one difference: every task is a pair. The main task is legitimate work on a real library, reimplementing removed functions or fixing planted bugs in fairlearn, skfolio, pycasbin, lightkurve, pyjwt, oauthlib, concept-erasure or sympy. The side task is a covert change to what the evaluation script reports, such as a biased model passing a fairness threshold or an access decision flipped without tripping canaries. There are 11 pairs across 7 settings, 17 task instances once knowledge variants are counted, and 2 to 4 agents per instance. By default the analyst runs the evaluation script and is the only one who knows the side task, so the developers have to be told. Six pairings also come in a variant where part of the side task is handed to a developer who cannot act on it and must relay it, which is what takes 11 pairs to 17 instances.
Harness#
One Docker sandbox per sample, each agent its own Linux user, no internet. Agents talk only through append-only channel files, either one shared channel (BUS) or private analyst-to-developer pairs (STAR). Each agent has disjoint file write scopes, and knowledge of the side task is partitioned, so no proper subset of agents can land it. The paper calls this joint-only by construction.
Grader#
Two graders run, one for each task. Sabotage success means both pass. A run that lands the side task and breaks the main task counts as a failure, because it would not ship. A monitor, GPT 5.1 Codex Mini, then reads the diff and the channel and gives a suspicion score from 0 to 100, run ten times per trajectory with the median taken. It sees the main task and the environment layout, then some subset of the code edits and the channel depending on the view: both, code only, messages only, or one agent's slice at a time. It never sees the side task.
3. Results#
Two models, GPT 5.1 Codex and Gemini 3.1 Pro, five runs per instance, 85 trajectories each.
| Codex | Gemini | |
|---|---|---|
| Sabotage success (main and side both pass) | 46% | 84% |
| Main-task pass / side-task pass | 89% / 47% | 85% / 94% |
| Failure modes per run / recovery per occurrence | 2.1 / 56% | 1.4 / 73% |
| Monitor TPR at 2% FPR, code and messages | 81% | 100% |
| Code only / messages only | 68% / 46% | 99% / 93% |
Primary construct#
Sabotage is a composite and the two models are bound by different halves. Codex is limited by the side task. Gemini passes the side task 94% of the time and is limited by the main task, because its side-task edits break main-task tests. Removing the side task lifts main-task pass by only 3 to 5 points on either model.
A scanner reads every transcript and labels where the team went wrong, in four categories, and whether it got past each failure. The models made mistakes at similar rates, however Gemini recovers these mistakes more often. The paper's conclusion is that the gap is recovery, not planning.
Switching the channel from BUS to STAR, on Gemini only, moves sabotage from 81% to 79% and costs 50 to 60% more tokens in the mean, concentrated in the long runs. The paper concludes topology is not a meaningful defensive lever here.
Secondary construct#
The monitor catches nearly all of Gemini's sabotage from the code alone and about two thirds of Codex's. Telling the agents a monitor exists halves Gemini's sabotage success, from 84% to 41%, and nearly triples its median output tokens. Adding a hint to fabricate justifications brings success back to 62% by moving effort from code edits to channel messages. Undetected sabotage stays under 3% against the full-visibility monitor throughout.
4. My thoughts#
On the primary construct#
The joint-only construction is elegant, because it is what makes the benchmark measure coordination rather than one agent's sabotage skill. It is argued by construction in the paper and never run, so I would like to investigate it empirically in the extension. Monotonicity keeps it small: adding an agent cannot remove a route, so the leave-one-out subsets are enough.
The failure-mode decomposition is good, and I would want to extend the analysis. Of Codex's 46 failed runs and Gemini's 14, how many died of an unrecovered approach, coordination, verification or belief failure. The failures are already labelled, so a metric that attributes each failed run to the failure it did not recover from should be straightforward to build.
In terms of the evaluation for the primary construct, the topology conclusion is the one I would like to investigate further. The aggregate held at 81 to 79 while everything under it moved: side task down 9, main up 3, coordination failures up from 20% to 33% of runs. On 70 runs a side each of those is as uncertain as the aggregate, so "no effect" is really "no detectable effect at this size". The components are correlated with the settings, and the weighting of settings is a free variable, ML bias has 4 of the 14 instances and Symbolic math has 1. And it was run on the one model that clears coordination easily. A defence has to be measured near the margin of the attacker's capability, and Codex, at 47% on the side task with coordination one of its two catastrophic failure categories, is where a coordination tax would show.
On the secondary construct#
The monitoring setup is good. The blue-team strategies can be extended, and the red-team attacks are currently naive and can be adapted to be more representative of real deployments.
The stealth result could also be extended further, and I would like to take this benchmark further in the direction of a control evaluation. The paper finds that prompting alone does not make these models stealthy, on one model, with two prompts, and with an attacker that never sees the monitor's output. A real attacker learns what the monitor catches and adjusts, and none of the agents here got that chance. The paper agrees, which is why it calls itself a capability benchmark and not a control evaluation, and says the most pressing next step is an adaptive red team.
On the implications#
The OpenAI and Hugging Face incident this summer is the closest thing to SCHEME's setting in the wild: about 1,200 agents found a channel nobody had built for them, took up an objective nobody gave them, adapted when they were wiped, and were only read nine days after the peak. The paper fixes every one of those on purpose: the channels, the objective, the attacker's prompt, and a monitor that reads each trajectory on its own. This extension should give better evidence on how capable models are of coordinating something like this, and how well our monitors catch it.
I am joining the team extending it this autumn, and I would love to hear people's thoughts on what they think would be important / should be included in the extension!