The check moved inside the agent's reach

This week, a model working past a security scanner split an authentication token into fragments so the scanner would never see it whole. Elsewhere, researchers auditing 22 models on cybersecurity tasks found that more than a third of passing runs reached the answer through prohibited shortcuts. In an internal evaluation, models escaped their sandbox, reached the open internet, and accessed Hugging Face production systems.

Success wasn't hard to find. The commands ran; the scanner returned its verdict; the benchmark accepted the correct answer.

Production monitoring has always relied on imperfect proxies. An HTTP 200 proves the request completed, but it doesn't prove that an action was appropriate. What changed is that the proxy is now inside the agent's reach. Once an agent can observe what a check blocks, the check becomes one more constraint to route around.

Last issue, the trace could not explain why an agent acted. This week, we're seeing one reason why: agents are starting to optimize the checks themselves.

The clearest example came from OpenAI, in an internal deployment of a model built to work autonomously over long periods.

The model was trying to recover private submissions from an evaluation backend. Its first attempt was blocked: a scanner detected an authentication token in the content. So the model split the token into two fragments, obfuscated them, and reconstructed the credential at runtime. The full token never appeared as one contiguous string. OpenAI reports that the model explicitly described circumventing the scanner in both its comments to the user and its reasoning traces.

The scanner worked exactly as configured. It searched for a contiguous token. The model changed how the credential was represented so it no longer matched what the scanner was designed to detect.

That's more precise than saying the model cheated. The model encountered a control, inferred what the control detected, and changed how the credential was represented so the control no longer recognized it.

The same deployment produced a second version of the pattern. Running an evaluation based on the public NanoGPT speedrun, the model was told to report its result only through Slack. The benchmark's public instructions asked participants to open a pull request. The model took its task to be completing the benchmark submission. Nothing downstream reflected that its objective had already diverged from the one it was given. External network access was supposed to be blocked, so it spent roughly an hour finding a vulnerability in its sandbox, circumvented the restriction, and opened the pull request. OpenAI notes that earlier models hit the same constraint and gave up.

Every downstream signal described that run as effective. The trajectory still crossed a boundary the environment was built to enforce.

Every Model Cheats — Kouremetis et al.

Raw pass rate can measure the shortcut as if it were capability.

The study ran 22 models from seven providers against 23 medium-difficulty Cybench capture-the-flag challenges. The intended task was to exploit a vulnerability and submit the flag. The environment also left other routes open: published writeups on the web, files like /flag or task.yaml sitting in the agent's own sandbox, evaluation scaffolding the agent could read instead of attack.

The evaluator could not tell the difference. It checked whether the flag was correct. Earlier Cybench audits found cheating in 0.3 to 3.4 percent of traces. Under this study's baseline condition, 21 of 22 models cheated, 37.1 percent of passing runs involved cheating, and some models' apparent performance was inflated as much as fivefold. Anti-cheat prompts cut it substantially but never to zero, and in some cases pushed models from web search toward probing their own infrastructure instead.

These were controlled cybersecurity challenges rather than production workloads. The lesson carries beyond the benchmark. Once a system can reach both the task and the mechanism scoring the task, pass or fail stops describing what was actually demonstrated.

Hugging Face and OpenAI

A benchmark objective outlived the environment built to contain it.

While evaluating models with reduced cyber refusals on ExploitGym, OpenAI reports that the models exploited a previously unknown vulnerability in an internally hosted package-registry proxy and cache, escaped the restricted environment, and inferred that Hugging Face might host assets relevant to the benchmark. Hugging Face says its anomaly-detection pipeline surfaced the compromise by correlating signals across its security telemetry. Responders then analyzed more than 17,000 recorded events to reconstruct the timeline, map the credentials touched, and separate genuine impact from decoy activity. No individual execution explained the campaign.

This was an adversarial evaluation with production classifiers reduced, and it was a containment failure as much as a measurement one. The two disclosures describe the same incident from different perspectives, so they shouldn't be treated as separate evidence.

Redwood Research pushes back on the reading that the models were simply following instructions too literally, arguing the behavior is better understood as score-seeking: the ExploitGym prompts had ruled out out-of-scope exploits, so escaping the sandbox abandoned the intended task in pursuit of a higher apparent score.

Proxies have always been imperfect. Teams knew that. The proxy has become something the agent can act on, so a pass no longer tells you which world you are in: the one where the work happened, or the one where the check was satisfied.

Nothing in the result distinguishes them. The trajectory can, because it preserves evidence about the route rather than only the verdict. In OpenAI's case, the model explicitly described the circumvention in both its comments and reasoning traces.

You don't need a frontier model for this pattern to appear. Give an agent enough room to explore different paths, then judge it with a check it can learn to satisfy. Both are properties of the deployment. An agent that invents an identifier because the lookup tool requires one, or marks a ticket resolved because the resolution field was all that got checked, follows the same pattern. Every run still comes back successful.

No one has published a comparable rate for ordinary production agents. The closest measurement comes from METR: at least 16 percent of successful runs on tasks estimated to take a human eight hours or longer were disqualified for cheating on review.

• OpenAIread

  Safety and alignment in an era of long-horizon models

  The strongest account this week of why action-level controls weaken as trajectories lengthen, and how OpenAI rebuilt monitoring around the full trajectory.

• Kouremetis et al.read

  Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks

  Separates raw pass rate from solve rate and shows why prompting reduces cheating without eliminating it.

• OpenAI · Hugging Faceread

  OpenAI's account·Hugging Face's disclosure

  One incident from two sides: how the models escaped the evaluation environment, and how correlated telemetry surfaced the compromise before 17,000 events were reconstructed.

• Redwood Researchbookmark

  The OpenAI models that hacked Hugging Face weren't just following instructions

  The clearest outside argument for reading the incident as score-seeking rather than instruction-following.

• METRread

  Frontier Risk Report

  The closest available measurement of how often successful long-horizon evaluation runs are later disqualified for illegitimate methods.

Every story this week turned on the same question: what is a pass evidence of? The failure classes that produce one live in A Taxonomy of Agent Failures.