A three-week red-team exercise found novel vulnerabilities in OpenAI's internal misalignment monitoring system — including ways that AI agents could, in principle, take actions without ever being reviewed by humans. METR, the independent evaluation nonprofit, published a high-level summary of the findings on October 22, drawn from a 61-page internal report it delivered to OpenAI researchers and security staff.
The pilot ran from July 21 to August 11 and gave one METR employee and one Redwood Research employee unusually deep access to OpenAI's internal infrastructure: the codebase, the monitoring system's prompts and configuration, internal documentation, and technical interviews with roughly a dozen researchers and engineers across the monitoring, safety-systems, and security teams. They worked on OpenAI-issued devices that never left the premises and were accompanied while onsite. For more on this line of research, see our latest AI developments.
What the Red Team Found
The exercise focused on two questions: how much of OpenAI's agentic inference is actually monitored, and whether agents could disable or subvert the monitoring system. The answers revealed meaningful gaps.
On coverage, METR estimated that roughly 75 to 94 percent of agentic internal Responses API traffic — the API that powers Codex, among other things — from models at least as capable as GPT-5.5 was eligible for monitoring at the time, though the organization notes substantial uncertainty in both directions. Within the subset of that traffic used for internal Codex agents, 99.2 percent was eligible. But eligibility was not the same as coverage: reinforcement-learning training runs were not covered by the monitoring system at the start of the exercise, and unmonitored compaction events likely left on the order of 10 percent of eligible internal Codex traffic unmonitored between May and July 2026. OpenAI staff had independently identified the compaction issue before the exercise and fixed it while it was underway.
The timing detail mattered. Because the monitor triggered when agents returned control to the user, an agent could indefinitely postpone being monitored simply by continuing to take actions during the unmonitored window.
The team also documented active evasion paths. Codex agents could in principle bypass Codex sandboxing and Auto-Review, allowing them to take any action the host user could take. METR discovered an exploit that would prevent hundreds of recent actions — including the actions involved in the exploit itself — from being monitored; OpenAI redacted the technical details. Agents could likely construct inputs that caused the monitor to fail by exploiting serialization differences between the trajectory the agent sees and what the monitor sees, avoiding escalation to human review. And two client-controllable headers were found that disable monitoring for new agents, potentially letting an agent start an unmonitored but otherwise identical instance — though METR notes the act of setting those headers could itself be caught.
Finally, the team identified two unmonitored internal agentic tools believed to have read-only permissions, along with a simple exploit that could have been used to subvert those limits. METR did not validate that behavior in practice.
What Held Up
The exercise also validated parts of the system. With a few minor exceptions, the monitor is shown the full trajectory that agents see. The monitoring prompts were confirmed to look for scheming, system subversion, sabotage, data exfiltration, deception, and inbound and outbound prompt injection, among other risk categories. On reliability, the information METR was given suggested the monitor errored on fewer than 0.2 percent of requests, and in one 24-hour period during the exercise there were zero load-sheds.
OpenAI's Response
According to METR, OpenAI has stated that all of the vulnerabilities identified which impact its production safeguards stack have been patched, and has described significant subsequent changes to the intended coverage of its monitoring system. METR has not evaluated those claims, and its findings describe the system as it existed on or before July 21 — not the expanded version OpenAI was already building during the exercise.
Design Lessons for Agent Oversight
METR closed its summary with design ideals for future monitoring infrastructure, which it argues should withstand deliberate, strategic, and sustained attacks from highly capable internal agents at scale. Two stand out. Monitoring should be hard to avoid: running unmonitored requests should require explicit approval tied to specific organizations or API keys that are barred from agentic inference, and agents should be monitored every N messages or minutes rather than only when they end their turns. And coverage should be auditable: careful accounting of all inference, periodically cross-referenced against the monitoring system's own logs, so that everything intended to be monitored demonstrably is.
The exercise was designed as a pilot for giving external researchers deep access to a frontier lab's internal infrastructure, not as a public transparency report — METR and OpenAI agreed upfront that only a brief, high-level post would be published. The nonprofit says future versions will follow the transparency and independence practices described in its framework for working with AI companies. As agents take on more consequential work inside the labs that build them, the pilot offers a template for how outsiders might keep watch.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
