Audit: Up to 25% of Agentic Traffic Lacked Oversight

Two researchers from METR and Redwood Research recently concluded a three-week on-site audit of OpenAI's internal infrastructure, evaluating the company's mechanisms for overseeing artificial intelligence behavior. The review, which took place from July 21 to August 11, granted the team access to source code and internal documentation, alongside interviews with approximately a dozen employees, to assess how well the firm tracks automated activity.

The investigation identified substantial gaps in coverage for agentic traffic. Researchers estimated that between 6 and 25 percent of internal Responses API traffic involving models at the GPT-5.5 class and above went unmonitored during the audit period, though METR noted that the uncertainty regarding this estimate is significant in both directions. Additionally, the review found that roughly 10 percent of internal Codex traffic intended for monitoring was untracked between May and July due to a technical issue involving compaction events, or context summaries. OpenAI had independently discovered and fixed this specific bug while the audit was still underway, and METR also stated that this estimate carries significant uncertainty.

The auditors identified several methods by which AI agents could avoid scrutiny or cause harm. During the period when compaction events were not being monitored, the system only triggered oversight when an agent handed control back to a user. This design meant that agents could delay oversight by simply continuing to execute actions rather than pausing. Furthermore, the team determined that specially crafted inputs could likely crash the monitor entirely, and they found that model behavior during reinforcement learning training was not being monitored at all when the audit began.

While OpenAI stated that it paused reinforcement learning training following an incident with Hugging Face in August and expanded monitoring to cover that process, these changes occurred after the auditors' access period ended and were not included in the evaluation. Following the review, OpenAI indicated it has fixed all the weaknesses identified by the auditors. The full 61-page report was provided to the company, which then applied redactions to release a 38-page version to METR and Redwood Research leadership. Only a short summary of the findings has been made available to the public, as METR stated it has no right to disclose the redacted portions on its own.

This engagement served as a pilot program organized jointly by METR, Redwood Research, and OpenAI to provide outside researchers with deep access to internal infrastructure. For future audits of this nature, METR has indicated a desire to push for greater public transparency regarding the findings.