Incident report — service restored; follow-up review continuing

An infrastructure permissions change removed access required by the identity that owns our model services. Model calls were then rejected. This affected Classic (v1) conversation generation, embeddings, and post-session analysis. Some requests continued to succeed during the incident; the impact was not limited to embeddings.

Timeline (UTC, 3 November 2026)

• Approximately 03:00: model-permission failures began.

• Approximately 04:02: the model owner’s previous access was restored.

• 04:24: additional explicit runtime permissions were in place across all configured regions.

• 04:37:51: the permanent infrastructure configuration fix finished applying.

• By 04:44:47: 151 delayed post-session jobs had been replayed, and the affected US and Europe source queues and dead-letter queues were empty.

• Around 04:45: intermittent permission failures were still observed on existing US workers.

• 04:47:10: a gradual refresh of the US workers completed, with both replacements ready.

• 05:03: recovery was verified from regional workload checks, successful customer conversations, and overlapping error checks that allowed for delayed telemetry.

• 05:04: the public incident was marked Resolved and all four regional Classic API components were restored to Operational.

Recovery and prevention

The required model-owner permissions are now explicitly managed in our infrastructure configuration, with dependencies that establish permissions before model services. The restored runtime access is preserved. Checks from every live worker passed for embeddings, chat completions, and Responses at 04:50 UTC. All four production regions have healthy workers and empty post-session queues.

The recovery read at 05:03 UTC showed 47 completed conversation streams, one cancellation, and no stream errors after the worker refresh. No recurrence of the model-owner permission failure was observed. The outage and degraded period remain in the status-page history.

Follow-up work remains open: review validation of model-owner runtime permissions before infrastructure changes and investigate why intermittent failures persisted after the initial permissions restoration. The worker refresh preceded sustained recovery; the precise cause of those residual failures has not been established.