For support, please contact us at contact@concurrence.com or reach out to your account representative.
Write-up published
Resolved
The Classic (v1) conversation and embedding issue is resolved. Recovery was verified at 05:03 UTC on 3 November across production regions using workload checks and successful customer conversations. No further model-permission failures have been observed since the US worker refresh completed at 04:47 UTC.
The permanent permissions fix is deployed. Delayed post-session jobs have been replayed, and the affected queues are empty. The outage history and incident write-up remain available. Follow-up review of the permissions dependency and intermittent failures after the initial mitigation is continuing.
Please contact support if you continue to experience errors.
Monitoring
The permanent permissions fix has been deployed across all production regions. Delayed post-session work has been replayed, and the affected processing queues are empty.
We observed intermittent conversation and embedding permission failures after the initial mitigation. We have completed a rolling refresh of the US Classic API workers and are checking customer traffic for recurrence. Successful requests are being observed, but full recovery is not yet confirmed. The incident remains in Monitoring with degraded performance.
We will provide the next update by 05:15 UTC on 3 November.
Monitoring
The issue has been mitigated and we are monitoring recovery. Starting at approximately 03:00 UTC on 3 November (22:00 ET on 2 November), Classic (v1) conversation creation and response generation experienced elevated failures. Embedding requests and post-session analysis were also affected.
We identified a permissions change in the model-serving infrastructure and restored access across production regions. Current recovery checks are succeeding. We are continuing to monitor customer traffic, verify delayed background processing, and complete the permanent configuration fix.
We will provide the next update by 05:00 UTC.