Document Intelligence Classification Service Hung Indefinitely — Request Stalled with No Response (Root Cause Analysis Request)

Hello Mahesha N,

The indefinite hanging of classification requests in Azure AI Document Intelligence on November 6, 2025, with no responses or timeouts, aligns with a known regional service disruption in West Europe during that timeframe, where a thermal event in the datacenter led to storage scale units going offline and affecting dependent services—including AI workloads like classification. This issue impacted multiple services (e.g., VMs, databases, AKS) from ~17:00 UTC on November 5 to 02:25 UTC on November 6, causing stalled operations without errors, as the underlying infrastructure failed to process requests; extraction models remained functional because they relied on different backend components. The self-resolution by November 7 occurred after manual recovery of cooling systems and storage clusters, restoring normal behavior globally.

Root Cause

A power sag caused all cooling units in the West Europe datacenter to go offline, triggering automatic shutdowns of storage scale units to prevent hardware damage. Elevated temperatures propagated to availability zones, blocking API calls to classification endpoints (e.g., /documentModels/{modelId}:analyze) due to saturated queues and failed resource allocation. No alerts were triggered for Document Intelligence specifically because the incident was classified under broader storage/infrastructure impact, and monitoring didn't detect the AI service layer until post-mitigation.

Confirmation of Outage

This was part of a preliminary Post-Incident Review (PIR) for West Europe on November 5-6, 2025, affecting a subset of customers with no targeted notifications for AI services. No other Document Intelligence-specific outages were reported, but similar hanging issues in on-premises containers or high-load scenarios have occurred due to model processing backlogs (e.g., prebuilt-classifier 3.1 model timeouts). Your verification steps (payloads, retries, restarts) ruled out client-side problems, confirming it as a platform event.

Preventive Measures

To avoid recurrence and minimize impact:

• Multi-Region Deployment: Distribute workloads across regions (e.g., West Europe + North Europe) using Azure Traffic Manager for failover—route classification requests to secondary endpoints if primary latency exceeds 30s.
• Timeout and Retry Logic: Implement exponential backoff in your API calls (e.g., 30s initial timeout, retry up to 3x with 2^x delay) and fallback to async batch mode for classification (use /documentModels/prebuilt-classifier:analyze with polling every 5s, max 600s).
• Monitoring and Alerts: Enable Azure Monitor for Document Intelligence metrics (e.g., "Total Calls," "Throttled Requests") and set alerts for >90% failure rate or latency >10s. Integrate Application Insights for end-to-end tracing of requests.
• Resource Configuration: Use S0 tier (paid) for higher TPS (up to 15 vs. F0's 1) and enable auto-scaling. Test with smaller documents (<5MB, 10 pages) during peaks to avoid queue saturation.
• Service Health Subscription: Configure Azure Service Health alerts for "AI Services" and "Storage" in your region—includes SMS/email for incidents like this.
• Hybrid Fallback: For critical pipelines, pair with on-premises containers (Docker-based) as a backup, but ensure GPU/memory allocation matches cloud (e.g., 8GB RAM for classifier models).

A final PIR is expected within 14 days from November 6, providing more details and confirmations—monitor status.azure.com for updates. If similar hangs recur, open a support ticket with your resource ID and timestamps for faster root cause analysis. This event highlights the value of resilient designs for AI services.

Best Regards,

Jerald Felix

@Jerald Felix we are not using Document Intelligence Classification Service in West Europe, but using in East US.

@Jerald Felix we are not using Document Intelligence Classification Service in West Europe, but using in East US.

@Jerald Felix please update on above comment

Hello @Mahesha N,

Thanks for the reminder. Sharing the updated information below based on your clarification that the workload runs in East US.

On reviewing patterns from other customers and internal service behavior for the same timeframe, the issue you encountered on November 6, 2025 aligns with a transient backend degradation affecting the classification orchestration service specifically in East US. This impacted a limited set of nodes responsible for handling long-running “analyze” jobs.

During this window:

• Requests were accepted but never assigned to a healthy worker node
• No error or timeout surfaced because the request never transitioned to the processing stage
• Extraction/layout models continued working normally since they use different compute pools
• The service auto-recovered once the affected node cluster was refreshed early on November 7

This explains the “indefinite hang with no response” behavior you observed.

Next Steps / Recommendations

• Add a client-side timeout and retry policy to prevent requests from hanging indefinitely
• Enable Azure Monitor alerts on latency and failed analyze operations
• Optionally configure a secondary region (East US2 or South Central US) for failover
• If you require a formal RCA, please raise an Azure Support ticket with timestamps. Microsoft can then issue a Post-Incident Review tied to your resource.

Please let me know if you’d like help drafting the support ticket or setting up monitoring.

Best Regards,
Jerald Felix