Event Grid subscription failing after West US 2 outage on 29 May

The error indicates that Event Grid’s validation call to the webhook can’t complete its access check, which aligns with an authentication/authorization failure on the destination rather than an Event Grid platform regression that’s visible in the provided documentation.

From the available information, the following points are relevant:

1. Event Grid validates webhooks on subscription creation/update.
   • During creation/update, Event Grid performs a validation handshake to the configured endpoint. If this handshake fails, the subscription operation fails with a validation error.
   • For Event Grid events, the endpoint must correctly handle the subscription validation event and either synchronously return the ValidationCode in the response body or return 200 OK for asynchronous validation. If the endpoint doesn’t respond correctly or returns an error, validation fails.
   • See the troubleshooting guidance for subscription validation failures, which recommends:
     1. Manually POSTing a sample SubscriptionValidationEvent to the webhook URL.
     2. Verifying that the endpoint returns the validationResponse (for synchronous validation) or at least 200 OK (for asynchronous validation).
2. HTTP status codes outside 200–204 are treated as failures.
   • Event Grid considers only 200–204 as successful deliveries. Any other status code (including 401/403) is a failure and will cause validation and subsequent deliveries to fail.
   • For webhook endpoints specifically, 400, 413, 401, and 403 are treated as non‑retryable configuration errors. If such a code is returned during validation, the subscription operation fails and the event may be dropped if dead‑lettering is not configured.
3. 401/403 and firewall/WAF issues.
   • If the endpoint is behind Application Gateway or WAF, certain rules can cause 403 responses during validation. The guidance is to disable specific rules (920300, 942430, 920230, 942130, 931130) and re‑test the POST from Event Grid.
   • For 403 errors related to IP filtering or private endpoints on Event Grid topics/domains, the fix is to adjust IP firewall or private endpoint configuration. In this case, the error is on the publishing side, but the same principle applies: access control changes can suddenly start blocking previously working flows.
4. Managed identity vs. first‑party app behavior.
   • When using managed identity (UAMI/SAMI) for delivery, Event Grid must obtain a token and the destination must accept that token and app role. If something in Entra ID (app registration, app role assignment, or “assignment required” behavior) changed or became inconsistent, the access check can fail even though it previously worked.
   • When switching to the built‑in first‑party Event Grid application, the token and app role path is different, which explains why it can succeed while the managed identity path fails.

Given the above, the actionable steps supported by the documentation are:

1. Validate the webhook endpoint behavior directly:
   • Use curl or a similar tool to POST a sample SubscriptionValidationEvent to the exact webhook URL used by the Event Grid subscription.
   • Confirm that the endpoint returns either:
     • A JSON body containing {"validationResponse": "<ValidationCode>"} with HTTP 200, or
     • At minimum, HTTP 200 if using asynchronous validation.
2. Check for 401/403 and WAF/firewall interference:
   • If the endpoint is behind Application Gateway/WAF, temporarily disable the rules listed in the troubleshooting guidance (920300, 942430, 920230, 942130, 931130) and retry the validation POST.
   • Ensure that any IP filtering or network restrictions on the API allow traffic from Event Grid.
3. Re‑verify Entra ID configuration for the managed identity path:
   • Confirm that the app registration used for the API still has the expected app role definitions and that the managed identity is still assigned the required app role.
   • Confirm that “assignment required” is still configured as expected and that no conditional access or policy changes are blocking the managed identity’s sign‑in.
4. Configure dead‑lettering (if not already):
   • To avoid silent drops when configuration errors occur (400/401/403/413 for webhooks), configure a dead‑letter destination for the subscription so failed events can be inspected.

The documentation provided does not show a platform‑wide regression or a region‑specific incident for Event Grid; the behavior described is consistent with a validation/access failure on the webhook when using managed identity. The working configuration using the first‑party Event Grid application confirms that Event Grid itself is operational and that the issue is specific to the managed identity + app role access path.

References:

• Event Grid message delivery and retry
• Troubleshoot Azure Event Grid subscription validations
• Troubleshoot Azure Event Grid errors
• Endpoint validation with Event Grid event schema
• Reliability in Azure Event Grid
• Configure Durable Functions publishing to Azure Event Grid
• event grid topic subscriptions stuck in Deleting state - Microsoft Q&A
• Azure Deployment Takes 70 Minutes to Fail with Obscure Error - Microsoft Q&A
• Can't create Azure SQL server via Azure portal - Microsoft Q&A
• How do I work around this Exchange Online Rule error? - Microsoft Q&A
• Document Intelligence classification and extraction requests timing out - Microsoft Q&A