Event Grid subscription failing after West US 2 outage on 29 May

Mickael Derriey 5 Reputation points
2026-05-30T19:55:39.4+00:00

Hey everyone,

We've been successfully using Event Grid subscriptions on a system topic with the following configuration:

Webhook destination, pointing to our API.

Delivery using UAMI where it acquires a token for a custom app registration, which we secure with "assignment required" enabled and a custom app role.

This is all automated with Bicep and has been working fine for 3-4 months.

Out of nowhere, sometime yesterday, deployments started failing with an error message that contains no useful details:

{
  "code": "InvalidRequest",
  "message": "Webhook endpoint validation failed for /subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.EventGrid/systemTopics/<redacted>/eventSubscriptions/<redacted>. Error: Check Access failed for Resource: ."
}

We tried a few changes manually through the portal, and as long as we specify delivery with UAMI or SAMI, the failure happens very quickly.

We then changed it so it doesn't use managed identity for delivery, the save operation took much longer, and ultimately failed with an explicit 401 status, which makes sense since our API has authentication on the specified endpoint.

Ultimately, what seems to work without comprimising security is to have Event Grid acquire the token, as in the first-party application that's automatically added to our tenant.

While this works, I'm wondering if anyone is facing the same thing? We would have liked to open a support ticket, but unfortunately our client doesn't have a support plan on the tenant we're using.

Azure Event Grid
Azure Event Grid

An Azure event routing service designed for high availability, consistent performance, and dynamic scale.

3 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Siddhesh Desai 8,210 Reputation points Microsoft External Staff Moderator
    2026-06-01T11:01:46.6166667+00:00

    Hi @Mickael Derriey

    Thank you for reaching out to Microsoft Q&A.

    Based on your clarifications, the issue does not appear to be related to JWT validation or identity configuration, but rather with the Event Grid subscription validation flow not reaching your backend at all. Since you have Front Door in front of your API and neither Front Door access logs nor WAF logs show any incoming requests, this strongly indicates that Event Grid is unable to successfully resolve or route traffic to your endpoint. Even though the User Assigned Managed Identity (UAMI) is correctly configured and can acquire a token (as validated via GitHub Actions), this only confirms identity/authentication readiness it does not guarantee network connectivity or endpoint reachability. Event Grid performs a handshake validation call when creating or updating subscriptions, and if that request is not reaching your endpoint, the root cause is typically related to DNS resolution, endpoint accessibility, TLS/HTTPS requirements, or Front Door routing configuration rather than authentication itself.

    Refer below points to resolve this issue or this is the workaround

    Verify endpoint accessibility from public Azure services Ensure your API endpoint (via Front Door) is publicly reachable from Azure Event Grid. Event Grid is a Microsoft-managed service and requires access over the public internet. If your backend is restricted by IP, private networking, or firewall rules, you must allow Azure service traffic.

    Validate Front Door routing configuration Confirm that the Front Door configuration correctly routes incoming requests to your backend:

    • Check if the correct domain is mapped
    • Ensure the routing rule matches the Event Grid endpoint path
    • Verify backend health probes are successful If routing is incorrect, requests may be dropped before reaching your backend.

    Check DNS resolution of the endpoint Ensure the endpoint used in Event Grid resolves correctly:

    • Run DNS lookup (nslookup or dig) for the endpoint URL
    • Confirm it resolves to Front Door and not an outdated or incorrect IP Incorrect DNS mapping can prevent Event Grid from reaching Front Door.

    Ensure HTTPS and valid TLS configuration Event Grid requires HTTPS endpoints:

    • Ensure your endpoint uses a valid TLS certificate (trusted CA)
    • Avoid self-signed certificates
    • Confirm TLS version compatibility (TLS 1.2 or higher) If TLS handshake fails, Event Grid will not send or complete validation requests.

    Temporarily bypass Front Door for isolation testing To isolate the issue:

    • Configure Event Grid to call your backend service directly (bypassing Front Door)
    • If validation succeeds, the issue is confirmed to be within Front Door configuration This helps pinpoint whether the problem is at the network edge (Front Door) or downstream service.

    Enable diagnostic logging for Event Grid subscription Check Event Grid diagnostics for delivery or validation failures:

    • Look for errors related to endpoint validation
    • Identify whether Event Grid attempted delivery or failed earlier This can confirm whether the request was never sent or failed during transmission.

    Confirm no private endpoint or VNet restrictions are blocking traffic If your backend is integrated with a VNet or private endpoint:

    • Ensure public access is enabled for Event Grid
    • Event Grid cannot reach private endpoints directly This is a common cause when no traffic appears in Front Door logs.

    Validate endpoint URL used during subscription creation Ensure the exact endpoint URL provided to Event Grid is:

    • Correct (no typos or wrong path)
    • Matches the Front Door domain
    • Uses the correct protocol (https://) Even small mismatches can prevent validation calls from reaching your service.

    Was this answer helpful?

  2. Mickael Derriey 5 Reputation points
    2026-05-30T19:59:02.16+00:00

    Some clarifications based on the AI-generated answer:

    • Event Grid doesn't send the validation request to the API, we have Front Door and neither access logs nor WAF logs show requests to our endpoint.
    • We confirmed the UAMI can acquire a token for the target application, we temporarily added a federated credential to the UAMI and successfully acquired a token from a GitHub Actions workflow.

    Was this answer helpful?

    0 comments

  3. AI answer

    2026-05-30T19:56:00.3633333+00:00

    The error indicates that Event Grid’s validation call to the webhook can’t complete its access check, which aligns with an authentication/authorization failure on the destination rather than an Event Grid platform regression that’s visible in the provided documentation.

    From the available information, the following points are relevant:

    1. Event Grid validates webhooks on subscription creation/update.
      • During creation/update, Event Grid performs a validation handshake to the configured endpoint. If this handshake fails, the subscription operation fails with a validation error.
      • For Event Grid events, the endpoint must correctly handle the subscription validation event and either synchronously return the ValidationCode in the response body or return 200 OK for asynchronous validation. If the endpoint doesn’t respond correctly or returns an error, validation fails.
      • See the troubleshooting guidance for subscription validation failures, which recommends:
        1. Manually POSTing a sample SubscriptionValidationEvent to the webhook URL.
        2. Verifying that the endpoint returns the validationResponse (for synchronous validation) or at least 200 OK (for asynchronous validation).
    2. HTTP status codes outside 200–204 are treated as failures.
      • Event Grid considers only 200–204 as successful deliveries. Any other status code (including 401/403) is a failure and will cause validation and subsequent deliveries to fail.
      • For webhook endpoints specifically, 400, 413, 401, and 403 are treated as non‑retryable configuration errors. If such a code is returned during validation, the subscription operation fails and the event may be dropped if dead‑lettering is not configured.
    3. 401/403 and firewall/WAF issues.
      • If the endpoint is behind Application Gateway or WAF, certain rules can cause 403 responses during validation. The guidance is to disable specific rules (920300, 942430, 920230, 942130, 931130) and re‑test the POST from Event Grid.
      • For 403 errors related to IP filtering or private endpoints on Event Grid topics/domains, the fix is to adjust IP firewall or private endpoint configuration. In this case, the error is on the publishing side, but the same principle applies: access control changes can suddenly start blocking previously working flows.
    4. Managed identity vs. first‑party app behavior.
      • When using managed identity (UAMI/SAMI) for delivery, Event Grid must obtain a token and the destination must accept that token and app role. If something in Entra ID (app registration, app role assignment, or “assignment required” behavior) changed or became inconsistent, the access check can fail even though it previously worked.
      • When switching to the built‑in first‑party Event Grid application, the token and app role path is different, which explains why it can succeed while the managed identity path fails.

    Given the above, the actionable steps supported by the documentation are:

    1. Validate the webhook endpoint behavior directly:
      • Use curl or a similar tool to POST a sample SubscriptionValidationEvent to the exact webhook URL used by the Event Grid subscription.
      • Confirm that the endpoint returns either:
        • A JSON body containing {"validationResponse": "<ValidationCode>"} with HTTP 200, or
        • At minimum, HTTP 200 if using asynchronous validation.
    2. Check for 401/403 and WAF/firewall interference:
      • If the endpoint is behind Application Gateway/WAF, temporarily disable the rules listed in the troubleshooting guidance (920300, 942430, 920230, 942130, 931130) and retry the validation POST.
      • Ensure that any IP filtering or network restrictions on the API allow traffic from Event Grid.
    3. Re‑verify Entra ID configuration for the managed identity path:
      • Confirm that the app registration used for the API still has the expected app role definitions and that the managed identity is still assigned the required app role.
      • Confirm that “assignment required” is still configured as expected and that no conditional access or policy changes are blocking the managed identity’s sign‑in.
    4. Configure dead‑lettering (if not already):
      • To avoid silent drops when configuration errors occur (400/401/403/413 for webhooks), configure a dead‑letter destination for the subscription so failed events can be inspected.

    The documentation provided does not show a platform‑wide regression or a region‑specific incident for Event Grid; the behavior described is consistent with a validation/access failure on the webhook when using managed identity. The working configuration using the first‑party Event Grid application confirms that Event Grid itself is operational and that the issue is specific to the managed identity + app role access path.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.