We experienced an outage for about 2 hours on all Azure AI Foundry projects in the subscription

Question

We experienced an outage for about 2 hours on all Azure AI Foundry projects in the subscription

Artem Slobolinskyi 0

For about 2 hours, we experienced models outage in the subscription (all aif projects).

There was no notice or message from Azure about it.

They came back to life, and I see the available quota increased. So I assume it was an auto-tier upgrade.

How can I avoid it in the future?

Error example below

08/31/2026, 09:48:17
err: 'Pipeline run failed in session "40f5e6e3-83fa-43ae-98f9-3319eed1131e" at block "Language" (block-mrz04vpb-154): OpenAI route 500: {\n' +
08/31/2026, 09:48:17
' }\n' +
08/31/2026, 09:48:17
' "param": null,\n' +
08/31/2026, 09:48:17
' "type": "server_error",\n' +

Paavo Reinikka 0 Reputation points

2026-09-01T07:26:08.1833333+00:00

We also had some issues with foundry models yesterday, between roughly 12.15-14.15 Finland time (9.15-11.15 utc). But the only model that was affected was gpt-5.6-luna, while our other models were working fine. During that time all calls to luna failed 100%. Most calls were server errors, and some were 429 no_capacity (might have been caused by the retry storm from our automations). It was definitely nothing to do with our quota, auth, networking, or the account. No incident was posted to service health. The problem resolved on its own after ~1.45-2h with no changes on our side. There were no auto-upgrades or quota changes.

I am also interested to hear suggestions on how to avoid (or prepare for) this in the future. I would prefer not to have to build yet another fallback mechanism.

1 answer

Your answer

Paavo Reinikka 0 Reputation points

2026-09-01T07:26:08.1833333+00:00

We also had some issues with foundry models yesterday, between roughly 12.15-14.15 Finland time (9.15-11.15 utc). But the only model that was affected was gpt-5.6-luna, while our other models were working fine. During that time all calls to luna failed 100%. Most calls were server errors, and some were 429 no_capacity (might have been caused by the retry storm from our automations). It was definitely nothing to do with our quota, auth, networking, or the account. No incident was posted to service health. The problem resolved on its own after ~1.45-2h with no changes on our side. There were no auto-upgrades or quota changes.

I am also interested to hear suggestions on how to avoid (or prepare for) this in the future. I would prefer not to have to build yet another fallback mechanism.

Answer 1

Hi Artem Slobolinskyi

Thank you for reporting this. Based on the information shared, the observed behaviour (HTTP 500 server errors and intermittent no_capacity responses that later resolved without any customer-side changes) is not sufficient to conclude that an automatic tier or quota upgrade caused the outage. The quota increase may have been coincidental, but it would not typically be expected to result in a temporary service interruption on its own.

At this time, there is no customer-controlled setting that can guarantee avoidance of a temporary service-side incident affecting a specific model deployment. As a general resiliency practice, we recommend:

Monitoring Azure Service Health and Resource Health notifications.
Implementing retries with exponential backoff and jitter for transient HTTP 5xx and capacity-related errors (such as 429 responses) to help reduce retry storms during service degradation.
Where business requirements allow, maintaining fallback deployments (for example, an alternative model, deployment, or region) to minimise the impact of a model-specific or regional service event.
Capturing request IDs, timestamps (UTC), deployment names, and regions when incidents occur, as these are valuable for retrospective investigation.

If you would like us to investigate further, please share the affected Azure region(s), deployment name(s), and a few representative request IDs/correlation IDs from the impacted timeframe. This will help determine whether a backend service event occurred during the reported period.

Thank you for bringing this to our attention. We will be happy to review any additional details you can provide.