Azure api manger buffers the entire SSE stream before forwarding it to clients when the backend is an Azure OpenAI endpoin

Question

Azure api manger buffers the entire SSE stream before forwarding it to clients when the backend is an Azure OpenAI endpoin

Jeffrey Bakker 0

APIM's internal CacheModelProcessStreamHandlerLegacybuffers the entire SSE stream before forwarding it to clients when the backend is an Azure OpenAI endpoint, despite buffer-request-body="true" buffer-response="false" being explicitly set on forward-request. This defeats streaming and significantly increases TTFT. The "Legacy" suffix suggests a non-buffering implementation exists but is not being used. Even when removing our token limet and llm-emit metric policies we still get this buffering behavior. A month or 2 ago this was not the case.
* User's image

**

Mohammad Altaf 360 Reputation points Microsoft External Staff Moderator

2026-09-02T08:18:44.09+00:00
Hello @Jeffrey Bakker ,

Welcome to Microsoft Q&A,

We understand that Azure API Management is buffering the Server-Sent Events (SSE) response from the Azure OpenAI backend, resulting in the complete response being delivered only after generation is finished, even though buffer-response="false" is configured and the token limit/LLM metric policies have been removed.

To help further isolate the behavior, could you please provide the following details:

APIM SKU

APIM Region

Azure OpenAI Region

A sanitized APIM trace for an affected request

Confirmation whether the same request streams correctly when sent directly to the Azure OpenAI endpoint without APIM in the request path

According to Microsoft documentation, SSE responses should be relayed immediately when buffer-response="false" is configured. Additionally, certain diagnostics or response-processing configurations can impact streaming behavior.

Once we have the above information, we can further analyze where the buffering is occurring and advise on the appropriate resolution.

Thank you!
Mohammad Altaf 360 Reputation points Microsoft External Staff Moderator

2026-09-03T09:29:58.72+00:00

Hello @Jeffrey Bakker

We are following up on our previous request. Could you please share the requested details, including the APIM trace and the result of testing the Azure OpenAI endpoint directly without APIM?

This information will help us further investigate the SSE buffering behavior.

Thank you!
Mohammad Altaf 360 Reputation points Microsoft External Staff Moderator

2026-09-04T10:50:49.0633333+00:00

Hello @Jeffrey Bakker ,

I Just wanted to follow up on my previous comment. When you have a chance, could you please provide the requested details?

Thank you!

Your answer

Mohammad Altaf 360 Reputation points Microsoft External Staff Moderator

2026-09-02T08:18:44.09+00:00

Hello @Jeffrey Bakker ,

Welcome to Microsoft Q&A,

We understand that Azure API Management is buffering the Server-Sent Events (SSE) response from the Azure OpenAI backend, resulting in the complete response being delivered only after generation is finished, even though buffer-response="false" is configured and the token limit/LLM metric policies have been removed.

To help further isolate the behavior, could you please provide the following details:

APIM SKU

APIM Region

Azure OpenAI Region

A sanitized APIM trace for an affected request

Confirmation whether the same request streams correctly when sent directly to the Azure OpenAI endpoint without APIM in the request path

According to Microsoft documentation, SSE responses should be relayed immediately when buffer-response="false" is configured. Additionally, certain diagnostics or response-processing configurations can impact streaming behavior.

Once we have the above information, we can further analyze where the buffering is occurring and advise on the appropriate resolution.

Thank you!
Mohammad Altaf 360 Reputation points Microsoft External Staff Moderator

2026-09-03T09:29:58.72+00:00

Hello @Jeffrey Bakker

We are following up on our previous request. Could you please share the requested details, including the APIM trace and the result of testing the Azure OpenAI endpoint directly without APIM?

This information will help us further investigate the SSE buffering behavior.

Thank you!
Mohammad Altaf 360 Reputation points Microsoft External Staff Moderator

2026-09-04T10:50:49.0633333+00:00

Hello @Jeffrey Bakker ,

I Just wanted to follow up on my previous comment. When you have a chance, could you please provide the requested details?

Thank you!