Mistral-medium-3-5 on Azure AI Foundry is useless: context length capped far below spec, intermittent OOM under reasoning_effort=high, and reasoning output misrouted to reasoning_content (likely missing vLLM launch flags)

Question

Mistral-medium-3-5 on Azure AI Foundry is useless: context length capped far below spec, intermittent OOM under reasoning_effort=high, and reasoning output misrouted to reasoning_content (likely missing vLLM launch flags)

Jesper Alkestrup - The Tech Collective 25

Resource details:

Model: mistral-medium-3-5
Tested against both DataZoneStandard (Sweden Central, capacity 500 / 500k TPM) and GlobalStandard deployments — identical results on both, ruling out a region- or SKU-specific cause.
Also tested directly in the Foundry portal's own Chat playground (not just via API) — same underlying failure.

Issue 1 — Input length rejected at a few hundred tokens, far below the documented 128K–256K context window

Plain requests with no extra parameters fail once input exceeds roughly 500–600 tokens:

HTTP 400: Input length (674 tokens) exceeds the maximum allowed length (564 tokens). Use a shorter input or enable --allow-auto-truncate.

The reported ceiling isn't fixed — we've seen 564, 3606, and 3639 tokens across different requests. Reproduced identically via: Azures own unmodified docs sample code, a second independently GUI-created deployment, and confirmed on both DataZoneStandard and GlobalStandard. Also reproduces in the Foundry portal Chat playground itself — any prompt over ~600 tokens there returns "An unexpected error occurred. Check your model's config and try resubmitting the request."

Issue 2 — ~50% failure rate with "out of memory" errors, specific to reasoning_effort="high"

Passing extra_body={"reasoning_effort": "high"} (documented as supported for this model) succeeds roughly half the time; the rest fail with:

HTTP 500: Out of memory even after retracting all other requests in the decode batch. Aborting the last request.

Not seen on plain requests or reasoning_effort="none" — only when the model is asked to actually reason, consistent with decode-batch/KV-cache pressure under longer generated sequences.

Issue 3 — Reasoning output entirely misrouted to reasoning_content, content left empty — likely a vLLM launch misconfiguration

When reasoning_effort="high" succeeds, the full response (both the reasoning trace and the final answer) lands in reasoning_content; content is empty. Happens in both streaming and non-streaming modes. This matches vllm-project/vllm#30139 exactly, where Mistral reasoning models need specific vLLM server launch flags to correctly split reasoning from the final answer:

--reasoning-parser=mistral
--config_format=mistral
--load_format=mistral
--tool-call-parser=mistral

If this deployment's backend is missing one or more of these flags, that would explain both this symptom and plausibly relates to Issue 1/2's capacity behavior. Could you confirm the launch configuration for this model on your end?

Can you confirm whether these issues are known, if they arre currently-being-addressed and share a resolution timeline if its planned?

0 comments

1 answer

Your answer

Answer 1

Hello @Jesper Alkestrup - The Tech Collective ,
Greetings,
Issues Analysis :

Issue 1: Input length being rejected far below the documented 128K–256K context window
The root cause appears to be an incorrectly configured inference backend (likely a vLLM-style container) with a significantly reduced max_model_len or context limit. This configuration enforces much lower input token limits ranging from 564 to 3,600 tokens despite the model’s documented support for 256K context. These limits and the appearance of --allow-auto-truncate in error messages are artifacts of the server-side configuration, not something controlled via client parameters.

This limitation is not something you can fix from your client side; it must be addressed by updating or redeploying the backend model container with correct configuration. As noted in the Q&A, redeploying from the catalog sometimes picks up fixes if they’ve been applied in the backend.

Issue 2: ~50% failure rate with “out of memory” (OOM) errors when reasoning_effort="high"
When reasoning_effort="high" is used, you’re effectively invoking deeper, multi-stage reasoning which increases compute and memory pressure, especially on the K/V cache during decoding. The intermittent OOM errors are consistent with capacity constraints under heavier inference load. This aligns with the deployment not being scaled or configured to reliably support high-effort reasoning workloads.

Issue 3: Reasoning output being misrouted entirely into reasoning_content, leaving content empty
This symptom both reasoning trace and final answer landing in reasoning_content and none in content is indicative of missing vLLM launch flags. Mistral reasoning models require specific vLLM server startup parameters (e.g., --reasoning-parser=mistral, --config_format=mistral, --load_format=mistral, --tool-call-parser=mistral) to correctly split reasoning vs final output. If those flags are missing, the backend will mis-route output.

Recommended Next Steps

Redeploy the model cleanly from the Foundry catalog

In Azure AI Foundry, delete the current Mistral‑medium‑3‑5 deployment and create a fresh one. Backend configuration issues are sometimes resolved in new deployments when Microsoft updates the model container template.

Escalate to Microsoft Foundry support / engineering

Provide Microsoft support with the specifics of the behaviors you’re seeing (token limit mismatches, OOM failures, routing behavior). Ask if the backend container for Mistral‑medium‑3‑5 can be updated to:

Correctly set max_model_len to match the advertised context window.

Include the required vLLM flags for proper reasoning parsing.

Scale memory configuration under reasoning_effort="high" to avoid OOM.

Test behavior post-redeployment
After redeploying, run controlled tests:

Gradually increase prompt length and observe token limits reported.

Use reasoning_effort="high" and monitor memory usage and successful runs.

Verify that responses include output in both content and reasoning_content as expected.

Mitigate immediate impact
If high-effort reasoning is critical and OOM remains frequent, consider setting reasoning_effort="none" or "low" temporarily while backend capacity improvements are in progress.

Break long prompts into smaller segments or summarizations to stay under the current enforced limits

Help make this community better for everyone: if this answer resolved your issue, please accept it or upvote it. If not, share more details in a comment so we can continue the discussion and find the right solution.