AssetHoardingLatency: por que minha esteira de leads B2B no Azure está custando 22% mais sem motivo aparente?

The described behavior aligns with expected Azure OpenAI latency dynamics rather than hidden state or “token hoarding” between calls.

For Azure OpenAI, per-call latency is primarily governed by the latency formula:

TTLT = TTFT + (TBT × Tokens Generated)

Where:

• TTFT: Time to First Token
• TBT: Time Between Tokens
• Tokens Generated: output token count
• TTLT: total time to last token (end-to-end latency)

Key implications from the documented behavior:

1. Latency scales with tokens, not hidden state
   The documentation explicitly models latency as a function of prompt size, generated tokens, and load on the deployment. There is no documented persistent context or “token accumulation” across stateless calls. Increases in TTLT are often fully explained by increases in output tokens or by deployment load.
2. Load-driven degradation is expected at higher concurrency
   Latency of completion/chat requests varies with:
   • Model used
   • Prompt token count
   • Generated token count
   • Overall load on the deployment and system
   As batch processing progresses (for example, 5k requests/day in bursts), the deployment can experience:
   • Increased queueing delay and concurrent request volume
   • Increased Time Between Tokens (TBT) when the deployment is under load
   This can produce exactly the pattern described: early requests are fast, later ones in the same wave become progressively slower as utilization rises.
3. How to verify if this is load/throughput, not hidden state
   Use Azure Monitor metrics on the Azure OpenAI resource to correlate latency with tokens and utilization: For non-streaming workloads:
   1. In the Azure portal, open the Azure OpenAI resource → Monitoring > Metrics.
   2. Add Time to Last Byte (AzureOpenAITTLTInMS) and split by ModelDeploymentName.
   3. Add a second chart for Generated Completion Tokens (GeneratedTokens) for the same time range.
   4. Compare:
      • If TTLT and GeneratedTokens rise together, the latency increase is explained by token volume.
      • If TTLT rises without a token-count increase, check for capacity pressure:
        • On provisioned (PTU-managed) deployments: chart Provisioned-managed Utilization V2 (AzureOpenAIProvisionedManagedUtilizationV2).
        • On pay-as-you-go deployments: check Azure OpenAI Requests (AzureOpenAIRequests) for 429s and concurrent request volume.
   For streaming workloads:
   1. In the same Metrics blade, add Time to Response (AzureOpenAITimeToResponse) split by ModelDeploymentName for TTFT.
   2. Add Time Between Tokens (AzureOpenAINormalizedTBTInMS) to see per-token throughput.
   3. Add Processed Prompt Tokens (ProcessedPromptTokens) to track prompt size.
   4. Interpret:
      • If Time to Response rises while prompt size is flat, this points to deployment utilization / concurrency.
      • If Time Between Tokens rises, the deployment is likely under load.
   If these metrics explain the pattern, the behavior is load/throughput-driven, not stateful accumulation.
4. Why RAG + context reduction improved both latency and cost
   The documentation notes:
   • Latency is largely dependent on model type, prompt tokens, generated tokens, and load.
   • Each prompt token adds some time, but each incremental generated token typically adds more.
   • Larger prompts increase Time to First Token.
   By introducing a pre-qualification RAG step that reduces effective context by ~40%, the workload likely:
   • Reduced Processed Prompt Tokens per call (smaller prompts)
   • Potentially reduced Generated Completion Tokens (more focused outputs)
   • Reduced overall Total TPM (tokens per minute), lowering pressure on the deployment
   This directly explains:
   • Lower average latency (less work per request, less load)
   • Lower token cost (fewer input and/or output tokens)
   These effects are consistent with the documented guidance that optimizing prompt size and workload shape improves both performance and cost.
5. Configuration and architectural levers to consider
   Based on the official guidance, the following knobs are relevant for a high-volume B2B lead-qualification pipeline:
   • Separate workloads by deployment: Mixing different workloads on the same endpoint can hurt latency due to batching and cache contention. Use separate deployments for distinct workloads when possible.
   • Prompt size control: Keep prompts as small as possible while preserving quality. Large prompts increase TTFT and overall latency.
   • Throughput sizing:
     • For provisioned deployments, ensure PTUs are sized for the actual Total TPM (input + output) and call rate.
     • Use Azure Monitor’s Processed Prompt Tokens and Generated Completion Tokens over multi-week windows to estimate real throughput and capacity needs.
   • Streaming: For conversational or user-facing scenarios, streaming can improve perceived latency (tokens arrive earlier), though total TTLT remains governed by the same formula.
   • Content filters: Content filtering adds safety but also latency. For lower-risk use cases, requesting adjustments to content filtering policies can reduce overhead, if appropriate for the scenario.
6. How to distinguish real regressions from expected behavior
   The documentation explicitly recommends using the latency formula and metrics to decide whether there is a regression:
   • If p95/p99 latency and TTLT can be explained by token counts and utilization, the system is behaving as expected.
   • If latency increases without corresponding changes in tokens or load, that is when a deeper investigation or support case is warranted.

In summary, the observed monotonic latency increase across a batch is consistent with load and token-driven behavior described by Azure OpenAI’s latency model, not with undocumented persistent context. The RAG-based “Protocolo Hidra” approach is aligned with recommended practices: reducing prompt size and total tokens per request to improve both latency and cost.

References:

• Performance and latency
• AI App Architecture for Startups