Production Latency Issue – Elevated Azure AI Search Query Latency from AKS (Request for Investigation + Call)

The latency gap between local calls and AKS calls to Azure AI Search can be broken down and investigated using the built‑in metrics and techniques for Azure AI Search, plus standard AKS diagnostics. The key is to separate:

1. Time spent in the Azure AI Search service
2. Time spent on the network path (AKS → Search)
3. Time spent in the client (container, SDK, retries, etc.)

Below is a concrete investigation path and optimization guidance based on the documented behavior of Azure AI Search.

1. Measure server vs. network latency per query

Use the Azure AI Search REST response headers to distinguish service processing time from end‑to‑end latency:

1. From an AKS pod, run the exact same REST query using a REST client (for example, curl or a simple test container) and inspect the HTTP response headers.
2. In the response, locate the elapsed-time header (milliseconds). This is the time Azure AI Search took to execute the query inside the service.
3. Compare it to the total round‑trip time reported by the client (for example, your HTTP client’s timing or the REST tool’s reported duration).

The difference between round‑trip time and elapsed-time is the network + client overhead. For example, if:

• elapsed-time ≈ 150 ms
• Round‑trip from AKS ≈ 500 ms

Then ~350 ms is outside the search service and must be investigated in AKS networking or client behavior.

This technique is explicitly recommended to isolate network latency from service processing latency.

“Azure AI Search includes in every response the number of milliseconds it takes to complete the query, visible in the Headers tab, in elapsed-time… Using these two values… we see the search service took 21 ms… and the entire client round-trip request took 125 ms. By subtracting these two numbers we can determine that it took 104-ms additional time to transmit the search query to the search service and to transfer the search results back to the client.”

Repeat this measurement:

• From local environment → Search
• From AKS → Search

This will show whether the extra ~300–400 ms is in the service or in the network/client path from AKS.

2. Check query performance and throttling in Azure AI Search

In the Azure portal for the search service:

1. Go to Monitoring → Metrics.
2. Inspect Search Latency:
   • Look at average, max, and distribution during the time windows when AKS traffic is active.
   • If Search Latency is low (similar to local tests), the service is not the bottleneck.
3. Inspect Throttled queries (ThrottledSearchQueriesPercentage):
   • If throttling is occurring, clients may be experiencing retries and backoff, which increases effective latency.

“Service-wide, query performance is measured as search latency and throttled queries. These metrics are also available on the Monitoring tab.”

If you see:

• Low Search Latency but high end‑to‑end latency from AKS → focus on AKS networking and client behavior.
• Elevated Search Latency during AKS load windows → the service is under heavier load when AKS is active.

3. Analyze load, indexing, and background processing

Higher latency from AKS may coincide with higher query volume or indexing activity coming from production workloads.

1. Use the Kusto‑based queries from the Azure AI Search performance analysis guidance to visualize:
   • Average Query Latency over time
   • Queries Per Minute (QPM)
   • Indexing Operations Per Minute (OPM)
2. Correlate spikes in latency with:
   • Spikes in QPM (query load)
   • Spikes in OPM (indexing load)

“Indexing uses the same resources as search queries. If you're indexing a large amount of content, you can expect to see latency grow as the service tries to accommodate both workloads.”

“Spikes might occur in response to indexing or high query rates, but could also occur during merge operations… Merging shards is fast, but also resource intensive and thus has the potential to degrade service performance.”

If AKS production is driving:

• Higher sustained QPS than local tests, or
• Concurrent heavy indexing,

then the service may be closer to capacity under AKS load, causing higher latency.

Actions if this is observed:

• Temporarily pause or reschedule heavy indexing away from peak query times.
• Consider capacity adjustments (replicas/partitions or tier) per the capacity planning guidance referenced in the performance article.

4. Distinguish AKS networking issues

Once service‑side metrics show whether the search service itself is fast or slow, focus on AKS networking if elapsed-time is low but end‑to‑end latency is high from AKS only.

Use the same measurement pattern as for AKS API server diagnostics: measure latency per client and over time.

From AKS:

• Run repeated REST calls to the search endpoint from a test pod and log:
  • Start time and end time
  • elapsed-time header
  • Any HTTP status codes (especially 429 or 503)

From local:

• Run the same script and compare distributions.

This mirrors the approach used in AKS API server diagnostics, where Kusto queries are used to compute average latency per user agent over time.

“To identify the average latency of API server requests per user agent, as plotted on a time chart, run the following query… extend latency… summarize avg(latency) by UserAgent, bin(start_time, 5m).”

While that specific query is for AKS API server, the same pattern (compute per‑client average latency over time) can be applied to your own logs for calls from AKS to Azure AI Search.

If AKS shows consistently higher network latency to the search endpoint than local, this indicates a networking path or configuration issue (for example, private endpoint routing, SNAT, or other infrastructure). That part would be investigated with Azure networking support.

5. Plan for query spikes and client behavior

If AKS production traffic is bursty, ensure that client behavior is not amplifying latency:

• Check for 429 responses from Azure AI Search (Too Many Requests). When the system reaches capacity, it returns 429s and expects clients to back off.
• Ensure the client implements proper backoff and retry logic (for example, exponential backoff with jitter) so that retries do not create a “retry storm” that increases latency.

“Latency stays low when the load is moderate, and gradually increases as you approach the maximum QPS capacity. When the system reaches 100% QPS capacity, it starts returning 429 errors… As a best practice… If you use the REST API, implement exponential backoff with jitter.”

If AKS clients are aggressively retrying without backoff, effective latency will be much higher than the underlying service latency.

6. Best‑practice checklist for optimizing latency from AKS

Based on the documented behavior of Azure AI Search:

1. Measure and monitor
   • Use elapsed-time vs. client round‑trip to separate service vs. network latency.
   • Monitor Search Latency and Throttled queries in the portal.
   • Use Kusto‑based queries for QPM and OPM to correlate load with latency.
2. Control indexing and background load
   • Avoid heavy indexing during peak query periods.
   • Be aware that shard merge operations after indexing can cause short latency spikes.
3. Stay below critical QPS limits
   • Monitor QPM/QPS from AKS workloads.
   • Implement exponential backoff with jitter for REST clients to avoid retry storms.
4. Validate AKS networking path
   • Compare elapsed-time and round‑trip from AKS vs. local.
   • If service latency is low but AKS round‑trip is high, engage networking diagnostics (VNet, private endpoint routing, SNAT, etc.) with Azure support.

7. Live troubleshooting session

For a live call with engineering/support to walk through architecture, timings, and logs, open a support request through the Azure portal for the Azure AI Search resource and attach:

• Example queries and headers (including elapsed-time).
• Metrics screenshots for Search Latency and Throttled queries.
• QPM/OPM charts around the time windows with elevated latency.
• AKS‑side measurements showing the difference between local and AKS round‑trip times.

Support can then correlate your correlation IDs and timestamps with backend telemetry and, if needed, involve networking teams to investigate the AKS → Search path.

References:

• Analyze performance in Azure AI Search
• Monitor query requests in Azure AI Search
• AI Search performance guide
• Troubleshoot API server and etcd problems in AKS