HorizontalPodAutoscaler Logs in Log Analytics

What you are observing is consistent with known behavior in Azure Monitor Container Insights:

• By default, only Warning-level Kubernetes events are collected into KubeEvents
• Normal events (such as SuccessfulRescale) are not collected unless explicitly enabled
• Even with collect_all_kube_events = true, there are scenarios where Normal events may still not appear, due to agent behavior, configuration application, or platform limitations

1. Expected Behavior for Normal HPA Events

• The KubeEvents table supports both Normal and Warning event types (for example via the KubeEventType field) [learn.microsoft.com]
• However, Container Insights does not collect Normal events by default
• Microsoft guidance states: “By default, Normal event types aren't collected … unless the collect_all_kube_events ConfigMap setting is enabled.” [Query cont...soft Learn | Learn.Microsoft.com]

Normal HPA events (e.g., SuccessfulRescale) are expected to be ingested only when collect_all_kube_events = true is correctly applied

2. Known Limitations / Observed Behavior

Default filtering behavior

• Container Insights is designed to prioritize warning/failure events
• This is explicitly done to reduce ingestion volume and cost
• As a result, environments commonly show:
  • Warning events (FailedGetResourceMetric, etc.)
  • Missing Normal lifecycle events (e.g., scale-up, scheduling) [medium.com]

Normal events may still not appear even after enabling

From available documentation and observed behavior:

• Enabling collect_all_kube_events does not guarantee immediate or complete ingestion of all Normal events
• Possible contributing factors (supported by documentation context):
  • ConfigMap changes require time to propagate and agent restart [learn.microsoft.com]
  • Event ingestion is agent-driven and subject to pipeline filtering / volume controls
  • Kubernetes events are inherently ephemeral (short-lived) and can be missed during collection windows [learn.microsoft.com]

3. Verifying collect_all_kube_events is Applied

Step 1 - Please Confirm ConfigMap applied

You already validated:

• ConfigMap deployed to kube-system
• AMA pods restarted

This is with required behavior: Config update triggers rolling restart of AMA pods [learn.microsoft.com]

Step 2 — Validate agent-side application - The definitive validation is via AMA pod logs, not just ConfigMap presence

You should check:

kubectl logs -n kube-system <ama-logs-pod>

Always Look for:

• Config parsing success
• Event collection settings applied
• Errors related to event scraping

If the configuration is invalid or not applied, errors are surfaced in:

• KubeMonAgentEvents table (agent diagnostics) [github.com]

Step 3 — Validate ingestion query

Run:

KubeEvents

| where KubeEventType == "Normal"

| order by TimeGenerated desc

If no results, it confirms: Normal events are not being ingested (not just filtered in query)

4. Important Clarification on Design Behavior

• Kubernetes generates:
  • Normal events = high volume, transient
  • Warning events = actionable signal

Azure Monitor Container Insights prioritizes:

• Signal over volume (Warnings)
• Not all Normal events are guaranteed to be persisted

This explains:

• Why kubectl describe hpa shows SuccessfulRescale
• But Log Analytics does not retain or ingest it

5. Recommended Approach (Supported and Defensible)

If your goal is: “Surface HPA scaling reasons in Azure Monitor”

Then:

Option A (Supported within Container Insights)

• Continue using collect_all_kube_events = true
• Validate agent logs + ingestion as above
• Accept that coverage of Normal events may not be complete

Option B (Explicitly reliable approach)

• Use a custom event pipeline (e.g., Fluent Bit / custom collector)
• This allows:
  • Full capture of Kubernetes events (including Normal)
  • Control over ingestion into Log Analytics

This workaround is consistent with patterns described where default pipeline does not capture all events [medium.com]

Final Answer to Your Questions

1. Are Normal HPA events expected to be ingested?

• Yes, only when collect_all_kube_events = true is enabled
• Not guaranteed in all cases due to ingestion behavior

2. Known limitations / delays?

• Default behavior filters out Normal events
• Config changes take time and require agent restart
• Events are short-lived and may not always be captured
• Platform prioritizes Warning events over Normal

How to verify AMA is applying the setting?

• Check AMA pod logs (kubectl logs)
• Query KubeMonAgentEvents for config/parse errors
• Validate ingestion using KubeEvents | where KubeEventType == "Normal"

You are seeing (only HPA warnings in Log Analytics) is:

• Expected by design in many cases, even after enabling full collection
• Not necessarily a misconfiguration unless agent logs confirm otherwise

Correct me if I'm wrong, but you're basically saying that even with collect_all_kube_events = true set, KubeEvents is not a reliable place to collect Normal HPA logs?

If so, I'm going to have to go the custom pipeline route.

I want to surface HPA scaling reasons in a Log Analytics workbook - seems like this would be quite a standard request....

Greg Pakes - Yes even with collect_all_kube_events = true, KubeEvents is not a fully reliable source for capturing all Normal HPA events (like SuccessfulRescale).

1. By design, Azure Monitor Container Insights prioritizes Warning events and may filter or miss Normal events to reduce ingestion volume and cost.
2. Even when collect_all_kube_events = true is enabled:
   • It allows collection of Normal events, but
   • It does NOT guarantee complete or consistent ingestion of those events.
3. Additionally: (https://learn.microsoft.com/en-us/azure/aks/events?tabs=azure-cli)
   • Kubernetes events are ephemeral (~1 hour retention), so short-lived events can be missed during collection.

For reliable HPA scaling insights, use one of the below:

Recommended for consistency

Use a custom event pipeline, such as:

• Kubernetes event exporter
• Fluent Bit
• OpenTelemetry collector

This gives:

• Full capture of both Normal + Warning events
• Deterministic ingestion into Log Analytics

(These approaches explicitly collect events from the Kubernetes API instead of relying on Container Insights filtering