Memory cgroup out of memory: Killed process 1828809 (systemctl)

The log entry: Memory cgroup out of memory: Killed process <pid> (systemctl) indicates that the Linux kernel OOM (Out‑Of‑Memory) killer was invoked at the node level after a memory cgroup exceeded its enforced limit. This is a host‑level memory exhaustion event, not an application bug in systemctl itself.

In Azure Kubernetes Service (AKS), both Kubernetes pods and node system services run inside Linux memory cgroups. When total memory usage on the node grows beyond what is available for that cgroup, the kernel selects one process in the affected cgroup to terminate in order to reclaim memory.

Microsoft’s AKS OOM troubleshooting guidance -https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/availability-performance/troubleshoot-oomkilled-aks-clusters

Why systemctl was terminated - systemctl is part of systemd, which manages core services on the Linux node. Its termination in this case does not indicate a defect in systemd. Instead, it confirms that:

• The node experienced sustained memory pressure, and
• The kernel OOM killer selected a system process as the victim, which can occur once pod eviction and kubelet safeguards are insufficient to relieve memory pressure in time.

Microsoft documentation explicitly describes this scenario from a normal container‑level OOMKilled event. A container‑level OOM affects only the container that exceeded its limit, whereas node‑level memory exhaustion can impact system processes and overall node stability -https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/availability-performance/identify-memory-saturation-aks?tabs=browser

Based on Microsoft Learn and Azure Q&A guidance, this condition typically occurs when aggregate memory consumption on a node exceeds allocatable memory, which can happen due to one or more of the following supported scenarios:

• Pod memory overcommitment: Total pod memory usage (including bursts) exceeds the node’s allocatable memory. AKS reserves part of the node’s memory for the OS and Kubernetes components, so pods cannot consume full node capacity. [learn.microsoft.com]
• High‑memory workloads or memory leaks: One or more pods steadily grow in RSS memory usage without sufficient limits, increasing overall node pressure. [learn.microsoft.com]
• System or agent memory growth: Node‑level services (for example, logging, monitoring, or security agents) can contribute to memory saturation if not properly constrained. Microsoft recognizes this as a valid contributor to node memory pressure. [learn.microsoft.com]

The following validation steps are aligned with Microsoft’s documented AKS troubleshooting approach:

1. Confirm a host‑level OOM event
   • Review kernel logs on the affected node for Memory cgroup out of memory or oom-kill entries.
   • These logs confirm a kernel‑initiated OOM rather than a Kubernetes eviction. [learn.microsoft.com]
2. Check node memory pressure
   • Run kubectl describe node <node-name> and review:
   • Allocatable vs Capacity
   • Conditions: MemoryPressure
   • Nodes reporting sustained memory pressure are at risk of host‑level OOMs. [learn.microsoft.com], [learn.microsoft.com]
3. Identify high memory consumers
   • Use:
   • kubectl top node
   • kubectl top pod -A --sort-by=memory
   • Correlate top memory‑consuming pods to the affected node. [learn.microsoft.com]

To prevent recurrence, Microsoft supports the following corrective actions:

• Right‑size pod memory requests and limits Ensure that total requested memory per node remains well below allocatable memory to absorb usage spikes and prevent node exhaustion. [learn.microsoft.com]

Adjust node sizing or scale out If workloads legitimately require more memory, use larger VM SKUs or increase node count so that allocatable memory aligns with real usage patterns. [learn.microsoft.com]

Monitor and manage node memory saturation Use AKS Diagnostics and Container Insights to track memory consumption trends and detect saturation before it triggers kernel OOM events. [learn.microsoft.com]

Please check if provided response was helpful! please let me know if you have any queries.