An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
Application pods running on a system node pool can contribute directly to the high memory utilization being seen on aks-agentpool-38991572-vmss000003.
The supported guidance is:
- Separate system and user workloads User workloads should run in a user node pool, not in the system node pool. A system node pool should be present, and user workloads should be isolated to a user node pool.
- Investigate which pods are consuming memory on the system node
Run:
Then inspect pods on the affected node:kubectl top node
This identifies the pods on that node with the highest memory usage.kubectl get pods --all-namespaces --output wide \ | grep aks-agentpool-38991572-vmss000003 \ | awk '{print $1" "$2}' \ | xargs -n2 kubectl top pods --namespace \ | awk 'NR==1 || NR%2==0' \ | sort -k3n \ | column -t - Review requests and limits on pods scheduled there
Check the node allocation details:
Then verify whether thekubectl describe node aks-agentpool-38991572-vmss000003fipropods have realistic memory requests and limits. In AKS, unrealistic or missing limits can contribute to memory saturation and OOMKilled behavior. - Move application workloads off the system node pool
Since
fiprois an application workload, it should be scheduled onto the user node pool instead ofaks-agentpool-*. The documented recommendation is to use user node pools for user workloads and keep system node pools for system pods. - If pods are already restarting or being evicted, check for memory pressure symptoms
Common signs include:
- unschedulable pods
- pod eviction
- node not ready
- OOMKilled events
- Prevent recurrence
Apply the AKS best practices from the documented guidance:
- set realistic memory requests and limits
- avoid overcommitting node memory
- use cluster autoscaler
- use horizontal pod autoscaler where appropriate
- isolate memory-heavy workloads to dedicated nodes by using node selectors, affinity, or anti-affinity
The fact that kubectl describe node ... | grep Taint returns Taints: <none> explains why application pods can land on that system node. With no taint shown, nothing in that output is preventing scheduling there.
References: