Application Pods are running on AKS System Node Causing High Memory Utilization.

2026-07-28T17:35:24.86+00:00

System node pool : aks-agentpool-38991572-vmss000002, aks-agentpool-38991572-vmss000003

User node pool: aks-workerpool-38991572-vmss000003,000004,000005

The system node aks-agentpool-38991572-vmss000003 is showing high memory utilization.

Application (fipro) pods are running on the system node.

kubectl describe node aks-agentpool-38991572-vmss000003 | grep Taint returns Taints: <none>

Azure Kubernetes Service
Azure Kubernetes Service

An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.


3 answers

Sort by: Most helpful
  1. SUNOJ KUMAR YELURU 18,736 Reputation points MVP Volunteer Moderator
    2026-07-31T16:38:32.3533333+00:00

    Hello @Jena, Tapas Kumar (893) (EXT),

    Thank you for using Q&A forum.

    AKS does not automatically taint system node pools — you have to do it explicitly. Here's how to fix it properly.

    Fix 1 — Add the Standard AKS Taint to the System Node Pool (Recommended)

    bash

    az aks nodepool update \

    --resource-group <your-rg> \

    --cluster-name <your-cluster> \

    --name agentpool \

    --node-taints CriticalAddonsOnly=true:NoSchedule

    This is the standard AKS taint. System components like CoreDNS already carry the matching toleration, so they won't be affected. Your fipro pods don't have that toleration, so they'll be evicted and move to the user node pool automatically.

    Verify the taint was applied:

    bash

    kubectl describe node aks-agentpool-38991572-vmss000003 | grep Taint

    Expected: Taints: CriticalAddonsOnly=true:NoSchedule

    1. Apply the taint to the system node pool via az aks nodepool update
    2. Drain vmss000003 manually to evict existing pods immediately (don't wait for natural eviction)
    3. Update the fipro deployment with nodeSelector or nodeAffinity
    4. Uncordon the system node
    5. Do the same taint check on vmss000002 — if it also has Taints: <none>, apply the same fix preemptively

    The taint alone solves the immediate problem.


    If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

    Was this answer helpful?

    0 comments No comments

  2. kagiyama yutaka 5,330 Reputation points
    2026-07-29T05:38:30.3666667+00:00

    I think this is happening because the system node shows Taints: <none>, and Kubernetes can place regular application pods there. Checking the fipro pod’s tolerations and nodeSelector is the right way to see why it landed on that node, and adding a taint to the system node and directing fipro to the user pool will keep it off the system node.

    Was this answer helpful?

    0 comments No comments

  3. AI answer

    2026-07-28T17:35:46.1066667+00:00

    Application pods running on a system node pool can contribute directly to the high memory utilization being seen on aks-agentpool-38991572-vmss000003.

    The supported guidance is:

    1. Separate system and user workloads User workloads should run in a user node pool, not in the system node pool. A system node pool should be present, and user workloads should be isolated to a user node pool.
    2. Investigate which pods are consuming memory on the system node Run:
         kubectl top node
      
      Then inspect pods on the affected node:
         kubectl get pods --all-namespaces --output wide \
             | grep aks-agentpool-38991572-vmss000003 \
             | awk '{print $1" "$2}' \
             | xargs -n2 kubectl top pods --namespace \
             | awk 'NR==1 || NR%2==0' \
             | sort -k3n \
             | column -t
      
      This identifies the pods on that node with the highest memory usage.
    3. Review requests and limits on pods scheduled there Check the node allocation details:
         kubectl describe node aks-agentpool-38991572-vmss000003
      
      Then verify whether the fipro pods have realistic memory requests and limits. In AKS, unrealistic or missing limits can contribute to memory saturation and OOMKilled behavior.
    4. Move application workloads off the system node pool Since fipro is an application workload, it should be scheduled onto the user node pool instead of aks-agentpool-*. The documented recommendation is to use user node pools for user workloads and keep system node pools for system pods.
    5. If pods are already restarting or being evicted, check for memory pressure symptoms Common signs include:
      • unschedulable pods
      • pod eviction
      • node not ready
      • OOMKilled events
    6. Prevent recurrence Apply the AKS best practices from the documented guidance:
      • set realistic memory requests and limits
      • avoid overcommitting node memory
      • use cluster autoscaler
      • use horizontal pod autoscaler where appropriate
      • isolate memory-heavy workloads to dedicated nodes by using node selectors, affinity, or anti-affinity

    The fact that kubectl describe node ... | grep Taint returns Taints: <none> explains why application pods can land on that system node. With no taint shown, nothing in that output is preventing scheduling there.


    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.