How to fix AKS pod transitioning into completed state due to preemption by another critical pod

Question

How to fix AKS pod transitioning into completed state due to preemption by another critical pod

faaz faaz 0

We have a service running in our cluster with its deployment configured for 2 replicas. After some time, the pods of this service get preempted to make room for higher-priority pods. When this happens, the pod transitions into a Completed state, and a new pod is created to replace it.

As a result, we continuously receive notifications indicating that the pod is not in a Running state (it shows as Completed in ArgoCD).

This behavior was initially observed only in only one service, but has now started affecting 2 more microservices as well. No other microservices or pods are exhibiting this issue.

Current status (debugging)

We are enabling AKS logs in the Log Analytics workspace to better understand which higher-priority pod(s) are preempting these affected services.

Reason for concern

The issue has become predictable — only these three services are consistently impacted.
We suspect a potential misconfiguration, but the root cause has not yet been identified.

Error message:

Pod was preempted by Kubelet to accommodate a critical pod.

Nikhil Duserla 9,950 Reputation points Microsoft External Staff Moderator

2025-09-29T16:21:37.4566667+00:00

Hello faaz faaz,

When you try to perform a new operation on an AKS cluster, you receive one of the following error messages:

Code: "AKSOperationPreempted"

This error usually occurs when an in-progress operation is interrupted by a subsequent operation that was issued before the in-progress operation is finished. The error will indicate the subsequent operation, which can be a delete or any other operation.

To resolve this issue, use one of the following methods. Once there are no operations running, you can try to run your operation again- https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-codes/aksoperationpreempted-error?source=recommendations#solution

If you have any further queries, do let us know.
Nikhil Duserla 9,950 Reputation points Microsoft External Staff Moderator

2025-10-01T20:23:51.7366667+00:00

Hello faaz faaz,

Checking to see whether you have resolved the issue with the solution provided above, or if you need any further assistance, please let us know. We are happy to help.
Anonymous

2025-10-01T20:40:53.9366667+00:00

@Nikhil Duserla , I don't think this is about a deployment operation being pre-empted by a previously running operation. This is about Kubernetes kicking out a running pod due to resources needed for a higher priority pod. A pod's default is "low priority", and so can be pre-empted.

The pod is then told to gracefully terminate, and then it's state transitions to Completed, which is setting off Argo. I suspect Argo needs to be configured to ignore this normal (yet, perhaps personally annoying) K8s behavior, but I'm not an Argo-dude.

Or bludgeon the problem by setting the deployment to never be pre-empted ;), but there's a reason a higher priority pod needs the resources, k8s can't find it, and K8s is kicking it out the existing lower priority pod. But the complaint seems to be Argo is alerting on this.
Nikhil Duserla 9,950 Reputation points Microsoft External Staff Moderator

2025-10-07T17:21:40.1366667+00:00

Hello faaz faaz,

Thank you for your response. ArgoCD is not a Microsoft product, and we do not provide support for it. Please reach out to the ArgoCD customer support team for further assistance. Thank you!
Joseph Tran 4,095 Reputation points Independent Advisor

2025-10-09T17:44:52.27+00:00
Hi, There

Your services have 2 replicas each, but their pods keep getting preempted — Kubernetes is evicting them to make room for higher-priority pods. When this happens, they show up as Completed in ArgoCD, triggering alerts.

This now affects three specific services, likely due to:

Lower priority classes or no priority set.

Resource pressure on the nodes.

What to check:

Compare priority classes of these pods vs. others.

Review resource requests/limits — they might be too low.

Check node usage (kubectl top nodes).

Create a slightly higher PriorityClass for these services.

Add capacity or enable autoscaling if the cluster is tight on resources.

Your answer

Nikhil Duserla 9,950 Reputation points Microsoft External Staff Moderator

2025-09-29T16:21:37.4566667+00:00

Hello faaz faaz,

When you try to perform a new operation on an AKS cluster, you receive one of the following error messages:

Code: "AKSOperationPreempted"

This error usually occurs when an in-progress operation is interrupted by a subsequent operation that was issued before the in-progress operation is finished. The error will indicate the subsequent operation, which can be a delete or any other operation.

To resolve this issue, use one of the following methods. Once there are no operations running, you can try to run your operation again- https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-codes/aksoperationpreempted-error?source=recommendations#solution

If you have any further queries, do let us know.
Nikhil Duserla 9,950 Reputation points Microsoft External Staff Moderator

2025-10-01T20:23:51.7366667+00:00

Hello faaz faaz,

Checking to see whether you have resolved the issue with the solution provided above, or if you need any further assistance, please let us know. We are happy to help.
Anonymous

2025-10-01T20:40:53.9366667+00:00

@Nikhil Duserla , I don't think this is about a deployment operation being pre-empted by a previously running operation. This is about Kubernetes kicking out a running pod due to resources needed for a higher priority pod. A pod's default is "low priority", and so can be pre-empted.

The pod is then told to gracefully terminate, and then it's state transitions to Completed, which is setting off Argo. I suspect Argo needs to be configured to ignore this normal (yet, perhaps personally annoying) K8s behavior, but I'm not an Argo-dude.

Or bludgeon the problem by setting the deployment to never be pre-empted ;), but there's a reason a higher priority pod needs the resources, k8s can't find it, and K8s is kicking it out the existing lower priority pod. But the complaint seems to be Argo is alerting on this.
Nikhil Duserla 9,950 Reputation points Microsoft External Staff Moderator

2025-10-07T17:21:40.1366667+00:00

Hello faaz faaz,

Thank you for your response. ArgoCD is not a Microsoft product, and we do not provide support for it. Please reach out to the ArgoCD customer support team for further assistance. Thank you!
Joseph Tran 4,095 Reputation points Independent Advisor

2025-10-09T17:44:52.27+00:00

Hi, There

Your services have 2 replicas each, but their pods keep getting preempted — Kubernetes is evicting them to make room for higher-priority pods. When this happens, they show up as Completed in ArgoCD, triggering alerts.

This now affects three specific services, likely due to:

Lower priority classes or no priority set.

Resource pressure on the nodes.

What to check:

Compare priority classes of these pods vs. others.

Review resource requests/limits — they might be too low.

Check node usage (kubectl top nodes).

Create a slightly higher PriorityClass for these services.

Add capacity or enable autoscaling if the cluster is tight on resources.