Preempted Batch node stays in the pool, misleads Batch autoscaling

Question

Preempted Batch node stays in the pool, misleads Batch autoscaling

JeffreyCMI 116

I run Azure Batch pools using spot nodes. Occasionally a node is preempted, which itself is not a problem. But I struggle to wrap my head around what happens to that node after it's preempted.

I observe is that the preempted node remains in the pool, and I have three questions about this:

What happens to a Batch pool node after it gets preempted, and when? Do they ever leave the pool? Do they ever try to "recover" out of their Preempted state to Running again? I have not observed either of these; it seems that they just remain Preempted forever.
Am I being billed for the preempted node as long as it remains part of the Batch pool?
Autoscaling does not discount this preempted node when it checks if it has enough nodes or if it needs to scale-out. In my case, my Job had 1 active Task, the autoscaling rule therefore correctly set $TargetLowPriorityNodes -1But no new nodes were added because there was already 1 node in the pool; it didn't account for the fact that it was preempted so nothing could run on it. Assuming preempted nodes remain in that state indefinitely, is there a workaround to tell the scaling not to count them?

Here's max Preempted node count over the span of 2 days. On Tue 21, the count rose from 0 -> 1, when a very large pool had just 1 (or 2?) nodes preempted. But then that preempted node remains forever, until I manually deleted it. I can't explain the intermittent observations of count=2, but I wasn't actively watching the pool at those times.

User's image

VarunTha 15,090 Reputation points Microsoft External Staff Moderator

2026-07-22T23:48:41.67+00:00
Hello JeffreyCMI,

This is a great question—when you use Spot VMs in an Azure Batch pool, preemption has a specific lifecycle, and it can definitely confuse both how the pool looks and how autoscaling behaves.

Here are direct answers to your 3 questions, based on the provided docs.

1) What happens to a Batch pool node after it gets preempted? Do they leave the pool / recover to Running?

When preemption happens on a Spot VM, Batch does the following:

The VM’s node state is updated to Preempted (in some scenarios it may go through Unusable first, then to Preempted).

The VM is effectively deleted, which means any data stored locally on the VM is lost.

A list nodes call for the pool still returns the preempted nodes.

The Batch service continually tries to reach the target number of Spot nodes in the pool. When replacement capacity is found, nodes are reinitialized and go through Creating and Starting before becoming available for scheduling.

Importantly: preempted nodes stay in the pool output/state (at least from what you’re observing and per the “list nodes still returns preempted nodes” behavior). The docs don’t describe automatic “recovery” of the same VM back to Running; instead, Batch replaces capacity by provisioning new nodes.

So your observation—preempted nodes remaining Preempted forever until manually removed—matches the documented behavior that preempted nodes are still returned as nodes on the pool while the service works to maintain the target via replacement nodes.

2) Am I being billed for the preempted node as long as it remains in the Batch pool?

Per the Batch best practices doc:

You are billed for every compute node in the pool, regardless of the state it’s in.

So if the node remains counted in the pool as a compute node (including being listed as preempted), it’s consistent with being billed while that node is still part of the pool’s compute inventory.

3) Why doesn’t autoscaling discount the preempted node? Any workaround?

Autoscale behavior here can feel surprising, because the preempted node still exists as a node entry in the pool (and list nodes returns it).

From the provided docs, we don’t have a specific statement that autoscale formulas/logic exclude preempted nodes from the node count when evaluating the target. What we can say from the docs:

Spot preemption leads to the preempted node being in Preempted state.

Batch continually attempts to reach the target number of Spot nodes by finding replacement capacity.

But if your pool already shows “enough nodes” because the pool still contains the preempted node entry, it’s plausible the autoscale evaluation won’t reduce the target or will think capacity already exists.

Workaround (based on what’s documented)

The provided documentation doesn’t include a direct switch to “exclude preempted nodes from autoscale counts.” The docs do note that:

Pools/compute-node lifetime is something you manage (explicitly or via service features like autoscale/autopool).

You can also resize pools.

So the practical workaround would be to:

manually manage node cleanup for preempted nodes (since you’re already deleting them to clear the stuck preempted count), or

adjust your scaling/target strategy so that replacement capacity is still provisioned even when preempted nodes remain listed.

However, since the docs provided don’t specify an autoscale parameter that discounts preempted nodes, I can’t give a guaranteed “do X setting” answer from the provided material.

References :

Use Spot VMs with Batch workloads (Spot preemption behaviors; states; replacement attempts; list nodes returns preempted nodes)

Azure Batch best practices (billing: billed for every compute node regardless of state; pool lifetime guidance)

Nodes and pools in Azure Batch (pool/compute node lifetime concepts; pool management)

Azure Batch pool and node errors (autoscale evaluation/resizing considerations; use Evaluate Pool Auto Scale / resize complete events).
JeffreyCMI 116 Reputation points

2026-07-23T01:39:36.69+00:00

Thanks. I checked and confirmed we continue getting billed for preempted spot nodes. It sounds like the ONLY resolution to that is that we need to do our own monitoring of the pool and manually delete preempted nodes. I see no benefit to keep them around, and 2 reasons it hurts us to keep them around.
VarunTha 15,090 Reputation points Microsoft External Staff Moderator

2026-07-23T23:34:59.4433333+00:00

Hello JeffreyCMI,

Thank you for confirming. I understand the concern, as the preempted node remains listed, may continue to incur charges, and can affect the expected autoscale behaviour.

One clarification: Azure Batch may try to restore a preempted Spot node for up to 48 hours, although restoration is not guaranteed. If capacity becomes available, the node can be reinitialised using the same node ID.

Manual removal is a possible workaround, but the documentation does not state that it is the only supported option. Autoscale also provides the $PreemptedNodeCount variable, which may be considered when designing the autoscale formula.

3 answers

Your answer

JeffreyCMI 116 Reputation points

2026-07-23T01:39:36.69+00:00

Thanks. I checked and confirmed we continue getting billed for preempted spot nodes. It sounds like the ONLY resolution to that is that we need to do our own monitoring of the pool and manually delete preempted nodes. I see no benefit to keep them around, and 2 reasons it hurts us to keep them around.
VarunTha 15,090 Reputation points Microsoft External Staff Moderator

2026-07-23T23:34:59.4433333+00:00

Hello JeffreyCMI,

Thank you for confirming. I understand the concern, as the preempted node remains listed, may continue to incur charges, and can affect the expected autoscale behaviour.

One clarification: Azure Batch may try to restore a preempted Spot node for up to 48 hours, although restoration is not guaranteed. If capacity becomes available, the node can be reinitialised using the same node ID.

Manual removal is a possible workaround, but the documentation does not state that it is the only supported option. Autoscale also provides the $PreemptedNodeCount variable, which may be considered when designing the autoscale formula.

Answer 1

hi JeffreyCMI & thx for sharing urs issue here at Q&A portal,

What u're seeing is generally expected for Spot nodes. When a Spot VM is preempted, the Batch node transitions to the Preempted state and the tasks running on it are terminated. The node doesn't immediately become usable again, and it won't automatically recover to Running. Whether it's eventually removed or replaced depends on the pool configuration and autoscale evaluation. (learn.microsoft.com) u aren't billed for compute while the underlying Spot VM has been evicted, but the presence of the Batch node object in the pool is separate from compute billing. The node record can remain even though there is no active VM behind it.

The autoscale behavior is the part that stands out. A permanently Preempted node continuing to satisfy $TargetLowPriorityNodes so that no replacement node is allocated isn't what I'd expect. Normally the pool should converge back to the target number of usable Spot nodes. As a workaround, many customers periodically remove nodes that remain in the Preempted state so the autoscaler can provision replacements. If the node stays preempted indefinitely and continues to count toward the target, I'd treat that as a potential Batch service issue rather than expected behavior.

I'd recommend opening a Batch support case with the pool ID, account name, region, autoscale formula, and timestamps where the node remained Preempted while $TargetLowPriorityNodes was still 1 and no replacement was allocated. The Batch team can verify from backend telemetry whether the autoscaler is incorrectly counting preempted nodes toward the target or whether the pool has become stuck.

rgds,

Alex

&

If my answer was helpful pls mark it and additional thx if u follow me at Q&A portal

and at my blog https://ctrlaltdel.blog/

JeffreyCMI 116 Reputation points

2026-07-27T21:59:59.86+00:00

Thanks for your response.

I re-visited by billing for the dates when this happened. Yes I was billed for the following components of the preempted batch node:

Load Balancer - Standard Included LB Rules and Outbound Rules
Virtual Network - Standard IPv4 Static Public IP
Storage - P6 LRS Disk

but I was NOT billed any VM Compute charges for the preempted node.
JeffreyCMI 116 Reputation points

2026-07-29T13:57:11.1766667+00:00

the pool should converge back to the target number of usable Spot nodes

I do not observe this behavior, at least not from the autoscaling feature. What I do see is Batch trying to re-provision a preempted node, but that is independent from pool scaling.

Answer 2

As an observer watching my Batch pools, I was able to confirm the following behaviors, all of which address my questions. I posted this as a new comment because my empirical observations have a lot of overlap with, but also slight differences from, the other comments posted here.

What happens to a batch node after it gets preempted?

In this case, my observations completely match VarunTha's response. Batch will periodically attempt to re-provision them (this is where I believe Alex Burlachenko is mistaken). The entire time -- even in Preempted state -- the node counts toward the pool size.

The only thing that would remove a Preempted node from the pool is an autoscaling rule doing general scale-in.

Am I being billed for the preempted node?

The VM is gone, so you are not billed for compute. But I observe the hard disk and networking for the node remain while the node is preempted, so you do get billed for those. This is typically a tiny fraction of what the compute cost would have been: In my experience we pay about $1/day/preempted-node (USD).

Autoscaling does not discount this preempted node when it checks if it has enough nodes.

This statement is correct. And it's consistent with the observation that Batch is trying to re-provision the preempted node, so from the autoscaler's point of view, that preempted node might yet do some more work!

One workaround to prevent starvation (when the preempted nodes never recover) may be to over-provision the pool. For example I use this:

lookbackMinutes = 3;

// Nodes preempted in the pool count toward pool size but run nothing,
// so we add additional capacity to avoid task starvation.
$preemptedNodes = max($PreemptedNodeCount.GetSample(TimeInterval_Minute * lookbackMinutes));
$TargetLowPriorityNodes = $targetNodes + $preemptedNodes;

I can't really explain the supply-and-demand of why this works. i.e. How can the autoscaler provision new nodes when existing nodes remained preempted? Oh well, no problem.

JeffreyCMI 116 Reputation points

2026-07-29T14:00:06.4466667+00:00

(comment deleted)

Answer 3

Hello JeffreyCMI,

Thank you for the clarification. You are correct that the re-provisioning attempt for the preempted Spot node is separate from the Batch autoscale evaluation.

From the Compute side, after the Spot VM is preempted, the underlying VM is deleted, so VM compute charges stop. However, associated resources such as managed disks, public IP addresses, and load-balancer rules may continue to incur charges while they remain allocated.

Azure Batch keeps the node record in the Preempted state and may attempt to restore the Spot capacity for up to 48 hours. The preempted node can therefore continue to count towards the pool target, even though it cannot run tasks.

Your use of $PreemptedNodeCount to add replacement capacity is a reasonable workaround for avoiding task starvation. The exact interaction between the Batch autoscale calculation and the retained preempted node is owned by the Azure Batch service rather than the Compute platform.

Thank you for sharing the detailed observations and billing breakdown. This distinction will also help other customers using Spot nodes with Batch autoscaling.

Preempted Batch node stays in the pool, misleads Batch autoscaling

1) What happens to a Batch pool node after it gets preempted? Do they leave the pool / recover to Running?

2) Am I being billed for the preempted node as long as it remains in the Batch pool?

3) Why doesn’t autoscaling discount the preempted node? Any workaround?

Workaround (based on what’s documented)

3 answers

What happens to a batch node after it gets preempted?

Am I being billed for the preempted node?

Autoscaling does not discount this preempted node when it checks if it has enough nodes.

Your answer