VMSS instances are in error state, but still running.

Question

VMSS instances are in error state, but still running.

Asmita Yadav 40 Microsoft Employee

VMSS instances are in error state, but still running. The SF nodes are healthy but the corresponding VM instances are in error state. The extension are also in falied state. Not able to delete from Portal. Tried Deploying but its failing with the following error: Error: Code: CannotAllocateRemainingVMsInAvailabilitySet Message: Delete/Deallocate operation on VM '[RESOURCE NAME REDACTED]' failed because the remaining VMs in the Availability Set '[RESOURCE NAME REDACTED]' cannot be allocated together. Changes in Availability Set allocation need to be executed atomically. Please deallocate or delete some or all of these VMs before retrying the current operation. Details: The VM allocation failed due to an internal error. Please retry later or try deploying to a different location. Target: [RESOURCE NAME REDACTED]

Anonymous

2026-03-23T12:10:10.3433333+00:00

Hello Asmita,

This seems to be an issue in the underlying VMSS resources for your managed service fabric cluster. I've reached out to you over Private Message for more details to discuss the case in detail over a Teams call.

Answer accepted by question author

Anonymous

Hello Asmita,

Thank you for providing the information during the call. We have reviewed the cluster operations and found that since January 2026 -> the VMSS operations, such as updates for VMSS (Infra deployments), have been failing. This is due to the Regional CPU limit for the subscription being set at 200, which is insufficient for VMSS operations. There are also VM-Extension provisioning failures, so once we resolve the VM provisioning issues, we can address those next

To increase quota,

Go to Quotas - Microsoft Azure
Select your subscription "sub-id" and then region as “East US” and then change the quota for "Total Regional vCPUs" Click on that edit icon on under adjustable column and then increase it to a higher value.

Once the increase is made, please try your Ev2 deployments again, as these handle the Infra deployment for your cluster. After that, we can review the VM extension failures once more.

0 comments

1 additional answer

Your answer

Anonymous

2026-03-23T12:10:10.3433333+00:00

Hello Asmita,

This seems to be an issue in the underlying VMSS resources for your managed service fabric cluster. I've reached out to you over Private Message for more details to discuss the case in detail over a Teams call.

Answer 1

The error indicates a VM allocation failure inside the availability set that backs the VM scale set/Service Fabric node type. Azure cannot place the remaining VMs in that availability set together on the cluster, so delete/deallocate operations on individual instances fail and must be done more atomically.

Use these steps to remediate:

Understand the cause
The message CannotAllocateRemainingVMsInAvailabilitySet and The VM allocation failed due to an internal error matches the documented allocation failure pattern: the original cluster that hosts the availability set no longer has enough capacity or cannot support the requested VM sizes. Operations that change allocation (start, delete, deallocate, resize) must succeed for the whole availability set or fail.
Stop/deallocate all VMs in the availability set together
Because changes must be atomic, deallocating or deleting a single VM instance can fail while others are still running. Instead, deallocate all VMs in the availability set at once so Azure can reallocate them from any available cluster:
1. In the Azure portal, go to Virtual machines.
2. Use Add filter and filter by the availability set backing the VMSS/Service Fabric node type (the one shown in the error: _vmss_d1036650cb5f464d9b0ad3e4be9e191e).
3. Select all VMs in that availability set.
4. Select Stop and wait until all show Stopped (deallocated).
5. If the goal is to keep the node type, select Start to reallocate all VMs together. If the goal is to remove them, delete them after they are deallocated.
This matches the documented workaround for allocation failures in availability sets and for starting/deallocating VMs when the original cluster has capacity issues.
If deallocation of all VMs is not possible immediately
- Retry the operation later, as the error message suggests: capacity can change over time.
- If the workload allows, consider scaling down the VMSS/node type (fewer instances) so fewer VMs need to be allocated together.
If the node type/VMSS must be recreated
If the availability set continues to hit allocation failures, consider:
- Creating a new VMSS/node type in a different availability set (or different region/zone) that has capacity.
- Migrating the Service Fabric workload to the new node type and then deleting the old one once all VMs are deallocated.
General guidance for allocation failures
- Consider resizing to a different VM size that has better availability in the region.
- For new deployments, use a different availability set or region if possible.

These steps align with the documented guidance for allocation failures when managing VMs in availability sets and when restart/deallocate operations fail due to cluster capacity.

References: