Service Fabric cluster - upgrade service unreachable and VMSS stuck in updating state

Question

Service Fabric cluster - upgrade service unreachable and VMSS stuck in updating state

Jemuel Manalo (Accenture International Limited) 0 Microsoft External Staff

I am trying to deploy a new change in our service, but I noticed that in the SF cluster it has an error 'Upgrade service unreachable' and when I checked the VMSS the instances there are nodes that stuck in updating state.

Anonymous

2026-05-05T09:33:47.85+00:00

The issue is not fully resolved and remains under investigation. A temporary mitigation has been applied by reverting the certificates to AME, which has helped stabilize the environment for now. However, the underlying problem persists and is currently being tracked as a compatibility concern between CCME and the OS upgrade.

Additionally, cluster operations are constrained by the EnsureSeedNodeQuorum safety check, which is preventing node deactivation to maintain cluster stability. Additionally, user also checking with their manager continuing tracking the compatibility aspects

1 answer

Your answer

Anonymous

2026-05-05T09:33:47.85+00:00

The issue is not fully resolved and remains under investigation. A temporary mitigation has been applied by reverting the certificates to AME, which has helped stabilize the environment for now. However, the underlying problem persists and is currently being tracked as a compatibility concern between CCME and the OS upgrade.

Additionally, cluster operations are constrained by the EnsureSeedNodeQuorum safety check, which is preventing node deactivation to maintain cluster stability. Additionally, user also checking with their manager continuing tracking the compatibility aspects

Answer 1

Jemuel Manalo (Accenture International Limited) - Fix the VMSS instances stuck in Updating - Service Fabric depends on VMSS health. Nodes stuck in Updating must be recovered first. -https://stackoverflow.com/questions/44257426/service-fabric-cluster-status-upgrade-service-unreachable

Microsoft‑recommended actions (in order):

From VM Scale Set > Instances, select the stuck instances
Perform Redeploy (preferred) or Reimage
If that fails, Deallocate then Start the affected instances

Ensure majority of seed nodes are healthy and running. If seed nodes are unhealthy or all were restarted together, the cluster ring collapses, triggering UpgradeServiceNotReachable. Minimum 3 healthy seed nodes must be available

Do not restart or update all nodes at once.

Once VMSS is healthy:

Open Service Fabric Explorer
Validate that nodes appear and are Up
Confirm System services (UpgradeService) are reachable

After nodes are reachable: - https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-application-upgrade-troubleshooting

Resume the upgrade or redeploy your service
If upgrade is blocked due to health policies, investigate via: Get-ServiceFabricClusterUpgrade

Last‑resort (only if cluster ring is broken) and If seed quorum cannot be restored: https://github.com/Azure/Service-Fabric-Troubleshooting-Guides/blob/master/Cluster/Cluster%20Not%20Reachable%20%20UpgradeServiceNotreachable.md

Recreate the Service Fabric cluster
Reusing the same VMSS without fixing quorum will not recover the cluster

The issue is not fully resolved and remains under investigation. A temporary mitigation has been applied by reverting the certificates to AME, which has helped stabilize the environment for now. However, the underlying problem persists and is currently being tracked as a compatibility concern between CCME and the OS upgrade.

Additionally, cluster operations are constrained by the EnsureSeedNodeQuorum safety check, which is preventing node deactivation to maintain cluster stability. Additionally, user also checking with their manager continuing tracking the compatibility aspects