Service Fabric: Cluster Stuck in UpgradeServiceUnreachable State — Certificate Renewal Issue

Dear @AzureCloud-3784 ,

Your cluster is in a known “blocked upgrade” scenario: the primary certificate expired, the ARM update started, but the Fabric Upgrade Service lost quorum (missing seed nodes) before it could apply the new certificate. In that state the cluster can’t complete the certificate rollover and stays in UpgradeServiceUnreachable with provisioningState Updating.

Below are the supported steps to recover.

1. Confirm certificate and ACLs on every node

On each VM in the primary node type:

1. RDP in and run certlm.msc → Local Computer → Personal.

Verify:

• The new cluster certificate is present and not expired.
  • The old/expired certificate is still present (the cluster may still be trying to use it).

1. For the new certificate:

• Right‑click → Manage Private Keys.
  • Ensure NETWORK SERVICE has Full Control.

If the new cert is missing or not ACL’d to NETWORK SERVICE, the Upgrade Service can’t use it and the cluster will stay stuck.

2. Restore seed‑node quorum

Your diagnostics already indicate missing seed nodes. Until quorum is restored, no configuration upgrade (including certificate swap) can complete.

Follow Microsoft’s guidance for missing seed nodes:

• How to fix missing seed nodes (automated script) https://github.com/Azure/Service-Fabric-Troubleshooting-Guides/blob/master/Cluster/How%20to%20fix%20missing%20seednodes%20with%20Automated%20script.md
• How to fix one missing seed node https://github.com/Azure/Service-Fabric-Troubleshooting-Guides/blob/master/Cluster/How%20to%20Fix%20one%20missing%20seed%20node.md

In brief:

1. Determine how many seed nodes are missing (from Service Fabric Explorer or diagnostics).
2. Scale up the primary VM Scale Set to add enough nodes to replace the missing seed nodes (often +2 or more depending on reliability tier).
3. Run the Microsoft troubleshooting script to “fake”/repair the missing seed node configuration so the ring can form quorum again.

Once quorum is back, the cluster should move out of the pre‑safety‑check failure and allow upgrades to proceed.

3. Use the expired‑certificate recovery guidance

Because the cluster became unreachable after the primary certificate expired, also follow the dedicated recovery guide:

• How to recover from an Expired Cluster Certificate https://github.com/Azure/Service-Fabric-Troubleshooting-Guides/blob/master/Security/How%20to%20recover%20from%20an%20Expired%20Cluster%20Certificate.md

Key points from that doc:

• If the cluster is secured with thumbprint‑based self‑signed certs, you may need to temporarily set:

    "fabricSettings": [
    {
      "name": "Security",
      "parameters": [
        {
          "name": "AcceptExpiredPinnedClusterCertificate",
          "value": "true"
        }
      ]
    }
  

]

  
  via **https://resources.azure.com** on the Service Fabric cluster resource, then apply. This lets the cluster start even while it still references the expired cert, so you can connect and complete the rollover.
  
- After you can connect with PowerShell (`Connect-ServiceFabricCluster`), run the provided **FixExpiredCert‑*.ps1*** script (or equivalent manual steps) to:

- Update the cluster manifest with the new thumbprint.

- Restart the FabricHostSvc service.

  - Force the Upgrade Service to pick up the new certificate state.
  
Microsoft’s issue tracker also confirms that an expired cert can block the Upgrade Service specifically, and that deleting the expired certificate from all nodes and/or moving to **common‑name** based configuration is the long‑term fix.

## 4. If the cluster resource is stuck in “Updating”

If, after restoring quorum and fixing certs, the ARM resource still shows:

- `provisioningState: Updating`

- `properties.certificate.thumbprint` unchanged

then:

1. Ensure **AcceptExpiredPinnedClusterCertificate = true** is set first (as above).

1. Re‑apply the cluster ARM template with the **new certificate thumbprint** (or common name) only after the cluster is healthy enough to connect via PowerShell.

1. Monitor upgrade status with:

   ```yaml
   Get-ServiceFabricClusterUpgrade

and verify the upgrade completes instead of failing at PreSafetyCheck due to seed nodes.

If the resource remains stuck despite a healthy cluster, open a support request (even with Basic support you can file a Subscription Management case) and reference:

Help make this community better for everyone: if this answer resolved your issue, please accept it or leave an upvote. If not, share more details in a comment so we can continue the discussion and find the right solution.