Service Fabric cluster upgrade failure when converting certificate from thumbprint to common name

The failure is caused by nodes being unable to mutually authenticate during the upgrade when switching from thumbprint-based to CN-based declarations. During a rolling upgrade, upgraded and not-yet-upgraded nodes must both be able to validate each other’s certificates at all times. If the configuration jumps directly from thumbprint-only to CN-only (or CN with issuer pinning) without following the supported transition pattern, the UpgradeService and other system services can no longer establish secure connections, which leads to UpgradeServiceDown.

To resolve this, follow the supported two-phase conversion pattern and ensure a valid starting state:

1. Ensure a valid starting state using thumbprints only
   The cluster must already be using the goal certificate (the Let’s Encrypt certificate) by thumbprint before converting to CN. Valid starting states include:
   • Thumbprint: GoalCert, ThumbprintSecondary: None, or
   • Thumbprint: GoalCert, ThumbprintSecondary: OldCert1, or
   • Thumbprint: OldCert1, ThumbprintSecondary: GoalCert (with GoalCert having a later NotBefore date than OldCert1).
     If the cluster is not in one of these states, adjust using one or two thumbprint-only upgrades as described in the “Achieve a valid starting state” table. For example, if currently Thumbprint: OldCert1, ThumbprintSecondary: None and OldCert1 has a later NotBefore than GoalCert, first move to Thumbprint: GoalCert, ThumbprintSecondary: OldCert1, then to Thumbprint: GoalCert, ThumbprintSecondary: None.
2. Verify certificates are installed on all nodes
   Before changing declarations, ensure both the current and target cluster certificates are provisioned to every VM in each VM scale set (LocalMachine\My). In Azure, this is typically done via Key Vault integration with VMSS. The documentation explicitly requires that all relevant certificates be installed on all nodes before changing the cluster’s certificate declarations.
3. Choose the CN validation scheme correctly
   When switching to CN-based declarations, decide whether to:
   • Rely on the trusted root CA (leave certificateIssuerThumbprintList empty), or
   • Pin the direct issuer(s) via certificateIssuerThumbprintList (recommended).
     If issuer thumbprints are specified, the certificate is only accepted if its direct issuer’s thumbprint matches one of the values in the list. If the Let’s Encrypt certificate is issued by a different intermediate than expected, or if not all relevant issuer thumbprints are listed, the certificate will be rejected and mutual authentication will fail, causing system services (including UpgradeService) to go down. Ensure that all expected issuer thumbprints for that CN are included.
4. Perform the conversion from thumbprint to CN in two phases
   The safe pattern is:
   • Phase 1: Cluster uses the goal certificate by thumbprint only, and validation rules are extended to accept both thumbprint and CN. Conceptually:
     • Initial state: all nodes present and validate by thumbprint A only.
     • After first upgrade domain: upgraded nodes still present A, but accept A (thumbprint) and B (CN) for validation; other nodes still present and accept A only.
   • Phase 2: Change the declaration so that the same certificate is declared by CN (with optional issuer thumbprints) and remove the thumbprint-based presentation. Because validation already accepts CN, nodes can mutually authenticate during the rolling upgrade.
   In ARM terms, this means:
   • First upgrade: keep the thumbprint-based certificate as-is (GoalCert as primary), but extend validation to include the CN (and issuer thumbprints) according to the rules in the certificate configuration documentation.
   • Second upgrade: update the cluster resource to use certificateCommonNames with commonNames (and certificateIssuerThumbprintList if pinning issuers), and remove the certificate property. Also update the Service Fabric node extension in each VMSS to match the CN-based configuration. Then redeploy the updated template.
5. Update and redeploy the ARM template correctly
   • Export the current template from the resource group if the original is not available.
   • Update each VM scale set’s Service Fabric node extension definition and the Microsoft.ServiceFabric/clusters resource: add certificateCommonNames with the correct CN and issuer thumbprints, and remove the old certificate block only in the second phase.
   • Redeploy using New-AzResourceGroupDeployment with the updated template and parameters.

If the upgrade is failing now, roll back to the last healthy thumbprint-based configuration, verify that the cluster is in a valid starting state with GoalCert as the active thumbprint, confirm that all nodes have the certificate installed, then reattempt the conversion using the two-phase pattern and correct issuer thumbprints.

References:

• Convert cluster certificates from thumbprint-based declarations to common names
• X.509 Certificate-based authentication in Service Fabric clusters
• Manage certificates in Service Fabric clusters