Live migration over RDMA failing with operation timed out error

Jack 5T 20 Reputation points
2026-09-04T16:34:27.1933333+00:00

Can do standard TCP migrations fine.

The only time it blows up is when trying to run Live Migration across the 100GbE RDMA links. Network team checked the top-of-rack switches and says it's due to Priority Flow Control (PFC) mismatches between the ports and the hosts, but nothing on their side looks wrong to them.

RoCEv2 setup.

Mostly hitting "The operation timed out" errors during migration.

How do we actually validate and align DCBX and PFC settings across our host NICs to get this working?

Windows for business | Windows Server | Networking | Software-defined networking
0 comments
Answer accepted by question author
Tracy Le 11,535 Reputation points Independent Advisor
2026-09-04T17:31:07.2166667+00:00

Hello Jack 5T,

The operation timed out error during RoCEv2 Live Migration typically points to packet drops caused by asymmetric Priority Flow Control (PFC) or unsynchronized DCBX states between the host physical adapters and your top-of-rack switches. When PFC priorities do not match the DSCP markings mapped to SMB Direct traffic, the switches drop packets instead of pausing transmission, stalling RDMA operations until the migration fails.

To validate your host configuration, open PowerShell as administrator and run Get-NetQosTrafficClass along with Get-NetQosPolicy to ensure the SMB priority, typically priority 3 for RoCEv2, matches the PFC setting returned by Get-NetQosFlowControl. Check whether hardware DCBX willingness is active using Get-NetAdapterQos. If your top-of-rack switches are configured manually, you must disable willingness by executing Set-NetQosDcbxSetting -Willing 0 to prevent the adapters from ignoring local QoS policies. You can test end-to-end RDMA connectivity using Test-NetConnection -CommonTCPPort SMB or the Mellanox nd_send tool. If this resolves your timeouts, please accept this answer.

Tracy Le

Was this answer helpful?

1 person found this answer helpful.
0 comments

1 additional answer

Sort by: Oldest
Most helpful Newest Oldest

  1. AI answer

    2026-09-04T16:34:48.7666667+00:00

    For RoCEv2, the most direct validation path is to confirm whether RDMA is operational but DCB is not, then correct the host-side DCB configuration and any VLAN settings that prevent PFC from working.

    1. Check whether the host is reporting RDMA without DCB
      • If RDMA is working but PFC/ETS are not active, that indicates Data Center Bridging is not active on the Windows hosts.
      • Windows does not automatically enable PFC/ETS just because switch QoS/DCBX is configured. The host must have the Data-Center-Bridging feature installed and QoS policies defined for the OS to negotiate and enforce DCB.
    2. Validate for known host-side RDMA blockers In Network ATC environments, check for these conditions:
      • RDMANotOperational can occur if the NIC is using an inbox driver.
        • Resolution: update the adapter driver.
      • RDMANotOperational can also occur if SR-IOV is disabled in BIOS.
        • Resolution: enable SR-IOV in BIOS.
      • RDMANotOperational can also occur if RDMA is disabled in BIOS.
        • Resolution: enable RDMA in BIOS.
    3. Validate the RoCE VLAN configuration
      • For RoCE RDMA, a nonzero VLAN is required so that PFC markings can be added to the frame.
      • A valid VLAN must be in the range 1 - 4094.
      • If VLAN 0 or another invalid override is being used, Network ATC reports InvalidIsolationID.
      • If supported by the adapter, switching to iWARP avoids the RoCE nonzero-VLAN requirement, but for RoCEv2 the fix is to use a valid VLAN.
    4. Clear conflicting QoS/DCB policy on the host if already preconfigured If the host already has traffic classes or QoS policies that conflict with the intended DCB configuration, Network ATC can report ConflictingTrafficClass. Clear the existing DCB configuration, then retry:
    Get-NetQosTrafficClass | Remove-NetQosTrafficClass
Get-NetQosPolicy | Remove-NetQosPolicy -Confirm:$false
Get-NetQosFlowControl | Disable-NetQosFlowControl

    Risk: this removes existing QoS/DCB policy from the host. Apply only during a controlled maintenance window.

    1. Reapply the intended DCB configuration and retry intent deployment
      • After removing conflicting policy, run Set-NetIntentRetryState if Network ATC is managing the configuration.
      • If the environment is not using Network ATC, the key requirement remains the same: install Data-Center-Bridging and define the required QoS policies so the host can participate in DCB/PFC.
    2. Validate the physical switch alignment
      • For Azure Local switched and disaggregated deployments, top-of-rack switches must support Priority Flow Control (PFC) for lossless RDMA traffic.
      • Also ensure the switch port MTU matches the host-side jumbo frame design. Azure Local applies the host MTU during deployment, but physical switches must be configured separately. A common pairing is MTU 9000 on the host and 9216 on the switch.
    3. Check live migration transport ports during failure analysis For live migration timeout scenarios, verify clustering-related ports are listening on both source and destination:
    netstat -ano | findstr /I /C:"6600"

    Also verify TCP ports 6600 and 3343 are listening on both sides.

    1. If the issue persists, collect traces from both nodes
      • Run TSS rOn VML:verbose on both source and destination nodes.
      • Reproduce the live migration failure.
      • Then collect logs with:
    .\GetpsSDP.ps1 HyperV -localNodeOnly

    For the scenario described, the most likely alignment checks are:

    • host has RDMA enabled but not Data-Center-Bridging
    • RoCE VLAN is invalid or zero
    • existing host QoS/DCB policy conflicts with the intended configuration
    • switch supports PFC, but host-side DCB/PFC negotiation is not actually enabled
    • MTU is inconsistent between host and switch

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.