AVS Gen2 HCX RAV migration stuck in MIGRATION_CANCELLING state for 48 hours

Question

AVS Gen2 HCX RAV migration stuck in MIGRATION_CANCELLING state for 48 hours

Nimish Todi 0 Microsoft Employee

Summary

A Replication-Assisted vMotion (RAV) migration on an AVS HCX manager has been stuck in MIGRATION_CANCELLING for ~48 hours. It does not transition to a terminal state, and several REST cancel/cleanup paths return HTTP 500. Has anyone seen this on AVS HCX, and does anyone know the supported way to clear it without a manager-side restart?

Environment

• AVS (Azure VMware Solution) HCX Manager.

• Authenticating as cloudadmin@vsphere.local.

• All requests use the dual-header form returned by POST /hybridity/api/sessions (x-hm-authorization session token plus Authorization: Bearer JWT for the AVS Apache proxy).

Stuck migration

• migrationType: RAV (Replication-Assisted vMotion)

• state: MIGRATION_CANCELLING (unchanged for ~48h)

• progress.state.isDone = false, progress.isCancelled = 1

What was tried

Call 1: POST /hybridity/api/migrations?action=query (filter by migrationId) Result: HTTP 200. Record returned in MIGRATION_CANCELLING.

Call 2: POST /hybridity/api/mobility/migrations/cancel Body: {"migrationIds":["..."], "options":{"force":true}} Result: HTTP 202. Body {"items":[]}. NO state change observed on subsequent ?action=query polls.

Manish Deshpande 8,135 Reputation points Microsoft External Staff Moderator

2026-05-14T00:31:50.3733333+00:00

Hello Nimish

Discussed in the call to restart the HCX manager
1.Go to Azure Portal → Your AVS Private Cloud → Operations → Run Commands
2. Select the Microsoft.AVS.HCX package
3. Choose Restart-HcxManager
4. Enable the Force parameter — this is specifically designed to bypass the "active migration detected" check that would otherwise block the restart
5. Allow 5–10 minutes for it to come back online

After the restart, re-run the forceCancel API call

https://learn.microsoft.com/en-us/azure/azure-vmware/use-hcx-run-commands
Nikhil Duserla 9,945 Reputation points Microsoft External Staff Moderator

2026-05-19T11:40:03.36+00:00
Hello @Nimish Todi ,

Hey Nimish—this is a known HCX bug where a RAV job can hang in MIGRATION_CANCELLING and no public REST call will drive it to a terminal state. Unfortunately the only supported “clean‐up” path today is to restart the HCX Manager via Azure Portal Run Commands and then re-issue the forceCancel. There isn’t an in-product API or alternative supported move to clear a stuck RAV without a manager restart.

Here’s what you can do:

In the Azure portal go to: Your AVS Private Cloud → Operations → Run Commands

Select the Microsoft.AVS.HCX package

Choose Restart-HcxManager and enable the Force flag

Wait ~5–10 minutes for the manager to come back up

Re-run your POST /hybridity/api/mobility/migrations/cancel with { "migrationIds": […], "options":{ "force":true } }

If you absolutely cannot restart the manager yourself, you’ll need to open a support ticket so the Azure backend team can do a deeper cleanup. Before opening a case, it’ll help to have:

• HCX Manager version (GET /hybridity/api/versions)

• Exact HTTP 500 payloads you’re seeing on cancel/cleanup calls

• Any HCX Manager logs around the time you retried the cancel

• Confirmation whether other migrations succeed post-restart

Hope this helps get you unstuck!

Reference docs

Run Commands for AVS HCX Manager: https://aka.ms/AVS-HCX-RunCommands

HCX Migration Event overview: https://aka.ms/AVS-HCX-MigrationEvent

Note: This content was drafted with the help of an AI system. Please verify the information before relying on it for decision-making.

1 answer

Your answer

Manish Deshpande 8,135 Reputation points Microsoft External Staff Moderator

2026-05-14T00:31:50.3733333+00:00

Hello Nimish

Discussed in the call to restart the HCX manager
1.Go to Azure Portal → Your AVS Private Cloud → Operations → Run Commands
2. Select the Microsoft.AVS.HCX package
3. Choose Restart-HcxManager
4. Enable the Force parameter — this is specifically designed to bypass the "active migration detected" check that would otherwise block the restart
5. Allow 5–10 minutes for it to come back online

After the restart, re-run the forceCancel API call

https://learn.microsoft.com/en-us/azure/azure-vmware/use-hcx-run-commands
Nikhil Duserla 9,945 Reputation points Microsoft External Staff Moderator

2026-05-19T11:40:03.36+00:00

Hello @Nimish Todi ,

Hey Nimish—this is a known HCX bug where a RAV job can hang in MIGRATION_CANCELLING and no public REST call will drive it to a terminal state. Unfortunately the only supported “clean‐up” path today is to restart the HCX Manager via Azure Portal Run Commands and then re-issue the forceCancel. There isn’t an in-product API or alternative supported move to clear a stuck RAV without a manager restart.

Here’s what you can do:

In the Azure portal go to: Your AVS Private Cloud → Operations → Run Commands

Select the Microsoft.AVS.HCX package

Choose Restart-HcxManager and enable the Force flag

Wait ~5–10 minutes for the manager to come back up

Re-run your POST /hybridity/api/mobility/migrations/cancel with { "migrationIds": […], "options":{ "force":true } }

If you absolutely cannot restart the manager yourself, you’ll need to open a support ticket so the Azure backend team can do a deeper cleanup. Before opening a case, it’ll help to have:

• HCX Manager version (GET /hybridity/api/versions)

• Exact HTTP 500 payloads you’re seeing on cancel/cleanup calls

• Any HCX Manager logs around the time you retried the cancel

• Confirmation whether other migrations succeed post-restart

Hope this helps get you unstuck!

Reference docs

Run Commands for AVS HCX Manager: https://aka.ms/AVS-HCX-RunCommands

HCX Migration Event overview: https://aka.ms/AVS-HCX-MigrationEvent

Note: This content was drafted with the help of an AI system. Please verify the information before relying on it for decision-making.

Answer 1

Hi Nimish Todi,

Yes, this behavior has been seen occasionally with HCX RAV migrations where the workflow becomes orphaned internally and remains stuck in MIGRATION_CANCELLING indefinitely. In AVS environments, especially behind the Apache proxy/auth layer, some cleanup APIs may return HTTP 500 even though authentication is valid.

A few important points:

• If:

progress.state.isDone = false

progress.isCancelled = 1

cancel requests return HTTP 202 but nothing changes

cleanup/removal APIs return HTTP 500

then the migration workflow is usually stuck inside the HCX Mobility Service database/task engine rather than actively processing.

What is typically safe to try before restarting HCX Manager:

Verify there are no active replication sessions/tasks still attached:

Check Replication Service state

  Check vSphere recent tasks/events
  
     Verify no active VR/NFC sessions remain
     
     Try querying the mobility group/task state:
     
        /hybridity/api/jobs
        
           /hybridity/api/migrations?action=query

Sometimes the migration object is blocked by a child workflow still marked RUNNING.

Attempt force cleanup from the Mobility API:

POST /hybridity/api/mobility/migrations/cleanup

or:

POST /hybridity/api/mobility/migrations/remove

However, on AVS these frequently fail with HTTP 500 once the state machine is already corrupted/stuck.

Check whether the migration still appears in:

HCX UI Mobility tab

  /common/services
  
     replication statistics APIs

If everything else is already torn down, the issue is likely only metadata persistence.

Unfortunately, on AVS there is usually no supported customer-side database cleanup access. VMware/Microsoft support generally handles this by:

restarting specific HCX services

clearing orphaned workflow entries

or performing backend DB cleanup

A full HCX Manager reboot is usually considered last resort because it impacts ongoing migrations and tunnels.

At this stage, if the migration has remained in MIGRATION_CANCELLING for ~48 hours with repeated no-op 202 responses and HTTP 500 cleanup failures, opening a Microsoft AVS support ticket is probably the supported path rather than attempting unsupported manager-side modifications.

If my answer is useful for you, please hit Accept Answer to support me.

Thank you so much.