How to fix a stuck custom vision training iteration?

Question

How to fix a stuck custom vision training iteration?

Victoria 0

I am trying to train a custom vision model, I put the model onto quick train (which had an estimated time of one hour) and it's now been 24 hours and the iteration is still training. There doesn't seem to be any option to delete, stop, or restart the training. What are the options to fix this?

SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-09-12T10:51:16.31+00:00

Hi Victoria,

We sincerely apologize for the inconvenience this has caused. At this time, we are aware that several customers are reporting similar issues with the service.

We will be reaching out to our Product Group (PG) team for further investigation and will share an update with you as soon as we have more information. Thank you for your patience and understanding.
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-09-18T07:01:42.58+00:00

Hi Victoria,

Can you check on your side and let me know if you run into any problems? My PG team cleared the issue.

Thank you!
Victoria 0 Reputation points

2025-09-18T07:15:08.4233333+00:00

Hi @SRILAKSHMI C , the model is still stuck in training, the issue is still the same.
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-09-22T16:50:08.28+00:00

Hi Victoria,

I’m sorry for the inconvenience caused. I understand the model is still stuck in training and the issue persists. I will check with our product group (PG) team for further insights and will get back to you with the next steps as soon as possible.

Thank you!
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-10-01T18:12:44.1833333+00:00

Hi Victoria,

Can you check on your side and let me know if you run into any problems? My PG team cleared the issue.

Thank you!
Victoria 0 Reputation points

2025-10-02T07:09:38.38+00:00

Hi @SRILAKSHMI C ,

The model is still stuck in training unfortunately.

Thanks
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-10-02T14:18:54.1966667+00:00

Hi Victoria,

Thank you for the update. Since the model is still stuck in training, I would recommend cancelling the current training run in some cases, it does not complete successfully if it gets stuck for too long. You may also need to delete the model currently in the “training” state before starting a new training session.

It’s a known scenario we’ve seen before, and following these steps has resolved similar issues. Please give this a try and let me know how it goes.

https://learn.microsoft.com/en-us/answers/questions/5549515/azure-custom-vision-training-session-stuck

Thank you!
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-10-03T09:18:55.55+00:00

Hi Victoria,

Did you check the above response. Do let me know if you have further queries.

Thank you!
Victoria 0 Reputation points

2025-10-03T09:40:24.52+00:00

Hi @SRILAKSHMI C ,

There does not appear to be an option to cancel the current training run (deleting the iteration is greyed out), and the linked question does not have any further information. I am happy to delete this project as I've had to remake it in another project but is there a solution to delete the training run?

Thanks!
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-10-08T17:50:58.3066667+00:00

Hi Victoria,

I sincerely apologize for the delayed response.

Thank you for the update, and I completely understand how frustrating it is to be stuck without an option to cancel or delete the training run.

At this point, if the “Delete Iteration” button remains greyed out and there’s no visible option to stop the training job from the portal, it typically means the iteration is in a zombie state on the backend something that unfortunately can’t be resolved from the portal side.

Here are your best next steps:

Since you’ve already recreated the project elsewhere, deleting the current one will automatically terminate any stuck iterations associated with it.

Thank you!

Your answer

SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-09-12T10:51:16.31+00:00

Hi Victoria,

We sincerely apologize for the inconvenience this has caused. At this time, we are aware that several customers are reporting similar issues with the service.

We will be reaching out to our Product Group (PG) team for further investigation and will share an update with you as soon as we have more information. Thank you for your patience and understanding.
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-09-18T07:01:42.58+00:00

Hi Victoria,

Can you check on your side and let me know if you run into any problems? My PG team cleared the issue.

Thank you!
Victoria 0 Reputation points

2025-09-18T07:15:08.4233333+00:00

Hi @SRILAKSHMI C , the model is still stuck in training, the issue is still the same.
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-09-22T16:50:08.28+00:00

Hi Victoria,

I’m sorry for the inconvenience caused. I understand the model is still stuck in training and the issue persists. I will check with our product group (PG) team for further insights and will get back to you with the next steps as soon as possible.

Thank you!
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-10-01T18:12:44.1833333+00:00

Hi Victoria,

Can you check on your side and let me know if you run into any problems? My PG team cleared the issue.

Thank you!
Victoria 0 Reputation points

2025-10-02T07:09:38.38+00:00

Hi @SRILAKSHMI C ,

The model is still stuck in training unfortunately.

Thanks
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-10-02T14:18:54.1966667+00:00

Hi Victoria,

Thank you for the update. Since the model is still stuck in training, I would recommend cancelling the current training run in some cases, it does not complete successfully if it gets stuck for too long. You may also need to delete the model currently in the “training” state before starting a new training session.

It’s a known scenario we’ve seen before, and following these steps has resolved similar issues. Please give this a try and let me know how it goes.

https://learn.microsoft.com/en-us/answers/questions/5549515/azure-custom-vision-training-session-stuck

Thank you!
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-10-03T09:18:55.55+00:00

Hi Victoria,

Did you check the above response. Do let me know if you have further queries.

Thank you!
Victoria 0 Reputation points

2025-10-03T09:40:24.52+00:00

Hi @SRILAKSHMI C ,

There does not appear to be an option to cancel the current training run (deleting the iteration is greyed out), and the linked question does not have any further information. I am happy to delete this project as I've had to remake it in another project but is there a solution to delete the training run?

Thanks!
SRILAKSHMI C 19,735 Reputation points Microsoft External Staff Moderator

2025-10-08T17:50:58.3066667+00:00

Hi Victoria,

I sincerely apologize for the delayed response.

Thank you for the update, and I completely understand how frustrating it is to be stuck without an option to cancel or delete the training run.

At this point, if the “Delete Iteration” button remains greyed out and there’s no visible option to stop the training job from the portal, it typically means the iteration is in a zombie state on the backend something that unfortunately can’t be resolved from the portal side.

Here are your best next steps:

Since you’ve already recreated the project elsewhere, deleting the current one will automatically terminate any stuck iterations associated with it.

Thank you!