Custom vision ai is stuck in training days after starting it with just 8h budget

Advanced training jobs in Custom Vision can sometimes take much longer than expected due to high training traffic, long queues, and constrained GPU resources. In some cases, the job may remain in the queue and not finish within the time frame originally budgeted, but the service guarantees that charges will not exceed the reserved time budget.

However, there is no way to cancel a training job once it has started. While that iteration is in training, it effectively blocks progress on that specific training run, and a stuck job can prevent moving forward with that iteration.

Given this behavior, the practical options are:

1. Wait for the current advanced training job to complete or fail. When service load decreases, queued jobs typically resume and finish without additional charge beyond the reserved budget.
2. If the job remains stuck for an extended period (as in this case, several days), open an Azure support request so the service team can inspect and clear the stuck training job on the backend.
3. For future iterations, consider using fast training for quick experiments and only use advanced training when necessary, keeping in mind that advanced training relies on shared GPU capacity and can be delayed during high-usage periods.

Because cancellation is not supported from the portal or API, direct intervention from Azure Support is required if the training state does not change after a reasonable time.

References:

• Custom Vision frequently asked questions
• What's new in Custom Vision
• Characteristics and Limitations of Custom Vision