An Azure machine learning service for building and deploying models.
Machine Learning: GPU deployment fails after hours — No container logs available
Problem description
I am deploying an Azure Machine Learning managed online endpoint using the GPU SKU Standard_NV36ads_A10_v5 via Terraform. The deployment runs for several hours and then fails, with no container logs available. Deploying the same model to a CPU SKU (Standard_F8s_v2) succeeds, indicating the issue is specific to GPU deployment.
Environment
Azure Machine Learning managed online endpoint in the target region, using GPU SKU Standard_NV36ads_A10_v5, deployed through Terraform and also attempted via Azure ML Studio UI.
What I've already tried
I deployed the model to a CPU VM (Standard_F8s_v2) successfully. I also attempted deployment directly through Azure ML Studio UI without using Terraform, selecting the same GPU SKU, but the deployment still fails after several hours with no logs from the container. I verified that my AML quota in the region supports the GPU SKU, and no OutOfQuota errors are reported. I have not retrieved logs via CLI or SDK, nor checked deployment events or quota details beyond what is mentioned.
Current status
The deployment continues to fail silently without container logs. I am seeking assistance to investigate whether the issue is due to infrastructure allocation failures, quota limitations, SKU support, or other backend provisioning issues. I would appreciate escalation to the Azure ML backend team or guidance on further diagnostics.