ML Workspace 401 pulling model/environment from ML Registry

Question

ML Workspace 401 pulling model/environment from ML Registry

Phillip Stenger 10

I am trying to deploy an online endpoint using a model/environment from an ML Registry. The deployment is failing and in the logs I am seeing 401 errors. The online endpoint is using a user assigned id which I have verified has AzureML Registry User role. It seems like maybe a networking issue, but I'm not sure. The ACR backing the registry has public access enabled as does the registry itself, though the registry does have a private endpoint configured. Not sure if that matters.

Kind: Pod, Name: ImagePullFailed, Type: Warning, Time: 2026-08-14T00:19:57.025589Z, Message: Image pull failed, retrying. Please check image-fetcher log for detail.
Kind: Pod, Name: DownloadFailed, Type: Warning, Time: 2026-08-14T00:20:10.550807Z, Message: Model download failed, retrying. Please check storage-initializer log for detail.
Kind: Pod, Name: Pulling, Type: Normal, Time: 2026-08-14T00:20:13.724089Z, Message: Start pulling container image
Kind: Pod, Name: DownloadFailed, Type: Warning, Time: 2026-08-14T00:20:21.557221Z, Message: Model download failed, retrying. Please check storage-initializer log for detail.
Kind: Pod, Name: Downloading, Type: Normal, Time: 2026-08-14T00:20:35.694766Z, Message: Startdownloading models

Container logs:
+ '[' '!' -S /run/containerd/containerd.sock ']'
+ ./mir-imagefetcher
{"level":"[INFO]","ts":"Aug  14 00:20:13","logger":"VMAgent.ImageFetcher","caller":"imagefetcher/image_fetcher.go:131","msg":"Use environment: AzurePublicCloud"}
{"level":"[INFO]","ts":"Aug  14 00:20:13","logger":"VMAgent.ImageFetcher","caller":"imagefetcher/image_fetcher.go:157","msg":"Parse image name","ImageUrl":"b7869157b0a.azurecr.io/azureml/30aa9c9f-f60e-5330-b189-17a6f8fa1499/statements:11","repo":"b7869157b0a.azurecr.io/azureml/30aa9c9f-f60e-5330-b189-17a6f8fa1499/statements","tag":"11","digest":""}
{"level":"[INFO]","ts":"Aug  14 00:20:13","logger":"VMAgent.ImageFetcher","caller":"imagefetcher/image_fetcher.go:290","msg":"Start getting image credential","ImageUrl":"b7869157b0a.azurecr.io/azureml/30aa9c9f-f60e-5330-b189-17a6f8fa1499/statements:11"}
{"level":"[INFO]","ts":"Aug  14 00:20:13","logger":"VMAgent.ImageFetcher","caller":"imagefetcher/image_fetcher.go:320","msg":"Found user ACR image","ImageUrl":"b7869157b0a.azurecr.io/azureml/30aa9c9f-f60e-5330-b189-17a6f8fa1499/statements:11","IdentityType":"XDS"}
{"level":"[INFO]","ts":"Aug  14 00:20:13","logger":"VMAgent.ImageFetcher","caller":"imagefetcher/image_fetcher.go:454","msg":"Request parameters","Registry":"b7869157b0a.azurecr.io","clientid":"f831ae94-ac82-435f-96e2-e5e88c20c870"}
{"level":"[WARN]","ts":"Aug  14 00:20:13","logger":"VMAgent","caller":"common/http_util.go:197","msg":"Http status code of response is not 2xx","HTTP status code":401}
{"level":"[INFO]","ts":"Aug  14 00:20:13","logger":"VMAgent.ImageFetcher","caller":"imagefetcher/image_fetcher.go:509","msg":"Get challenge header","registry":"b7869157b0a.azurecr.io","Www-Authenticate":"Bearer realm=\https://b7869157b0a.azurecr.io/oauth2/token\,service=\"b7869157b0a.azurecr.io\""}
{"level":"[INFO]","ts":"Aug  14 00:20:13","logger":"VMAgent.ImageFetcher","caller":"imagefetcher/image_fetcher.go:514","msg":"Update login server","registry":"b7869157b0a.azurecr.io","service":"b7869157b0a.azurecr.io"}
{"level":"[WARN]","ts":"Aug  14 00:20:13","logger":"VMAgent","caller":"common/http_util.go:197","msg":"Http status code of response is not 2xx","HTTP status code":401}
{"level":"[WARN]","ts":"Aug  14 00:20:15","logger":"VMAgent","caller":"common/code_util.go:75","msg":"After attempt # 1, got error: exchange refresh token return non-success status: 401, details: "}

Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-08-14T14:13:03.6333333+00:00
Hi Phillip Stenger ,

Thank you for the additional information. The deny-assignment error is expected for Microsoft-managed resources that back an Azure ML Registry. Since those resources are managed by Azure Machine Learning, customers cannot directly modify RBAC assignments on the backing ACR or Storage Account.

Combined with the deployment logs showing: "exchange refresh token return non-success status: 401"

this suggests the failure is occurring during authentication to the registry artifacts rather than due to a missing role assignment that you can configure yourself.

To help narrow this down, could you please confirm:

Whether the Azure ML Registry is configured with a private endpoint or network isolation.

Whether the same deployment succeeds when using assets registered directly in the workspace instead of assets referenced from the Registry.

Whether the behavior is reproducible across multiple registries and workspaces.

Since the backing registry resources are Microsoft-managed and protected by deny assignments, there may be a platform-side dependency involved here. Additional investigation may be required to determine whether this is a networking configuration issue or a service-side authorization problem.
Phillip Stenger 10 Reputation points

2026-08-14T16:10:17.2166667+00:00

The ML Registry is configured with a private endpoint, but it has public network access enabled. I did notice that the backing storage account was configured with public network access disabled. I enabled it to try and eliminate remaining network issues. After doing so I am no longer getting the 401 error. Instead I am getting generic error: "Message: Internal error."

Phillip Stenger 10

I tried to creating the deployment again and got this error:

(InferencingClientCallFailed) {"error":{"code":"Validation","message":"{\"errors\":{\"\":[\"Specified deployment [blue] failed during initial provisioning and is in an unrecoverable state. Delete and re-create.\"]},\"type\":\"https://tools.ietf.org/html/rfc9110#section-15.5.1\",\"title\":\"One or more validation errors occurred.\",\"status\":400,\"traceId\":\"00-f342e680c8905c90697f6d7675c08be1-6825a90a453103d0-01\"}"}}

The error does not tell me what validation error is. I tried to deleting the deployment to re-create as suggested by the message but the deployment has been stuck in deleting for about 30 minutes now.

Phillip Stenger 10 Reputation points

2026-08-14T17:24:24.5733333+00:00

I am not able to test your second two bullet points due the deployment being stuck in "Deleting" and needing more quota
Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-08-15T13:08:50.4033333+00:00

Hi Phillip Stenger ,

Thank you for sharing the details. I will look into this and will get back to you shortly.
Phillip Stenger 10 Reputation points

2026-08-15T15:46:59.84+00:00

Thank you. Is there any update on this?
Phillip Stenger 10 Reputation points

2026-08-16T12:57:18.56+00:00

Any update or guidance?
Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-08-19T09:03:58+00:00

Hi Phillip Stenger ,

Could you please share the details requested over Private message.

2 answers

Your answer

Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-08-14T14:13:03.6333333+00:00

Hi Phillip Stenger ,

Thank you for the additional information. The deny-assignment error is expected for Microsoft-managed resources that back an Azure ML Registry. Since those resources are managed by Azure Machine Learning, customers cannot directly modify RBAC assignments on the backing ACR or Storage Account.

Combined with the deployment logs showing: "exchange refresh token return non-success status: 401"

this suggests the failure is occurring during authentication to the registry artifacts rather than due to a missing role assignment that you can configure yourself.

To help narrow this down, could you please confirm:

Whether the Azure ML Registry is configured with a private endpoint or network isolation.

Whether the same deployment succeeds when using assets registered directly in the workspace instead of assets referenced from the Registry.

Whether the behavior is reproducible across multiple registries and workspaces.

Since the backing registry resources are Microsoft-managed and protected by deny assignments, there may be a platform-side dependency involved here. Additional investigation may be required to determine whether this is a networking configuration issue or a service-side authorization problem.
Phillip Stenger 10 Reputation points

2026-08-14T16:10:17.2166667+00:00

The ML Registry is configured with a private endpoint, but it has public network access enabled. I did notice that the backing storage account was configured with public network access disabled. I enabled it to try and eliminate remaining network issues. After doing so I am no longer getting the 401 error. Instead I am getting generic error: "Message: Internal error."
Phillip Stenger 10 Reputation points

2026-08-14T16:13:43.8133333+00:00

I tried to creating the deployment again and got this error:

(InferencingClientCallFailed) {"error":{"code":"Validation","message":"{\"errors\":{\"\":[\"Specified deployment [blue] failed during initial provisioning and is in an unrecoverable state. Delete and re-create.\"]},\"type\":\"https://tools.ietf.org/html/rfc9110#section-15.5.1\",\"title\":\"One or more validation errors occurred.\",\"status\":400,\"traceId\":\"00-f342e680c8905c90697f6d7675c08be1-6825a90a453103d0-01\"}"}}

The error does not tell me what validation error is. I tried to deleting the deployment to re-create as suggested by the message but the deployment has been stuck in deleting for about 30 minutes now.
Phillip Stenger 10 Reputation points

2026-08-14T17:24:24.5733333+00:00

I am not able to test your second two bullet points due the deployment being stuck in "Deleting" and needing more quota
Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-08-15T13:08:50.4033333+00:00

Hi Phillip Stenger ,

Thank you for sharing the details. I will look into this and will get back to you shortly.
Phillip Stenger 10 Reputation points

2026-08-15T15:46:59.84+00:00

Thank you. Is there any update on this?
Phillip Stenger 10 Reputation points

2026-08-16T12:57:18.56+00:00

Any update or guidance?
Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-08-19T09:03:58+00:00

Hi Phillip Stenger ,

Could you please share the details requested over Private message.

Answer 1

A 401 during image pull and model download in an online endpoint usually points to an authorization problem first, with networking as a secondary possibility.

Based on the failure details:

The image pull path is failing at registry token exchange.
- The log shows exchange refresh token return non-success status: 401.
- For online deployments, Azure uses managed identities to access the container registry and storage.
- When the endpoint uses a user-assigned identity, that identity must have the required permissions on the backing resources.
For the container image pull, verify the user-assigned identity has AcrPull on the container registry that stores the environment image.
- The documented requirement for user-assigned identity is AcrPull on the workspace container registry.
- A registry-specific role such as AzureML Registry User is not sufficient for ACR image pull by itself.
For the model download, verify the same identity has the required storage access.
- The documented requirement is Storage blob data reader on the workspace storage account.
- If MDC is enabled, the requirement changes to Storage Blob Data Contributor.

Check whether the model and environment artifacts are actually reachable.

Get deployment logs:

     az ml online-deployment get-logs -e <endpoint-name> -n <deployment-name> -l 100

If model download is failing, verify the model exists in the workspace and check the storage initializer logs:

     az ml model show --name <model-name> --version <version>

     az ml online-deployment get-logs --endpoint-name <endpoint-name> --name <deployment-name> –container storage-initializer

Networking can also cause this if the registry is private or otherwise inaccessible.
- The documentation calls out container registry authorization failures when the registry cannot be accessed with the current credentials.
- If the registry is behind a virtual network or private endpoint, verify that the virtual network is set up properly.
- If using legacy network isolation for managed online endpoints, image download failures are expected when public egress is disabled and the registry visibility is private.
If the endpoint was created with system-assigned identity instead, RBAC is granted automatically. For user-assigned identity, the role assignments must be added manually.

If the image is not in the expected registry or tag, verify it exists:

   az acr repository show-tags -n <acr-name> --repository <repository-name> --orderby time_desc --output table

Most likely fix for this case:

Add AcrPull to the user-assigned identity on the ACR that backs the environment image.
Add Storage blob data reader on the storage account that holds the model artifacts.
If private endpoint or VNet routing is involved, validate that the endpoint can reach the registry and storage paths.

References:

Troubleshoot online endpoint deployment and scoring

Answer 2

Phillip Stenger 10

I am unable to add roles to the storage/acr which back the ML Registry as they are Microsoft managed. I am getting this error if I try: "The client *** with object id *** has permission to perform action *** on scope *** however the access is denied because of the deny assignment with name 'Azure machine learning services user RG access denier"

0 comments