Azure ML HOSTTOOLS 403 – ExperimentRun logs not written to workspaceartifactstore

Antonieta Pinto Rebelo 0 Reputation points
2026-09-03T16:07:12.3633333+00:00

Hello,

I am investigating an Azure Machine Learning logging issue inside an Azure Managed Application and would appreciate help identifying which identity or permission Azure ML expects for the workspaceartifactstore.

The Azure ML workspace and compute cluster are deployed inside the Managed Application’s managed resource group. The compute cluster is connected to a managed VNet/subnet and uses only the user-assigned managed identity.

The compute identity has:

Storage Blob Data Contributor on the workspace storage account

AzureML Data Scientist on the Azure ML workspace

AcrPull on the private container registry

The workspace system-assigned identity has:

Storage Blob Data Contributor on the workspace storage account

Storage Account Contributor on the workspace storage account

A Key Vault access policy for the workspace’s linked Key Vault

Both workspaceblobstore and workspaceartifactstore were changed through Azure ML Studio to identity-based access. They now show:

Authentication type: None

The training job itself completes successfully. Input mounting works, and the declared training and model outputs are successfully written through workspaceblobstore.

However, Azure ML reports:

Component HOSTTOOLS_CAPABILITY unhealthy.

Service HOSTTOOLS_CAPABILITY returned health check response with code: 4,
message: 403 Forbidden.

Unable to upload outputs and logs to blob storage.
Check if you have lost access to storage account or workspace.

When the following environment variable is set to ignore, the training job completes but the Azure ML system logs are missing:

AZUREML_CR_HOSTTOOLS_CAPABILITY_OUTPUT_MANAGER_FAILURE_MODE=ignore

If it is set to fail, Azure ML marks the job as failed even though the training executable finishes and creates its declared artifacts.

The expected default output location is:

azureml://datastores/workspaceartifactstore/ExperimentRun/dcid.good_vinegar_fwr5g45fn1

Storage diagnostic logs show that the workspace identity successfully wrote:

azureml/ComputeRecord/dcid.good_vinegar_fwr5g45fn1/compute_record.txt

That request used OAuth and returned HTTP 201, with authorization granted through Storage Blob Data Contributor.

In my customer Azure ML, it works fine. But, with an MRG Azure ML, it causes problems.

What I'm I suppose to do here?

Azure Machine Learning
0 comments
Answer recommended by moderator
Antonieta Pinto Rebelo 0 Reputation points
2026-09-04T11:01:56.76+00:00

Hi there @Allan Solomon Mejia

I solved the issue

It cames down to how I set up the Deny Assignments of Managed Resource Group

If anyone out there is trying to do a Managed Application with Azure ML make sure that for logs to be produced

You must add these on the Allowed control actions

Microsoft.MachineLearningServices/workspaces/listStorageAccountKeys/action Microsoft.MachineLearningServices/workspaces/resynckeys/action Microsoft.MachineLearningServices/workspaces/listKeys/action

And then it will work!

Was this answer helpful?

0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Allan Solomon Mejia 7,995 Reputation points
    2026-09-03T20:06:06.57+00:00

    Hello @Antonieta Pinto Rebelo

    Your diagnostics already rule out the simplest RBAC explanation. The workspace system-assigned identity can successfully write compute_record.txt to the same storage account using OAuth with Storage Blob Data Contributor, and normal job outputs through workspaceblobstore are also succeeding.

    That means focus on the specific identity/credential HOSTTOOLS uses for the workspaceartifactstore/ExperimentRun/... upload, rather than blindly adding more roles. Enable Storage diagnostic logging and correlate the exact HOSTTOOLS failure timestamp. Check the requester/object ID, authentication type, request URI, and authorization failure reason for the failed Blob request. That will show whether HOSTTOOLS is actually using the workspace identity, compute UAMI, SAS/key credentials, or another service identity.

    The fact that the same setup works in a normal customer workspace but fails when deployed inside the Managed Application managed resource group is also important. You previously had a very similar HOSTTOOLS 403 case even after the expected workspace identity roles were assigned, so this may be specific to Azure ML orchestration/credential handling in the MRG scenario rather than ordinary storage access.

    Also, AZUREML_CR_HOSTTOOLS_CAPABILITY_OUTPUT_MANAGER_FAILURE_MODE=ignore is only suppressing the job failure. It doesn't fix the 403; it just lets the executable finish while Azure ML system logs remain missing. Azure ML normally stores job logs and artifacts in the workspace-associated storage account and exposes them under Outputs and logs.

    If the Storage logs show the expected workspace identity is authorized, or there is no corresponding denied Blob request, open an Azure ML support case and provide the job ID, HOSTTOOLS logs, Storage diagnostics, MRG details, and a working non-MRG comparison. Ask Azure ML engineering to trace which credential is being issued specifically for the ExperimentRun artifact upload.

    Also remove the subscription ID and managed identity object IDs from the public Q&A post and share them only through the support case.

    Help make this community better for everyone: if this answer resolved your issue, please accept it or upvote it. If not, share more details in a comment so we can continue the discussion and find the right solution.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.