Azure Machine Learning Pipeline job failing during MLflow model logging on custom Docker environment

Question

Azure Machine Learning Pipeline job failing during MLflow model logging on custom Docker environment

Lac Oha 20

I am running an Azure ML v2 CLI pipeline job utilizing a custom curated environment built from Docker file base

mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04. The compute cluster uses Standard_NC6s_v3 instances.

When the training script reaches mlflow.pytorch.log_model(), the pipeline fails with a permission denied error attempting to

write to /mnt/azureml/cr/. The log error reads:

PermissionError: [Errno 13] Permission denied: '/mnt/azureml/cr/...'

Is there a specific environment variable or non-root user setup required in custom Dockerfiles for AML pipeline outputs to

successfully write MLflow artifacts?

0 comments

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Hi,

This PermissionError: [Errno 13] occurs because Azure ML v2 pipeline runners mount the container runtime directory /mnt/azureml/cr/ with root ownership, while custom Dockerfiles using non-root USER directives lack write permissions to mounted paths.

Here is how to resolve it:

Grant Permissions in Dockerfile: Pre-create and grant write access to the mount path in your custom Dockerfile before switching users:
```
   RUN mkdir -p /mnt/azureml && chmod -R 777 /mnt/azureml
```
Redirect MLflow Temp Directory: In your Python script, redirect MLflow's local staging directory to /tmp before calling mlflow.pytorch.log_model():
```
   import os
   os.environ["MLFLOW_TMP_DIR"] = "/tmp/mlflow"
```
Avoid Restrictive Non-Root Users: Ensure your custom base image preserves default execution rights for Azure ML runtime mounts.

Ref: Create & Manage Custom Environments in Azure Machine Learning v2

If this helped resolve your issue, please mark as Accept Answer!