Edit

Verify the connection to the container registry

This article is part of a series. Start with the overview.

To successfully deploy containerized applications in your Azure Kubernetes Service (AKS) cluster, verify the connectivity between the cluster and the container registry. This step ensures that your worker nodes have the necessary permissions to pull the required container images from the registry.

Identify symptoms

When the kubelet that runs on an agent node creates the containers for a pod, one or more containers might end up in the waiting state because of an ImagePullBackOff or ErrImagePull error.

  • ImagePullBackOff is an error message in Kubernetes that indicates a failure to pull the required container image from a public or private registry. The BackOff part of the status signifies that Kubernetes continuously attempts to pull the image, with an increasing delay between each attempt. The delay gradually increases until it reaches a predetermined limit, which is typically set to 300 seconds in Kubernetes.

  • ErrImagePull often precedes ImagePullBackOff and indicates that the initial pull attempt failed.

Various factors can cause these errors, including network connectivity problems, an incorrect image name or tag, insufficient permissions, or missing credentials. Verify the registry URL, image name, and tag, and confirm that the cluster identity has the necessary permissions to pull from the registry.

Role assignments

When you attach a container registry to an existing AKS cluster, the AcrPull role is automatically assigned to the Microsoft Entra managed identity that's associated with the agent pools in your cluster. For more information, see Authenticate with Container Registry from AKS.

Run the following command to retrieve the kubelet managed identity of a Kubernetes cluster and its current role assignments:

# Get the kubelet managed identity.
ASSIGNEE=$(az aks show -g $RESOURCE_GROUP -n $NAME --query identityProfile.kubeletidentity.clientId -o tsv)
az role assignment list --assignee $ASSIGNEE --all -o table

Run the following command to assign the AcrPull role to the kubelet managed identity:

AZURE_CONTAINER_REGISTRY_ID=$(az acr show --name <container-registry-name> --query id --output tsv)
az role assignment create --assignee $ASSIGNEE --scope $AZURE_CONTAINER_REGISTRY_ID --role acrpull

ABAC-enabled registries

If your Container Registry uses Azure attribute-based access control (ABAC) repository permissions, it doesn't honor the classic AcrPull role. ABAC-enabled registries also don't support the az aks create --attach-acr and az aks update --attach-acr integrations. Instead, you must manually assign the Container Registry Repository Reader role to the kubelet managed identity. For more information, see Azure ABAC repository permissions.

If you recently migrated your registry to ABAC-enabled mode and image pulls fail, verify that the kubelet identity is assigned an ABAC-compatible role.

Validate connectivity

To check if your AKS cluster can reach the container registry, run the following command:

az aks check-acr --name $NAME --resource-group $RESOURCE_GROUP --acr <container-registry-name>.azurecr.io

This command checks Domain Name System (DNS) resolution, network routing, and authentication between the AKS cluster and the registry. Use it as a first diagnostic step when you suspect connectivity or permission problems.

Troubleshoot Container Registry problems

Use the following guidance to diagnose networking, sign-in, and performance problems.

Troubleshoot networking problems

If you encounter problems accessing Container Registry because of virtual network restrictions, firewall rules, or proxy server configurations, consider the following solutions:

Troubleshoot sign-in problems

If you encounter authentication and authorization problems when you sign in to Container Registry, consider the following solutions:

Troubleshoot performance problems

If you encounter performance problems with a container registry, consider the following solutions:

If pull latency increases suddenly on a geo-replicated registry without any AKS configuration changes, check Azure Resource Health for the registry. When a regional replica degrades, Container Registry failover automatically reroutes traffic from that replica to a healthy replica through the global endpoint (<registry>.azurecr.io). The global endpoint can route AKS nodes to a replica in a more distant region. Pull latency returns to baseline after the local replica recovers.

Integrate a non-Microsoft container registry

When you use a non-Microsoft container registry, you need to create the appropriate ImagePullSecret credentials for the registry so your AKS cluster can securely access the container images. For more information, see Create an image pull secret.

Ensure that you set up the correct permissions and credentials so that you can verify the connection to the container registry and your AKS cluster can successfully pull the required container images during deployments.

Contributors

Microsoft maintains this article. The following contributors wrote this article.

Principal authors:

Other contributor:

  • Sam Cogan | Senior Cloud Solution Architect

To see nonpublic LinkedIn profiles, sign in to LinkedIn.

Next steps

  • Last updated on