alb-controller failing in AKS after update to 1.11.1_rc14 - Application Gateway reports "no healthy upstream"

Paul Wysocki 0 Reputation points
2026-06-26T18:11:19.8433333+00:00

3 days ago ALB in my AKS cluster (thankfully in my dev environment) was automatically updated to version 1.11.1 from 1.10.27.

the logs for alb-controller show:

{"level":"info","version":"1.11.1","Timestamp":"2026-06-26T17:58:19.846993381Z","message":"Starting alb-controller version=1.11.1-rc14 build=prod"}

{"level":"info","version":"1.11.1","Timestamp":"2026-06-26T17:58:19.87380044Z","message":"Starting alb-controller version=1.11.1-rc14 build=prod"}

{"level":"fatal","version":"1.11.1","error":"reconciler failed to watch resources: no matches for kind "ReferenceGrant" in version "gateway.networking.k8s.io/v1"","Timestamp":"2026-06-26T17:58:19.946964828Z","message":"Internal error: Reconciler client failed to start in ALB Controller."}

My cluster does NOT have a crd for ReferenceGrant with version "v1". Instead it has version "v1beta1" only. I did not install the crds directly, all of this was installed using the managed extension for Application Gateway for Containers using the following bicep snippet:

    ingressProfile: {
      applicationLoadBalancer: {
        enabled: true
      }
      gatewayAPI: {
        installation: 'Standard'
      }
    }

Why didn't the automatic update provide the proper crd for ReferenceGrant as part of the update? And how do I fix this for both my dev environment and prevent it from happening in my production environment.

Azure Load Balancer
Azure Load Balancer

An Azure service that delivers high availability and network performance to applications.

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Paul Wysocki 0 Reputation points
    2026-07-01T20:23:32.73+00:00

    In the end, my solution was to wait. It looks like AKS adjusted their deployment strategy to deploy alb version 1.10.30 instead of (the too-early) 1.11.1-rc14.

    I received temporary relief by doing

    kubectl rollout undo deployment/alb-controller -n kube-system

    But, I did need to do this two days in a row as my 1.10.27 pod stopped - leaving only a couple of 1.11.1 pods (that were continually restarting).

    In my case, this was a dev environment, and I could leave it in this precarious state for a bit.

    6 days after the update to 1.11.1, my AKS cluster was 'downgraded' by the managed services to 1.10.30

    This version works and is content with ReferenceGrants CRD at the v1beta1 version.

    Was this answer helpful?

    0 comments
  2. Sina Salam 31,456 Reputation points Volunteer Moderator
    2026-06-27T18:05:44.8+00:00

    Hello Paul Wysocki,

    Welcome to the Microsoft Q&A and thank you for posting your questions here.

    I understand that your ALB Controller in AKS failed after the managed add-on upgraded from 1.10.27 to 1.11.1 / 1.11.1-rc14, causing Application Gateway for Containers to report “no healthy upstream.”

    The failure was caused by a Gateway API CRD version mismatch. The upgraded ALB Controller 1.11.1 requires Gateway API v1.5.1, but the cluster still exposes ReferenceGrant only as gateway.networking.k8s.io/v1beta1, while the controller is trying to watch gateway.networking.k8s.io/v1.

    The fact is that ALB Controller 1.11.1 uses Gateway API v1.5.1, and AKS Managed Gateway API CRD versions are managed separately from the ALB Controller add-on. - https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/alb-controller-release-notes, https://learn.microsoft.com/en-us/azure/aks/managed-gateway-api

    Therefore the resolution is to:

    • Upgrade or reconcile the Gateway API standard CRDs so that ReferenceGrant is available as gateway.networking.k8s.io/v1.
    • Restart the alb-controller deployment after the CRDs are corrected.
    • Do not rely on kubectl rollout undo for the managed add-on, because AKS managed add-on updates are automatically reapplied.
    • For production, open an Azure support case to confirm or force alignment between the managed ALB Controller version and the managed Gateway API CRD bundle before the same update reaches production.

    After the Gateway API CRDs are aligned with the ALB Controller requirement, the controller can start successfully, reconcile routes again, and restore Application Gateway for Containers configuration.

    Use the below resource links for more reading and steps:

    I hope this is helpful. Please! Do not hesitate to let me know if you have any other questions, steps or clarifications.

    Please do not close the thread by upvoting and accepting the answer if any part of it is helpful.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.