alb-controller failing in AKS after update to 1.11.1_rc14 - Application Gateway reports "no healthy upstream"

Question

alb-controller failing in AKS after update to 1.11.1_rc14 - Application Gateway reports "no healthy upstream"

Paul Wysocki 0

3 days ago ALB in my AKS cluster (thankfully in my dev environment) was automatically updated to version 1.11.1 from 1.10.27.

the logs for alb-controller show:

{"level":"info","version":"1.11.1","Timestamp":"2026-06-26T17:58:19.846993381Z","message":"Starting alb-controller version=1.11.1-rc14 build=prod"}

{"level":"info","version":"1.11.1","Timestamp":"2026-06-26T17:58:19.87380044Z","message":"Starting alb-controller version=1.11.1-rc14 build=prod"}

{"level":"fatal","version":"1.11.1","error":"reconciler failed to watch resources: no matches for kind "ReferenceGrant" in version "gateway.networking.k8s.io/v1"","Timestamp":"2026-06-26T17:58:19.946964828Z","message":"Internal error: Reconciler client failed to start in ALB Controller."}

My cluster does NOT have a crd for ReferenceGrant with version "v1". Instead it has version "v1beta1" only. I did not install the crds directly, all of this was installed using the managed extension for Application Gateway for Containers using the following bicep snippet:

    ingressProfile: {
      applicationLoadBalancer: {
        enabled: true
      }
      gatewayAPI: {
        installation: 'Standard'
      }
    }

Why didn't the automatic update provide the proper crd for ReferenceGrant as part of the update? And how do I fix this for both my dev environment and prevent it from happening in my production environment.

Paul Wysocki 0 Reputation points

2026-06-26T18:17:54.0933333+00:00

Running: kubectl rollout undo deployment/alb-controller -n kube-system
gave me some relief yesterday, but the 1.11.1 pods kept running - one 1.10.27 pod started up and this was able to configure my routes. However, that 1.10.27 pod stopped somewhere between then and now.
Paul Wysocki 0 Reputation points

2026-06-26T18:20:35.2366667+00:00
Here's the version of the ReferenceGrant

$ kubectl get crd referencegrants.gateway.networking.k8s.io -o=jsonpath="{.spec.versions[*].name}" v1beta1
Shree Hima Bindu Maganti 7,590 Reputation points Microsoft External Staff Moderator

2026-06-26T18:34:21.44+00:00
hi @Paul Wysocki
sounds like your upgrade to ALB Controller 1.11.1-rc14 (running in the AKS add-on) hit a Kubernetes API compatibility issue.

What the logs are telling us

Your alb-controller pod is failing like this:

reconciler failed to watch resources: no matches for kind "ReferenceGrant" in version "gateway.networking.k8s.io/v1"

Then you confirmed:

The cluster has the CRD referencegrants.gateway.networking.k8s.io with version v1beta1 only

There’s no v1 version installed.

So when the controller tries to watch ReferenceGrant at gateway.networking.k8s.io/v1, it can’t find that CRD version and the controller fails to start its reconciler. That also explains the downstream symptom you mentioned (“Application Gateway reports no healthy upstream”), because the controller never fully comes up to configure/repair the routing to your backends.

Why the automatic update didn’t “just add” the right CRD

Based on the information provided, the add-on update brought in a newer ALB Controller that expects Gateway API-related CRDs/versions to include ReferenceGrant at gateway.networking.k8s.io/v1. However, your cluster only had ReferenceGrant at v1beta1.

The docs you provided don’t include the specific mechanism that the managed Application Gateway for Containers extension uses to upgrade/align CRDs during ALB controller updates—so from the given materials we can’t conclusively say why the CRDs were not updated alongside the controller. But the outcome is clear: controller and CRD API versions are mismatched, causing the controller to crash loop/fail its reconciler.

How to fix (for dev + prevent in prod)

The relevant docs you provided don’t include an explicit “CRD version migration” procedure for ReferenceGrant, so the safest concrete guidance I can provide (from the provided troubleshooting materials) is:

Confirm which ALB Controller version is deployed

Use the troubleshooting guide command to ensure you’re really on what you think:

kubectl get deployment -n azure-alb-system -o wide

Ensure the CRD API versions the controller expects exist

Since your controller log indicates it specifically needs ReferenceGrant under:

gateway.networking.k8s.io/v1

And your cluster only has v1beta1, you’ll need to align the cluster CRDs to include the expected API version.

Apply the “last known good” rollback while you fix CRDs

kubectl rollout undo deployment/alb-controller -n kube-system

That corresponds to the troubleshooting theme: restore controller functionality while you correct the underlying mismatch.

Prevention in production

The docs provided say that AKS add-on updates are automatic (“If using the AKS add-on, updates are automatically applied to the cluster.”).

Practically, to prevent recurrence, you’d want to confirm before/after automatic ALB controller updates that the needed Gateway API CRDs/versions are present for that controller release.

Also, since the controller release notes show the ALB Controller version corresponds to a Gateway API version, you can use that mapping as a checklist during change windows:

Current “Latest Release (Recommended)” in your docs shows ALB Controller 1.11.1 uses Gateway API v1.5.1.

That’s consistent with the idea that CRD expectations may evolve.
https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/alb-controller-release-notes
https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/troubleshooting-guide?tabs=active-pod-linux
Let me know if you have any further assistances needed.

Paul Wysocki 0

Here's the output from that kubectl command:

PS C:\src\az\CloudAgent> kubectl get deployment -n kube-system -o wide
NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS                          IMAGES                                                                                                                                 SELECTOR
alb-controller                        1/2     2            1           61d    alb-controller                      mcr.microsoft.com/application-lb/images/alb-controller:1.11.1                                                                          app=alb-controller
azure-wi-webhook-controller-manager   2/2     2            2           425d   manager                             mcr.microsoft.com/oss/v2/azure/workload-identity/webhook:v1.5.1-9                                                                      azure-workload-identity.io/system=true,kubernetes.azure.com/managedby=aks
coredns                               2/2     2            2           425d   coredns                             mcr.microsoft.com/oss/v2/kubernetes/coredns:v1.13.1-12                                                                                 k8s-app=kube-dns,version=v20
coredns-autoscaler                    1/1     1            1           425d   autoscaler                          mcr.microsoft.com/oss/v2/kubernetes/autoscaler/cluster-proportional-autoscaler:v1.9.0-15                                               k8s-app=coredns-autoscaler
ingress-appgw-deployment              1/1     1            1           425d   ingress-appgw-container             mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.9.10                                                                  app=ingress-appgw
konnectivity-agent                    2/2     2            2           425d   konnectivity-agent                  mcr.microsoft.com/oss/v2/kubernetes/apiserver-network-proxy/agent:v0.32.1-5                                                            app=konnectivity-agent
konnectivity-agent-autoscaler         1/1     1            1           425d   autoscaler                          mcr.microsoft.com/oss/v2/kubernetes/autoscaler/cluster-proportional-autoscaler:v1.9.0-15                                               app=konnectivity-agent-autoscaler
metrics-server                        2/2     2            2           425d   metrics-server-vpa,metrics-server   mcr.microsoft.com/oss/v2/kubernetes/autoscaler/addon-resizer:v1.8.23-17,mcr.microsoft.com/oss/v2/kubernetes/metrics-server:v0.8.0-13   k8s-app=metrics-server

Note that my system has it in kube-system as the namespace.

Paul Wysocki 0

And, for what it's worth - it looks like the add-on deployed gateway api v1.3.0, not 1.5.1:

apiVersion: apiextensions.k8s.io/v1

kind: CustomResourceDefinition

metadata:

  annotations:


api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/3328

app.kubernetes.io/managed-by: aks

app.kubernetes.io/part-of: 640e285647d025d1

gateway.networking.k8s.io/bundle-version: v1.3.0

gateway.networking.k8s.io/channel: standard  (snipped)

2 answers

Your answer

Paul Wysocki 0 Reputation points

2026-06-26T18:17:54.0933333+00:00

Running: kubectl rollout undo deployment/alb-controller -n kube-system
gave me some relief yesterday, but the 1.11.1 pods kept running - one 1.10.27 pod started up and this was able to configure my routes. However, that 1.10.27 pod stopped somewhere between then and now.
Paul Wysocki 0 Reputation points

2026-06-26T18:20:35.2366667+00:00

Here's the version of the ReferenceGrant

$ kubectl get crd referencegrants.gateway.networking.k8s.io -o=jsonpath="{.spec.versions[*].name}" v1beta1
Paul Wysocki 0 Reputation points

2026-06-26T18:42:53.7866667+00:00

Here's the output from that kubectl command:

PS C:\src\az\CloudAgent> kubectl get deployment -n kube-system -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR alb-controller 1/2 2 1 61d alb-controller mcr.microsoft.com/application-lb/images/alb-controller:1.11.1 app=alb-controller azure-wi-webhook-controller-manager 2/2 2 2 425d manager mcr.microsoft.com/oss/v2/azure/workload-identity/webhook:v1.5.1-9 azure-workload-identity.io/system=true,kubernetes.azure.com/managedby=aks coredns 2/2 2 2 425d coredns mcr.microsoft.com/oss/v2/kubernetes/coredns:v1.13.1-12 k8s-app=kube-dns,version=v20 coredns-autoscaler 1/1 1 1 425d autoscaler mcr.microsoft.com/oss/v2/kubernetes/autoscaler/cluster-proportional-autoscaler:v1.9.0-15 k8s-app=coredns-autoscaler ingress-appgw-deployment 1/1 1 1 425d ingress-appgw-container mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.9.10 app=ingress-appgw konnectivity-agent 2/2 2 2 425d konnectivity-agent mcr.microsoft.com/oss/v2/kubernetes/apiserver-network-proxy/agent:v0.32.1-5 app=konnectivity-agent konnectivity-agent-autoscaler 1/1 1 1 425d autoscaler mcr.microsoft.com/oss/v2/kubernetes/autoscaler/cluster-proportional-autoscaler:v1.9.0-15 app=konnectivity-agent-autoscaler metrics-server 2/2 2 2 425d metrics-server-vpa,metrics-server mcr.microsoft.com/oss/v2/kubernetes/autoscaler/addon-resizer:v1.8.23-17,mcr.microsoft.com/oss/v2/kubernetes/metrics-server:v0.8.0-13 k8s-app=metrics-server

Note that my system has it in kube-system as the namespace.
Paul Wysocki 0 Reputation points

2026-06-26T18:45:40.1866667+00:00

And, for what it's worth - it looks like the add-on deployed gateway api v1.3.0, not 1.5.1:

apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/3328 app.kubernetes.io/managed-by: aks app.kubernetes.io/part-of: 640e285647d025d1 gateway.networking.k8s.io/bundle-version: v1.3.0 gateway.networking.k8s.io/channel: standard (snipped)

Answer 1

Sina Salam 31,456 Volunteer Moderator

Hello Paul Wysocki,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that your ALB Controller in AKS failed after the managed add-on upgraded from 1.10.27 to 1.11.1 / 1.11.1-rc14, causing Application Gateway for Containers to report “no healthy upstream.”

The failure was caused by a Gateway API CRD version mismatch. The upgraded ALB Controller 1.11.1 requires Gateway API v1.5.1, but the cluster still exposes ReferenceGrant only as gateway.networking.k8s.io/v1beta1, while the controller is trying to watch gateway.networking.k8s.io/v1.

The fact is that ALB Controller 1.11.1 uses Gateway API v1.5.1, and AKS Managed Gateway API CRD versions are managed separately from the ALB Controller add-on. - https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/alb-controller-release-notes, https://learn.microsoft.com/en-us/azure/aks/managed-gateway-api

Therefore the resolution is to:

Upgrade or reconcile the Gateway API standard CRDs so that ReferenceGrant is available as gateway.networking.k8s.io/v1.
Restart the alb-controller deployment after the CRDs are corrected.
Do not rely on kubectl rollout undo for the managed add-on, because AKS managed add-on updates are automatically reapplied.
For production, open an Azure support case to confirm or force alignment between the managed ALB Controller version and the managed Gateway API CRD bundle before the same update reaches production.

After the Gateway API CRDs are aligned with the ALB Controller requirement, the controller can start successfully, reconcile routes again, and restore Application Gateway for Containers configuration.

Use the below resource links for more reading and steps:

I hope this is helpful. Please! Do not hesitate to let me know if you have any other questions, steps or clarifications.

Please do not close the thread by upvoting and accepting the answer if any part of it is helpful.

Lasse Mammen 0 Reputation points

2026-06-27T20:49:49.42+00:00

Thanks for this Sina, however, according to the support matrix on the Gateway API only Kubernetes 1.36+ is supported, but on my cluster on 1.34 the ALB controller newer version was also used. Does that effectively mean that the only path forward is either not using Bicep to deploy and use helm instead or upgrade Kubernetes. It sounds to me like a bug that that newer version was pushed to lower Kubernetes versions given the compatibility mismatch.

Thanks,
Lasse
Sina Salam 31,456 Reputation points Volunteer Moderator

2026-06-28T10:36:16.7366667+00:00

Hi Lasse Mammen,

Thank you for your feedback.

I agree with your interpretation. On AKS 1.34 the Managed Gateway API support matrix maps the standard Gateway API bundle to v1.3.0, while ALB Controller 1.11.1 expects Gateway API v1.5.1. Since Managed Gateway API v1.5.1 is only listed as supported on Kubernetes 1.36.0+, a 1.34 cluster cannot be reconciled to the required managed CRD version through the supported AKS managed path.

So, this does not look like a user-side Bicep issue. It appears to be a managed add-on version skew though, the ALB Controller was upgraded to a version that requires newer Gateway API CRDs than the AKS Managed Gateway API installation supports for that Kubernetes minor version.

For a managed add-on deployment, you might need Microsoft Internal Azure support for safety purpose and reference the ALB Controller/Gateway API mismatch, or upgrade the AKS cluster to a Kubernetes version that supports Gateway API v1.5.1 when available. Moving to Helm/self-managed ALB Controller may allow pinning versions, but that changes the support and operational model.

Left to me I would not manually force-install v1.5.1 CRDs on an AKS 1.34 managed Gateway API cluster unless Microsoft support explicitly recommends it.

cheers.
Paul Wysocki 0 Reputation points

2026-07-01T20:17:28.9166667+00:00

My system is now working. It appears to have been updated with alb-controller v1.10.30. on 6/29 at 22:47Z

The CRD for ReferenceGrants is still v1beta1. I think the downgrade from 1.11.x alb is what did the fix.

Answer 2

In the end, my solution was to wait. It looks like AKS adjusted their deployment strategy to deploy alb version 1.10.30 instead of (the too-early) 1.11.1-rc14.

I received temporary relief by doing

kubectl rollout undo deployment/alb-controller -n kube-system

But, I did need to do this two days in a row as my 1.10.27 pod stopped - leaving only a couple of 1.11.1 pods (that were continually restarting).

In my case, this was a dev environment, and I could leave it in this precarious state for a bit.

6 days after the update to 1.11.1, my AKS cluster was 'downgraded' by the managed services to 1.10.30

This version works and is content with ReferenceGrants CRD at the v1beta1 version.