Persistent "Failed to allocate pool: Failed to delegate" Errors Across 160 Clusters

Question

Persistent "Failed to allocate pool: Failed to delegate" Errors Across 160 Clusters

Li Zhao 20 Microsoft Employee

Summary

We are experiencing a platform-wide, persistent issue where pods fail to start due to the Azure CNI plugin (azure-vnet) being unable to allocate IP addresses. The error message is:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "...":
plugin type="azure-vnet" failed (add): Failed to allocate pool: Failed to delegate:

A subset of nodes also report a CNI plugin crash:

Failed to delegate: netplugin failed with no error message: signal: bus error

This issue affects all 160 clusters in our fleet, across all node pool types, and has been running at high volume for ~10 months without resolution.

Impact

Metric	Value
Affected clusters	160
--------	--------
Affected clusters	160
Affected cluster-pool combinations	724
Events in last 48 hours	53,626
Affected nodes in last 48 hours	43,479
Events per month (recent)	500,000 – 780,000
Duration	~10 months (since July 2025)

Pods that hit this error fail to start, causing service degradation, scheduling delays, and reduced cluster capacity. The issue affects all pool types including default pools, spot pools, GPU pools, and memory-optimized pools.

Timeline

The issue was first observed at low frequency in August 2024, then escalated dramatically in July 2025:

Period	Events/Month	Clusters Affected	Notes
2024-08 (first seen)	184	31	Sporadic, low impact
--------	--------	--------	--------
2024-08 (first seen)	184	31	Sporadic, low impact
2024-09 – 2025-06	600 – 1,700	60 – 91	Low frequency, occasional
2025-07 (inflection)	73,681	142	46x increase — suspected platform change
2025-08	437,182	133	Further 6x increase
2025-09 – 2026-05	497,000 – 784,000	128 – 160	Sustained high volume, no improvement

Error Details

Primary Error (majority of events)

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "...":
plugin type="azure-vnet" failed (add): Failed to allocate pool: Failed to delegate:

The delegate error message is empty, indicating the CNI plugin failed silently.

Secondary Error (subset of nodes)

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "...":
plugin type="azure-vnet" failed (add): Failed to allocate pool: Failed to delegate:
netplugin failed with no error message: signal: bus error

The signal: bus error indicates the azure-vnet CNI plugin binary is crashing, possibly due to memory alignment issues or binary incompatibility with the node OS.

Concurrent high volumes of NetworkNotReady events (10,000–30,000/hour) and FailedCreatePodSandBox events were also observed, confirming widespread CNI plugin instability.

Request

Root cause analysis: Investigate the azure-vnet CNI plugin's IP allocation delegation failure, particularly the signal: bus error crash.
Identify the change: Determine what platform change occurred in late June / early July 2025 that caused the 46x increase in these errors.
Fix or workaround: Provide a CNI plugin fix or a recommended workaround to eliminate these allocation failures.

Anonymous

2026-05-07T07:39:42.8033333+00:00
Hello Li Zhao,

The errors you’re seeing (Failed to allocate pool, Failed to delegate, pod sandbox failures) indicate that the Azure CNI plugin is unable to assign IP addresses to pods.

This type of failure is consistent with AKS networking/IP allocation issues, where the CNI plugin cannot successfully allocate or manage pod IPs.

Key Important Points

Azure CNI is responsible for assigning IP addresses to pods.

Pod creation will fail if the CNI plugin cannot allocate an IP address.

In standard (flat network) Azure CNI, both nodes and pods consume IPs from the virtual network, which requires careful capacity planning.

As clusters scale, insufficient IP space or high churn can lead to repeated allocation failures.

Possible Root Causes :

Below are the most likely documented causes for this class of issue:

IP Address Exhaustion (Most common)

Subnet does not have enough free IPs for pods or nodes.

Scaling, upgrades, or surge operations may temporarily increase IP demand.

This is a known limitation of traditional Azure CNI designs.

Incorrect or insufficient IP planning

Max pods per node and node count not aligned with subnet size.

Leads to inability to allocate new pod IPs as load increases.

High pod churn or allocation pressure

Frequent pod creation/deletion can increase stress on IP allocation.

May surface as intermittent allocation or delegation failures (no explicit RCA documented, but consistent with IP resource pressure).

Networking model limitations

Flat network Azure CNI requires large contiguous IP space.

If not designed for scale, allocation failures can persist across clusters.

Recommended Next Steps

Validate available IP capacity in node and pod subnets

Review maxPods configuration and subnet sizing

Check if clusters are leveraging:

Azure CNI Pod Subnet (dynamic IP allocation), or

Azure CNI Overlay (recommended for large-scale scenarios)

Collect network and CNI logs if IP capacity is sufficient but failures continue

References

https://learn.microsoft.com/en-us/azure/architecture/operator-guides/aks/troubleshoot-network-aks

https://learn.microsoft.com/en-us/azure/aks/concepts-network-ip-address-planning
Li Zhao 20 Reputation points Microsoft Employee

2026-05-07T08:55:06.6966667+00:00
We are using Azure CNI Node Subnet.

The IPV4 of k8s-subnet is 10.1.0.0/16 which means we have 65536 ips.

We set max pod per agent pool. Sum of preserved ip is around 36000 for this moment. Below shows we only use 25968 ips. ===== K8s IP Usage Summary ===== Used by Nodes: 704 Used by Pods: 24403 Used by Services: 861

ClusterIP Services: 860

LoadBalancer IPs: 1

Total Used: 25968

Li Zhao 20 Microsoft Employee

We found 1 node which was created since April 30th.

Property	Value
Created	2026-04-30 15:08 UTC (~8 days ago)

Created	2026-04-30 15:08 UTC (~8 days ago)
First IP exhaustion	2026-04-30 15:18 UTC (10 minutes after creation)
Last IP exhaustion	2026-05-08 03:27 UTC
Total IP exhaustion events	1,476
Current pod count	45
Pods without IP	31 (69%)
Pods with IP	14 (all hostNetwork system components)

That means this node has issue since it was created.

Answer accepted by question author

0 additional answers

Your answer

Li Zhao 20 Reputation points Microsoft Employee

2026-05-07T08:55:06.6966667+00:00

We are using Azure CNI Node Subnet.

The IPV4 of k8s-subnet is 10.1.0.0/16 which means we have 65536 ips.

We set max pod per agent pool. Sum of preserved ip is around 36000 for this moment. Below shows we only use 25968 ips. ===== K8s IP Usage Summary ===== Used by Nodes: 704 Used by Pods: 24403 Used by Services: 861

ClusterIP Services: 860

LoadBalancer IPs: 1

Total Used: 25968
Li Zhao 20 Reputation points Microsoft Employee

2026-05-08T04:30:47.07+00:00

We found 1 node which was created since April 30th.

Property Value

Created 2026-04-30 15:08 UTC (~8 days ago)

Created 2026-04-30 15:08 UTC (~8 days ago)

First IP exhaustion 2026-04-30 15:18 UTC (10 minutes after creation)

Last IP exhaustion 2026-05-08 03:27 UTC

Total IP exhaustion events 1,476

Current pod count 45

Pods without IP 31 (69%)

Pods with IP 14 (all hostNetwork system components)

That means this node has issue since it was created.

Answer 1

Thank you for your patience while we worked closely with the AKS engineering team to investigate this issue.

Based on the investigation conducted by the AKS and AzSecPack engineering teams, the issue has now been isolated and the root cause identified. Although the symptoms appeared as Azure CNI failures with errors such as "Failed to allocate pool: Failed to delegate" and "signal: bus error," the underlying cause was not related to IP exhaustion or networking capacity constraints.

The investigation determined that, under specific timing conditions, a race condition within the AzSecPack CertsInUse component could leave the SymCrypt OpenSSL provider configuration file in an invalid state during node initialization. This subsequently impacted OpenSSL initialization on affected Azure Linux 3 FIPS-enabled nodes and manifested as Azure CNI failures during pod startup.

The engineering team has developed a fix that makes the configuration update process atomic, preventing the configuration file from becoming corrupted even if the process is interrupted. In addition, the Azure Linux team is evaluating further improvements to increase resiliency during the FIPS initialization process.

As a mitigation, the engineering team recommends setting AzSecKeysinuseEngineInstall=False prior to node pool creation until the permanent fix is broadly available. This mitigation has been validated by the product group and is currently considered the preferred workaround.

Anonymous

2026-06-02T18:40:55.65+00:00

Hello Li Zhao,

Could you please accept the answer and upvote it.

Persistent "Failed to allocate pool: Failed to delegate" Errors Across 160 Clusters

Summary

Impact

Timeline

Error Details

Primary Error (majority of events)

Secondary Error (subset of nodes)

Related Symptoms

Request

0 additional answers

Your answer