Started intermittent SSL handshake issues for outbound calls from private AKS cluster hosted services

Question

Started intermittent SSL handshake issues for outbound calls from private AKS cluster hosted services

cargobuddy 30

Hello,

Suddenly some SSL handshake issues started surfacing intermittently, for the outbound calls from my AKS hosted services. My AKS cluster is a private cluster and exposes the APIs via Azure Application Gateway as the ingress. This was not the case for the same services hosted in an elastic Kubernetes hosted in AWS. The detailed exception is here:

"System.ApplicationException\n ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.\n ---> System.IO.IOException: Unable to read data from the transport connection: Connection timed out.\n ---> System.Net.Sockets.SocketException (110): Connection timed out\n at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)\n at System.Net.Sockets.NetworkStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)\n at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)\n at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)\n at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()\n at System.Threading.ThreadPoolWorkQueue.Dispatch()\n at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()\n at System.Threading.Thread.StartCallback()\n--- End of stack trace from previous location ---\n\n --- End of inner exception stack trace ---\n at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)\n at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)\n at System.Net.Security.SslStream.ReceiveHandshakeFrameAsync[TIOAdapter](CancellationToken cancellationToken)\n at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)\n at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)\n --- End of inner exception stack trace ---\n at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)\n at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\n at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\n at System.Net.Http.HttpConnectionPool.InjectNewHttp11ConnectionAsync(QueueItem queueItem)\n at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)\n at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\n at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\n at System.Net.Http.SocketsHttpHandler.<SendAsync>g_CreateHandlerAndSendAsync|115_0(HttpRequestMessage request, CancellationToken cancellationToken)\n at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>gCore|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)\n at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>gCore|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)\n at System.Net.Http.HttpClient.<SendAsync>gCore|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

Need urgent fix as it is a production issue. My HTTP client configurations are verified as per the standard .net practice recommendations.

Saraswathi Devadula 16,040 Reputation points Microsoft External Staff Moderator

2026-08-06T16:08:46.81+00:00

Hello cargobuddy

Kindly please share the details requested in the private message for further troubleshooting.

3 answers

Your answer

Saraswathi Devadula 16,040 Reputation points Microsoft External Staff Moderator

2026-08-06T16:08:46.81+00:00

Hello cargobuddy

Kindly please share the details requested in the private message for further troubleshooting.

Answer 1

SUNOJ KUMAR YELURU 18,736 MVP Volunteer Moderator

Hello @cargobuddy,

That SSL connection could not be established is misleading. The root cause is not SSL. It's TCP.

SocketException (110): Connection timed out

at SslStream.EnsureFullTlsFrameAsync

TCP SYN went out, TCP connection was never established, so TLS ServerHello never came back. SSL is the victim.

In AKS private clusters this is almost always SNAT exhaustion or hairpin routing via App Gateway. This explains why it worked in EKS - AWS NAT Gateway has 55k ports per destination and auto-scales, Azure LoadBalancer SNAT gives you ∼1024 ports per node per public IP by default.

1. Add outbound capacity immediately:

If you are on LB outbound type:

az aks update -g <RG> -n <AKS> \

--load-balancer-managed-outbound-ip-count 3 \

--load-balancer-managed-outbound-ports 16000 \

--load-balancer-idle-timeout 4

This triples your ports from ∼1024 to ∼48k per node. Node reboot not needed but takes 5-10 min.

Best long term: switch to Managed NAT Gateway:

az aks update -g <RG> -n <AKS> --outbound-type managedNATGateway --nat-gateway-managed-v4-ip-count 2

2 IPs = 128k SNAT ports.

2. Harden .NET client - this is mandatory on Azure:

Default Azure LB idle timeout is 4 min. If .NET reuses a connection after 4 min, LB already dropped the mapping -> timeout on next handshake

Also set DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP2SUPPORT=false temporarily to test if HTTP/2 multiplexing is aggravating it.

If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

cargobuddy 30 Reputation points

2026-08-07T05:07:47.2666667+00:00

@SUNOJ KUMAR YELURU just to let you know, I applied the solution 1 as you suggested. But it gave me the below error:

unrecognized arguments: --load-balancer-managed-outbound-ports 16000

Then, I removed that argument, and it worked. Hope this should work in my case.
SUNOJ KUMAR YELURU 18,736 Reputation points MVP Volunteer Moderator

2026-08-07T08:02:10.55+00:00

If you remove it, AKS uses defaults (∼1024 ports per node).

If you have 10 nodes and want 16000 ports per node, you need 4 IPs: 10 nodes * 16000 ports = 160k ports needed / 64k per IP = 3 IPs minimum

Don't use 16000 with 1 IP, it will fail validation. Use the calculation above.

For a permanent fix for a private cluster with heavy outbound, migrate to NAT Gateway:

To follow-up, Please let us know if you have further query on this and don’t forget to Accept the answer
cargobuddy 30 Reputation points

2026-08-07T08:11:57.1566667+00:00

@SUNOJ KUMAR YELURU I just got an update from the team that this issue is found even if there is very light load, in which case there are sufficient ports available.

Answer 2

Hello @SUNOJ KUMAR YELURU

After applying the above command, now we have started getting the below exception:

ExceptionMetadata { Error: "System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Unable to read data from the transport connection: Connection timed out. ---> System.Net.Sockets.SocketException (110): Connection timed out at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) at System.Net.Sockets.NetworkStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute() at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() at System.Threading.Thread.StartCallback() --- End of stack trace from previous location --- --- End of inner exception stack trace --- at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at System.Net.Security.SslStream.ReceiveHandshakeFrameAsync[TIOAdapter](CancellationToken cancellationToken) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.InjectNewHttp11ConnectionAsync(QueueItem queueItem) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.SocketsHttpHandler.<SendAsync>g__CreateHandlerAndSendAsync|115_0(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

Can you help get this fixed urgently? If required, please setup a call, as it is crucial for our production deployment to get resolved. The above exception is found with almost no traffic, with just one person triggering the APIs, so it is not related to the request load.

SUNOJ KUMAR YELURU 18,736 Reputation points MVP Volunteer Moderator

2026-08-07T10:39:08.5333333+00:00

This is not SSL. It's TCP timeout..NET just reports it during the TLS handshake because that's where it hangs.
Quick test from inside the cluster:

kubectl run -it --rm debug --image=mcr.microsoft.com/azure-cli -- bash

inside pod

for i in {1..100}; do curl -w "%{http_code} %{time_total}\n" -o /dev/null -s https://your-external-dependency.com & done

If many curls hang, it's SNAT/firewall, not code.

Send me the output of: az aks show... --query networkProfile.loadBalancerProfile

and your total node count
cargobuddy 30 Reputation points

2026-08-07T10:54:55.89+00:00

Hi @SUNOJ KUMAR YELURU Just now I moved from load balancer to NAT gateway for outbound traffic. So now, we are not facing the SSL issue, but the following issue which is totally different:

System.Exception: An exception occurred during backend call: Response entity too large: DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)

HTTP/1.1 200

server-timing: intid;desc=693c7d22039ac7d6

vary: accept-encoding

content-type: application/xml

content-length: 2747460

date: Fri, 07 Aug 2026 10:46:37 GMT

---> System.Exception: Supplier API 500 Error: An exception occurred during backend call: Response entity too large: DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)

HTTP/1.1 200

server-timing: intid;desc=693c7d22039ac7d6

vary: accept-encoding

content-type: application/xml

content-length: 2747460

date: Fri, 07 Aug 2026 10:46:37 GMT

Do you think switching to NAT gateway as you suggested earlier have resolved the problem ? If so, then what can I do to resolve the above one?

Answer 3

cargobuddy 30

Please close this incident, as the root cause of this issue was not configuring the NAT gateway for the cluster. Earlier, we had load balancer for outbound traffic, which we have now substituted with NAT gateway, and after that we are not facing any issue.

0 comments