APIM not able to connect to Load Balancer even though they all are on the Public IP

Question

APIM not able to connect to Load Balancer even though they all are on the Public IP

Kartik Mahajan CONSULTANT 0

APIM Public IP .

LB frontend Public IP 20.***.***31

Backend pool -- 10.*.4 / 5 / 6 --- All healthy via probe Service Listening on '8096' --- netstat -tulnp | grep 8096 confirms it

Firewall rules -> Inbound - az_sitaA_dev_apim_to_LB_rh9 (APIM → LB) az_sitaA_dev_api_lb_rh9_to_rh9_docker

(LB → backend servers) Test from backend VM → LB public IP --- times out --- curl http://LB Public IP:8096(Note it is not working from RHEL 7 servers too)

Test from backend VM → APIM public IP --- works --- curl -v http://APIM public IP

When doing the GET API call Trace test from APIM gets error - forward-request (21,044.392 ms) { "messages": [ "Unable to connect to the remote server", "Error occured while calling backend service.", "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond LB Public IP:8096" ] }

Anonymous

2025-11-17T18:47:27.9966667+00:00
Hello @Kartik Mahajan,

We have requested some details in a private message. Could you please check and share them in message with us?

And also, please share the the below information

Could you confirm the exact load‑balancing rule you have on the LB for port 8096 (frontend port, backend port, probe, and whether any Inbound NAT rule also uses 8096)?

Are there any particular NSGs assigned to your Load Balancer or backend VMs? If so, what are the current inbound rules?

Regard's
Jeevan S
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-19T11:34:36.1366667+00:00

Hi Jeevan,
I have shared the asked details with you in Private message.
Also for the asked Question -
Can we connect and then I can show you the results. Not sure if sharing the details is safe in Public community.
Anonymous

2025-11-19T17:49:16.5066667+00:00
Hello @Kartik Mahajan,

Thank you for sharing the details in a private message. Please avoid sharing personal information in public channels.

Thank you for the details about the issue between Azure API Management (APIM) and the Load Balancer (LB).

The problem seems to be a network connectivity issue rather than APIM itself. Even though APIM and the LB have public IPs, APIM cannot reach the LB on port 8096. Your backend VMs are healthy and listening on port 8096, but tests from these VMs to the LB’s public IP on that port time out.

Here are some key steps to check:

Verify firewall and Network Security Group (NSG) rules to make sure port 8096 is open between backend VMs and the LB.

Confirm the LB health probe matches your service port and protocol correctly.

Check if routing allows backend VMs to access the LB’s public IP; sometimes using the LB’s private IP or an Internal Load Balancer works better.

Test if the LB’s public IP and port 8096 are reachable externally to isolate the issue.

Review APIM logs to ensure no outbound port exhaustion is happening, which can cause connection failures.

Reference:

Troubleshooting Azure Load Balancer backend traffic and connectivity: Troubleshoot Azure Load Balancer | Microsoft Learn

Troubleshooting connectivity and timeout errors in Azure API Management: https://learn.microsoft.com/en-us/azure/api-management/troubleshoot-response-timeout-and-errors

Kindly let us know if the above helps or you need further assistance on this issue.
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-20T07:11:18.66+00:00

All the above steps mentioned were already done and was shown to you over the call.
Regarding the Last point mentioned - Review APIM logs to ensure no outbound port exhaustion is happening, which can cause connection failures.

I am not able to see the diagnostics logs the way it was stated by one of your support person yesterday.
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-20T09:33:10.6+00:00

All the above steps were already done. The Last step of collecting APIM logs that part we looked into it tomorrow and the way you asked to look after the 15 min of SYNC is still not active.
Please instead of going in circles lets connect and give it a sure shot.
Anonymous

2025-11-20T19:50:53.06+00:00

Hello @Kartik Mahajan,
Thank you for the update.
API Management doesn’t perform proactive probing of backend services. You can manage outgoing request timeouts using the timeout property in the forward-request policy:

https://docs.microsoft.com/en-us/azure/api-management/api-management-advanced-policies#ForwardRequest

If you’re using the set-backend-service policy, include logic within the sf-resolve-condition property to make sure API Management re-resolves the backend when a timeout occurs. It’s usually better not to trigger re-resolution on every timeout, since timeouts may also happen when the service is just slow, not unreachable. To handle this smartly, apply the condition only for specific APIs, operations, or URLs where you really need it.

Kindly let us know if the above helps or you need further assistance on this issue.
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-21T14:38:46.27+00:00

Let me look. I will get back to you if it still fails
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-21T16:42:28.5633333+00:00

Seems like this document is not of a big help.
Can you guide us over the call to implement it?
Anonymous

2025-11-21T17:59:20.5966667+00:00

Hello @Kartik Mahajan,
I have posted my comment in a private chat.
Ganesh Patapati 12,170 Reputation points Microsoft External Staff Moderator

2025-11-26T09:22:23.3566667+00:00

Hello Kartik Mahajan CONSULTANT

I have initiated a private message with you. Please share the required details there so I can further investigate the issue.

Your answer

Anonymous

2025-11-17T18:47:27.9966667+00:00

Hello @Kartik Mahajan,

We have requested some details in a private message. Could you please check and share them in message with us?

And also, please share the the below information

Could you confirm the exact load‑balancing rule you have on the LB for port 8096 (frontend port, backend port, probe, and whether any Inbound NAT rule also uses 8096)?

Are there any particular NSGs assigned to your Load Balancer or backend VMs? If so, what are the current inbound rules?

Regard's
Jeevan S
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-19T11:34:36.1366667+00:00

Hi Jeevan,
I have shared the asked details with you in Private message.
Also for the asked Question -
Can we connect and then I can show you the results. Not sure if sharing the details is safe in Public community.
Anonymous

2025-11-19T17:49:16.5066667+00:00

Hello @Kartik Mahajan,

Thank you for sharing the details in a private message. Please avoid sharing personal information in public channels.

Thank you for the details about the issue between Azure API Management (APIM) and the Load Balancer (LB).

The problem seems to be a network connectivity issue rather than APIM itself. Even though APIM and the LB have public IPs, APIM cannot reach the LB on port 8096. Your backend VMs are healthy and listening on port 8096, but tests from these VMs to the LB’s public IP on that port time out.

Here are some key steps to check:

Verify firewall and Network Security Group (NSG) rules to make sure port 8096 is open between backend VMs and the LB.

Confirm the LB health probe matches your service port and protocol correctly.

Check if routing allows backend VMs to access the LB’s public IP; sometimes using the LB’s private IP or an Internal Load Balancer works better.

Test if the LB’s public IP and port 8096 are reachable externally to isolate the issue.

Review APIM logs to ensure no outbound port exhaustion is happening, which can cause connection failures.

Reference:

Troubleshooting Azure Load Balancer backend traffic and connectivity: Troubleshoot Azure Load Balancer | Microsoft Learn

Troubleshooting connectivity and timeout errors in Azure API Management: https://learn.microsoft.com/en-us/azure/api-management/troubleshoot-response-timeout-and-errors

Kindly let us know if the above helps or you need further assistance on this issue.
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-20T07:11:18.66+00:00

All the above steps mentioned were already done and was shown to you over the call.
Regarding the Last point mentioned - Review APIM logs to ensure no outbound port exhaustion is happening, which can cause connection failures.

I am not able to see the diagnostics logs the way it was stated by one of your support person yesterday.
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-20T09:33:10.6+00:00

All the above steps were already done. The Last step of collecting APIM logs that part we looked into it tomorrow and the way you asked to look after the 15 min of SYNC is still not active.
Please instead of going in circles lets connect and give it a sure shot.
Anonymous

2025-11-20T19:50:53.06+00:00

Hello @Kartik Mahajan,
Thank you for the update.
API Management doesn’t perform proactive probing of backend services. You can manage outgoing request timeouts using the timeout property in the forward-request policy:

https://docs.microsoft.com/en-us/azure/api-management/api-management-advanced-policies#ForwardRequest

If you’re using the set-backend-service policy, include logic within the sf-resolve-condition property to make sure API Management re-resolves the backend when a timeout occurs. It’s usually better not to trigger re-resolution on every timeout, since timeouts may also happen when the service is just slow, not unreachable. To handle this smartly, apply the condition only for specific APIs, operations, or URLs where you really need it.

Kindly let us know if the above helps or you need further assistance on this issue.
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-21T14:38:46.27+00:00

Let me look. I will get back to you if it still fails
Kartik Mahajan CONSULTANT 0 Reputation points

2025-11-21T16:42:28.5633333+00:00

Seems like this document is not of a big help.
Can you guide us over the call to implement it?
Anonymous

2025-11-21T17:59:20.5966667+00:00

Hello @Kartik Mahajan,
I have posted my comment in a private chat.
Ganesh Patapati 12,170 Reputation points Microsoft External Staff Moderator

2025-11-26T09:22:23.3566667+00:00

Hello Kartik Mahajan CONSULTANT

I have initiated a private message with you. Please share the required details there so I can further investigate the issue.