Traffic between Databricks and Private Endpoints in Hub and Spoke Architecture

Question

Traffic between Databricks and Private Endpoints in Hub and Spoke Architecture

Sahil Chaudhry 20

I am working on deploying some of my workloads in hub and spoke in Azure where I have deployed Azure firewall and Private Endpoints for storage accounts in hub vnet and in the Spoke Vnets, I have my databricks workspace. I have setup peering between hub and spoke Vnets. I was able to access storage accounts using databricks but I wanted to provide selective access to few storage accounts from databricks and during the research for a solution for it I discovered that traffic between databricks and storage account PE was not travelling via firewall and this is the default behaviour with PEs and to overide this we need to enable network policies for private endpoint subnet and we need to create a route to force the traffic via firewall and create a allow network rule in the firewall policy to allow selected private endpoint ip addresses and deny other databricks traffic but after implementing this I am not able to reach those storage accounts at all from databricks whose ips are allowed in azure firewall network policy so I need some guidance how can this issue be resolved?

Ravi Varma Mudduluru 12,625 Reputation points Microsoft External Staff Moderator

2025-11-03T12:48:31.6933333+00:00
Hello @Sahil Chaudhry,

Thanks for reaching out to Microsoft Q&A

Could you please share the information requested below so I can better understand the issue?

Have you checked if the UDRs in the Databricks subnet are set to direct traffic to the Azure Firewall for the storage account private endpoint IP ranges, and whether network policies are currently enabled for the private endpoint subnet?

Did you add explicit allow rules for the private endpoint IPs in the firewall policy?

Has the private DNS zone for privatelink.blob.core.windows.net been linked to both the hub and spoke VNets, and have you confirmed that name resolution works from the Databricks workspace?

Are NSGs applied to the private endpoint subnet and Databricks subnet allowing traffic on port 443?

After implementing the UDR and firewall rules, what error do you see when Databricks tries to access the storage account?
Sahil Chaudhry 20 Reputation points

2025-11-03T16:36:41.4733333+00:00

@Ravi Varma Mudduluru Please find answers to the questions you asked -

Have you checked if the UDRs in the Databricks subnet are set to direct traffic to the Azure Firewall for the storage account private endpoint IP ranges, and whether network policies are currently enabled for the private endpoint subnet?

Yes UDRs in the Databricks subnet are set to direct traffic to the Azure Firewall for the storage account private endpoint IP ranges and network policies are enabled for PE subnet

Did you add explicit allow rules for the private endpoint IPs in the firewall policy?

Yes tried with both network and application rules

Has the private DNS zone for privatelink.blob.core.windows.net been linked to both the hub and spoke VNets, and have you confirmed that name resolution works from the Databricks workspace?

Yes

Are NSGs applied to the private endpoint subnet and Databricks subnet allowing traffic on port 443?

There are no NSGs on PE subnet but there are NSG on Databricks allowing traffic on port 443

After implementing the UDR and firewall rules, what error do you see when Databricks tries to access the storage account?

There is no error as the databricks jobs keeps running with no output but in the firewall logs I can see following - HTTPS request from 10.1.96.5:47264 to xxxxx.dfs.core.windows.net:443. Action: Deny. No rule matched. Proceeding with default action"

I have the application and network rule to allow this but still the rule is not getting matched. Please suggest!!
Ravi Varma Mudduluru 12,625 Reputation points Microsoft External Staff Moderator

2025-11-05T14:06:25.53+00:00

Hello @Sahil Chaudhry

Could you please check the private message and share us the requested details in the private message?

Your answer

Ravi Varma Mudduluru 12,625 Reputation points Microsoft External Staff Moderator

2025-11-03T12:48:31.6933333+00:00

Hello @Sahil Chaudhry,

Thanks for reaching out to Microsoft Q&A

Could you please share the information requested below so I can better understand the issue?

Have you checked if the UDRs in the Databricks subnet are set to direct traffic to the Azure Firewall for the storage account private endpoint IP ranges, and whether network policies are currently enabled for the private endpoint subnet?

Did you add explicit allow rules for the private endpoint IPs in the firewall policy?

Has the private DNS zone for privatelink.blob.core.windows.net been linked to both the hub and spoke VNets, and have you confirmed that name resolution works from the Databricks workspace?

Are NSGs applied to the private endpoint subnet and Databricks subnet allowing traffic on port 443?

After implementing the UDR and firewall rules, what error do you see when Databricks tries to access the storage account?
Sahil Chaudhry 20 Reputation points

2025-11-03T16:36:41.4733333+00:00

@Ravi Varma Mudduluru Please find answers to the questions you asked -

Have you checked if the UDRs in the Databricks subnet are set to direct traffic to the Azure Firewall for the storage account private endpoint IP ranges, and whether network policies are currently enabled for the private endpoint subnet?

Yes UDRs in the Databricks subnet are set to direct traffic to the Azure Firewall for the storage account private endpoint IP ranges and network policies are enabled for PE subnet

Did you add explicit allow rules for the private endpoint IPs in the firewall policy?

Yes tried with both network and application rules

Has the private DNS zone for privatelink.blob.core.windows.net been linked to both the hub and spoke VNets, and have you confirmed that name resolution works from the Databricks workspace?

Yes

Are NSGs applied to the private endpoint subnet and Databricks subnet allowing traffic on port 443?

There are no NSGs on PE subnet but there are NSG on Databricks allowing traffic on port 443

After implementing the UDR and firewall rules, what error do you see when Databricks tries to access the storage account?

There is no error as the databricks jobs keeps running with no output but in the firewall logs I can see following - HTTPS request from 10.1.96.5:47264 to xxxxx.dfs.core.windows.net:443. Action: Deny. No rule matched. Proceeding with default action"

I have the application and network rule to allow this but still the rule is not getting matched. Please suggest!!
Ravi Varma Mudduluru 12,625 Reputation points Microsoft External Staff Moderator

2025-11-05T14:06:25.53+00:00

Hello @Sahil Chaudhry

Could you please check the private message and share us the requested details in the private message?