Azure Databricks Ephemeral Worker Nodes Evaluated as Non-Compliant in Defender for Cloud Networking Policies

Marisa GG 40 Reputation points
2026-06-25T15:50:15.28+00:00

We are currently reviewing Microsoft Defender for Cloud networking policies and their impact on Azure Databricks environments.

During our analysis, our team observed that several built-in Defender for Cloud networking recommendations are reporting non-compliance for Azure Databricks worker nodes.

The recommendations involved include:

  • All network ports should be restricted on network security groups associated to your virtual machine
  • Internet-facing virtual machines should be protected with network security groups
  • Non-internet-facing virtual machines should be protected with network security groups

We have noticed that a significant portion of the non-compliant resources corresponds to Databricks ephemeral worker VMs, which are:

  • short-lived / dynamically created compute instances
  • fully managed by Azure Databricks
  • not directly customer-managed IaaS resources

Additionally, when analyzing the recommendation evaluation details in Defender for Cloud, some of these resources appear with states such as “AssessmentNotFound” or “NotApplicable / NotFound”, particularly for Databricks-related worker infrastructure.

From our understanding, this may be related to the ephemeral nature of Databricks compute nodes and their service-managed networking model.

However, the same VM networking recommendations continue to be evaluated against these resources, resulting in a significant portion of the overall subscription-level non-compliance being driven by Databricks worker infrastructure.

Given this behavior, we would like to understand:

  • whether this is expected behavior in Defender for Cloud when evaluating managed compute platforms such as Azure Databricks
  • how these “AssessmentNotFound / NotApplicable” states should be interpreted in this context
  • and what the recommended operational or remediation approach is for handling these findings in Databricks environments

We would appreciate any clarification or official guidance on how these recommendations should be interpreted for Azure Databricks workloads.

Microsoft Security | Microsoft Defender | Microsoft Defender for Cloud
Answer accepted by question author
Jyotishree Moharana 1,945 Reputation points Microsoft External Staff Moderator
2026-06-29T08:45:58.54+00:00

Hello @Marisa GG,

The behaviour you are observing Defender for Cloud flagging Azure Databricks worker nodes as non-compliant for VM-level NSG recommendations is generally expected due to the nature of Azure Databricks architecture.

Azure Databricks uses service-managed, ephemeral compute infrastructure, where:

  • Worker VMs are dynamically created and destroyed.
  • Networking is governed by Databricks-managed constructs (e.g., Network Intent Policies, service-managed NSGs) rather than traditional customer-managed VM NSGs.

As a result, Defender for Cloud VM-based recommendations may not accurately reflect the control boundaries for these resources, leading to:

  • False-positive/non-actionable non-compliance
  • States like “AssessmentNotFound” or “NotApplicable”

This happens due to:
Mismatch between Defender evaluation model and Databricks architecture

  • Defender for Cloud networking recommendations are designed for IaaS virtual machines where: NSGs are explicitly attached to NICs/subnets Customers control inbound/outbound rules
  • In contrast, Databricks worker nodes: Are not directly customer-managed VMs Are provisioned and governed by the Databricks control plane Use predefined network intent policies instead of user-controlled NSGs

Therefore, Defender applies VM policies to resources that do not expose full policy evaluation signals.

Ephemeral lifecycle of worker nodes

  • Worker VMs are: Short-lived and Frequently recycled
  • Defender relies on agent-based or configuration-based signals, which may: Not be collected in time Be removed before assessment completes

Platform-managed networking and NSG abstraction. Databricks automatically creates and manages NSGs and required rules.

 

Recommendations:

1. Treat Databricks worker findings as “platform-managed exceptions”

  • These findings are typically non-actionable for customers
  • Validate: Whether the resource belongs to a Databricks-managed resource group Whether it is part of worker/driver compute If yes classify as expected / informational

2. Use Defender Exemptions

Where appropriate:

  • Create policy exemptions (scoped): Resource group (Databricks-managed RG) Specific recommendations

This is consistent with Defender guidance allowing exemptions for non-applicable resources

3. Shift compliance focus to supported control points

Instead of VM-level NSG compliance, focus on:

a) Workspace-level networking

  • VNet injection configuration
  • Private endpoints
  • Managed VNet vs customer VNet

b) Subnet-level NSG and routing controls

  • Validate that required rules exist (Databricks → control plane, storage)
  • Ensure no conflicting custom rules override platform policies

c) Network Intent Policy alignment

  • Avoid modifying NSGs in a way that conflicts with Databricks requirements

Kindly do let us know if you have any questions or query.

Was this answer helpful?

3 people found this answer helpful.
0 comments

0 additional answers

Sort by: Newest
Most helpful Newest Oldest

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.