Recurring unexplained VM deallocation

Joel Zhu 0 Reputation points Microsoft Employee
2026-08-09T06:36:38.6233333+00:00

Availability Status: Degraded

Title: Customer Nodes Power Off

Reason Type: Customer Initiated

Summary: Multiple nodes could not be found in running/powered on state.

What we've verified:

  • Confirmed via az vmss get-instance-view that the affected instances show PowerState/deallocated (not just NotReady in Kubernetes) — this is a genuine Azure-level VM power-off, not a Kubernetes/kubelet issue.
  • Fix applied each time: az vmss start on the affected instance IDs — this reliably brings nodes back to Ready within minutes, confirming the VMs themselves are otherwise healthy.
  • Checked az monitor activity-log list for the affected resource group AND subscription-wide, across the full day of each incident, filtering for any deallocate/stop/powerOff operation — found zero matching entries at any scope.
  • Ruled out: DevTestLab auto-shutdown schedules (Microsoft.DevTestLab/schedules) and Azure Automation Accounts (Microsoft.Automation/automationAccounts) — none exist anywhere in this subscription (checked via Azure Resource Graph).
  • Ruled out: Spot VM eviction — confirmed scaleSetPriority: Regular on both affected node pools.
  • Considered but not confirmed: AKS's nodeOsUpgradeChannel: NodeImage auto-upgrade setting — found one coincidentally-timed manualupgrade action attributed to the AKS control plane's own managed identity ~7 minutes before one incident, but this doesn't explain the other occurrences and doesn't have a corresponding entry for most of them.

What we need from support:

  1. Please check internal telemetry (beyond what's exposed via Activity Log/Resource Graph) for what triggered the VM deallocation at each timestamp above — specifically the actual caller/operation that set PowerState to deallocated on these specific VMSS instances.
  2. If this is tied to a subscription-level policy, quota-management action, or some other Azure-internal process we're not aware of, please advise how to identify or disable it.
  3. If it is something on our end that isn't surfacing in the tools available to us, guidance on how to find it would be appreciated.
Azure Kubernetes Service
Azure Kubernetes Service

An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.

0 comments

2 answers

Sort by: Oldest
Most helpful Newest Oldest
  1. Manasa Akula 770 Reputation points Microsoft External Staff Moderator
    2026-08-09T07:30:31+00:00

    Hello Joel Zhu,

    Thank you for providing the detailed investigation results and timestamps.

    We have reviewed the information shared and agree with your findings that this is not a Kubernetes-level node readiness issue. The affected AKS node pool instances were observed in PowerState/deallocated state at the Azure VM Scale Set level, and manually starting the instances successfully restored the nodes to a healthy Ready state.

    Based on the validation already completed, the following potential causes have been ruled out:

    • Spot VM eviction
    • DevTest Lab auto-shutdown schedules
    • Azure Automation Account runbooks
    • Kubernetes kubelet or node readiness issues

    At this time, the available evidence does not indicate a platform-wide Azure service issue. However, additional backend investigation is required to determine the exact operation or caller responsible for transitioning the VMSS instances into the deallocated state.

    Recommended Validation Steps:

    Review Azure Activity Logs for all identities, service principals, managed identities, and automation tools that may have permissions against the AKS-managed resource group.

    Verify whether AKS Node OS Auto Upgrade or Node Image Upgrade is enabled by checking:

    Shell

    az aks show
    --resource-group <RG>
    --name <AKSCluster>
    --query autoUpgradeProfile

    1. Review node pool configuration and maintenance settings:

    Shell

    az aks nodepool show
    --resource-group <RG>
    --cluster-name <AKSCluster>
    --name <NodePool>

    Check for external deployment pipelines, Terraform jobs, GitHub Actions, Azure DevOps pipelines, or custom automation that may be issuing VMSS lifecycle operations.

    Continue monitoring future occurrences and capture:

    • Exact UTC timestamp
      • VMSS name
        • Instance ID
          • Correlation IDs, if available

    These details will help correlate backend telemetry and identify the originating operation.

    Microsoft Documentation

    At this stage, identifying the exact initiator of the deallocation requires correlation with internal backend telemetry for the affected timestamps. Once the occurrence timestamps and affected instance details are provided, we can proceed with deeper backend validation to determine the source of the deallocation event.

    Please let us know if any new occurrences are observed and share the corresponding timestamps so that further investigation can be performed.

    Was this answer helpful?

    0 comments
  2. SUNOJ KUMAR YELURU 18,736 Reputation points MVP Volunteer Moderator
    2026-08-10T08:10:54.1133333+00:00

    Hello @Joel Zhu ,

    Your AKS cluster’s node pools (VMSS instances) are intermittently deallocating at the Azure VM level

    Step 1: Verify Azure Resource Health

    Why? Azure Resource Health tracks platform-related issues (e.g., maintenance, capacity constraints) that might not appear in Activity Logs.

    Actions:

    Check VMSS Resource Health:

    Navigate to your VMSS resource in the Azure Portal → Resource Health.

    Look for:

    Planned Maintenance (e.g., host upgrades).

    Unplanned Maintenance (e.g., hardware failures).

    Platform Issues (e.g., network/storage faults).

    Check Subscription-Level Health:

    Go to Subscription → Resource Health → Events.

    Filter for Service Health incidents (e.g., region-wide outages).

    Step 2: Dive into Azure Monitor Logs

    Why? Activity Logs only show user-initiated operations. Azure Monitor Logs (Log Analytics) capture deeper platform events, including VM lifecycle changes.

    2.1 Query VM Lifecycle Events

    Use Log Analytics to search for VirtualMachineEvents and AzureActivity tables:

    // Query 1: VM deallocation events (last 7 days)

    VirtualMachineEvents

    | where TimeGenerated > ago(7d)

    | where VMEventCode == "instance-retirement" or VMEventCode == "deallocate"

    | project Timestamp, VMName, ResourceGroup, VMEventCode, VMEventCodeDetails

    | sort by Timestamp desc

    // Query 2: Cross-reference with Azure Activity

    AzureActivity

    | where TimeGenerated > ago(7d)

    | where OperationNameValue has "deallocate" or OperationNameValue has "stop"

    | project Timestamp, OperationNameValue, SUBSCRIPTIONID, RESOURCEGROUP, RESOURCENAME, CALLER, CORRELATIONID

    | sort by Timestamp desc

    Key Fields to Investigate:

    VMEventCode:

    deallocate: Explicit user action (should appear in Activity Log).

    instance-retirement: Azure-induced deallocation (e.g., spot VM eviction, maintenance).

    CALLER: Often shows system or azure-resource-manager for platform actions.

    2.2 Enable Detailed Diagnostics

    If not already enabled:

    VMSS Diagnostic Settings:

    In your VMSS → Insights → Diagnostic Settings → Add Diagnostic Setting.

    Enable VirtualMachineAgentEvents and AzureActivity streams.

    Send logs to Log Analytics or Event Hub.

    Scope to Specific Timestamps:

    Use the exact timestamps of past incidents to filter logs and identify hidden triggers.

    Step 3: Investigate Subscription Quotas & Limits

    Why? Azure may silently deallocate VMs if you hit subscription quotas (e.g., vCPU limits, regional capacity).

    Actions:

    Check Quotas:

    In Azure Portal → Subscription → Quotas → Select the affected region.

    Verify limits for:

    Total Core Quota

    Specific VM Size Quota (e.g., Standard_DS3_v2)

    Monitor Usage:

    Use Azure Usage and Estimated Costs to track core usage across subscriptions/resource groups.

    Contact Support if Quotas Are Hit:

    Azure may deallocate VMs to enforce quotas without logging it as an explicit operation.

    Step 4: Review AKS-Specific Configurations

    4.1 Node OS Upgrade Channel

    AKS can trigger node upgrades via Node OS Upgrade Channel (e.g., NodeImage). While you ruled out most upgrades:

    Check Upgrade History:

    az aks get-upgrades --name --resource-group

    Look for recent Upgraded events coinciding with incidents.

    Temporarily Disable Auto-Upgrades:

    az aks update --name --resource-group --os-upgrade-type None

    4.2 AKS Add-On Health

    Some AKS add-ons (e.g., Azure Policy for AKS, Defender for Containers) can trigger VM reboots/deallocations.

    Check Add-On Status:

    az aks show --name --resource-group --query "addons"

    Disable Suspicious Add-Ons (e.g., AzurePolicy):

    az aks disable-addons --name --resource-group --add-ons AzurePolicy

    Step 5: Inspect VMSS Extensions & Dependencies

    Why? Malfunctioning VM extensions (e.g., Azure Monitor Agent, Network Watcher) can force deallocations.

    Actions:

    List VMSS Extensions:

    az vmss list-extensions -g -n

    Check for failed extensions (e.g., ProvisioningState: Failed).

    Disable/Uninstall Problematic Extensions:

    az vmss extension delete --resource-group --vmss-name --name

    If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

    Was this answer helpful?

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.