Compute VM Machines are not being terminated after job completes.

Question

Compute VM Machines are not being terminated after job completes.

Anonymous

One of our Slurm cluster using SpotVMs are able launch VMs on job excecution, however the VMs are keep running and not being terminated automatically even when the shutdown policy is specified.

Anonymous

2026-01-12T17:35:16.2533333+00:00

This issue only occurs when many jobs are submitted simultaneously; when a small number are executed, everything works fine.

Answer accepted by question author

0 additional answers

Your answer

Anonymous

2026-01-12T17:35:16.2533333+00:00

This issue only occurs when many jobs are submitted simultaneously; when a small number are executed, everything works fine.

Answer 1

Anonymous

Root Cause:

The cluster deletion was failing because the underlying VMSS was stuck in an extension upgrade state, which prevented the Scale Set from being deleted. This behavior has been identified as a CycleCloud product bug, and the product team has already acknowledged the issue and targeted a fix for it in CycleCloud version 8.8.2.

0 comments

Answer 2

Hello @Anonymous

Thank you for contacting us about the VM issue.

In this scenario, the compute Spot VMs were not being terminated after job completion because the underlying Virtual Machine Scale Set (VMSS) was stuck in an extension upgrade state.

When a VMSS is in an extension upgrade or provisioning‑blocked state:

The scale set cannot process delete or deallocation operations

The cluster cleanup process fails silently

As a result, VMs continue running even though shutdown or termination policies are configured

This behavior has been identified as a known product issue in CycleCloud, where a stuck VMSS extension prevents the Scale Set from being deleted, leaving compute nodes active even after jobs complete.

Links :
https://learn.microsoft.com/en-us/azure/cyclecloud/?view=cyclecloud-8

Thanks,
Manish Deshpande.