S2D storage pool emergency recovery

Question

S2D storage pool emergency recovery

Anjali Desai 40

Hi!

I’m running into a pretty rough issue multiple NVMe drives across different nodes suddenly crashed at the same time because of a buggy firmware state. Now my S2D storage pools are stuck in a read‑only degraded mode, and I can’t get them back online normally. I’m trying to figure out how to force‑mount these pools in emergency mode just long enough to perform rolling firmware updates safely on each disk.

Has anyone dealt with this kind of situation before or found a reliable way to recover without risking data integrity ? Any guidance from experienced engineers would be really appreciated! Thank in advance

0 comments

1 answer

Your answer

Answer 1

Hi,

What you are facing is a known failure mode in Storage Spaces Direct when multiple NVMe drives hit a firmware bug simultaneously. When S2D detects widespread disk errors, it will flip the pool into read‑only degraded mode to protect data integrity. There is no supported “force‑mount” switch to override that protection, because doing so risks corruption. The only safe way to proceed is to bring the pool back online in a controlled recovery state.

The recommended path is to place the cluster into maintenance mode for the affected nodes, then update firmware one disk at a time. You can use PowerShell to suspend the cluster node (Suspend-ClusterNode -Drain) and then apply the firmware update offline. Once the disk firmware is corrected, re‑enable the node and let S2D resync. If the pool remains stuck, you can attempt Set-StoragePool -FriendlyName <poolname> -IsReadOnly $false but this should only be done after confirming all disks are stable, because forcing read/write on a degraded pool can cause unrecoverable damage.

To check the exact state, run Get-StoragePool | ft FriendlyName, OperationalStatus, HealthStatus, IsReadOnly and Get-PhysicalDisk | ft FriendlyName, OperationalStatus, HealthStatus, FirmwareVersion. This will show you which disks are failing and whether the pool is locked. If you see multiple drives in “Lost Communication” or “Failed,” you must not attempt to write until firmware is updated.

In practice, the safest recovery is: isolate nodes, update firmware sequentially, verify disk health, then clear the read‑only flag. If the pool refuses to come back online even after disks are healthy, at that point you should open a case with Microsoft Support, because unsupported force‑mounting can compromise the cluster metadata.

I hope you've found something useful here. If it helps you get more insight into the issue, it's appreciated to accept the answer. Should you have more questions, feel free to leave a message. Have a nice day!

Domic Vo.