Based on everything I read here, I believe the main issue is linked to the mobile phone app.
I ran some tests, and I couldn't completely remove my XM5 from my PC without first removing it from the App on my phone.
*I agree that this is something Sony needs to look at.
But, as a workaround, I recommend completely removing the Headset from your PC and turning off the feature to change between devices quickly.
After finishing the steps above, re-connect your headset to PC by Bluetooth, and you should have it working again.
That's what worked for me, and as a plus for you, read until here:
I tried to disable space sound and some features on W11, and didn't go well. The headset started to reproduce many noises instead of the audio, plus the volume was a bit decreased, so it was better to keep that on.