So what is your configuration? I find this very odd.
I have three monitors with audio in them and an RME Babyface high end audio device I use for recording , mixing and mastering.
I think its as most folks are saying. Windows does not lock in the audio device you choose.
Lets say you get a new driver for Nvidia Graphics card. Nvidia will have an audio driver included. Even if I uncheck the box for the audio, in my setup I can be guaranteed that after installing the graphics driver, Windows will change my audio device to one of the monitors. When I go into the sound settings to change back all three monitors audio are enabled again. This is after I have choose don't allow them under sound settings. This also happens when I have them disabled in Device Manager.
So this is and there is a problem, and always has been a problem for me. WIN10/11