Jason, have you resolved this yet? I updated to the newest Win 11 which caused my Realtek driver to update. The problem returned so I backed down the driver again. This fixed it but then my Realtex Audio Console would not work with that driver. I then installed the latest Reatek driver from my PC's manufacturer's (Lenovo) website and followed the regedit on this page...
Solved: 3 seconds delay to play sounds through headphone - Page 2 - HP Support Community - 6631359
Specifically, I set these keys to these values in the proper section...
ConservationIdleTime = ff ff ff ff;
IdlePowerState = 00 00 00 00;
PerformanceIdleTime = ff ff ff ff.
That fixed the problem again... for now.