Same issue here, icons become invisible when closing a fullscreen app or game or when switching between desktops, started to happen after windows 11 update 23H2, worthy to mention that:
1- did a reset pc with keep files and apps.
2- did icon cache rebuild.
3- did scan system using (sfc & dism).
4- did reinstall using iso.
and all above solutions didn't help, this is officially a BUG in windows 11.
fyi