I had the same issue.
SOLUTION for me was uncheck "Show badges (unread messages counter) on taskbar apps" which appears directly below "Automatically hide the taskbar" in preferences.
You can check that this has worked for you by restarting Windows Explorer in Task Manager.