I think I have found a solution to this. In the advanced power settings, go to USB settings and under USB selective suspend setting, disable both for power and battery.
This seems to fix my limited issue. At least, I haven't seen the issue coming back for a while now.
I've tried updating the driver as well and that did not fix my issue.
Thanks there.
I've changed my settings as you suggested.
I will report back my findings.
I hope, it fixed.