**** Does not work ****
I think I have found a solution to this. In the advanced power settings, go to USB settings and under USB selective suspend setting, disable both for power and battery.
This seems to fix my limited issue. At least, I haven't seen the issue coming back for a while now.
I've tried updating the driver as well and that did not fix my issue.
***EDIT***
Nevermind... it finally came back and haunt me. :( I'm still having this issue.