I called Microsoft and they sent out a new charger. So far it looks like it's fixed the problem for me fingers crossed but I've only had it two days so I probably need to test it out a bit longer. Give them a call and see if that fixes it for you or pop into a store and see if you can swap the charger on the spot.
Super annoying though. Mine happened like that straight out of the box which isn't a great experience because I spent time formatting thinking it was a software problem.