Surface headphone connected but no audio

This issue started a couple of days ago. When I turn on the headphone, it says: "Hi, ... you are connected to (my phone name)". On the phone, it shows that the Bluetooth is connected to the headphone. But the audio output is still the phone speaker. I have to open the Cortana app, tap on the headphone and try to navigate to setting and go back and it randomly gets fully connected. 

Sometimes, I have to turn off the headphone, and tap lets connect on the Cortana app and turn on the headphone. Sometimes it works in this way.

I bought this headphone a couple of weeks ago. It used to get connected with redirecting the audio to the headphone immediately after turning on. I wonder what is wrong now.

What I tried:

1. Restarting the phone.
2. Restarting the headphone.
3. Factory resetting the headphone (solved the issue for a couple of days).

I'm not sure why a factory reset is working for a period of time. This is reoccurring.

Any Idea?