My Surface Earbuds will not connect to more than one device

It won't do it at all. Even in the Surface Audio app the only option is to connect to one device and I can't even switch from there. I have to go to Bluetooth settings and disconnect it from the first, and then they will connect to the next. With my Surface Headphones it is not like that at all, and I can regularly have them playing audio from two sources at the same time. 

Were the Earbuds designed to work like this or is there something wrong? 

Surface earbuds can pair with upto 7 devices. However they can connect to only one device at a time. Manual intervention is required to connect to a device after the first is connected. This is a severe limitation as it makes using the earbuds on your Surface (or other laptop) and phone simultaneously virtually useless since you must manually connect to the non-connected device.

I have since been able to got to the Surface Audio app and just select connect from the device I want to switch to. But this still takes about a minute to do most of the time, which is still about 58 seconds to long.