Just found this thread, and it looks like no one from Microsoft has commented yet. Unfortunately, I am in the same camp as the OP. I just got my buds yesterday. I ordered them after checking the reviews, which all commented on how comfortable they were to wear, and how they were great for outdoor exercise since they were fitted snugly into the ear. The only negative seemed to be that they didn't have active noise cancellation, which I was okay with since I wanted them for road cycling and I need to be able to hear the traffic overtaking me.
I tried all of the tips, then settled on the mediums as the best fit. I found it was quickly very painful to use gestures on my right ear. Just the force of swiping to adjust volume hurt!
Today was the first day riding with the buds. I found that I needed to crank the volume but I could hear fine and still hear traffic. Unfortunately the left bud fell out after only about 10 miles and it was just a fluke that I felt it slip and was able to catch it while riding. Before I return these I wanted to find out - is this just what I will get with these buds or am I doing something wrong?