I have the same problem with all the Surface tablets that I purchased and returned in the past two months (three Surface 3, two Surface Pro 4) for work. I really need the SD drive to work reliably, and yet, not one of them will. I am certain to conclude that there is something fundamentally wrong that Microsoft needs to fix. Doing a Google search, I could see reports of this problem tracking back in 2014 on the Surface / Surface Pro line of tablets. I am very dissatisfied with the lack of effort in addressing this long-known problem.
This is the exact kind of things that put a bad reputation to Windows.