Go too default apps under settings and make your browser anything BUT Chrome. This fixed it for me.
That's nice, but I actually want Chrome as my default browser, so this isn't a good solution for me. How to I keep Chrome as my default Windows browser and stop it from opening on startup? The solutions marked earlier in this thread have not worked for me.