I read through the earlier replies and didn't see anyone claim it was impossible, but I would come close :-)
Have you checked: https://account.live.com/Activity to see if there is any suspicious activity?
If I know your email address and find out your password one way or another then I can say that we both have the same email account - we would both have access to it.
You can go further... in your account settings turn on two-step verification and change your password. A hacker would have a much more difficult job breaking into your account.
Of course, none of these steps would stop other people sending you email.