I think I may have actually found the solution here.
Step 1. Go to your computers settings.
Step 2. Click system.
Step 3. Click sound.
Step 4. Scroll down to App volume and preferences
Step 5. Notice that for some really bizarre reason there are two separate Microsoft Teams App volumes - I don't know why, there just are.
Step 6. As counterintuitive as this is, the two Microsoft Teams App Volumes should NOT be set to the same thing. ONE of them needs to be set to "Default" and the other should be set to whatever your headset earpiece is (I would have expected them to need to be the same but nope). Changing the top Microsoft Teams App volume to "Default" seemed to fix the problem.
I don't know why they set it up this way, but it really does seem to work.