Sadly only uninstalling IE11 didn't work for me. I guess IE10 needs to be uninstalled as well. The only temporary fix I've found is disabling the network adapter. It instantly lowers the memory usage for svchost.exe. It does just continue to rise after enabling it again though. I was really hoping uninstalling and reinstalling the network adapter would work.
I just don't understand why it is affecting a small number of us. I've asked friends and they have no issues with it. With IE10 or IE11 installed. I'm considering doing a repair using the Windows 7 SP1 disc tomorrow. Maybe a fresh install of all the updates and services will fix something.