You might all want to have a look at this link which refers to the identical problem but with Teams as against Link to Windows: Fixes or workarounds for recent issues on Outlook.com - Microsoft Support
Run the EXE file and it deletes the spurious contacts folder created by Link to Windows/Teams with reference to your phone. Choose that folder. Let the app take its time to run. Be patient. It eliminated thousands of duplicate contacts in my case.
What remains unclear is whether you also need to sync the Samsung accounts contacts folder. My understanding is that the Outlook sync is one way.
Someone might address that please.