How to find and delete duplicate files in Microsoft 365 sharepoint

you can automatically find and delete duplicate files in SharePoint using:

1. Microsoft Graph QuickXorHash–based PowerShell scripts ( https://jamesachambers.com/cleaning-sharepoint-find-duplicates-trim-versions-retention/ )
2. Third‑party deduplication apps, such as deDup for SharePoint by Nonlinear SRL from Azure Marketplace https://www.sharepointmigrator.com/blog/sharepoint-duplicate-analysis-tool/

All of these identify duplicates by comparing file content hashes, not filenames - which is essential for GDPR cleanup.

Warning: Before using any third‑party scripts or tools, always review the source code carefully and verify what it does. Make sure it does not delete, overwrite, or modify data in an unintended way. Test it first in a non‑production environment and only run it with the minimum required permissions.

There is no built-in SharePoint feature I'm aware of that automatically finds and removes duplicate documents across an entire tenant so your approach would depends on how accurate you need the matching to be and how much automation you want.

For starters, the reliable method would be to compare the file hash (such as SHA-256 or MD5) of every document. A PowerShell script using the Microsoft Graph API or PnP PowerShell can enumerate SharePoint libraries, download file metadata (or the files themselves if necessary), calculate hashes, and produce a report showing duplicate files. This approach minimizes false positives and works for Word, Excel, PowerPoint, PDFs, images, and most other file types.

If this is impractical because of the volume of data, you can first narrow the candidates by comparing metadata such as file size, last modified date, and filename. Only files with matching metadata would then be hashed. This could reduces processing time in large environments.

If you have Microsoft Purview, note that Purview Data Lifecycle Management and Retention Policies determine how long information should be kept, but they do not deduplicate documents. Likewise, Microsoft Search and SharePoint Search can locate similar files, but they are not designed to identify exact duplicates for deletion.

There are also commercial products, such as ShareGate, AvePoint, Quest Metalogix, and SharePoint Essentials, that include duplicate analysis and cleanup capabilities. These tools often provide reporting, approval workflows, and safeguards to prevent accidental deletion, making them suitable for large enterprise SharePoint environments.

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin