Duplicates are being imported

Question

Duplicates are being imported

LeighHarding 1

Hello, I am using the New-MailboxImportRequest to import .PST files into existing mailboxes. These mailboxes have emails in them that are also included in the .PST file being imported. The import request completes successfully, some needing the BadItemLimit to be set. After importing the .PST file, I am noticing that duplicates have been created for emails and contacts. From this site, https://learn.microsoft.com/en-us/powershell/module/exchange/new-mailboximportrequest?view=exchange-ps I understand that duplicates are not imported. This is an example of the PowerShell command used: New-MailboxImportRequest -FilePath "\ServerName\FolderLocation\UserName.pst" -BadItemLimit 10 -Mailbox "UserName"
This is an on premiss Exchange 2016 server.
So, to my questions, How do I stop the duplicates from being imported? And How do I remove the duplicates for those already imported?

Locked Question. You can vote on whether it's helpful, but you can't add comments or replies or follow the question.

0 comments

3 answers

Answer 1

This is normal behavior from your description. The old database had 1 name the new database had another. The database name is part of the metadata for the message. So you created an empty mailbox, restored from backup, which put 1 copy of a message into the new database and restored from pst which put another copy in. So either 1 of 2 things happened. First veem backup/restore was being helpful by changing the name of the metadata to the new name during it's import and the import from pst via powershell was not, so the server saw these as not duplicates. Second, veem restore didn't change the metadata of the database name, but the outlook pst file didn't have the database name in it's metadata and so the powershell import added the new database name as part of the metadata.

The basic takeaway here is you cannot trust imports/exports from psts from either different sources or using different tools to not gracefully handle conflicts. The most common example often cited is importing a pst file created by powershell export on an old mailserver to a new mailserver, then importing a pst created by export from an .ost file into a .pst file that covers the same date range. Remember that the .pst file format has never been "officially" documented by Microsoft, and the only 2 known tools out there that create .pst files are Outlook, and the API's within exchange that are used by some backup/restore tools that have the capability of restoring from backup into a pst, and by the powershell import/export commands. All the backup/restore tools I have ever seen that can "create" pst's depend on hacks like loading outlook on the machine you are running the backup restore tool from or having an exchange server somewhere (such as the target) so they clearly are using APIs inside of Outlook or in Exchange to create the PST

Answer 2

LeighHarding 1

@KyleXu-MSFT
Thanks for your suggestions.
I can confirm there is only 1 email in the .pst file used during the import process. After the import, there are 2 showing in the mailbox.
The Outlook Clean Up function is not removing the duplicated emails, both are still present.
I have checked this with multiple PST files and mailboxes.
The 2 emails in the mailbox show the same time and date, they both have the same content.
Checking the Internet Headers for the duplicated emails shows both are the same.

KyleXu-MSFT 26,406 Reputation points

2021-06-25T08:29:40.037+00:00
It sounds weird, I would confirm more information with you:

How do you export data to the PST file?

Do you export data from from one mailbox, then import to another mailbox?

Whether all emails show twice?

II think this phenomenon may related with the way that you export data. If you export data from mailbox A, then import A again, whether this phenomenon occurs?
LeighHarding 1 Reputation point

2021-06-28T04:39:25.757+00:00
@KyleXu-MSFT
Thanks again for your response.

Some background first, the Exchange database became corrupted. A new empty database was created to allow email access and flow. Some users had cached Exchange mode enabled prior to the database corruption. These cached emails were exported to .pst using Outlook. These locally exported .pst files were then imported into the empty Exchange database and respective mailbox. Some users had only 3 months, while some had longer periods cached.

To answer your questions:

Veeam Backup & Replication was used to export mailboxes to .pst from the database at a time prior to the corruption, it was while importing these recovered .pst files that I am getting duplicates

Data was exported from a users mailbox, then imported into the same user's mailbox

Only those emails or contacts that were cached and exported using Outlook are showing duplicates, ie 3 months is some cases, longer in others
KyleXu-MSFT 26,406 Reputation points

2021-06-29T08:58:03.297+00:00

I think the export tool change the attributes of emails, then Exchange import command think those emails are different from original emails. So, duplicates occurs.

A little doubt: since database broken, why there still exist emails in mailbox? Since there exist emails in mailbox, why do you import data again? I would suggest you empty mailbox, then import just once.
KyleXu-MSFT 26,406 Reputation points

2021-07-02T09:00:45.47+00:00

Any update about this thread now?
LeighHarding 1 Reputation point

2021-07-05T03:39:39.433+00:00

@KyleXu-MSFT
No further progress. Emptying the mailbox and then importing is not possible. There are current emails in the mailbox, as I have many this is happening too I am not able to follow this process.

Answer 3

@LeighHarding

I think there exist copy in the PST file that you used to import. I would suggest you have a check about it before importing.

By default, the import will could merger PST file with target mailbox.

Based on my testing, if there exist duplicate in the PST file, the import operation will not delete the duplicate from target mailbox.

Such as:

If there exist email "Tes54321" and a copy "Test54321" in a mailbox. If I export data and import data to this mailbox, there will exist 4 emails which called "Test54321".
If there only exist an email "Test54321" and doesn't exist copy. If I export data and import data to this mailbox, there will only exist 1 email which called "Test54321"

As a solution, you can use the Outlook "Clean Up“ function to delete all duplicates from this mailbox directly:

If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.