Is there a way to identify duplicate messages?

User Help for Mozilla Thunderbird
Post Reply
fixer1234
Posts: 15
Joined: July 15th, 2021, 9:32 am

Is there a way to identify duplicate messages?

Post by fixer1234 »

I had to recover my mail from a large collection of mbox files. The result contains extensive duplication of messages, often in different folders. I'm looking for a way to identify duplicates. If someone knows of a tool to do that, that would be great.

Short of that, I'm looking for a way to compare message lists. I can sort two lists the same way, and identify duplicates by the same subject, correspondents, and date/time (the basic summary information that shows in the main window). If there's a way to output the list that shows in the main window to a file, I could compare those lists offline, as time permits, even use a comparison tool to spot them for me. Is there a way to do that? Or even to open two instances of Thunderbird using the same profile (I can put the windows side-by-side and scroll through them).

(I'm using Linux. The system with the recovered folders has TB 68; TB 78 is available on another computer.)
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Is there a way to identify duplicate messages?

Post by tanstaafl »

There are add-ons that test the Message-Id: header (it should have a unique value unless there are duplicates) to see if there is only one copy of a message in a folder tree, and remove any duplicates.

One potential problem is that version 68 uses legacy add-ons (XPCOM/XUL based) while version 78 uses add-ons that have been rewritten to use the WebExtension API. So you need to use different add-ons if you're going to do this on both machines unless they share the same profile.

https://addons.thunderbird.net/en-US/th ... es/?src=ss (68)
https://addons.thunderbird.net/en-US/th ... es/?src=ss (78 and later)
fixer1234
Posts: 15
Joined: July 15th, 2021, 9:32 am

Re: Is there a way to identify duplicate messages?

Post by fixer1234 »

tanstaafl, that sounds promising. I'll check it out. Thanks.
fixer1234
Posts: 15
Joined: July 15th, 2021, 9:32 am

Re: Is there a way to identify duplicate messages?

Post by fixer1234 »

tanstaafl, just to follow-up, your suggestion of the Remove Duplicate Messages add-on was a great one. That's a nicely-done add-on. It found and fixed thousands of duplicates for me, which would have taken forever with the methods I was trying. Thanks again.
Post Reply