many duplicate messages

[georgesgiralt]

Hello,
I'm a very very long time user of Thunderbird (even when it were a part of the web browser...) So I moved/copied/transferred mail folders from computers to computers and kept a lot of backups.
Recently I discovered that in some folders some messages are duplicated. Sometimes with 4 or 5 copies. So I checked quite all my folders, and found this to be a general problem.
So I've a question :
As my mail and it's history is very important to me, how could I automatically remove all the duplicate messages ?
Many thanks in advance for your help and answer !
Have a nice and bright day.

[DanRaisch]

I'm guessing that this is a POP account but please confirm that. In Thunderbird the menu path is Tools->Account Settings->Server Settings->Server Type at the top right of the dialogue.
How have you been transferring messages from machine to mache? That is, what procedure have you been using?

BTW, Thunderbird (as Thunderbird) has never been "part of a web browser". Perhaps you used SeaMonkey, the suite, in the past?

[georgesgiralt]

Hello,
Thank you for your answer !
Yes, they are pop accounts. (about 6 of them). I never leave any mail on the POP server and store everything on my computer.
And at that time it was the embedded email client in Netscape browser.... (when I said it was a long time ago )
I transfer files in the Mail folder (using an old app called Unison to check which one is current and synchronize them). This way i copy whole files so no editing takes place in the moved files.

[DanRaisch]

You could try an extension like this one -- https://addons.thunderbird.net/en-US/th ... es/?src=ss

[georgesgiralt]

Thank you DanRaisch !
I downloaded it and it did not work. So I read the manual and the comments on the Web page, and I restarted Thunderbird, and ran it.
I choose to find messages with all the criteria matching (if they are _real_ duplicates, everything is identical, isn't-it ? )
And before running the extension, I made a copy of my Mail folder with Thunderbird shut dow, just in case (once bitten, twice shy... )
IT ran for hours (literally hours). But my Mail folder is many giga bytes big.
Now, it is moving the duplicates to the trash folder.... Will take time I guess.
Next compress all folders and see if I still have some mail remaining...
Will report back if the computer survives

[georgesgiralt]

So, the job finally ended. I reclaimed around 2GB of space... Must have been an awful lot of duplicates.
But the question remains : Why did I get duplicates ? What did I did wrong ?
I would be delighted and thankful to know that in order to avoid it in the future !
Have a nice day and a lovely Sunday !

[DanRaisch]

How many messages were stored in the folder(s) where the duplicates appeared? Was the Inbox a major culprit? If so...
Storing messages in the Inbox of any POP type email account is a very bad practice. As the busiest area of the system, the Inbox is most at risk of data corruption and potential data loss. That could, in some cases, take the form of duplication.

[georgesgiralt]

No, DanRaisch,
I leave into Inbox folder only the messages awaiting treatment. All other are stored in a folder relating to the subject. (I file the question and the sent response in order to be able to find the whole thread, so even the Sent folder is "empty" ).
IMHO, it is when I copy the Mail folder during a "sync" that the problem arise.
The Unison software compare time stamps and checksums on the various files in the folder and put the most recent one in place of the "not in sync". As each Thunderbird folder is one file, I thought this would work provided I do this without Thunderbird running.
When I was working, I used Unison a lot to sync software files between various developers platform (this way, each individual could make huge mistake and do not risk the whole project). So I was quite confident about the behaviour of this piece of kit..
So the problem is on the Thunderbird side. I'm doing something Thunderbird does not like and it says so by duplicating the messages. But what and why ? That I would like to know because as I said, keeping all my emails is important to me as is the fact that they are stored on MY computer, not in the cloud somewhere...
Thank you very much for your help !

[DanRaisch]

You're perhaps missing the fact that moving or even copying and then deleting messages from a message files does not immediately remove the data from the message file. It simply causes Thunderbird to flag the message as deleted and not display that message. Until that message folder is compacted ( http://kb.mozillazine.org/Thunderbird_: ... ng_Folders ) the old data remains in the folder. At some point the .msf summary file or the actual message file (for instance Inbox, with no file name extension) can become corrupted and that can build up to the point where the problems because noticeable.

[georgesgiralt]

No, I did not.
This is why I copy both files for a given folder. The folder itself AND the .msf file from one computer to the other. The only parameter is which one is current, the source or the destination.
Now, since I only use one computer at a time, I copy the files from the computer I used to the computer I intent to use. And I do this with Thunderbird shut down on both machines.
And as the folder I've copied are here for safekeeping of the emails, I do not delete nor move messages from them. A normal message travel is from Inbox to a said folder where it stays forever, being followed by the reply i sent moved from the sent folder to the exact same folder...
So I still do not understand where the error is. On my side or on Thunderbird behaviour ?
It is not a big deal, but it make me itch when I do not understand why some software is not doing what it is supposed to do. Perhaps 40 years in the computer business explain that
Thank you anyway for your help which was very very helpful !
Have a bright and nice day.