Duplicates

User Help for Mozilla Thunderbird
Post Reply
IanGreen
Posts: 15
Joined: February 17th, 2017, 11:06 am

Duplicates

Post by IanGreen »

Hi,
I'm a "newbie" here, so if I have posted incorrectly, please let me know.

I think I have a similar problem to many others, judging by some posts on here.

I have installed both available Duplicate Email Remover Add-ons, and configured them, where possible, to do really intensive checking ----- but they still fail to correctly identify duplicates.

Does anyone know of any other Add-ons, or even a full blown commercial (Pay money for) programme ---- that will do the job effectively?

Thank you,

Ian
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Duplicates

Post by tanstaafl »

http://kb.mozillazine.org/Duplicate_messages_received

Your post is fine.

I haven't noticed it being a wide spread problem. You can find a few examples for any problem you think of with over 21 million users. Any possibility that your folder is corrupt? If it is, that would explain why the add-ons didn't remove the duplicates. The inbox is most vulnerable to corruption. Most users drastically overestimate the effectiveness of pressing the "repair folder" button. See http://kb.mozillazine.org/Compacting_folders#Real_fix

I suggest you use view -> message source or Control-U and look at the Message-Id header for a few cases of the duplicates and verify they really do have the same value for the Message-Id, and that the message isn't fragmented. Message-ID: <1d306b1b-a4c6-1b99-7658-739e955915f9@yk.rim.or.jp> is an example of a Message-Id: header. Its supposed to have a unique value, and is what is used to identify the message.

Does this problem occur in a POP or IMAP account?

https://removedupes.mozdev.org/bugs.html is apparently the bug reports for both add-ons. You'll need to add a temporary exception (you'll get prompted) to read the bug reports. You could always file a bug report. If you do that please let us know what sort of response you got. I suggest you read https://www.mozdev.org/bugs/show_bug.cgi?id=26325 before filing the bug report.

Its possible the add-ons don't work correctly due to some API change. You could install an older version of Thunderbird in a different directory and use your existing profile with it. if you're lucky that would workaround the problem, and then you could go back to using the recent version of Thunderbird. See http://kb.mozillazine.org/Go_back_to_an ... hunderbird . The add-on page saying supports "Works with Thunderbird 3.0 - 31.*" doesn't mean it doesn't work with later versions, that usually just means the last update was released when TB 31.* was the current version and the author didn't want to release another version just to bump the maximum supported version information in the manifest file. In general, any add-on that supports version 10 or later is compatible with any later version, by default.

Is this a one-time problem where the add-on aren't helping you as much as you'd like, or a re-occurring problem?
IanGreen
Posts: 15
Joined: February 17th, 2017, 11:06 am

Re: Duplicates

Post by IanGreen »

Hi tanstaafl, and thanks for your response.

Following your advice, I carried out these experiments --

Ran the "Remove app" but it returned a "Zero files for deletion" message.
However, the folder listing shows <Subject> <From> <To> and <Date-Time> as being identical for the three (at least) files I chose to examine.

Viewed "Message Source" for each of the three files and saved the results (a) in "raw state" and then (b) in .txt format after opening and saving "raw" file.

All six files (3 x raw & 3 x .txt) displayed as being identical in regard to file size.

Ran Windows DOS "File compare", using the "/lb2" switch, and received "Resync failed" and "Files are too different" error messages.

A difference in time was reported between the files as --- e.g. "19:27:29 and 20:40:44", this, despite the two files each showing the same date and time in the folder listing.
Similar differences were noted for all files, and it appeared irrelevant as to whether they were "raw" or ".txt".

In the "Setting Options" of "Remove Duplicate", I unchecked Message ID., Date in seconds., Compare date and Line count, leaving only Subject., From., Message Body and To for use in comparison.
However, running the Add-on again with only those parameters, whilst it did find the files detailed earlier, there were a dozen or more other files "un-found" yet all displaying identical Subject, From, To and Date headings, with the contents too appearing the same.

I ran "File Compare" over a pair of these "identical files", using the greatly reduced number of parameters, and again received time variations.

I am bemused as to how, with only " Subject., From., Message Body and To" being the only comparison criteria, these messages are not being found.

I have screen-shots of the comparisons, but there doesn't appear to be the facility to upload them here.

I have just run "Remove" with only the "Message body" parameter selected for comparison, and have received notification that 12 out of 22 messages should be deleted.

In answer to your last question, this is a one-time problem which has come about following my transfer to TB and my wish to have all historic emails contained therein. (Several thousand involved -- hence my trying to avoid a manual comparison.)

Regards,

Ian
User avatar
LIMPET235
Moderator
Posts: 39932
Joined: October 19th, 2007, 1:53 am
Location: The South Coast of N.S.W. Oz.

Re: Duplicates

Post by LIMPET235 »

Hi Ian,
Here's how to post images here...

> http://kb.mozillazine.org/Posting_a_scr ... _the_forum

You should see a "Print Screen" key near the top RHS of the keyboard.
Use that & the image will be copied to your Clipboard.
Reduce them to around 600 x 500 or so in your favourite image editor, (Be sure to remove any personal info.)
& upload them to eg: > Photobucket or your favourite image hosting site.
> http://photobucket.com/
> https://imgur.com/
> http://postimg.org

(I use Photobucket....so...)
In photobucket...
Open the image by clicking on it.
On the RHS, there are a few options. "Links to share this photo."
Choose the [img] link at the bottom & it will be copied to the clipboard.
Paste that link in here.
[Ancient Amateur Astronomer.]
Win-10-H/64 bit/500G SSD/16 Gig Ram/450Watt PSU/350WattUPS/Firefox-115.0.2/T-bird-115.3.2./SnagIt-v10.0.1/MWP-7.12.125.

(Always choose the "Custom" Install.)
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Duplicates

Post by tanstaafl »

I selected a dozen messages in a IMAP account and used "message -> copy to" to copy them three times to a child folder of the inbox in Local Folders that already had a dozen other messages in it. I installed the alternate version of the remove duplicates add-on v0.3.12 and ran it against that folder using "remove duplicates". It correctly found all of the duplicates. I told it to only keep the first copy and that's what it did. I'm using TB 45.7.1 under Windows 10. The add-on has default settings. Its comparing based on From, Message Id, send time , size and subject. I then changed the settings to have it only use Message Id and repeated the test. It worked fine.

I suggest you do something similar to see if you can ever get the add-on to work. If it does, that would seem to confirm that the problem is folder specific.
User avatar
DanRaisch
Moderator
Posts: 127186
Joined: September 23rd, 2004, 8:57 pm
Location: Somewhere on the right coast

Re: Duplicates

Post by DanRaisch »

Moving to Thunderbird Support.
IanGreen
Posts: 15
Joined: February 17th, 2017, 11:06 am

Re: Duplicates

Post by IanGreen »

Hi LIMPET235
Thanks for the info, although a bit surprised at the lack of straight uploading on the site.

Ian
IanGreen
Posts: 15
Joined: February 17th, 2017, 11:06 am

Re: Duplicates

Post by IanGreen »

Hi tanstaafl,

I too can copy and dedupe successfully in the manner which you describe, and am more than certain that all folder structures are good.

However, I suspect my problem is down to the fact that many of the emails have seen life, either as an "original creation", or as an "imported from", by other email clients and have probably suffered alteration during the process of import/export and conversion to the differing formats demanded by the various software packages.

Some of the emails go back a million years or so, to the days when I first used Outlook, and have since undergone change to suit Claws, EmMail, Postbox and Thunderbird, during my search to find a decent alternative email client to anything Microsoft.

Along the way too, some of them will have been archived into Mailstore, later being recreated for transfer to a new client (very probably in a different format too) ---- who said there was no demand for conformity and interchangeability between packages?

Anyway, it looks as if I have something of a long, drawn out, manual -- or at best semi-manual, task ahead of me.
Still, an opportunity to REALLY decide if an email is needed to be kept, I suspect.

Thanks for all your help, interest, and advice in this.

Best wishes,

Ian
User avatar
LIMPET235
Moderator
Posts: 39932
Joined: October 19th, 2007, 1:53 am
Location: The South Coast of N.S.W. Oz.

Re: Duplicates

Post by LIMPET235 »

Hi Ian,
Can you imagine the type of images that would be posted here if that (direct image posting), feature was available?

No siree. Not going to happen.
[Ancient Amateur Astronomer.]
Win-10-H/64 bit/500G SSD/16 Gig Ram/450Watt PSU/350WattUPS/Firefox-115.0.2/T-bird-115.3.2./SnagIt-v10.0.1/MWP-7.12.125.

(Always choose the "Custom" Install.)
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Duplicates

Post by tanstaafl »

Think about exporting the messages as .eml files using the ImportExportTools add-on, and then importing them into a new folder using the same add-on. You could try it with just one folder to see if it helps. If you are lucky, that might sanitize any changes made due to moving them between so many email clients. If that doesn't help, then I'm afraid I'm out of ideas.
IanGreen
Posts: 15
Joined: February 17th, 2017, 11:06 am

Re: Duplicates

Post by IanGreen »

Hi LIMPET235,

I should imagine the majority of contributors would respect imagery in the same way as they do speech, with only the foolhardy and stupid submitting anything of an "anti-social" or "undesirable" nature -- such idiots then falling foul of the Moderating Rules accordingly.

Ian
IanGreen
Posts: 15
Joined: February 17th, 2017, 11:06 am

Re: Duplicates

Post by IanGreen »

Hi tanstaafl,

I shall certainly give that a go, but as I shall be "on the move" over the next few hours, it might be tomorrow before I have a chance to get back to you.

Thanks,

Ian
User avatar
DanRaisch
Moderator
Posts: 127186
Joined: September 23rd, 2004, 8:57 pm
Location: Somewhere on the right coast

Re: Duplicates

Post by DanRaisch »

IanGreen wrote:Hi LIMPET235,

I should imagine the majority of contributors would respect imagery in the same way as they do speech, with only the foolhardy and stupid submitting anything of an "anti-social" or "undesirable" nature -- such idiots then falling foul of the Moderating Rules accordingly.

Ian
Ian, it's not regular users/members who are the problem with images embedded in forum posts but spammers, particularly some posting truly disgusting porn. We went through periods where we were moving dozen's of messages at a time to locked, all obviously sent by 'bots. They've made things more difficult for everyone who just needs to post an image of the problem they are having.
IanGreen
Posts: 15
Joined: February 17th, 2017, 11:06 am

Re: Duplicates

Post by IanGreen »

Hi tanstaafl,
Sorry my few hours became a few days but had a bit of an urgent demand on my time at this end.

However, I have (two or three times now) done the Import/Export routine you suggested, but to no avail -- it still records the same number of files/dupes as it did before.

I guess the answer is ---- don't keep swapping your emails about from client to client -- unless you are prepared to do a lot of manual remedial work further down the line.

I have actually found a load of emails, from the 2009-2011 period, which have somehow managed to "re-date" themselves as 2071 when displayed in the folder.
All dates within the email appear correct, but it really makes it tough to sort in chronological order.

By the way, do you have any idea who thought of making the default "sort" to show the oldest first?

Hope all is well with you.

Ian
Post Reply