'auto compact folders' feature resulting in mail duplication

User Help for Mozilla Thunderbird
Post Reply
mikeymike
Posts: 2832
Joined: April 24th, 2003, 6:00 am
Location: UK
Contact:

'auto compact folders' feature resulting in mail duplication

Post by mikeymike »

A business customer of mine with multiple users is using Thunderbird; each user has an IMAP account hosted on a non-local-based server, and I've configured 'Local Folders' to access a file share on the LAN (mapped network drive) as well as configuring the 'archive' feature to point towards 'Local Folders'. I've configured the archive feature with the following options: yearly archived folders, and keep existing folder structure of archived messages.

The business generates a fair amount of e-mail with large attachments dating back about twenty years, so archiving older e-mails to the LAN-based file server makes more sense than paying increasing fees for IMAP storage. However, when Thunderbird wants to do a compact, it looks like some of the larger mail folders stored in 'Local Folders' (ie. the mapped network drive) are approximately doubling in size immediately after the compact.

Two users are experiencing this problem, and naturally they're the users who store/generate the most mail. This issue has been going on for some years but it's one of those issues that tends to get put on the back-burner and with an add-on in TB to delete duplicate messages it seemed like something that we would eventually get on top on. One of the users is less competent than the other so I tend to do more hand-holding of organising that user's e-mail. Yesterday I noticed something interesting in that their 'Sent' archive folders hadn't generated any duplicate messages, but when I visited them today, TB decided that it wanted to compact yet again and one of those Sent folders (we're talking 2011 so there's no reason to expect any other activity in that folder) was 1.17GB in size and after the compact had approximately doubled in size (figure taken from the file system).

By multiple accounts the issue has never occurred in the IMAP account, just in 'Local Folders'. Yesterday I was going through each folder, detecting duplicate messages and deleting the duplicates and it seemed to be doing the job correctly, and if I ran the process again it would say 'no duplicates'. It's quite time-consuming because some of these mail folders are >1GB in size and it's only a 100mbit network link. Once I spotted the problem today, I checked back to other mail folders that I had already ran duplicate detection on and TB had since compacted, and further duplication had occurred in some but not all the folders. I have the feeling that the issue is only affecting mail folders beyond a certain size (some that were affected had approximately 3k messages in).

This add-on, in case that helps: https://addons.thunderbird.net/en-GB/th ... movedupes/

For now I've completely disabled the auto compact feature and shown users how to only compact the IMAP folders.

The customer won't mind me visiting again to continue investigating this problem, but it would be useful to have some pointers for what I ought to look out for. Having TB compact the entire archive for one of these users takes ages, but I imagine I could grab a copy of their archive, sit on another PC, set up a dummy TB profile and the networked 'Local Folders' setup and just play with a few mail folders to investigate the problem further.

Operating systems involved: Historically Win8.1 64-bit, maybe Win10 64-bit, but presently Win11 64-bit for both PCs. Server is running Win2012 Essentials.

The only other issue being experienced is that roaming profiles are in use (the networked 'Local Folders' data is not included in the roaming profile), and sometimes the profiles balloon in size if TB leaves nstmp folders around (if this happens, it's in the IMAP account), and sometimes .tmp files are being left around in the roaming profile data on the server. I have the feeling that roaming profiles do not do well with large files like from the Thunderbird profile, so archiving old e-mail helps considerably with this situation.
User avatar
DanRaisch
Moderator
Posts: 127187
Joined: September 23rd, 2004, 8:57 pm
Location: Somewhere on the right coast

Re: 'auto compact folders' feature resulting in mail duplica

Post by DanRaisch »

Moving to Thunderbird Support as no specific bug has been identified here and multiple extensions are involved, clouding the source of the issues.
Post Reply